How an HTTP header caused time.gov to skew from UTC

# How an HTTP header caused time.gov to skew from UTC Source: [https://alexsci.com/blog/how-time-gov-works/](https://alexsci.com/blog/how-time-gov-works/) [Built on Shards of Silicon](https://alexsci.com/blog/) --- In the United States, the[National Institute of Standards and Technology \(NIST\)](https://www.nist.gov/)maintains the official U\.S\. time reference\. NIST distributes this reference to enable all sorts of applications from meteorology to GPS satellites\. Programmers are probably most familiar with distributing time using the network time protocol \(NTP\), which NIST supports by operating several NTP servers\. NIST also runs[the beautiful time\.gov website](https://time.gov/)which provides an official time reference via a web page\. Its an easy way to check the time if you don’t trust the clock on your computer’s taskbar\. On a recent project I needed a trustworthy clock and time\.gov was a convenient option\. To validate that the provided reference was accurate, I opened time\.gov in two browser windows side\-by\-side, but found that the provided clock offset estimates disagreed by a margin larger than I could tolerate\. When I compared to another source, an NTP client, I found even more disagreement\. [](https://alexsci.com/blog/how-time-gov-works/three-way-compare.png)#### Side\-by\-side browser windows hint of problems on time\.gov I dismissed time\.gov as an option, but the inconsistency of the estimates kept bugging me\. My intuition told me that more precision should be possible so I had to circle back to figure out what was broken\. This blog post explains the issue and how NIST fixed it\. *For the sake of clarity: the issue described in this post affects the clocks displayed on the time\.gov website only, not any of NIST’s other time services\.* ## NTP and half round\-trip time Before I dig into the time\.gov implementation, I’ll give an overview of how NTP works\. I’ll simplify things for the sake of brevity; there are great explainers elsewhere if you want an accurate deep dive\. In the NTP protocol, the server responds with the current timestamp\. This timestamp is accurate at the time it was generated, but the client doesn’t see it immediately\. It takes time for the response to reach the client, which causes the timestamp to grow stale\. The NTP client needs to estimate how much the time elapsed since the timestamp was generated\. It does this by measuring the round\-trip time \(RTT\) of the request, which includes both network latency and server processing time\. Adjusting the server\-provided timestamp using these metrics can produce a very good estimate of the current time\. While it is possible for the request and response to take different amounts of time to travel the network, its reasonable to predict that the network delay is the same in both directions\. As such, the network delay experienced by the NTP response can be estimated as half of the round\-trip time\. [](https://alexsci.com/blog/how-time-gov-works/Simplified-NTP.drawio.png)#### A simplified version of the NTP protocol There’s a lot more to know about NTP: it uses UDP, the response contains multiple timestamps, multiple requests are used, and statistical analysis is performed\. The time\.gov bug was significant enough that we can ignore those details and use this simpler model\. ## Distributing time over HTTP The time\.gov website synchronized time over HTTP,[using JavaScript to perform the requests](https://web.archive.org/web/20260308233044/https://time.gov/scripts/zzz__5c7ff74f782289d35dd3f9a4744ff31c221b34e9.js)\. The core functionality performed an HTTP request and collected timing information using “new Date\(\)”\. ``` d = new Date(); var xmlHttp = new XMLHttpRequest(); ... xmlHttp.send(null); ... var currentTimeObj = new Date(); ``` Like NTP, time\.gov sends multiple timestamps, but the difference between these is rounded to zero milliseconds and doesn’t impact calculation\. So we will continue to ignore that detail\. The calculated offset is displayed to the user and the website display is updated by adjusting the local time with the offset\. The JavaScript looks like a reasonable approximation of NTP and it’s believable that it could produce the correct time\. The issue time\.gov was facing happened at the network level\. ## What happened on the network? Opening time\.gov loads a bunch of resources from the time\.gov domain, including the HTML, JavaScript, and CSS\. The network request to fetch the timestamp also uses the time\.gov domain\. For performance reasons web browsers typically send multiple HTTP requests over a single connection\. However, the HTTP responses from time\.gov all contained the`Connection: close`header, which told the web browser to immediately close every connection\. Each time the web browser requested a resource from time\.gov, it did so over a new network connection, which required TCP handshake and TLS setup\. The JavaScript code assumed that, like NTP, a single network round\-trip occurred\. Unfortunately, the`Connection: close`header forced three round\-trips to occur\. This incorrect assumption was the root cause of the time\.gov issue\. [](https://alexsci.com/blog/how-time-gov-works/simple-3rtt.drawio.png)#### Simplified network timeline As the diagram shows, the server generated the timestamp after two\-and\-a\-half round trips while the client’s guess fell near one\-and\-a\-half\. This miscalculation caused a full round trip time of error, more error than if they hadn’t adjusted for network delay at all\! Interestingly, the estimate occurred before the HTTP part of the network request began\. We know the server didn’t tell us the time before we asked, so this can’t be correct\. In practice the situation could be even worse as each round trip can take a different amount of time\. In the following example, the TCP handshake and TLS setup took much longer than the HTTP round trip\. This caused time\.gov to estimate that the server timestamp was generated before the TLS session was established\. [](https://alexsci.com/blog/how-time-gov-works/400ms-network-timing.png)#### Showing network timing details Network round trips were often fast enough that the error was difficult to notice\. Close examination of the network behavior revealed the issue\. ## How can this be fixed? I see two approaches that could help fix this issue\. First would be to change the connection header to`keep\-alive`\. This would allow the web browser to keep the connection open longer, reducing the time\-synchronization HTTP request to a single network round trip in*most*circumstances\. This is the approach used by the[National Research Council \(NRC\) Canada web clock](https://nrc.canada.ca/en/web-clock/)\. [](https://nrc.canada.ca/en/web-clock/)#### The Canadian Government's web clock But it’s not a perfect solution\. During the initial synchronization, the NRC web clock app is able to use an existing connection\. However, this web clock resynchronizes periodically\. Later synchronizations are delayed long enough that the earlier connections have already closed, which means a fresh network connection must be created\. You can see this behavior by watching the clock: the offset estimate and the estimated network delay change somewhat dramatically after a minute or two\. So both the NIST and NRC web clocks are impacted by this bug\. The extra network round trips wouldn’t be a problem if we could just measure the HTTP portion of the network request\. The “new Date\(\)” approach is very coarse grained and measures much more than intended\. An alternate approach uses[PerformanceResourceTiming](https://developer.mozilla.org/en-US/docs/Web/API/PerformanceResourceTiming)to collect precise network timing information\. This interface was designed to provide accurate measurement of each part of the network request, allowing us to measure just the application layer; excluding DNS lookup, connection handshake, TLS initialization, and more\. I built an[experimental CDN\-based web clock](https://clock.alexsci.com/)to validate the process, which had promising results\. I reported the issue to both NIST \(on March 17th\) and NRC \(a month later\) with these suggestions\. Around April 24th time\.gov changed their HTTP headers, enabling keep\-alive\. [](https://alexsci.com/blog/how-time-gov-works/fixed-headers.png)#### Connection keep\-alive header added I consider the core issue to be fixed as the “round trip” measurement no longer measures multiple round trips\. In my testing, the time\.gov estimates are now closer to those of NTP\. ## Remaining precision concerns Unfortunately, there are still too many things being measured in time\.gov’s “new Date\(\)” approach\. Compare how the site estimates the network request duration as compared to Firefox’s network debugging tab and PerformanceResourceTiming’s measurement \(more on that in the next section\)\. [](https://alexsci.com/blog/how-time-gov-works/new-date-vs-network-tab.drawio.png)#### timeDotGov\.data\.responseTime \- timeDotGov\.data\.requestTime is still too long In this example, the original measurement of 63 ms is still larger than the 49 ms shown in Firefox’s network timing panel and calculated from PerformanceResourceTiming\. ## Further improvements Here’s a client\-side patch that changes time\.gov to use PerformanceResourceTiming to calculate the clock offset\. Pasting this JavaScript into the developer tools console will re\-calculate the round\-trip time based on a more precise measurement\. It also adds error bounds, which should be included when generating estimates\. Note: after ten minutes the page hard\-refreshes and this patch is removed\. ``` var requestTiming = performance.getEntries().filter((x) => x.name.includes("cgi?disablecache"))[0]; const serverDelay = timeDotGov.data.RThalf * 2 - timeDotGov.data.responseTime + timeDotGov.data.requestTime; timeDotGov.data.RThalf = ((requestTiming.responseEnd - requestTiming.requestStart) - serverDelay) / 2; const localTimeResponseEnd = new Date().getTime() - (performance.now()-requestTiming.responseEnd); timeDotGov.data.realTimeDif = timeDotGov.data.serverTime - localTimeResponseEnd; // time.gov also forgot RThalf here var diff = ((timeDotGov.data.realTimeDif + timeDotGov.data.RThalf)/1000) * -1; var diffDisplay = diff.toFixed(3); if (diffDisplay > 0 ) { diffDisplay = "+" + diffDisplay; } // add error bounds diffDisplay = diffDisplay + " ± " + (timeDotGov.data.RThalf/1000).toFixed(3); document.getElementById("realTimeDif").innerHTML = diffDisplay; ``` Here’s what that looks like: [](https://alexsci.com/blog/how-time-gov-works/fixed-example.png)#### Strong agreement between modified time\.gov and ntpdig *Note: time\.gov and ntpdig invert the sign of the computed offset\.* This example shows the modified time\.gov can provide better precision\. With that said, please do not consider my patch as a better “official time” as I provide no warranty\. NIST should consider adjusting how time\.gov measures network timing\. ## Closing thoughts These sorts of hard to diagnose issues lurk in every complex computer system\. You know your software isn’t performing the way it should, but you can’t locate the problem\. With so many architectural layers it can feel daunting to know where to start\. As a DevOps consultant, I love guiding my clients to a deeper understanding of their software\. Sometimes a single header is all that’s holding you back\. The time\.gov bug is a good example of how challenging things can get\. The web application made assumptions about network behavior that didn’t hold up in the production environment\. Local testing was unlikely to detect this issue as the development environment likely used different headers\. It seems possible that the HTTP Connection header was modified some time after time\.gov launched, perhaps by a network team changing load balancers or web application firewalls\. It’s even possible that this issue did not reproduce on NIST networks, as the network path may traverse different middle boxes which provide different HTTP headers\. A perfect storm of challenges that requires going back to first principles\. --- If you'd like to show your appreciation and help support my writing, you can do so via this link\. Statements are my own and do not represent the positions or opinions of my employer\.