I recently had to write a Blink layout test ensuring that a piece of code returns the current time. I am documenting the stages I went through, hoping to provide both entertainment and an appreciation for the importance of using code that has been thoroughly tested instead of rolling your own.
Let’s start by formalizing the task at hand: given a function
code, we want
to write a test function
test that returns true if
code’s return value is
the current time, and false otherwise.
Revision 1: Direct Comparison
The straight-forward comparison method is relatively straight-forward.
to convert Date instances into numbers, because different
Date instaces are
never equal. Conveniently,
valueOf() works the same way for both numbers and
dates, so I don’t have to worry about whether
Date.now() returns a
instance (which is true in some browsers) or a number, as the specification
This test will probably work on your machine, and will most likely pass a continuous integration suite. However, it is a flaky test, meaning that at some point it will fail. Let’s write some code to prove that.
On my machine, running this in the Chromium dev tools comes up with a number in less than a second.
Revision 2: Stubbing
In many cases, a great way to solve this problem is to stub the Date API used by the code being tested.
This code is a bit paranoid, and is probably overkill for specific cases. It
now() on both the stubbed
Date constructor and on the real
constructor, so all the
code() variants below would be recognized as correct.
return new Date();
return (new Date()).constructor.now()
The biggest advantage of this approach is that the stubbed current time is
consistent across tests, which makes for very robust tests. In return, we pay
the usual price of stubbing and mocking, namely our test doesn’t prove that
code() returns the current time, it merely proves that it calls some API and
passes down its return value. Assuming that the underlying APIs are solid and
will not change, the trade-off is usually worth it!
At the same time, this approach does not work if
code() uses an API that we
can’t stub, or if we really want to assert that it returns the current time.
Revision 3: Time is Monotonic
The clever code below takes advantage of the fact that time is monotonic.
Sadly, this test is still not bulletproof.
proveTestIsBroken() will terminate
if our assumption of monotonic time breaks down. Most computers use NTP to keep
their clock synchronized, so the clock might still be adjusted backwards.
test is still flaky.
Revision 4: Sometimes, Time is Monotonic
A straight-forward fix for the NTP issue is below.
The sight of
while (true) makes experienced programmers wary, as it can turn
into an infinite loop under the right cirumstances. In some cases this will be
catastrophic. Most test frameworks implement a timeout mechanism, so the
consequence of an infinite loop will be a cryptic error message, and possibly
slowing down other tests that are queued up to run on the same machine. Still,
test failures are better than timeouts.
Revision 5: Time is Usually Monotonic
We can get rid of the
while (true) if we assume that NTP updates are
infrequent, and just bail if the time keeps moving backwards.
We still have to assume that when the clock is adjusted backwards, the jumps
are relatively large. If there is a small jump backwards right after
startTime is read, but
code() takes long enough to run,
still be greater than
test() to return
code() might be correct.
I will stop here for now, but reserve the right to update the aricle if the code above proves insufficient.
All the clever bits in this article are lifted from code review comments contributed by various Chromium reviewers. The mistakes are all mine.