Measure first

1A guess written as a fact

One scene, the Noise library (an example scene of rows of shapes, each covered in a different noise relief: marble, granite, leather), rendered at well under two samples a second. A sample is one noisy try at the whole picture, a path of light traced through every pixel; the picture on the screen is the average of many (chapters 12 and 23), so the time one sample takes is how fast the renderer is. A plan to speed ShadyTracer up called this scene's time "this architecture's full-resolution cost", and set out to change the architecture. Here is that time in the benchmark's first full run, on the user's GTX 1080 Ti (an NVIDIA card of the generation called Pascal, which is what the documents call it): 736.5 ms a sample at 1280 × 720.

The same card, the same size, the same code, the other bundled scenes: 61.8 to 177.2 ms, the demo 7.7 times faster. The user had said as much, "about 6x faster at the same resolution", before the plan was written. A cost of the architecture would be paid by every scene. One scene paying it is a fact about that scene.

And the scene's own rows, each changing one thing, say which. Without its six reliefs the Noise library renders in 81.1 ms, beside Surface materials, the same layout without them, at 80.0. The rows marked "only relief" keep one of the six, named by how high its bumps reach and how far apart they are (1 mm high, features about 10 mm across): each alone adds 72 to 91 ms. They do not simply add up: alone, the six add 484 ms summed, and together 655 ms, a difference the run's notes record and do not explain. Either way the time was the reliefs: the noise they evaluate at every step near a surface (chapter 26 is what was done about it). The plan was reverted, and the repository's instructions gained the rule this tutorial stamps every number with:

2Where the time goes

A sample's time on the card (the GPU) is spent in its kernels: small programs, each run over thousands of rays at once. A page reaches the card through WebGPU, the browser's way of handing it work, and WebGPU takes work in compute passes; ShadyTracer gives every run of a kernel a pass of its own, about 200 in this sample. They are chapter 22's: genrays makes the camera's rays; then, round after round (an iteration), march moves every ray up to 32 steps, surface and shade work out what each ray found and where it goes next, occlude traces the shadow rays, and four small kernels (three prep_ ones and flip) only size the next kernel's work or swap its queues; resolve, last, adds the sample into the picture. The sample is chapter 22's room, a small room in the flat world (this tutorial's two-dimensional scenes) lit by the sun through a window, in 24 iterations.

To time the passes, ShadyTracer has the card write a timestamp, its own clock in nanoseconds, as each pass starts and as it ends, and reads them all back after the sample. Each row here is a kernel and each mark a pass at its timestamps; one march pass is marked with its two, in nanoseconds from the first. The times are a model's, made from how many rays and steps each pass handled in the flat world (the caption says how); what reads them is the engine's own code, kerneltime.js, run in this page. Point at a pass.

Added up by kernel (tally): the march takes half of this sample, shade a fifth, occlusion a seventh, the surface kernel a ninth. Each of the four small kernels runs 24 times, once an iteration, and its 24 passes together take about a tenth of a millisecond. The part of the sample's span no pass covers is the gaps between passes, here 0.4 ms of 88: what launching one kernel after another costs. Wall clock, the time the program waits for the sample, is neither: it adds everything outside the card (chapter 22 measured that part at 1.5 ms plus 2.35 ms an iteration).

By iteration (perIteration, every pass from a prep_march to its flip): the first eleven carry most of the work, the twelfth about half as much (chapter 22 says why it drops there), and the drained tail (the last iterations, when only a few slow rays are left) costs a little each. Timing by kernel and by iteration is how the benchmark found the budget that shrank by accident (section 6): its rows ran 15 iterations where row 1 ran 24.

One catch. A browser may round the timestamps, to keep a page from timing things it should not (a fine enough clock lets a page measure what other code on the machine is doing). Chrome's documentation gives 100 µs; the user's fifth run, in Chrome 154 on the Ampere card (a later NVIDIA generation; the documents do not name the model), read every small kernel in multiples of 32.768 µs, and the run's notes record the difference without explaining it. Rounded to 100 µs, 124 of the 194 passes here read 0, and a few read 0.1 ms, which no single pass took (marked above them). Added up, the big kernels' totals still come out near their true values (the march 44.6 ms against 44.3); the small ones' only on average, over more passes than one sample has: prep_march reads 0 in all where its passes took 0.1 ms, prep_occlude 0.2 where they took 0.1. The gaps move too, since they are the span less the passes, both read from rounded stamps: 0.20 ms here against 0.39.

Why on average: the start and the end are each rounded down to the tick before them, separately, so a pass reads the whole ticks between its two rounded stamps. A 4 µs pass reads 0 when no tick mark falls inside it and a whole tick, 100 µs, when one does. It holds a tick mark when its start lands in the 4 µs just before one: 4 µs in every 100. Placed anywhere, then, it reads a tick 4 times in a hundred, and its reading averages 4% of 100 µs: its own 4 µs. (Ten thousand random placements give 4.33: an average of random draws wobbles a little round the truth.) Drag the pass across a tick mark, and change its length: a rounded timestamp is right on average over many passes and wrong for any one of them.

3The same code, another number

The same kernels (their text the same to the byte), the same scene and settings, on the user's GTX 1080 Ti in one evening: 184.7, 184.8, 216.6 and 255.5 ms a sample of GPU time. Nothing in the code differed. Two measures of how far apart they are. The range: how far the highest is above the lowest, 38%. The spread: their standard deviation as a share of their mean, 16% (the readings' distances from their mean squared, added, divided by one less than the number of readings, and the square root taken: the usual way for a handful of readings). The card's own speed moves, with its heat and power state or other work on it (inferred: the readings cannot say which), and the review that collected these says a comparison between runs "is good to about ±20%, and sometimes not that".

So time a change in a run of its own, and the old code in another: the drift between the two runs lands in the difference. Here build B is truly 10% faster than A, and 40 comparisons are made under a model of that drift (each run's speed its own, spread by 16% as those four readings are): in 17 of them B reads no faster at all, and only 8 land within 5 points of the truth (a point: one percent, so between −5% and −15% here). A 10% change cannot be seen this way.

Interleave instead: A, B, A, B in one run, so both share the card's mood of the moment, and compare the mean of B's two rows with the mean of A's. Only the small noise between rows next to each other is left: about 1.5% in the model, from the documents' rows repeated straight after themselves (0.38%, and 1.3% in later runs). Rows far apart in one run drift further, as section 5's repeated row shows. All 40 comparisons read B faster, 37 of them within 2 points of the truth. The benchmark repeats its first row in the middle and at the end for the same reason, a drift check, and can run its rows, by number, in any order: BENCH.run(33, 35, 33, 35).