Part V · Light the camera can't find · Chapter 19
Photon mapping
Chapter 18 ended on a path neither side could make: a caustic seen through a pane of glass. The camera's shadow rays cannot aim through the ball, and a photon's straight line to the eye meets the pane. A photon map meets the two in the middle: photons are kept where they land, and the eye's path, arriving through the pane, asks how much light landed near it. This chapter is how that works, why it blurs, how the blur is made to fade, and how a GPU finds the photons near a point among millions.
1Light kept where it lands
Chapter 18 in short. A caustic is light that glass or a mirror gathers onto a surface. The camera's paths find the sun at every matte point by aiming a shadow ray at it, but not through glass, which bends the line: through glass they find it only by a lucky bounce. A lucky find counts for a lot, 85 here against the open floor's 0.39, because what a path finds is divided by the chance of finding it, and that chance is small. So light tracing sends photons from the sun through the glass and joins each landing point to the eye, and one rule gives every kind of path exactly one owner, the camera or the photons, so no light is counted twice. A sample is one path through every pixel; the picture is the average of the samples, and ± is its error bar (chapter 12).
Chapter 18's ball and sun, with a pane of glass standing on the floor between the eye and the ball. The strip below is the eye's picture; the pixels that look through the pane are bracketed, and the caustic is among them. Here the picture is made as chapter 18 left it, the camera's paths with light tracing. Light tracing cannot count the caustic through the pane (its line to the eye meets the pane), and the camera's bounces find the sun through the ball only by chance: 9 times in 300 samples over the pane's 14 pixels, each find worth 85. Point at the strip: pixel 24 reads 1.45 ± 0.63, a value as unsure as it is big, and pixel 25 beside it, where the caustic is brightest, 0.03. The readout's error bar over the pane, ±0.060, is that luck. Given enough samples the camera's paths get there (0.323 ± 0.060 is within its error bar of the map's 0.366 below, and the tutorial's test runs both long enough to agree); the map gets there in a few.
Now the photon map. Each sample's photons go first, as in chapter 18, and every one that has passed glass or a mirror is stored where it lands on a surface that scatters: where, how much light, which way the surface faces (the short ticks on the floor, the first sample's 181). Then the camera's paths. One that has met only glass since the eye, here through the pane and onto the floor, gathers there: it adds up the photons stored within a radius r of where it landed, 0.06 here (the dashed circles), and turns them into the light the floor sends back to the eye (the fold-out). The caustic is there from the first sample, and smooth: pixel 25 reads 0.72, and over the pane the pixels average 0.366 ± 0.003 after 300 samples. The path then bounces on, and any light it reaches through glass alone is the map's now, so it is not counted twice: chapter 18's rule of one owner for each kind of path, with the map a third owner.
The price is r. Make it 0.4: the gather now averages the light over 0.8 of floor, and the caustic's peak is spread over its surroundings. Pixel 25 reads 0.48 where it read 0.72. This is a bias, an error that does not average away: every sample makes the same blur. The pane's average hardly changes, though: a blur moves light from the peak to the pixels beside it, mostly inside the pane, rather than losing it. The error is in where the light is.
Make it 0.012, and the blur is gone, but each gather now finds only a photon or two, so every pixel's error bar is about twice as large (point at pixels 25 and 26: ±0.02 where 0.06 gave ±0.01; the pane's average, over 14 pixels, hides most of it). Wide is smooth and wrong, narrow is right and noisy. Section 2 has both.
The mathsWhat a gather adds
A photon carries Φ, its share of the sun's light (chapter 18). The photons that land on a stretch of floor of length ℓ, added up and divided by ℓ, are the light arriving per length there: the floor's irradiance E, measured. The matte floor sends back f × E towards the eye, f = ρ/2 in the flat world (chapter 8). A gather takes ℓ to be the stretch within r of the point, 2r long:
L ≈ f × Σ Φp2r in the flat world, L ≈ Σ f Φpπr² on a surface in 3D
where the sum is over the photons within r, and in 3D f depends on the direction each came from, so it sits inside the sum. What this measures is E averaged over the 2r (or the disc), not E at the point: that average is the blur. A photon whose surface faces more than about 25° away from the gather's is on another surface (a wall meeting the floor) and is not counted.
2A blur that fades
One point of the floor, under the ball where the caustic peaks. Its true brightness is 0.938, found without any randomness: 400,000 of the sun's rays sent evenly across the ball and followed through it exactly (the dashed line). Each sample sends 600 photons and gathers within r₁ = 0.25 of the point; the red curve is the average of the samples so far. It settles, but on 0.623, not on 0.938: on the light averaged over the 0.5 of floor round the point (the dotted line). More samples only make it surer of the wrong value.
Claude Knaus and Matthias Zwicker's way out (2011): each sample is a photon map of its own, so each can have its own radius, smaller every time, by the rule
rk = r₁ × 1 + α2 × 2 + α3 × … × k − 1 + αk
with α between 0 and 1 (the lower panel; α = 2/3 here, ShadyTracer's). The first samples are blurred and the late ones sharp, and the average of them all, the blue curve, climbs towards the true value: 0.905 after 300 samples, the radius down to 0.041. Slowly, because the blurred early samples stay in the average, but the blur's share of it falls towards nothing while the late samples' noise, though larger, is averaged over more and more of them. No pixel has to remember anything but its average, which every pixel keeps anyway (chapter 12): the paper's point. A renderer whose error fades to 0 with more samples, though every finite picture is a little wrong, is called consistent.
α sets the pace. At 0.25 the radius falls to 0.004 by sample 300, the blur goes sooner, and the average reaches 1.002, past the true value: a late sample's gather catches one photon or none, so each sample jumps and the curve jumps with them. ShadyTracer uses α = 2/3, the paper's recommendation. In 3D it is the disc's area πr² that shrinks by the rule, so r² does; on the flat world's line the "area" is 2r, so r itself does.
The mathsWhy the average converges
Two words first. "a ∝ b" says a grows in proportion to b: double one and the other doubles. Variance is the square of the noise, the spread of a sample's value about its mean (chapter 12); averaging k samples of equal variance divides it by k.
How the radius shrinks, with α = 2/3, in numbers:
| sample k | rk ÷ r₁ | photons a gather finds | its blur |
|---|---|---|---|
| 1 | 1 | 1 | 1 |
| 10 | 0.51 | 0.51 | 0.26 |
| 100 | 0.24 | 0.24 | 0.057 |
| 300 | 0.17 | 0.17 | 0.027 |
| 1000 | 0.11 | 0.11 | 0.012 |
(the last two columns as a share of the first sample's). For large k the product behaves like a power, rk ∝ k(α−1): here about k(−1/3), a tenth by sample 1000. Two errors, sample by sample:
- noise
- with N photons landing on each unit of floor, a gather finds about N × 2rk, give or take its square root (how counts of independent random events spread). The gather divides the count by 2rk, so its spread is √(N 2r) ÷ 2r = √(N ÷ 2r), and its square, the variance, is ∝ 1/rk: fewer photons, more noise, growing as k(1−α). The variance of an average of k independent samples is the sum of their variances divided by k², here about k × k(1−α) ÷ k² = k(−α): it falls, for any α above 0.
- blur
- a smooth curve averaged over ±r differs from its middle value by about its bend × r²/6, so the error grows as r² ∝ k(2α−2), the table's last column. Its average over k samples falls too, for any α below 1.
So both go, and α trades one against the other: small α kills the blur fast and the noise slowly, large α the reverse. At α = 1 the radius never shrinks (section 2's first step); at 0 it shrinks so fast that the noise never falls.
3Finding the photons near a point
A ShadyTracer sample keeps up to two million photons (for a 1920 × 1080 picture, more for a
larger one: section 5), and every camera ray seen through glass gathers: for each of the 921,600 pixels of a 1280 × 720 picture
to look at every one would be nearly two million million records a sample. So the floor is cut into cells
2r wide (the ticks on it). The photons within r of a point can then lie only in its own cell
or in the neighbour on the side of the cell's middle the point is on: two cells (shaded), and in 3D, two on each
axis, 2 × 2 × 2 = 8. Most cells hold no photon, so they are not kept as an array but in a
hash table: a cell's number times a large constant (73856093, the engine's), and the
remainder after dividing by the table's size, picks its list. (The large constant spreads neighbouring cells
over the lists. The product is kept as a 32-bit whole number without a sign, so a negative cell's wraps round to
a large positive one: cell −3 gives −221,568,279, which wraps to 2³² − 221,568,279 = 4,073,399,017; and since the
table's size is a power of two, its remainder is just the product's last bits, here 1: list 1. In 3D a cell is
three numbers, each multiplied by a constant of its own and the three combined into one, bit by bit. The page
runs the engine's JavaScript, hashCell; its WGSL twin, sppmHash, is the same formula.) Drag the gather point; the
blue ticks are the photons it counts, the blue bar its 2r. Here it walked
20 records and counted 10.
Two cells can pick the same list. With four lists (ShadyTracer has a million, so this is rare there), cell −3 shares list 1 with cells 1 and −11, and cell −2 shares list 2 with cell 2 (a list's label names the cells that have photons on it; −3 has none). A gather at −0.5 walks ten records of those other cells (red; one of them far off the figure's left edge) to count the one of its own. Counting every record a list holds would be wrong twice over: a record of cell 1, far away, would count for a gather at −0.5, and a gather whose two cells picked the same list would walk it twice and count its records twice. So each record keeps where it is, and is counted only on the visit to its own cell, and only within r. A collision costs time, never light.
Light piled into a line can put thousands of photons in one cell: 8,315 records in one list, counted on the dichroics scene. A gather walking them all takes that long, and the code bounds a thread's time, so until version 0.61.3 a gather stopped at 512, and the light it did not reach was lost where the photons were densest. Now every record is also put on one of a cell's 32 sub-lists, drawn at random as it is stored, and each list counts its records (all the records on it, whichever cells share it). Past its share of the budget, a gather walks only m of the sub-lists, from a random start, and every record it finds counts 32/m times. Step by step:
- The budget is the records one gather may walk: 512 in the engine. Shared over the 8 cells it visits, each cell's share is 512 ÷ 8 = 64. This figure uses a share of 6.
- A cell whose list holds no more than its share is walked whole.
- One that holds more is walked through m = 32 × share ÷ count of its sub-lists, rounded down: on average that is about a share's worth of records.
- Here list 1 holds 8 records: m = 32 × 6 ÷ 8 = 24 sub-lists, each find counting 32/24 = 1.33. List 0 holds 12: m = 32 × 6 ÷ 12 = 16, each find counting 2.
From start 0 the gather reads 0.738 against the whole walk's 0.615; press The next start to see the others. Averaged over all 32 starts it reads 0.615 exactly: each record is on the walked sub-lists in m of the 32 starts, and counts 32/m each time. And m is decided by the list's count alone, never by which sub-list a record drew, or the weights would favour some records. Right on average, a little noisier only where it acts.
4How ShadyTracer does it
In outline. Storing: a photon that has passed glass and lands on a surface that scatters takes the next record number (an atomic add on a counter), writes where it is, its light and its two directions there, and puts itself at the front of its cell's list and of one of its 32 sub-lists (an atomic exchange of the list's head). Gathering: a camera ray visits its 8 cells; for each it reads the list's count, walks the whole list or m of its sub-lists, and adds every record that is its own cell's, within r, on a surface facing the same way and within the bounce cap (and within Max Diffuse Bounces, chapter 13: the surface it lands on is a diffuse bounce, so a record is stored only with room for one), each times the surface's f; the sum is divided by πr². What the map is, in the file's own words:
The schedule, with the square root that makes it the area that shrinks:
And where r₁ comes from: a number of pixels at the distance the camera is focused on, so the blur starts at a few pixels whatever the scene's size. The setting is 3 pixels by default, and its own comment says why 3: "3 is a choice made without seeing it" ().
The grid, built as photons are stored, with no sorting:
The gather's walk, cell by cell, with the budget (the loop over the eight cells and its start are just above):
Why the budget is right on average, as the file states it:
And one factor a gather inside glass needs (the bottom of a pool, seen through the water's surface):
5Tricks and trade-offs
Each is stamped with how it is known; point at a stamp to see what it means.
What it costs. On the user's Ampere card, the dichroics scene (glass plates lit by a sheet of
light, the floor seen through them) at 1280 × 720 took 196.0 ms a sample with the map and 173.3 without: 22.7
ms, of which the map's own kernel was 16.2 (docs/PERFORMANCE.md, Tests > dichroics, version
0.61.2). That was before the sub-lists, which walk more where the old gather stopped; their cost on a card is
not measured yet.
Neither full nor cut, checked. On Dawn (Chrome's WebGPU run outside the browser) on an RTX 3080
Laptop, the dichroics scene (glass plates lit by a sheet of light): the light the map put in its pixels, as a
share of a map with room for everything (CHANGELOG.md, 0.61.3). A seed is the start of the random numbers, so two seeds are two
different runs of the same picture.
| 1280 × 720 | 1920 × 1080 | |
|---|---|---|
| the old gather, stopped at 512 | 0.355 | 0.274 |
| the sub-lists and kept stores | 0.965, 1.000 (two seeds) | 0.884 to 0.903 (three seeds) |
| the exact map, run again | 0.978 | 0.894 |
The exact map, run a second time, reads 0.978 and 0.894 of its own first run: that is how far two runs of the same thing differ, and the new gather is within that spread.
Photons first, in every sample. A map built in one sample and gathered in the next would cost no second pass, but what the two sides read changes between samples (which glowing objects are known, chapter 16), so the camera would hand over light the photons were drawn without. So a sample runs its photons, then its camera rays (). That comment still calls the second pass's cost "not measured"; the 22.7 ms above includes it.
Idle when nothing looks. A gather marks a word in the map's header. A sample in which no
camera ray gathered makes the next traces run without the map, until a new scene, size or mode, or every 16th
trace looks again; a camera move does not wake it (SPPM_PROBE; raytrace_main.js, the photon map this trace). A view with no
glass in it pays nothing.
Memory. A record is 11 words, 44 bytes: where, the power in three colours, the photon's direction
and its surface's normal packed into 32 bits each, the next record in its list and in its sub-list, and its
bounces. Two million photons, the default, are 88 MB (SPPM_REC; the setting
sppmCapacity). That is the room for a 1920 × 1080 picture: a larger one gets room in proportion, four
times at 3840 × 2160, 352 MB (sppmCapacityFor), because a larger picture sends more photons and
gathers over a smaller disc (the radius is in pixels), and at a fixed room each gather found fewer of them.
6What went wrong before
- Black, with white specks. The map exists because of one picture: a lamp, a glass slab on a white floor before a white wall, and through the slab, black with white specks (). Two earlier mechanisms each covered part of that path class; the specks were photons that happened to cross the camera's lens. The map replaced both.
- A mirror that was not a mirror to the map. The test for "can a gather here see anything" compared a surface's roughness with a floor set for alpha, roughness squared (chapter 9), so a silver mirror of roughness 0.02, a delta to the BSDF, stored every photon that reached it, and every camera ray seen through glass gathered there, all for an f of 0 ().
- Photons nobody could count. A photon stored on its last allowed bounce takes a record, its atomics and every gather's visit, but a camera ray gathers with at least two bounces of its own, so the bounce cap never lets it be counted (). It is no longer stored.
- A gather that stopped, and a buffer that filled. Section 3's budget and the keep probability: both lost light exactly where it was densest, which no number of samples brings back.
- Darker at 4K. A 3840 × 2160 save of a scene in the Optics view came out darker than the same scene at 1920 × 1080, and right with the room raised to 16 million. Nothing was dropped or cut: the 4K map kept 21% of its stores, the 1080p one about 85% (worked out from the 4K's demand of 8.1 million, a quarter of it at 1080p; not read), and with a disc a quarter the area each gather found far fewer photons. The keep probability is right on average, so the darkening is inferred to be noise: bright pixels clip at white and give back less than they took. A run on a software GPU agreed (the clipped mean fell as the keep fell, the linear mean only wandered), but the user's picture was not measured that way. The fix gives a larger picture room in proportion ().
- A keep planned from a reading that never came. The keep probability is planned from what the sample before stored. The host used to plan it from a readback, which arrives only when the GPU has finished and the page gets a turn. The test bench records all of a row's samples before the GPU finishes the first, so every sample ran on the first guess. On the user's card, a row seen inside milky glass read 0.546 of its reference: the guess was a surface's two stores a photon, where a photon walking the glass stores at every collision, and every sample overflowed. With a higher guess the same row read 1.01, and its keep at the 64th sample was still the guess's 0.00082, where 0.025 fits. Now the GPU plans the keep itself, in a one-thread kernel at the start of each sample, from the count the heads still hold (). It is the kind of bug the repository's own notes warn of: true for the one caller that existed (the live loop, which yields every frame), false for the second.
7Try it
- In the first figure, with the map on, move the radius from 0.01 to 0.4 and watch pixel 25 and its error bar. Where would you set it for 10 samples? For 300?
- In the second, set r₁ to 0.05. How close do the two curves end, and why does shrinking help less? Then set α to 0.95.
- In the third, make the radius 0.4. How many records does a gather walk, and how many does it count? What happens to the hash table's lists?
- In ShadyTracer, stand a glass slab on a white floor in front of a lamp, and look through it. Switch Seen through glass and mirrors (photon map) off and on (Render settings). With the Developer switch on, change Photon map: radius (pixels) and watch the first samples.