One eye, three renderers

Snoot is a companion robot inside a plush polar bear. It is in design. There is no physical prototype yet. Its face has two round screens for eyes, 360 × 360 pixels each, and those eyes do most of the acting.

The eyes are drawn three times, by three renderers:

1. The Python engine. It draws every eye you see in our films and on the site. It supersamples 3× and filters down, so it is slow and pretty.
2. The float C port. A line-for-line port for the robot's ESP32-S3 chip. It draws at 1× in strips of 360 × 40 pixels, ready for the screen.
3. The fixed-point C port. The same compositor with no floats per pixel, built for the chip's budget.

If these drift apart, the films promise an eye the robot can't draw. So a test holds all three together.

How we compare two eyes

Every case is one eye in one expression. The Python engine exports the static layers (the eye white, the iris and the catchlight sprites) and the expression's numbers. The C ports draw from those. So the C never re-draws the art. It only composites it.

Then the test converts both frames to CIE L*a*b* and takes the colour distance per pixel, ΔE (CIE76). A ΔE of about 1 is at the edge of what an eye can see. It splits the round glass into two regions:

• Field pixels, away from any edge. Here the two must agree. The gate is the 99th percentile of these: the worst 1 % must sit at ΔE ≤ 3.
• Edge pixels, found with a Sobel filter and grown by 2 px. These are reported, not gated. More on them below.

Here is the comparison, copied from the test with the file paths stripped:

c = np.fromfile(c888_file, np.uint8).reshape(N, N, 3) / 255.0
ref = np.asarray(Image.open(ref_png).convert("RGB")) / 255.0   # the intent
Lr, Lc = lab(ref), lab(c)
de = np.sqrt(((Lr - Lc) ** 2).sum(axis=2))
em = edge_mask(Lr[..., 0]) | edge_mask(Lc[..., 0])
em |= ~GLASS                     # the round glass shows nothing past r = 1
field = de[~em]
p99 = float(np.percentile(field, 99)) if field.size else 0.0
ok = p99 <= LIMIT
floor = np.sqrt(((Lr - lab(from565(to565(np.round(ref * 255).astype(np.uint8))))) ** 2).sum(axis=2))[GLASS]

Why the gate sits before RGB565

The screens take RGB565: 5 bits of red, 6 of green, 5 of blue. That rounding loses colour on its own. We measure it by rounding the Python engine's own frame to 565 and comparing it with itself. On this palette the worst 1 % reaches ΔE 3.0–3.5 (3.1–3.7 in the 13 Nov re-run).

That loss is as large as the whole gate. A gate on the 565 frame would test the screen, not the renderer. So the test compares the C colour before the 565 step. The 565 frame and the 565 floor sit beside it in the report, for scale.

Float C: the reference

The float port passes every case. When we wrote it on 18 October, it passed 24 of 24 cases (11 expressions × 2 eyes, plus a blink frame × 2) at ΔE ≤ 3. The worst 1 % ranged 0.84–0.97.

The expression set has grown since. In the 13 Nov re-run it passes 28 of 28, with the worst 1 % at 0.73–0.87. The art changed in between (a warmer eye white, a fur-coloured lid), so the two runs are not the same frames.

Speed on the Mac was 2.2 ms per eye (median; one core, -O2, median of 200 runs per case). The 13 Nov re-run gave 4.0 ms, on a different Mac under heavy load from other jobs. Neither number predicts the chip. The float port is a correctness reference. It does float maths on every pixel.

Fixed point: right, and too slow

The fixed-point port drops floats from the pixel loop:

• colour in 8.8 fixed point, coverage in Q8, shading in Q12;
• every curve (shadow, lid roll, fur taper, glass edge) from a lookup table;
• the per-frame setup for 360 columns stays in float, since the chip has a single-precision FPU.

It also skips work. Each column splits into spans: pure upper lid, pure lower lid, eye white, and the edge bands that need the full path. Each fast path does exactly what the full path does there. Span skip on and off give bit-identical frames.

It is correct. The worst 1 % against the Python engine was 0.80–1.00 on 18 October. In the 13 Nov re-run, all 36 of 36 cases pass, at 0.75–0.88.

It is too slow today. We have no board, so we estimate from a hand count of cycles per pixel, then add 50 % for spills and memory misses at the high end. On one core at 240 MHz:

In the 13 Nov re-run the common range is 27–57 ms EST; the low end is the wide-open surprised eye. Either way, most of the range is over budget.

The notes propose these fixes:

• Cache the crescent. It depends on very few inputs. The first frame is drawn into a buffer and later frames copy it: 1.5–2.9 ms EST. A ramp into the smile redraws only the arc's 8 % of pixels: 4.5–7.3 ms EST.
• Skip unchanged rows. A held frame draws nothing (0.3–0.6 ms EST). A glance with the lids held redraws about 71 % of rows: 20–30 ms EST.
• One eye per core. Then the frame time is the slower of render and transfer, not the sum for two eyes.
• Next candidates: update the lids at half rate during a lid move, and send only a column window to the screen on a glance.
• Held back: pre-baked lid textures. They are fast, but they change the look in motion.

Every lever above, except the held one, is gated as bit-identical to the frame drawn whole. A bench for the chip is written and compiles, but it has not run. The board decides. Until then, eye compute stays our top eye risk.

The edges, in the open

Edge pixels are 11–26 % of the glass. Their mean ΔE is 0.5–0.9 (0.4–0.8 in the 13 Nov re-run). Their max reaches ΔE 18.8 (18.4 in the 13 Nov re-run), on the grin, happy and sheepish lids.

Those are the 1-pixel lash and seam lines. The Python engine draws them supersampled and filtered. The C ports draw at 1× with an analytic 1-pixel ramp. One pixel off on a dark line over a light lid is a big ΔE, and it is expected. We report it so nobody finds it later and wonders what we hid.

The miss

On 18 October we added an eye smile: the squint Snoot makes when you boop its nose. The upper lid arches up and the lower lid pushes high. To keep the inner corner from dropping, the Python engine mirrored the lid curve's crown offsets for an arched (negative) bend.

That change landed in the Python engine only. Both C suites failed on the 2 new squint cases. The worst 1 % was ΔE 5.9 against the limit of 3, because both C builds still drew the old lid edge.

We mirrored the crown offsets for a negative bend in both C builds the same morning. The mainline was red for 24 minutes. The suites went back to 26 of 26 (float C) and 34 of 34 (fixed point). No other state moved, because the mirror only fires on a negative bend.

The test did its job. The process did not: we had reviewed the Python change without asking for the C run. So the rule since is simple. Every eye-engine change ships with its C mirror and both suites passing, in the same branch.

What we take from it

• Gate the renderer, not the screen. Measure the screen's own floor and put the gate before it.
• Show the pixels you don't gate. A max of 18.8 on lash lines is fine. A max nobody looked at is not.
• Correct and fast are separate results. The fixed-point port is correct today. It is not yet fast enough, and only a board will say how far off it is.

This post is about Snoot: A plush bear that performs. It sees you, talks, and makes a show out of your day.

Written by Zhao Lou.