Hyperdimensional computing for video

Hyperdimensional computing, also called vector symbolic architectures, represents things as very long random-looking vectors, thousands of bits wide, and combines them with three cheap operations: XOR binds two vectors into one that resembles neither, a majority vote bundles many into one that resembles all of them, and a rotation marks order. Similar things end up a short Hamming distance apart.

raea uses this as the layer between the models that look at footage and the questions an editor asks about it. We call it the sandwich: frozen encoders at the bottom, bits everywhere above them, and no network trained on video.

The layers

Layer 0 is per moment. SigLIP 2 embeds frames at 8 a second, YAMNet embeds the sound every 0.48 seconds, and a 64-band log-mel spectrum is taken every 10 ms. Each embedding becomes 8,192 sign bits through a seeded random projection. That projection is the one real cost: 4.3 ms a frame on a Mac, 0.86 ms on an iPhone 13 Pro Max, against roughly 60 ms for SigLIP itself.

Layer 1 binds each moment to the next (rotate the previous, XOR the next), so a vector stands for a change. Layer 2 bundles the changes in a two-second window, each bound to its position. Layer 2b keeps three change profiles beside them, picture change, sound onsets and loudness, as level hypervectors. Everything above layer 0 costs about 0.29 ms a window on one Mac core and 0.045 ms on the iPhone.

a:i%n
b:i!n
d:sum[k;64;pop[xor[word[H;(a*64)+k];word[H;(b*64)+k]]]]

Against V-JEPA

We measured the sandwich against V-JEPA 2-L on the same windows: 97 clips with sound, 8,738 two-second windows, split by creator so every number is on 34 creators the learned parts never saw.

Where bits lose

Predicting the next two seconds of an edited ad, nothing beat copying the current window: SigLIP 0.352, V-JEPA 0.298, bits 0.280. Filling a hidden two seconds is a question about content, and float embeddings win it: SigLIP 0.568, V-JEPA 0.480, bundled frame bits 0.469. Sign bits cost about 0.1 on content matching, so raea keeps the float SigLIP column beside the bits for “what is this” and uses the bits for “how is it changing”.

Next: optical flow from Apple Vision on the Neural Engine to close the picture-only gap, and a prediction test set after cuts, where copying the current window can't win.