Skip to content
Jason Peterson Jason Peterson
Go back

I Grouped a Film's Shots for Grading in 8 Seconds

I’ve been wanting to try a System 1 model, sometimes called a decision model, since reading about Jev from TypeSafe. Until this week Jev only took text. Version 1.14 (Oct 14) added image input, still flagged as experimental. So I tried it on a problem I care about: grouping a film’s shots so one color grade can cover each group.

124 shots of Tears of Steel sorting themselves into scenes, then into grade groups. Sped up 2x.

What a System 1 Model Does

It doesn’t write text at all. You send one request with some shared material (here, stills from the film) plus a list of questions. Each question has a fixed set of possible answers: yes/no, pick one of N, or a score on a scale. Every answer comes back as a probability.

TypeSafe hasn’t published how Jev works inside, but it behaves like a model trained to give calibrated probabilities instead of generated replies. The images are billed once per request no matter how many questions you ask about them, so asking thousands of questions in one go is cheap and fast.

The Experiment

I used Tears of Steel, the Blender Foundation’s open short film. Shot detection found 124 shots once titles and credits were removed. Jev got one still per shot, the middle frame, and never saw the edit order.

The usual way to do this is to run every frame through an image embedding model, then cluster the vectors. That works, but you can’t easily tell an embedding what “same” should mean. Here the question itself is the knob.

Pass 1: Scenes

Jev answered one yes/no question for every pair of shots, 7,626 of them:

Are shot A and shot B part of the same scene? Count them as the same scene only if they share the same location and the same lighting conditions: interior or exterior, time of day, weather and quality of light all match.

Each pair comes back with a probability. A standard clustering step then merges the shots Jev rates as likely the same, and a cutoff decides how sure it has to be.

The result I didn’t expect: it reassembled intercut scenes from across the whole film. The bridge argument between Thom and Celia keeps returning between other scenes, and Jev pulled all 30 of its shots into one group, from 0:25 to 8:37. The lab and Barley’s room came back together the same way.

My first wording on an earlier test asked about “the same location during the same stretch of story time.” It merged day and dusk scenes at the same location. Rewording it to stress lighting split them cleanly. No retraining, no re-tuning, just a better question.

Pass 2: Grade Groups

Within each scene, Jev answered a second question for every pair, 903 of them:

Could shot A and shot B take the same color grade? Answer yes when both shots look at the same subject from the same camera angle, so the background, the direction of the light and the skin tones match. Shot size can differ if the camera is on the same side and axis.

Clustering those answers splits each scene into groups one grade could cover. On the bridge, Celia’s seven close-ups form one group and Thom’s close-ups another: the two sides of a shot/reverse-shot conversation. On the sniper’s rooftop, the sniper, his scope view and the old man each get their own group.

See It

The board opens in edit order. Click Group shots for grading to watch the shots fly into scenes and grade groups. Drag the sliders to change how sure Jev has to be, or click a shot to see its probability against every other shot.

Open the board full screen

Cost and Time

PassQuestionsCostTime
Scenes7,626$0.0685.3 s
Grade groups903$0.0112.5 s
Total8,529$0.0797.8 s

Times are wall-clock, including uploading the stills.

How Good Is It?

I haven’t hand-labeled Tears of Steel, so judge it by eye on the board. On an earlier test with an 84-shot trailer that I did label, it never put shots from different scenes together and found 30 of the 34 same-angle pairs. Its main disagreement with my labels was merging wides with close-ups shot from the same side, which is arguably fine for grading.

One Pass or Two?

Two passes is just where I landed while experimenting. On that trailer, asking only the grade question across every pair gave the same grade groups as scenes-then-grades, because the grade question already covers background and light. Where two passes still earns its place is scale: pairs grow with the square of the shot count, so a full episode or feature wants a cheap scene pass first.

Caveats

Image input is experimental on Jev, and TypeSafe says it isn’t production-ready. This uses one still per shot. The real test of grade groups is whether one grade actually works across each group, which takes a colorist, not a model.

What’s Next

Jev runs off-site, but open-weight decision models with image support appeared this month. Cloudflare’s Clef and Clef-flash are Apache 2.0, accept images and use the same API as Jev, so they could run on your own hardware. Running the same two passes on one of those is my next test.

Stills from Tears of Steel, (CC) Blender Foundation | mango.blender.org, licensed CC BY 3.0.



Previous Post
I Built a Travel Map an Agent Fills From Your Old Emails
Next Post
I Built a Visual Fallback for Failed Conforms