How Shot Detection Finds Chapters Without a Model

Video chapters look like a job for a model. Something has to watch the clip, work out where one section ends and the next begins, and name them. Two of those three things are true, and the middle one is not a model's job at all.

Chapters and Scene Cuts finds shot boundaries with a measurement that predates deep learning by decades, runs in a few seconds on any device, and downloads nothing. It is worth understanding exactly how, because the algorithm's shape tells you precisely when to trust its answer and when to look harder.

The measurement

Sample roughly one frame per second, to a ceiling of 120 samples, so anything longer than two minutes is sampled more coarsely than that. For each sampled frame, bucket every pixel by colour into a three-dimensional histogram: sixteen levels each of red, green and blue, so 4,096 buckets, each holding the fraction of the frame's pixels that fell into it. A frame is now a list of 4,096 numbers that sum to one, and two frames can be compared without looking at where anything sits in the picture.

Compare consecutive frames with a chi-squared distance. For each bucket, take the difference between the two frames' values, square it, divide by their sum, and add it up. Halve the total. Identical frames score zero. Two frames with no colours in common score one.

A score above the threshold is a cut.

That is the entire detector. The default threshold is 0.2, and the sensitivity control on the tool moves it.

Why a histogram rather than a difference of pixels

The obvious alternative is to subtract one frame from the next and look at how much changed. It performs badly, and it fails in the direction that matters most.

Subtracting pixels measures motion as well as cuts. A locked-off shot of someone talking scores near zero, which is correct. Pan the camera across the same room and every pixel changes, so the frame difference spikes and the detector marks a cut that no editor made. The colours in the room did not change; only their positions did.

A histogram throws position away deliberately. Pan across a room and the distribution of colours barely moves, so the chi-squared distance stays low and no cut is reported. Cut to a different room and the distribution changes wholesale. The measurement is blind to exactly the thing that was producing false positives.

The guard that stops one shot becoming thirty chapters

A histogram is not blind to everything. A camera flash, a hard lighting change, or a fast cut back and forth between two angles all produce genuine, large distances, several seconds in a row.

So the detector carries a minimum scene length: once a cut is accepted, no further cut is considered until enough frames have passed. The floor is two sampled frames or five per cent of the clip, whichever is larger. A thirty-second clip cannot produce a chapter shorter than about two seconds; a ten-minute one cannot produce one shorter than about thirty.

This is a real trade. The guard is why a strobing sequence produces a handful of markers instead of sixty, and it is also why two quick cuts inside the same second collapse into one. Both behaviours come from the same line.

The three ways it is wrong

A dissolve is found late, or not at all. A cross-fade changes the colour distribution gradually over twenty or thirty frames. No single consecutive pair jumps far enough to clear the threshold, so nothing fires. Drop the threshold far enough to catch it and the detector starts marking camera moves instead. This is a limit of the measurement, not a tuning problem.

A cut between visually similar shots is missed. Two angles on the same person in the same room share a colour distribution almost exactly. There was an edit; there was no change for the detector to see.

A lighting change reads as a cut. A light switched on, a cut to black, a car headlight sweeping past. The colours genuinely changed wholesale, so the measurement is correct and the interpretation is wrong.

All three are visible in a second on a thumbnail strip, which is why the tool shows one rather than handing over a list of numbers. The review is not decoration; it is the part of the process that has judgement in it.

The timestamps are measured and the titles are not

The tool numbers its chapters. Chapter 1, Chapter 2, Chapter 3.

It would be trivial to generate something that reads better. It would also be a claim about the content of a video that nothing in this tool measured. There is no vision model, no transcript, and no language model anywhere in the path, so any title beyond a number would be invented, and an invented title next to a measured timestamp is worse than a number, because the number is honestly boring and the title is confidently wrong.

Type over them. The detector found the seams; you know what is inside them.

One rounding decision worth stealing

A boundary measured at 12.7 seconds is written 0:12, not 0:13.

Rounding to the nearest second puts the marker 0.3 seconds inside the shot that follows, so the chapter opens on a frame belonging to the previous one. Every viewer who clicks that chapter sees a flash of the wrong scene first. Flooring keeps the marker on or just before its own cut, which is always the safer side to be wrong on.

What YouTube will not tell you

A chapter list in a video description has three requirements, and YouTube enforces all of them by doing nothing. Fail any one and your description renders as ordinary text with no chapter menu and no explanation.

The first timestamp must be exactly 0:00. There must be at least three chapters. Every chapter must run at least ten seconds.

The tool forces the first line to 0:00 regardless of where the first boundary landed, and it checks the other two before you copy anything, naming whichever one failed. A clip with two long shots cannot have YouTube chapters at all, and it is better to learn that in the tool than from an empty menu on a published video.

The WebVTT chapter track has none of those constraints, and players that read chapter tracks accept it directly.

When a model would actually help

Not for finding the cuts. A learned shot-boundary detector improves on dissolves and on similar-looking cuts, which is a real gain, and it costs a download plus the inference time on every frame. For hard cuts, which are the large majority of edits, it would agree with the histogram.

Where a model earns its keep is the part this tool refuses to fake: naming what is inside each chapter. That needs to understand the picture or the speech, and it is a different tool with a different download and a different consent question. Auto Subtitles already transcribes locally, and a transcript beside a boundary list is a better way to write chapter names than any single-frame guess.

Until then, the honest division holds. The arithmetic finds the seams. You write the words.