There are good benchmarks for generating video and almost none for editing it. That gap matters, because the two jobs are not the same job. Generating asks a model to invent every pixel. Editing asks it to take footage somebody else shot and decide what to keep, where to cut, and how to make the pieces agree with each other.
So we built CutBench: nine families of editing task, media generated from a fixed seed so the right answer is known by construction, and scoring that happens on the rendered file rather than on the commands a model claims to have run.
Then we ran nine current models through it, and the headline finding is a negative one. On these tasks, at this sample size, the models are statistically indistinguishable. We are publishing that rather than the ordering, because the ordering is not real, and we nearly made the mistake of believing it.
01 / FIELD NOTESWhy editing needs a different benchmark from generation
HeyGen's Code2Video is the closest reference point. It hands a model a creative brief and a deterministic rendering environment, then scores the result on five axes — engagement, prompt-intent, composition, temporal, craft — entirely through pairwise human-preference Elo, using a judge model trained on screened annotators.
It has to work that way. When a model invents every pixel there is no right answer to compare against. HeyGen report that scoring against their own human reference gives near-zero for almost every model, which is why they moved to pairwise comparison instead.
Editing is the opposite situation, and that is the opportunity. If you construct the footage, most of what editors care about has an exact answer. Where a cut lands relative to a beat is a number of milliseconds. Whether dead air was removed is precision and recall against a known list of silences. Whether dual-system audio is in sync is the error against an offset you applied yourself.
None of that is taste. So CutBench is objective first, and sends only the genuine residue — shot choice, rhythm, feel — to preference judging. The measurable parts get measured.
02 / FIELD NOTESTruth you can construct
Every fixture is generated from a fixed seed with numpy and ffmpeg, and ships with a manifest of what it contains. A drum loop knows its own beat and downbeat times. A talking-head take knows exactly where its silences are. Two camera angles of the same moment know the offset between them, because we applied it.
The rule we inherited from the rest of our test suite is the one that makes this work: a fixture whose right answer lives only in a test's head is not a fixture.
The variants deliberately include the cases most likely to catch a direction error — a negative sync offset where the second recorder started later, a subject travelling right to left, a colour grade that goes cooler and darker rather than warmer and brighter. A fixture set that only ever leans one way cannot tell a model that understands the problem from one that has memorised the common case.
And the fixtures are verified against themselves. A separate pass measures the generated media back using methods that share no code with the generator — cross-correlation for the alignment fixtures, onset detection for the beat grid, column analysis for the pan. If the manifest and the media disagree, the fixture is wrong and the run is abandoned. A benchmark whose ground truth nobody re-checks can be confidently backwards with nothing to show it.
03 / FIELD NOTESThe headline metric is not craft. It is whether the file is usable.
On 22 September one of our own agent runs applied 53 of 54 commands. The ledger recorded 53 applied. The render job reported COMPLETED. Every colour command returned ok: true.
The exported video was black for 85 of its 92 seconds.
The agent had written brightness values of 0 to 5 on a scale where 100 is neutral, crushing almost every clip to nothing. Nothing in the stack noticed, because the document genuinely had changed — so even the guard that catches commands which change nothing stayed quiet.
A benchmark that scored that run on command receipts would have called it a near-perfect pass. That is the failure CutBench exists to make impossible to score well on, so every task is gated on the rendered file: does it have picture, sound where required, the requested runtime, and no silent damage. Fail the gate and the task scores zero on everything else, however elegant the edit decision list looks.
The gate is calibrated against those two real exports. Both are 92.00 seconds, 1920x1080, identical mean audio level, and both were reported COMPLETED. The only thing that separates them is looking at the picture.
agent's own grade (brightness 0-5) FAIL black 88.0% frozen 85.1%
brightness corrected (~100) PASS black 0.5% frozen 0.0%04 / FIELD NOTESWhat the nine families ask for
Each family is a job editors would happily hand to someone else, and each has constructible truth. Assembly turns many shots into a cut of target length. Dead air strips the silences from a talking head. Sync lines up a separate audio recorder with camera reference. Multicam aligns three angles and cuts a program. Music cuts picture to a tempo. Reframe follows a subject from 16:9 into 9:16. Colour matches two cameras. Continuity finds planted faults. B-roll covers jump cuts.
Models are given the measurements a real editor would already have — detected clap times, a silence list, a motion track, a tempo readout — and scored on whether they use them correctly. That line matters. Sync is told where the clap falls in each recording and must work out the shift itself, including its sign. Handing over the offset would turn the task into transcription.
We learned that boundary the hard way. An early version printed every beat time for the music task, and all nine models scored a perfect 1.00 with a median error of zero milliseconds. That is not a benchmark, it is a copying exercise.
05 / FIELD NOTESThe result: no measured difference
Nine models, 22 tasks, 198 runs, zero run failures. Accuracy is the mean over the seven families that can currently be scored from the document; two families are deferred until the render pass lands.
Read the intervals, not the order. Every one of them overlaps every other one. The gap between first and second is 0.4 sigma. The gap between first and last — the widest gap on the board — is 1.6 sigma, which is still not significant.
On these tasks, at this sample size, we cannot distinguish these models from each other. That is the finding.
model mean 95% CI
gemini-3.8-flash 0.83 [0.66, 1.01]
gemini-3.5-flash-lite 0.78 [0.62, 0.95]
qwen3.7-plus 0.76 [0.59, 0.94]
qwen3.6-flash 0.73 [0.54, 0.91]
nemotron-3-super-120b 0.72 [0.52, 0.91]
gpt-5.6-luna 0.71 [0.51, 0.90]
deepseek-flash 0.68 [0.47, 0.90]
gpt-5.6-sol 0.61 [0.39, 0.82]
grok-3 0.60 [0.38, 0.82]
1st vs 2nd: 0.4 sigma
1st vs last: 1.6 sigma06 / FIELD NOTESHow we know the ordering is noise
The statistics say it, but there is a blunter demonstration. We first ran the benchmark with one task instance per family — nine tasks, nine models — and got an ordering that looked perfectly reasonable. Then we added two or three variants per family and ran it again.
grok-3 moved from third to last. deepseek-flash moved from first to seventh. gemini-3.5-flash-lite moved from eighth to second.
Nothing about the models changed. The first ordering was noise that happened to look like a result, and it would have been published as one. The report now prints NOT POWERED TO RANK in as many words whenever the widest gap on the board is under two sigma, because a leaderboard without error bars invites a reader to believe an ordering the data does not support — and the reader it misleads first is its own author.
07 / FIELD NOTESTwo bugs we found in our own scorers
Three different models placed the scratch audio at exactly -0.750 seconds. That is the correct answer: the clap sits at 2.000s in the camera audio and 2.750s in the scratch recording, so the clip starts at 2.000 minus 2.750. Our scorer compared against positive 0.750 and marked all three 1500 milliseconds wrong.
Six of nine models had been getting sync exactly right and scoring zero for it. The multicam scorer had the same inverted sign, which is why all nine models sat at precisely 0.30 on that family — alignment scoring zero while the rest of the task scored full marks.
The part worth dwelling on is that both self-tests passed the entire time. They passed because they encoded the same inverted convention as the code they were testing. A test written from the same misunderstanding as the implementation does not test the implementation, it agrees with it.
Both tests now state the arithmetic independently — clap at 2.000, clap at 2.750, therefore minus 0.750 — rather than restating what the scorer does. If you are writing a benchmark, assume your scorers are wrong until something that does not share their assumptions says otherwise.
08 / FIELD NOTESWhat we are not claiming
The gate has not run on these numbers. Deliverable is measured on the rendered file and the render pass is not built yet, so the accuracy figures above are ungated by our own standard. They are reported as document-level scores and labelled as such.
Two of the ten families — dead air and colour matching — cannot be scored without that render pass, so they are excluded from the mean rather than silently folded in as zeros for everybody. Captions is omitted entirely, because it needs real intelligible speech and shaped noise will not transcribe.
Claude is absent from the board because no Anthropic key is configured on the machine that ran it. We are not implying anything by its absence. Gemini appears but is not a production path for us: we use it for video understanding, never as a planner, and it has a benchmark-local adapter rather than a place in our provider registry.
And the sample is small. Per-task scores are mostly zero or one, which is close to the worst variance a bounded score can have. Getting to a defensible ranking needs graded partial credit and roughly ten times the instances — not another run of this one.
09 / FIELD NOTESReproduce it, or disagree with it
The fixtures regenerate byte-identically from a fixed seed, the prompt every model receives is a file in the repository rather than an implementation detail, and the scorers are separate from the runner so a scoring bug can be fixed and re-applied without paying for the model calls again.
That last property is why we could correct the sign errors and re-derive the whole board in seconds rather than re-running 198 tasks. It is also why a disputed score can be argued about from the stored trace instead of re-litigated by re-running.
The benchmark is deliberately not scored through our own production planner. That carries thousands of tokens of house style and workflow guidance, and benchmarking through it would measure our prompt engineering while calling it a model's editing ability. Code2Video makes the same choice for the same reason: a brief and a documented environment, not the authors' internals.
Common questions
Which model should I use for AI video editing, based on this?
On this evidence, we cannot tell you. Every confidence interval on the board overlaps every other one, and the widest gap is 1.6 sigma. Anyone quoting a CutBench ranking as a reason to pick one of these models over another is reading an ordering the data does not support.
Why is the benchmark gated on the rendered file rather than on the edit?
Because a run can apply 53 of 54 commands, report COMPLETED, return ok on every colour command, and still export a video that is black for 85 of its 92 seconds. That happened to us. Command receipts describe intent; only the file describes what a person would actually receive.
Is this measuring the models or measuring Vidmoat?
The models. The prompt is a small published file, not our production planner, and the environment is our documented command schema — the same thing any developer gets from the API. Benchmarking through our real planner would measure our prompt engineering and mislabel it as model ability.
Why publish a negative result?
Because it is what we found, and because the alternative was publishing an ordering we had already watched reshuffle completely when we added more tasks. "Nine current models are indistinguishable on structured editing tasks" is a more useful claim than a ranking that would not survive its own next run.
What would make the ranking meaningful?
Graded partial credit instead of near-binary scoring, which is where most of the variance comes from, and roughly ten times the task instances. Plus the render pass, which would bring dead air and colour matching back into the mean and put the deliverable gate in front of every score.