We make an animated series with AI, and nobody signs off a picture before it goes into an episode. One model draws it. A second model looks at it and decides whether it is good enough to animate. If that second model lets a mistake through, the mistake is in the episode. If it complains about a good picture, we draw the picture again for nothing.

For a while two different models did that looking, and they kept disagreeing about the same pictures. When two judges disagree you cannot tell which one to believe. So we put ten models through one test, on our own pictures, to see which of them does the job best. This is the record of that test as it stands. It is written for people who make video, and the pictures we tested are in it.

How an episode gets made

Here is the line a picture travels before it reaches an episode. Each step is made by a model, and beside each step is the check that has to pass before the next one may start. The model this post is about is the one doing the looking in those checks.

Why we needed to choose

An episode is hundreds of shots, and a shot is often drawn several times, each drawing a take, before one comes out right. That is thousands of pictures an episode, and image models make exactly the mistakes a viewer notices at once: an extra hand, the same person standing in two places, a face in a shot that was meant to show someone only from behind. No person can look at all of them. A vision model can: it is an AI that takes a picture and a question and answers in words.

The two we had been using were a small one, Qwen3-VL-8B, and a large one, Qwen3.8-27B. We wanted one. It also has to fit our hardware, because a model must sit whole inside the memory of the graphics card that runs it: the card that does our checking today has 24 GB, the ones we are adding have 32 GB each, and we would like it to run on 16 GB cards too. The large model does not sit well in 16 GB. We went in hoping to find something smaller that checks as well.

Two things in a model’s name matter here. The number before the B is how many billion parameters it has, which is roughly how much it can hold of what it was trained on; a bigger model usually notices more and answers more slowly. And 4-bit or 8-bit says how hard the model was compressed to save memory, a little like saving a photo as a smaller file.

What we tested

The pictures are our own. They are stills our image model drew while we were making real shows: most come from Stories of Wulfa, an animated series, across fifteen of its episodes, and the rest from a short film called Flat Nine and from one short reel. None of them was made for a test. Each was kept because, at some point in the work, it had been looked at closely and judged right or wrong.

There are 1,862 pictures. One picture can be asked more than one thing, such as whether the right number of people are in it and whether a face is showing that the shot keeps hidden, so what we count is a picture together with one question about it. We call that a case, and there are 2,392.

The questions come in four kinds.

The face question is put only to shots that claim a hidden face, which is why 152 of its 413 cases were asked. And only the first three kinds count towards a model’s score. We still ask the fourth and show its results, but we no longer let it count for anything: a picture can differ from its description in ways no viewer would ever mind, and a checker that sends those back wastes more than it saves.

The four rows do not add up to 2,392, and the gap is 23 cases the answer key sets aside. They are about a moving clip or a script, or their right answer was never settled, so no model was scored on them.

That leaves 1,298 cases that count. In 584 of them the picture really has something wrong with it, which is 45 in every hundred. Hold on to that figure. It means a checker that never looked, and simply complained about every picture, would be right 45 times in a hundred.

Who decided what the right answer was

Every case needs a right answer, and somebody has to supply it. In our studio the producers who run a show from day to day are AI agents, each working through the same dashboard a human producer would use. For about two weeks, from 25 September to 10 October, one of their jobs was this: when a check passed or failed a picture, look at the picture closely, zoomed in, decide whether the check had been right, and write down the verdict with the reason for it. Each of those became a kept case.

On 10 October at 20:24, before the first of the ten models had been asked anything, that collection was frozen as the answer key. Nobody could move an answer after seeing a result.

Our director did two things on top of that. He set the standard for the hardest question, what counts as a face showing on someone seen from behind: an eye, the nose, the mouth or the chin counts, and an ear, hair or the curve of a cheek does not. The older face cases were read again under it. And he ruled on eleven takes himself; his rulings are shown and scored on their own further down.

There is a weakness in this, and it is better said than hidden. An answer key kept by an AI is itself a judgement. The agents that kept it are a different model from any of the ten, they looked more slowly and more closely than a checker can afford to, and they had to give a reason every time. They can still be wrong, and where they are, a model that disagreed with them was marked down for being right. That is one reason the director’s own pictures get a column to themselves, and it is why his own review of every shot comes before any of this is called a result.

How a model was asked

Every model got the same pictures and the same questions, in the same words, and its answers were read by the same code. A case is not answered in one breath. For each one the model is asked several small questions (how many people it sees, whether there is any writing in the picture, whether the picture matches the shot) and the verdict is read out of its answers. One pass through the whole test is at least 9,125 of those small questions.

Three pictures from the test

Numbers are easier to trust once you have seen what they count. Here are some of the cases, with what was wrong in each and what a few of the models wrote about that very picture.

The shot asks for a grandfather holding a small coffee cup at his chest. Our own description of that cup called it “a tiny round bowl the size of an egg”, and the image model took the comparison literally: there is an egg sitting in the cup. It is a real mistake, of the kind “is something drawn wrong”. Five of the ten models flagged it. (A finjan, in their answers, is the small handleless coffee cup.)

This shot has four people in it: a grandmother, a girl in green, a boy, and one girl in yellow. The picture has five. The girl in yellow was drawn twice and stands facing herself. Seven of the ten models flagged it.

Look at the two Qwen3.8-27B rows. They are one model, saved at two levels of compression, and on this picture one of them said five people and the other said four.

Nothing is wrong with this one. The shot asks for five people in front of a crowd in a lane, and all five are there. Five of the ten models flagged it anyway.

One of them objected that the characters in an animated series looked animated. Each flag like that costs us a redraw of a picture that was fine.

How to read the scores

Two questions come out of those boxes, and anyone making video with AI has reason to care about both. When this checker complains, how often is it right? A checker that does badly there sends good pictures back to be drawn again, which costs graphics-card time and does not always give a better picture. And of all the real mistakes, how many did it catch? A checker that does badly there lets mistakes into the finished episode. Engineers call the first precision and the second recall.

Neither is worth anything alone. Complain about everything and you catch every mistake while being right 45 times in a hundred, the figure from earlier. Complain about almost nothing and you are nearly always right, and nearly every mistake gets through.

One of the models did almost exactly that. On the questions of the kind “Does the picture make sense in the story?”, Qwen3.5-9B, 4-bit flagged 371 of the 378 cases. Only 233 of those 378 had a mistake in them, so flagging every one would have been right 62 times in a hundred, and Qwen3.5-9B, 4-bit was right 62 times in a hundred. No better than not looking. It caught nearly everything on that kind by complaining about nearly everything.

What the models scored

Scored on 11 October 2026 at 05:01 (UTC+4), over the three kinds of question that count. These are results so far. The right answers they are measured against are the ones our producing agents kept, and every score here is measured again once our director has given his own verdict on every shot.

On these results the most careful checker was Qwen3.8-27B, 8-bit. When it said a picture was wrong, it was right 77 times in a hundred. But it caught only 38 of every hundred real mistakes. The one that caught the most was Qwen3.5-9B, 4-bit, with 88 in a hundred, and it got there by complaining about 56 of every hundred pictures that had nothing wrong with them. No model was good at both.

The whole table so far, for anyone who wants every figure. “Good pictures flagged” is the share of pictures with nothing wrong that the model complained about, so lower is better there. The director’s column counts, of the sixteen questions asked about pictures he passed himself, how many the model complained about. He passed some of those pictures on one point only, the hidden face, and in two of the sixteen questions the answer key finds a real mistake on another point, so a complaint there is a catch and not a false alarm. Speed is how many small questions it answered a second on the card it ran on; the cards were not all the same, so treat it as a rough guide.

The last row is not a model. It is a checker that flags every picture: it catches every mistake, and it is right only 45 times in a hundred, because only 45 in a hundred of these cases really have a mistake.

This picture takes one kind of question at a time and names the model that has done best on it so far. It uses a single score that folds catching and false alarms together: nought is no better than chance, and one is perfect.

The two plain scores for each kind, so far (when it complains, right / real mistakes caught, each in a hundred):

We ran Qwen3.8-27B twice on purpose, compressed two ways, to see whether compression changes what a model sees. It does. At 8-bit it was right 77 times in a hundred when it complained and caught 38 of every hundred mistakes; at 4-bit, 72 and 42. So whatever form of a model we end up running is the form that has to be tested.

The 8-bit one also needed two graphics cards and was hardly faster for it (4.1 questions a second against 3.8 on one card). The second card was for memory: at 8-bit its weights alone take about 33 GB, more than one card can hold with room left to work.

What surprised us, and what we got wrong

The first surprise, on the results so far, was how little daylight there is between these models and a checker that does no looking at all. Flag every picture and you are right 45 times in a hundred. The best of the ten was right 77, and eight of them were right sixty times in a hundred or fewer. Put the other way round: when a typical one of these models complains about a picture, the picture is fine about four times in ten.

The second was the story question. We had hoped a vision model could tell us whether a picture makes sense in the story: the right place, the right period, the right people. On the single score in the picture above, where nought is chance and one is perfect, the best any model has managed on that kind so far is 0.20. That is too weak to act on. So the story question is leaving the picture check. A text model will read the script, scene by scene, before anything is drawn; that check is being added now.

Then there is what we got wrong ourselves. Six of the ten models did not start the first time we tried them. None of that was the models’ doing. Each time, the software that loads a model onto a graphics card stopped before the model had answered a single question:

Every one of these ran in the end. What we took from it: a model that has never been served on a card is not known to start on it, however good its published numbers are.

We also lost some of our own evidence that night. When a model was tried again on the same card, its new run record replaced the old one, failed attempts and all, so the run records no longer hold three of the first failures above. We recovered them from what else was kept that night. Run records now keep every earlier attempt.

The last surprise came from asking twice. A checker that gives a different verdict on the same picture tomorrow cannot be the last word on anything, so two of the models were put through the whole test a second time: same pictures, same questions, nothing changed.

Gemma 4 26B-A4B, 4-bit gave a different answer the second time to about 12 in every hundred of the small questions. Its verdict on the whole picture still came out the same in 1,248 of the 1,298 cases that count.

Molmo2-8B gave a different answer the second time to about 8 in every hundred of the small questions. Its verdict on the whole picture still came out the same in 1,275 of the 1,298 cases that count.

A verdict is read from several small answers, so one of them changing does not always change it. A changed answer and a changed verdict are different things, and it took the second run to show us how different.

The pictures our director ruled on

These are the pictures that went in front of our director, with the answer every model gave. Twelve of them carry a ruling he gave himself: his eleven takes, with a take shown twice where the answer key holds two drawings of it. The rest are marked as what they are: two pictures he has not ruled on yet, and one picture that a producer passed by the same reasoning as one of his rulings. A picture is often asked more than one question, so some have several rows. A tick means the model agreed with the answer key and a cross means it did not. The ruling under a picture can be about something none of its questions asked: it says what he judged, and each row is one question the checkers were asked.

Where the models disagree

In 1,520 cases the models did not all give the same answer. Here are 26 of them, a few from each kind of question that counts, with what every model wrote. Two of its answers are shown. The first is whether the picture matches the shot, which is one of the small questions and not the whole verdict, so a model can say the picture matches and still flag it, or the other way round. The second, whether the picture makes sense in the story, was asked separately and counts in no score; every model had the same short room to answer it, and one that wrote at length was cut off and its answer not kept. The face questions are asked in a numbered form that reads poorly, so for them only the second answer is shown.

One thing to know before the pictures. A kind belongs to the case and not to the wording of its question. Where someone stands in the frame is small print in most of the cases that ask it, and it counts in the few the answer key files as a mistake of drawing or of story, which are the ones shown here.

What happens next

Nothing above is the last word. Every score so far is measured against an answer key kept by the AI agents that produce our shows, and the next step is a person. Our director is going to watch every shot himself and give his own verdict on each. The ten models do not have to run again for that, because every answer each of them gave is kept; they are scored a second time, against his verdicts, and this post is updated with what changes.

After that comes the same kind of test on moving clips, where a checker has to read every frame of a clip where here it read one picture. It has not produced a result yet.