Written on 8 October 2026. Updated 8 October 2026, with the lessons of that day at the end.

Optra Studio makes an animated series with no crew. A script goes in; the studio boards it into shots, draws every still, animates every clip, voices every line in two languages, cuts the episode and checks all of it with a vision model, frame by frame. Generation runs on our own GPUs, the checks are the approval, and the studio’s software is written and run by a team of AI agents (how we build the studio with its agents is its own post).

The hard thing about a system like this is not that it fails. It is that it fails quietly. Every stage reports success. A count says every still is complete. A cut is made, and made again. A model answers in exactly the format it was asked for. Each failure below passed every number we had at the time, and was found by someone looking at the thing itself.

We write every lesson down the day it happens. Since the studio began on 22 September we have written 1,314 of them, up to 186 in a single day, out of 1,875 in the project’s memory. Most are the same few mistakes made in new places. Here are fourteen, each with the number that exposed it, the mechanism, and the change after which the same mistake cannot ship again, at any size, in a production that does not exist yet. That last part is the rule we hold ourselves to: a bug is closed when the structure cannot ship it again, not when it stops happening, and where part of that change is not built yet, we say so. The first two were found in the production pipeline the studio was built out of, in its first days; the rest in the studio itself.

The assumption behind most of them

Read the fourteen together and most share one assumption: that what a stage reports is what ships. A file exists, so it is current. A step is waiting, so it is progressing. A model was told a rule, so it keeps it. A setting says three, so it is three. The fixes share a shape too: stop asking whether a thing is there, and ask whether it is still made from what it should be.

Where the lessons fell says the same thing. The memory store files every lesson under a topic when it is written. Of the 1,314 lessons since the studio began, 459 are about process and tooling and 376 about the checks; 43 are about the pictures and 9 about the video clips. What failed was rarely a model. It was the plumbing between the models, and the checks we trusted to watch it.

1. A copy that never followed its source

Before a clip is rendered, each shot’s voice track is copied into the renderer’s input folder, padded to the clip’s length. The copy step handed back any file that already had the right name. The episode’s copies were made on 19 September and still held the previous draft’s words on 24 September, five days later. The voice tracks had been rebuilt and were correct; nobody heard them, because the renderer reads the copy. So the first repair, rebuilding the voices, changed nothing a viewer hears, and the second, re-rendering 62 clips, came back saying the identical wrong sentences.

Mechanism: a derived file named only by what it is for (this shot’s voice) is found and reused whatever it was made from. Fix: every derived input is now named by a hash of its source and settings, so a changed source gets a new file and the old one is never found. And a property every pipeline must answer for, implemented or exempt in writing: derived inputs match their source.

2. A re-board that kept the old pictures under the new numbers

An episode was re-boarded from scratch, every scene rewritten, and the studio then reported 582 of 582 stills complete. On disk there were 584 stills, 420 of them drawn two days before the new board existed, and 0 of the 194 new shot numbers had kept their old prompt. 288 of 375 voice files were stale the same way. Both stages said success throughout.

Mechanism: stills and voices were named by shot number, and a stage skipped any shot whose file existed. A re-board renumbers everything, so every old file matched a new name. Fix: a still’s name now carries a hash of the words, the reference pictures and the model it was drawn with, so a picture is current only against the shot it was made for. The vision check had flagged 77 of 193 shots as not matching their words, and 40% reads exactly like a broken gate. It was the gate correctly seeing pictures of other shots: a high flag rate is a reason to look, not a reason to override.

3. A freshness check that compared two lists

One morning an episode was cut 104 times, about every five minutes from 06:56 to 15:07: 8 hours and 11 minutes of whole, correct cuts, each thrown away after all 287 clips had been rendered and read. Nothing alarmed: the cut stage was running every cycle, so the stall watch saw progress.

Mechanism: the check asking whether the cut was current compared the list of background beds the cut laid with the list the editor picked. Those differ for good when a scene returns (its bed is laid twice) or a picked bed is silent. The cut could never be current, so the scheduler offered it again, forever. Fix: compare by key, at the grain of what is held: each scene of the cut says which bed it chose, and the check asks each scene. Two lists are only comparable if both are made the same way.

4. A wait that nothing could end

Six episodes were given new script versions in the small hours and froze for 7.5 hours. The page showed them as work in progress; the stall watch, which counts a stated wait as not stalled, said nothing. The agent running production found it by noticing that counts stood still.

Mechanism: a new script marks the episode as waiting for a re-board of the scenes it changed. That re-board had not been switched on yet; the full re-board refused while a script was pending; nothing could end the wait. Fix: the step that ends that wait was built: a new script is re-boarded in exactly the scenes it changed. And a wait on anything the engine cannot do yet now says so in its own words, that the studio has it to build and the project has nothing to fix, so the page and the stall watch can tell it from a wait on work. The question we now ask of every step’s condition: read it out loud and ask who changes it. If the answer is this step, or something waiting on it, it will deadlock.

5. Holding every shot for the slowest one

One episode had 226 shots passed by the QA and 0 of 399 clips made, because nothing was drawn until every shot had passed, and 30 shots were still being rewritten. Most of the render cards sat idle, across every episode at once.

Mechanism: a board-wide gate is the safe-sounding choice, and it ties every shot’s progress to the slowest shot’s rewrite loop. Fix: because job names are keyed to each shot’s own content, a passed shot’s still and clip are the same whatever its neighbours become, so a shot is drawn and animated the moment its own verdict passes. What reads every shot together, the cut and the full QA, still waits for all of them.

6. Stages that said nothing when they worked

A stage that adds new story to an episode ran and ended in the same minute with no word on the page. A run that took in eight scenes, a refusal, and “nothing left to do” all read the same. When we counted, 15 of the 25 stages the scheduler can start said nothing on success.

Fix: every stage ends with one outcome line (what it took in, what it refused and why, or that there was nothing to do), and the page shows the last one. A stage that does not is on a named list, with its reason, that a test only lets shrink.

7. A fix for one symptom that made the next

Narration was missing from the cut, so we let a narration cue set its own shot’s length. The words came back, and eight cues of 19-44 words each now held one picture for 9-17 seconds, on shots boarded at 3-5 seconds: a near-freeze, because the clip that carries narration moves only a mouth.

Mechanism: nothing asked whether a shot’s length came from its picture or from a voice laid over it. The fix for “the cue is missing” created “the picture stands still”, and both passed every check. Fix: a voice laid over a picture never stretches it more than one second past its boarded length. The cue spans the next pictures, or the board gives it pictures of its own, or it is refused as a finding before anything is drawn.

8. The model does what it can, not what the words say

Speaking shots are animated by a lip-sync model (Wan 2.2 S2V). It moves the mouth and small turns of the head. Watching the first finished episode, every beat whose meaning was a body movement had not happened: people stood where the story said they walk, sit down, run, look away, pour or tip a bowl.

Fix: a check now counts a movement only in a clip that can perform it, so a movement written into a lip-sync shot’s words is never counted as seen: it would be reading something that never ships. A project can send its wordless shots to a motion model instead; that is switched on project by project, not yet for every production.

9. The image model draws every noun

A prop’s description said a small cup was “the size of an egg”. The image model drew the comparison: an egg in the cup in 4 of the 25 redrawn pictures. The still screen passed all four. It asks about counts and period, never whether something is in the picture that the words do not name.

Mechanism: a prop’s look is pasted into every shot that shows it, so one comparison noun reaches every picture of it, and removing it costs every picture again. Fix: a rule in code reads every look and every shot’s words for a likening. The rule was measured before it was allowed to act, across 3,917 live shots: “the size of” fired 7 times and 6 were real; “like a” and “shape” fired 14 times and one was real, because 13 of them were a bare “like a”, which the board uses for manner (“sounds like a question”), not form. So it reads a likening of size to a drawable thing, not every simile. The still screen does not yet ask whether a picture holds something its words do not name; that question is next.

10. A rule written in a prompt is a wish

A local model picks which sentence of the script voices each shot. Its prompt said a shot that only holds a place “is not a beat: answer nothing”. It answered a sentence for all 97 shots, including the 13 setting shots, and never once answered nothing. The three runs were byte-identical, so they were one measurement, not three.

Fix: the prompt now says what the model was already doing: a shot that holds a place takes the sentences that describe what it shows. The opposite rule is one switch in code, and it is measured before it can be turned on. The lesson holds for any model: before building anything that depends on a model taking a branch, count how often it takes it, or do the branching in code.

11. A number measured in the wrong unit

Episodes must run fifteen minutes, and some came up short, so we write new scenes for them. A scene’s length is planned in the script’s seconds. We sized the first additions with a ratio measured on the board. But the script overstates the board, and the board overstates the cut: one episode went 1,250 script seconds to 920 board seconds to 650 seconds of cut, 250 short of fifteen minutes. Sized by the board’s ratio, the new scenes would have left the cut about 85 seconds short.

Fix: we size in the file’s own unit, measured: the cut runs about half the script’s planned seconds, so an addition is written at about twice the shortfall. The engine now predicts a story’s cut from its board before any cut is made, so a story that will come up short is known before the drawing starts. The code’s own check on an additions file still asks only for the shortfall plus a tenth; holding it to the measured ratio is next.

12. A matcher that answered by order

New scenes name a place, and the engine resolves the name to one of the episode’s locations so the right picture of it is used. On one episode, 4 of 8 place names resolved to the wrong location: “the lane” became a gate in the lane, “the majlis” became the wall behind it.

Mechanism: the matcher returned the first location, in list order, whose name contained the words, before it looked for an exact match. A loose matcher that stops at the first hit answers by order, not by fit. Fix: an exact name wins, then a contained one, and two equally good answers are refused with both names.

13. A limit that a setting could not hold

Our language model’s box takes at most 3 requests in flight from us, and the setting said 3. The box still saw 20 in flight. Four processes had started with an override that put them at 56 wide together.

Mechanism: the limit was enforced per process, and an override from the environment was applied above the file in whichever direction it pointed. Fix: the limit is held across every process by operating-system file locks, which the system releases when a process dies, so there is no stale slot; an override may only lower it. Our load is mostly reading long prompts, so the cap costs little throughput.

14. New scenes that had nothing to say

The studio drops any shot over which nothing is heard: no line, no narration, no song, because a picture nobody speaks over is dead air. We wrote new scenes for an episode that ran short, and four of them were quiet ones, action only, told in long sentences. The board made shots of them, and 10 of the 36 new shots were dropped. The engine can give a dropped shot back by reading a script sentence over it, but these sentences needed 5.5-8.4 seconds of narration and the shots ran 3-6. The minutes the scenes were written to add never reached the cut.

Mechanism: two rules that are each right, that a shot nobody speaks over is removed and that a narration must fit its picture, together leave no room for a silent beat. In this studio a wordless moment is not a pause; it is a shot that will be taken out. Fix: every beat of a new scene is a spoken line or one short sentence that fits a three-second shot in both languages. The next round was measured by hand with the engine’s own fit measure before the file shipped, and passed 113 of 113. A check in code that refuses an unvoiceable beat before anything is boarded, so the measure does not rest on the writer’s care, is next.

And one from the day this was written

To check new scenes against the whole series, a local model read all fifteen scripts and listed the facts a new scene could contradict, each with a verbatim quote that code checked against the script. That held: no quote was invented. But one field asked for no quote, the things the series never shows, and two of its items were invented, including a character whose face, it said, is never seen. She is on screen in several scenes. In a read that checks quotes, a field without a quote is a lead, not a fact.

What is not solved

• 3 shots are waiting as STUCK right now: a STUCK shot is one the QA kept failing and the rewrite loop could not fix in its allowed rounds. Each one is a gap in a check or in the board, and we fix them as structure, one kind at a time.
• We size new scenes by a ratio, not a model of the cut. We know the cut runs about half the script’s planned seconds; we do not yet predict it scene by scene, so an episode can need a second round, and one needed it on the day this was written.
• Some defects are still found by watching. Every one becomes a check, measured on a truth written by eye before it is allowed to block. Until that check exists, the finished episode is where it is caught.
• The checks themselves need checking. A vision check can be confidently wrong in whole categories, so every check’s flags and passes are sampled by eye whenever its question, model or input changes.

What we would tell anyone building with generative models

• “Is it there?” is never the question. Ask “is it still made from what it should be?”, and name every derived thing by a hash of what it was made from.
• Compare by key, at the grain of what you hold. Two lists made different ways never agree.
• Read every wait’s condition out loud and ask who changes it. If it is the step itself, or something waiting on it, it is a deadlock that will read as progress.
• Make every stage say what it did, including “nothing, and why”. Silence and success look the same.
• A prompt rule is a wish until you count the branch. Do in code what the model must not get wrong.
• A limit is real only where it is enforced. Find the place that enforces it, not the place that states it.
• A bug is closed when the structure cannot ship it again: in the smaller case, in productions not built yet, and in the thing the viewer actually sees.

Added on 8 October

On 8 October, after the post’s previous update, we wrote down 31 more lessons. The 23 below tell 24 of them; the other 7 stay in our own notes, each with the reason it is not here. We add each day’s lessons here at its end, newest first. Each says what happened and the number that showed it. Fix is a change the code now holds, and this post’s build refuses a fix it cannot find in the code; The rule we keep is how we work, not yet code; Still open is not built yet.

One waiting read moved two cards for hours

Two of our render cards are lent to the vision checks when the checks back up. The rule counted every check waiting anywhere, so a single read that had waited 60 seconds took both cards off the episode in production, and they stayed on checks for hours: with 15-16 other episodes’ reads in the queue, the way back (no read waiting for a while) could never come true.

Fix: The demand is measured on the episode in production alone, and only the same read waiting at every look across 5 minutes counts as a backlog; the way back is none of its reads waiting for 3 minutes, whatever the other episodes do. A sample taken before the rule existed is no measure at all, never a clear.

Green tests on a project shaped unlike the real one

A rule decided which pictures of an episode the checks should read before its cut. It passed seven tests, all on a test project that had a record brought in from the old pipeline and nothing else. The episode in production had that record and a cut made by the engine since, and the rule never offered its stills to the checks at all.

Fix: The rule asks whether the engine has cut the project since anything was brought in, and the tests carry both states. Before a test project stands for a live one, we list the live one’s states and give the test project every state the rule branches on.

A pass from a check that had not asked

An episode’s loop re-picked 12 of its 13 shots whose faces had failed, each to a take the full QA had never read. The still screen had passed those takes, but the screen does not ask the full QA’s blocking questions, so its pass meant only that nobody had asked yet.

Fix: A pick of a take the full QA has not read is provisional: its clip and the cut wait, and the full QA reads it within 1-2 minutes. Not yet asked is never passed.

The same answer worked out again from nothing

The process that builds each project’s page spent 17.8 seconds of a 136-second build asking the whole board gate again, from a cold start, about a board the engine had gated a minute earlier. The same checks cost 0.3 seconds in the engine’s own process. Timing each check found it; a guess had not.

Fix: The engine hands its verdict to the page builder beside the storyboard. When it gives none, the gate is asked as before, under a label that says so, so the slow path cannot quietly become the normal one. On the next build that step took 0.0 seconds.

A fairness limit only one side obeyed

Calls to our language model take turns by priority, and a low-priority call that had waited long enough was meant to stop yielding. It stopped on its own side only: the higher call still saw nobody ahead of it, so both went for the freed slot at once, and the test of the limit failed 3 of 10 runs.

Fix: A call past the limit counts as the top priority in the one order every waiter computes, so the higher call yields back, and ties go by arrival: 0 failures in 30 runs. A fairness limit belongs in the order everyone computes, never in a rule one side applies to itself.

A test that read an improvement as a break

Two tests held that a stage owed nothing claims no work left, by asserting that it said nothing at all. The day every stage learned to end with a line saying what it did (the sixth mistake above), those tests went red on the live tree, though the behaviour they guarded still held.

Fix: They assert the property now, that no work is claimed left, through one helper that accepts a silent stage and one that says nothing is left. A test pins what a stage must not claim, not the absence of a message.

Twins counted as separate evidence

A picture is drawn from a shot’s words and a take number, so shots with identical words draw the identical picture at each take. A set of 54 rows judged by eye held only 37 distinct wrong pictures and 10 distinct right ones, so a rate counted by rows weighed some pictures several times, and a fix measured on one shot’s redraw said nothing about its twins.

The rule we keep: A held-out set is counted by distinct picture, and both counts are reported. A redraw meant to change twins changes their words or their take, or it changes one picture.

A retry loop that would turn a network blip into a redraw

To save slow restarts, the still screen was made to repeat its pass within one run while a second opinion was still owed. A reviewer saw the hole before it bit: when the model does not answer at all, the repeats would spend all of a picture’s tries in seconds, fail the take and pay for a redraw on the render cards, where the old pacing of one ask per run let an outage of a few minutes cost nothing.

Fix: A pass in which any ask got no reply ends the run’s repeats; only a reply the reader could not read is asked again at once. A retry loop repeats only on a failure that more asking can cure, never on one that only time cures.

Still open: The counting screens ask the model by another path, which is not yet tallied this way.

A busy lock read as a refusal

A press on the page was refused because the queue’s lock had been held by another writer for 60 seconds: approving a board re-hashes it under that lock for tens of seconds, and many episodes were approving at once. The press had broken no rule. It had only waited.

Fix: A busy lock is a wait now: the request stays in hand and is tried again every 20 seconds, up to 12 times, before the page says, in words, that the lock was busy.

Still open: Why an approval holds the lock that long is not yet fixed.

A guard that forgot itself at every hand-over

The page builder rebuilds a project’s page out of turn when its board jumps, behind a guard so one project cannot take every turn. The guard was one timer for all projects, kept in the process’s memory. So the episode in production stood fifteen minutes behind its own new board while another episode’s jump held the guard; and since the builder hands over to a fresh copy at every code change, each fresh copy forgot the jumps already made.

Fix: The project served first has a short guard of its own and does not spend the others’. Every jump goes through one function that writes it to disk, and each new copy reads it once. A guard’s test now hands over between its steps, or it proves the guard only inside one process.

A ratio that carried an old board

New story was sized with a ratio of cut seconds to script seconds (the eleventh mistake above). Asked whether the episode in production would now reach its floor, the ratio said about 861 seconds, short of it. The cut’s own code, run on the voices already made, said 1230.5 seconds in one language and 1242.8 in the other, well above the floor. Checked first on the last cuts’ recorded shots, the same count matched the real cuts to a tenth of a second. The ratio had missed about 203 shots the board had gained since it was measured.

Fix: Whether a cut will reach its floor is counted before the cut is made, by the cut’s own code on the voices already made, after it matches the last cut’s recorded shots. The ratio is kept for when no voices exist yet.

A store that turned “not said” into “none”

An episode’s line on the page lost its words about the episode’s floor. The store wrote an empty floor on every film entry, including entries from an older feed that never sent the field, and the page read a stored empty value as an answer: this project has no floor. The page’s test was green because its stand-in handed the page a film with no field at all, a shape the store never produced.

Fix: A field with three meanings (a value, none, not said) is stored with all three: the key is written only when the feed sent it, and the test that watches it fail is on the store, not only on the page. In the same line, two writers rounded one length two ways, 3:46 beside 3:47; they round the same way now.

A version read from words that had already dropped it

The page states which storyboard an approval was for, read from the approval’s own event. But the page translates every event into plain words when it loads, and a board’s version becomes “the storyboard”, since the studio never shows a hash. By the time the line was drawn, the version it needed was gone, and the line said nothing.

Fix: The version is kept as a field of the event before its words are translated, and the line says whether the approval was of the current storyboard or an earlier one. Anything read later from a string is taken before the page rewrites that string.

A judgement attached to whatever held its place

A reviewer judged 54 pictures by eye and asked the page to keep each one as a check case. The form sent only the take number. The requests waited in line 20-40 minutes while the engine rewrote and redrew some of the shots. Six cases were kept with a verdict, and its words, on a picture nobody had judged.

Fix: A request that records a judgement names its picture by the still’s own name, resolved against the board the person was looking at; a take the board no longer holds is said, and nothing is sent. A take number is a position, not an identity.

The loudest writer took the whole window

The page’s activity list was the newest 150 events of the account. One evening the agents’ check-case records were written by the hundred, filled most of that window, and crowded out the films’ own lines, which are what the list is for.

Fix: The records are a read of their own, with their own limit and their true count, and the activity is everything else; what counts as a record is defined in one place. On a test database of 200 records and 20 film events, all 20 film events come back, where the old read returned 0.

A second definition that silently replaced the first

A helper was added to a module of thousands of lines under a name the module already used, 3000 lines below the first. Python replaces the first definition without a word, so every full page build failed for about 13 minutes, and no test reached that path.

Fix: A fast guard over every tracked Python file refuses a top-level name defined twice in one module and names both lines; it was watched to fail on the commit that broke. Whether a name is taken is a guard’s job, not a reviewer’s.

Three launchers and one log

Three launchers can start the page builder. Each named its log by the second it started and opened it fresh, so two launches in the same second wrote over one file, and the copy that won kept no line of its first round. The page stood 8.2 minutes old, and the log could not say why.

Fix: A launch takes a claim before any copy starts, so there is one launcher per hand-over; each launcher writes a log of its own, with its first line, before the copy runs; and the builder says each minute which step it is still on. The alarm’s threshold was replayed on the day’s 575 page builds before anyone touched it, and it stayed.

A hold that arrived after the work it meant to stop

A project’s owner wanted to see its stills before any clip was drawn, and an agent watched the page to press Hold when the stills were done. The farm drew all 21 stills and then six of the seven clips within minutes, while the page’s count trailed behind; the Hold was pressed after the clips it was meant to stop, and minutes later it was still only queued.

The rule we keep: A hold is pressed at the first sign of the stage before the one to stop, and a hold is reported only when the page says it is done; queued is a request, not a hold.

Still open: The structure is a gate the farm keeps itself: no clip is drawn until the owner has seen the stills.

A pause that paused nothing

When a picture on a customer’s project failed a safety question, the engine published a safety pause, and the page showed the project as paused. No step of the scheduler read that field, so the farm kept drawing every other shot.

Fix: The pause sets the project’s own hold, the one hold every step already honours, and only a person’s release, in their words, lifts it; a new picture that fails holds it again. Of every “paused” on a page we ask which step reads it.

The terms kept a number the price table had changed

The free allowance moved from 60 to 300 seconds in the plan table that billing reads. The terms still said 60 seconds as typed words, a test still pinned the old line, and the live tree went red, which stopped every other agent’s release until it was fixed.

Fix: The terms name the allowance by a placeholder that the page’s build fills from the table, and a test compares the built page with the table and refuses any placeholder left unfilled. A number the customer is told, typed beside the table it comes from, is a second copy.

Every agent releasing, and a release of the wrong branch

Every agent released its own work, and with sixteen of them the line of releases became hard to follow. The same night, a release run from another agent’s working copy shipped whatever that copy held at the moment it started: its owner had just switched it to new, empty work, so a fix for a failing check did not go live, and the release still reported success.

Fix: Releases go through one agent that does nothing else. The others commit, push and hand it a ready line; it runs a smoke test and releases when the test passes.

Still open: The release tool refusing any branch but the one it was asked to ship, the release agent working from a copy of its own, and its confirming that the branch is in the live tree rather than trusting the exit code, are next.

A test loop running yesterday’s code

The full test suite was meant to run on a separate test server, and that split had landed hours earlier. It was still running on the development machine, 1-3.5 hours a run, because a loop started the night before ran the code it had imported at its start, and held the port every newer launch needed, for 22 hours. On the server the same suite takes 187 seconds.

Still open: The loop was stopped by hand. A loop that ends and starts again when its own files change is being built.

A yes that had to quote the text

A screen we are building asks a model whether a customer’s script holds content we refuse, and a yes must quote the script, checked by code. The parser of its replies swallowed a line break and read the next line as this line’s quote, so 17 of 36 test texts came back unread on the first run. And asked for a quote, the model sometimes quoted the question’s words instead of the script’s.

Still open: The fix is built and measured, and goes live with the screen: the parser reads only within its line, and a yes without a quote is asked again, up to 3 times, naming the line, then held with its reason.

The studio as of this writing: 1,875 lessons in its memory, 8,836 automated tests, 51 properties every production must answer for, and 16 agents building and running it.