QA for AI Video: Catching Defects Before a Client Does

The 13-second local smoke detector

Our local QA runner cuts a clip into 2-second segments by default, samples 4 frames per segment, and gives each segment a PASS or FAIL. Its exit code is 1 when any segment fails, so it works as a gate in a shell script.

A 10-second clip splits into 5 segments and sweeps in about 13 seconds, with zero tokens and zero dollars. The footage never leaves the machine.

Automating quality assurance on a local workstation eliminates cloud API costs and keeps unpublished commercial assets secure. Before any synthetic clip enters the editing timeline or client review portal, running a rapid local check ensures that catastrophic generation glitches are caught immediately. By integrating the sweep as a scripted pre-commit check, bad takes are rejected automatically.

When scaling automated AI video generation across dozens of candidate scenes, human review quickly becomes the primary operational bottleneck. Editors spend hours scrubbing through hundreds of seconds of footage just to identify obvious rendering failures. A local smoke detector weeds out damaged takes in seconds.

Why polite QA prompts fail on severe artifacts

A polite prompt ('DEFECTS: ...or NONE') answered PASS on a frame with a huge lime-green box over the logo. Asked directly whether any solid colour block covers content, the same model on the same frame described the box and said FAIL.

Vision-language models exhibit a strong compliance bias toward approving imagery when asked open-ended or courteous evaluation questions. When presented with a prompt that gently inquires whether defects exist, the model tends to rationalize bizarre visual artifacts—such as bright geometric overlays or corrupted textures—as avant-garde stylistic choices or intentional graphic design.

To achieve reliable automated inspection, evaluation prompts must abandon neutrality. They must aggressively interrogate the image for specific failure modes, explicitly instructing the model that abnormal geometric occlusions represent structural defects.

Vision models trained with reinforcement learning are conditioned to be helpful and constructive, often praising the artistic composition of an image even when a bright lime-green rectangle covers the center of the frame. Removing subjective language from the evaluation prompt forces the model to perform objective forensic inspection.

Taxonomy of a defect-hunting prompt

The prompt that held names the artifact classes: solid colour blocks, torn lines, corrupted regions, missing textures, warped anatomy, extra fingers. It states that a solid bright rectangle over content is always a defect, never a design choice.

By establishing an explicit catalog of physical and digital rendering anomalies, the model's visual attention is directed toward specific edge cases. Rather than formulating a subjective aesthetic critique, the vision model functions as an automated checklist validator.

Demanding that solid colored boxes or garbled anatomical features be classified as immediate failures removes ambiguity, transforming subtle visual hallucinations into clear actionable signals.

Providing explicit categories of defects prevents the inspection model from overlooking subtle errors. When told to watch specifically for warped anatomy, corrupted line work, or solid color blocks, the model's visual attention aligns with actual human quality control standards.

Standardizing artifact taxonomy also allows engineering teams to track recurring model flaws across library updates. If a specific base model consistently generates distorted hands or blurred logos, the test logs provide empirical justification for re-tuning prompt embeddings or switching to alternate foundation models.

Comparing Gemma 4 12B and Qwen3-VL 8B

The default vision model is a QA-tuned Gemma 4 12B build. It judged 8 frames in one call in about 3 seconds. The Qwen3-VL 8B alternative silently saw only 3 of 5 frames when given several at once, so we use it one image at a time only.

Gemma answers in about 40 tokens and Qwen3-VL in about 470. Gemma uses 8.4 GB of VRAM, Qwen3-VL 6.5 GB. Gemma's concise output and stable multi-frame comprehension make it ideal for high-throughput batch evaluation in automated production sweeps.

Ollama endpoint gotchas: generate versus chat

Call the Ollama generate endpoint, not chat. On the chat endpoint these models wrote the whole answer into a thinking field and returned empty content, which looks exactly like a dead model.

API integration details often dictate pipeline reliability. In our tests with Ollama, the chat endpoint caused reasoning models to dump their entire analytical breakdown into internal scratchpads, leaving the public payload completely blank. Switching to the raw generate endpoint resolved this bug immediately.

num_predict must be at least 1200. At 500 the qwen model ran out of budget mid-thought and returned nothing; the runner reports that as INCONCLUSIVE instead of passing it. Ollama unloads a model after about 5 minutes idle, so the first call after a pause pays a reload. Inside one batch the model stays warm.

Token budget management is equally critical in headless automation. If an inspection model runs out of output tokens while formulating its internal reasoning, the runner receives empty strings, causing false test results. Setting generous token limits guarantees complete, conclusive evaluations.

Synthetic corruption benchmarks and gate integration

Validation on a deliberately corrupted clip with a lime box over 4 to 6 seconds: only segment 3 of 5 failed, with the model naming a large solid green rectangle. The clean original scored 5 of 5 PASS. We re-run both clips whenever the prompt or model changes.

Maintaining benchmark test assets is essential when maintaining local inspection software. Regularly testing the runner against known corrupted video files guarantees that prompt updates or engine patches do not degrade detection accuracy.

The rail answers 'is anything broken?', not 'is this good?'. Shippability and face-identity drift go to a stronger reviewer; the local sweep is a smoke detector, and a PASS is not a creative verdict.

Deploying an automated smoke detector creates a reliable boundary between structural integrity and creative review. Human supervisors should never waste attention identifying corrupted video files or missing textures; their focus must remain on acting nuances, color harmony, and emotional storytelling.