Local Voice Acting Speed Is Set by Layers, Not VRAM

One 16 GB Card, One Full Model

We run Fish Audio Drama 3 locally: the full 35B-A3B model in a Q4 GGUF file of 24,665,234,144 bytes, on one RTX 5060 Ti 16 GB. Everything in this piece comes from our own runs on that card. We did not benchmark a second machine or a cloud service.

The question was narrow. Can a designed voice come out fast enough to be useful for a short drama, and which settings decide that? We measured generation speed, take quality and pacing, and kept only the results we could check by number or by ear.

Two numbers frame the rest of the piece. Realtime for our material needs 21.5 frames per second. Our best setting reached 22.1.

Memory was the first thing we watched and the first thing we stopped watching. VRAM stayed close to full at every setting we tried, so it could not explain the differences in speed. That left the settings as the variable worth testing.

Speed Is Set by cpu_expert_layers

Speed is set by cpu_expert_layers, not by VRAM. VRAM sat near 15.6–16 GB at every setting we tried, so memory gave us no lever. The layer count did the work.

At 8 layers the same work ran at 2.5 frames per second, too slow for an editing loop. The curve is not monotonic. Past 20 layers the output got slower again, so the fastest setting is a point to find, not a number to push upward.

If you tune this on your own card, sweep the layer count and read frames per second at each step. Changing memory settings did not move the number in our runs.

Design the Voice, Then Clone the Fix

We design voices from a text description. We do not clone them as a starting point. Each design runs six seeds. We listen to the six, pick one by ear and lock it as the reference. The seed batch is the unit of work: one description gives six candidate voices, and a person chooses among them.

Accent-heavy and 'husky' design tags raised hiss. In one design, five of six takes hissed. Our working rule now is to design wide and then clone to refine. An accent swap also held pitch: it stayed at 165–169 Hz across 12 of 12 takes, so an accent change did not drag the voice out of its register.

The refinement step is where cloning helped. In one test, we cloned a good take as the reference and tagged only the fix. That gave 6 of 6 clean takes. The chosen take scored 1.92 on our crude nasal proxy, against 2.73 for the source take. The proxy is crude, so read it as a direction rather than a verdict. One test is also one test.

Design wide, then clone only to correct one thing. Tagging only the fix keeps the reference that already worked and changes the single flaw. Tagging everything again starts the search over.

Filter Takes by Measurement First

Every take is filtered by measurement before anyone listens. We drop takes with dropped words, pitch outside the target range, or hiss above about −52 dB. Ears come last, so a person only hears takes that already passed the numbers.

The failure rate is why we never judge a batch on one take. Measured against these filters, one to two of six seeds failed per character. That leaves four to five candidates per character, and the ear chooses among those.

Laughs are the exception. Whisper does not transcribe them, so a transcript-based check says nothing about a laugh. A person has to judge it by ear.

We also tested the pruned drama3-small. Its gaps measured about −51 dB of hiss, against −57 to −60 dB on the full model. We deleted it. Speed was never a VRAM problem for us, so the smaller file traded clean audio for nothing we needed.

Pacing Tags Overshoot

Pacing tags overshoot, so use one pacing word and no pause tags. A 48-word read with no pacing tag came out at 151 wpm. Add a slow tag plus pause tags and the same read fell to 36–81 wpm. Our target is 110–130 wpm.

The tags overshot in both directions. Untagged, the read ran too fast for a drama line. Tagged, it ran too slow. The target sits between those two results, which is why we ask for one pacing word and nothing else.

Stage directions have a sharper trap. A stop word is obeyed as a stop: a direction naming '[breaks off]' was treated as a stop and cut the line. We write the effect without the stop word, so the line keeps its ending and the engine does not end it early.

Pacing is the part of the pipeline where a plain instruction was least reliable. We treat every pacing tag as a hypothesis to check against the read, not as a setting we can trust.

Locks Are Model-Bound, and the Licence Has Terms

A voice lock belongs to one model. The same seed and the same description produced a different person on the small model and on the full model. A reference locked on one model is not a reference on another. If the model changes, the locked voices have to be re-checked and re-locked.

The practical rule is to keep the seed, the description and the model version together in the project file for every locked voice. A lock without its model is not reproducible, and a reproducible voice is the whole point of locking one.

The Fish Audio Community License sets the terms we work under. It is free while revenue stays under USD 500k a year. It requires 'Built with Fish Audio' attribution, and it bans cloning real people without consent. Check the attribution requirement before any voice from this pipeline is published.

Designing voices from descriptions keeps the pipeline on the designed side of that line. Any reference we clone from is one of our own designed takes, and the licence's ban on cloning real people is the reason we would stop there.