Building a Room Out of Holograms: Empty Set, Wireframe, Solid

What this adds to the hologram effect

Our first hologram test turned one object in a clip into a wireframe twin and back. It ended with a limit: on a moving camera the grid slid over the object.

This time the brief was bigger. The place starts empty. Furniture and people arrive as wireframes, then turn solid on a word in the dialogue. We built it twice:

• A locked shot: an Unreal Engine scene with a slow push-in, where a woman raises her hand and a choir appears on the risers behind her.
• A handheld shot: a 5.2 s MiniMax H3 clip, 1280x704, of a woman walking along a seaside restaurant terrace and talking to a friend's camera.

Everything ran locally except one 20-credit edit on Google Flow.

The locked shot: a choir out of nothing

In Unreal the empty version is free. We rendered the same camera three times: nobody on stage, the woman alone, everyone. Each grey render was turned into a finished frame with Qwen-Image 2.1 and then into video with LTX 2.5, and we built the stills up rather than painting people out:

1. The empty hall first. Qwen restyles the nobody render into the finished room. That still is the base for everything that follows.
2. Then the woman. We pasted the grey mannequin's pixels into the base and asked Qwen to replace it with a real person in the same place, size and pose.
3. Then the choir. The same step on top of that result.
4. Masks keep the layers identical. Each Qwen result was composited back only inside its own mask, so the room stays pixel-identical across all three layers.

Removing people from a full frame instead left ghosts and seams. Asking an inpainting model to add a person in a mask drew a thin side-on figure.

The engine also gives perfect mattes. Unreal's depth pass is in centimetres, so the difference between the woman-only and the full render, thresholded at 5 cm, is the choir's bodies with no shadows. A colour difference kept the shadows, and they turned into grid boxes on the risers. Depth for the grid came from the float EXR, not the 8-bit control video, which drew topographic rings.

Each of the four stages (Unreal, Qwen, LTX, hologram) took about five minutes on a 16 GB card.

The handheld shot: getting an empty terrace

A generated handheld clip has no second take with the tables removed. We tried two ways to make one.

The Omni edit still needed registering. After scaling, the static areas differed from the source by 15.6 on a 0-255 scale. A per-frame homography plus a per-channel colour fit brought that to 3.4. Smoothing the camera path over time brought it to 2.8 (see the flicker section).

The area to hide also needed care. The furniture masks from SAM3 miss the glasses, vases and cloth edges. When we wiped only the masks, those floated in the empty set. The fix is to wipe the region where the source differs from the empty plate, and to keep only the parts of that difference that touch a furniture or diner mask. Without that last filter, small alignment errors in the sea read as white statues.

Two layers, timed to the line

The build-up is two hologram passes on top of each other. The diners pass runs first. The furniture pass then runs on its output, so the furniture appears first and the guests after it. Word timestamps from Whisper set the timing.

The speaker is cut out of both passes with her own matte. She is never part of the effect.

The flicker, in three rounds

The first version flickered badly while things appeared, and it disturbed the woman's outline too. Our first fix aimed at the grid, and it was the wrong target.

We stopped trusting a plain frame-difference number early. It punishes a grid that correctly moves with the scene. What found the real causes was a heat map of each pixel's frame-to-frame zig-zag, minus the source's own.

One idea made it worse. Adding an optical-flow correction on top of the camera motion, to cover the nearest chairs' parallax, raised the build-up shot from 4.9 to 7.1, and still to 6.4 with heavy smoothing. The flow's own noise was larger than the parallax it fixed. We removed it.

Where we start next time

Three checks before calling it finished:

• Look at three to six consecutive frames at full resolution in each phase: empty, appearing, turning solid.
• Build the flicker heat map and look for anything bright outside the intended sweep.
• Check the speaker's outline. If she flickers, the plate or her matte is moving, not the hologram.