nemotron-halloween

What it claims

MonsterHouse is a talking garage-door character built by an NVIDIA employee on the NVIDIA Nemotron Voice Agent, running off-grid on one workstation. The author chose Nemotron 3.5 Bolt E4B (4-bit quantized release) after testing more than a dozen models. They report a smaller footprint than the other frontrunners, Qwen3 30B-A3B and Qwen 3.6 35B-A3B. The author had pre-release access to Bolt E4B. The MonsterHouse code itself is not released.

How it works

Nemotron Speech ASR (nemotron-speech-streaming-en-0.6b.q8_0.gguf) transcribes visitors. Bolt E4B writes the reply and emits an emotion tag that selects one of seven moods. Magpie TTS Zeroshot voices it from a 3-10 second reference sample per mood, recorded through a "Sample your Voice" UI block. Speakable text is released before the full answer finishes. Two shortcuts bypass work: a text bank of authored lines skips generation, and cached recordings skip synthesis.

Clean TTS speech drives Audio2Face-3D in Unreal Engine 5. The audience hears a second stream with real-time "demonic" effects. Audio2Face-3D's default weights work best on clean, neutral, mid-pitch voices, so effects are added after animation, not before. A 100 ms offset compensates for Audio2Face-3D buffering.

flowchart LR
  A[Visitor speech] --> B[Nemotron ASR]
  B --> C{Cached line?}
  C -- yes --> D[Saved audio]
  C -- no --> E[Bolt E4B + LoRA]
  E --> F[Magpie TTS]
  D --> G[Clean speech]
  F --> G
  G --> H[Audio2Face-3D / UE5]
  G --> I[Vocal effects to speakers]

The persona comes from a LoRA trained on more than 5,000 multi-turn conversations. These were generated and judged by a very large LLM, and a local agent trained and benchmarked them. Each LoRA took one to a couple of hours to train. Encounter state, such as whether a visitor said "please", lives in application logic rather than in the model. Nemotron 3 Nano Omni describes costumes, and YOLO plus ByteTrack count and track people so Omni is not asked to narrate every frame.

Evidence

The median end-of-speech-to-audio time is 1.70 s with Bolt Q4 and live TTS, and 0.72 s for cached responses. Both include a 300 ms silence wait for end-of-speech detection. Hardware is split by compute, not memory. The LLM and voice models run on an RTX PRO 6000 Blackwell Max-Q. Unreal Engine and Audio2Face-3D run on an RTX PRO 4000 Blackwell, so frame rate stays steady. The vision test used a synthetic video of five costumed sheep, and Omni labelled pirate, vampire, witch, pumpkin and ghost outfits.

How to use it

The author's suggested path uses only the public pieces:

1. Clone the Nemotron Voice Agent and run the Generic Assistant example. It includes ASR, LLM, Magpie TTS, VAD and end-of-utterance handling.
2. Use the Voice Agent's Bolt E4B profiles (cloud, server, single-GPU). Use conversational mode and disable reasoning for simple exchanges.
3. Start with a system prompt and example dialogues. Train a LoRA with the NeMo AutoModel recipe for Bolt E4B when prompting falls short.
4. Swap the default Magpie TTS Multilingual for Magpie TTS Zeroshot, which requires an access request.
5. Optionally add the Nemotron Omni Assistant Subagents example for webcam input.

For low compute, the author proposes ASR plus a Python router over pre-generated cached audio, with no live LLM or TTS.

Limits

The system has not been tested with real visitors, driveway groups or night lighting. The vision result is synthetic only, and the author expects costume identification and retorts to need post-training. The Magpie Zeroshot access path is left as an unresolved "[CONFIRM]" placeholder in the post. The model comparison gives no quality or latency numbers for the Qwen baselines.