Tutorial — Authoring a Q&A Experience with Ambient Idle

Augmented Traveler local tutorial. Back to Unreal — Tutorials · conversation system: AI Mind — Process · program: Tutorial — Authoring a Timed Experience

How to build the Q&A part of a program. It has two parts:

  1. The waiting figure looks alive: breathing, blinking, glancing, small shifts of weight, in period character, while it waits for a question.
  2. Answering: listening, a brief "thinking" beat, then speaking with lip sync, gestures and captions.

Draft. The real-time lip-sync route must be tested in a packaged kiosk build before we commit to it (section 4).

1. The Q&A states (inside the program's QA state)

Waiting (ambient idle) ──push-to-talk / wake──▶ Listening ──speech end──▶ Thinking
    ▲                                                                   │ first audio
    └────────────── answer ends ◀── Speaking (lip sync + gesture) ◀─────┘
State Figure behaviour System
Waiting Ambient idle (section 2); an occasional glance at the visitor Mic closed; "Ask me a question" caption hint
Listening Turns to the visitor; attentive face (slight brow lift); stops idle fidgets ASR streaming; a live transcript is shown
Thinking A short authored "considering" gesture (look aside, touch the chin, adjust spectacles). Never filler speech (AI Mind — Process) Retrieval + LLM; target under 1.5 s to first audio
Speaking Audio-driven face, body gesture from a gesture tag, gaze at the visitor with natural breaks Streaming TTS → lip sync; captions
Refusal / out of scope An authored, in-character redirect line (pre-recorded) Policy layer flag

Build these as a sub-tree of ST_AT_Program (QA) or as their own ST_AT_QA.

2. Ambient "living" idle

Layer the idle so it never loops visibly:

Layer How Notes
Base idle 3–6 idle Animation Sequences from mocap (standing, seated at a desk), randomly blended in an Animation Blueprint state (Random Sequence Player or a blend space) Captured on the capture days: "waiting" takes of 30–60 s
Weight shifts and fidgets Short additive clips (adjust shawl, turn a page, glance at the telescope) played at random intervals of 8–25 s Keep them period-appropriate. The historian approves the list
Breathing A slow additive on spine and clavicles (a looping additive clip, or Control Rig) Subtle at kiosk distance
Blinks and micro-expressions Random blinks (every 2–6 s, more when speaking); tiny brow or lip movements on the face rig Driven in the face Animation Blueprint or Control Rig
Gaze Look At node (head/neck) with a clamp and smooth interpolation, pointed at a target actor that moves between the visitor's head, points of interest and "unfocused" spots Visitor head position from the XR camera (headsets) or a depth camera/sensor (kiosk)
Hands Hand poses from the Pose Asset (Tutorial — Hand Pose Library for Gestures) E.g. rest_folded in Waiting, open_palm_present when explaining

Tools:

Timing rule: randomise intervals, avoid repeating the same fidget twice in a row, and never fidget while a visitor is speaking.

3. Hand-off between idle, listening and speaking

  1. Waiting → Listening: blend the head and eyes to the visitor over 0.3 s; body turn over 0.8 s. Pause fidgets.
  2. Listening → Thinking: play one of 4–6 "considering" clips, chosen at random.
  3. Thinking → Speaking:
    • start the face lip sync on the first audio;
    • start the body gesture 100–200 ms later, which looks natural;
    • the gesture tag from the conversation service chooses the clip (explain, recall, emphasis, humour, sorrow).
  4. Speaking → Waiting: finish the gesture, return the gaze to the visitor briefly, then resume the idle.

4. Lip sync for live answers: choosing a route

Route What it is Status (Sept 2026)
MetaHuman Audio Live Link Source (MetaHuman Live Link plug-in) Epic's real-time audio-driven face animation; 5.8 adds a new real-time model Official; GPU-heavy. Epic doesn't state whether it works in packaged builds. Test first
NVIDIA ACE plug-in (Audio2Face-3D) ARKit-style face curves from audio, with a MetaHuman pose-mapping asset; models are open source Supported UE versions unconfirmed; NVIDIA GPU on the kiosk
Runtime MetaHuman Lip Sync (Georgy Dev) Third-party runtime lip-sync plug-in Check the 5.8 version and licence
All-in-one character platforms (Convai, Inworld Unreal AI Runtime SDK) Speech-to-text, LLM, text-to-speech and lip sync in one plug-in Fast prototypes. Check data retention, cutoff and refusal control against the Ground Rules before any production use; 5.8 support unconfirmed

Plan:

  1. Prototype with the MetaHuman Audio source.
  2. In the same week, test Audio2Face-3D on the kiosk GPU.
  3. Pick by quality and latency in a packaged Shipping build on the target PC.
  4. Headsets use the lighter route, or pre-authored answers for the top questions.

Authored answers first. For the most-asked questions, play pre-recorded, fully animated answers (Sequencer). Generate only for the long tail (AI Mind — Process).

5. Captions and accessibility

Full style and standards: Tutorial — Captions UI. The network client is described in Tutorial — Conversation Client Plugin.

6. Practice exercise

  1. On a test MetaHuman, build an idle Animation Blueprint with:
    • 3 base idles;
    • 4 random fidgets;
    • breathing;
    • random blinks;
    • Look At pointing at a movable target sphere.
  2. Add a key press to toggle Listening (turn to the target, stop fidgets).
  3. Play a WAV through the MetaHuman Audio source and check the lip sync. Then package a Development build and check it still works.
  4. Time the Listening → Speaking hand-off and aim for no visible pops.

7. Check before a venue build

Sources