AI Mind — Process
Role in the pipeline: the character's conversation system. Retrieval first, generation only inside retrieved sources, with a policy layer enforcing cutoff and refusals. Follows Specification — Full-Body Historical Virtual Human §3 Stage 5.
Runtime loop
Visitor speech
→ ASR (speech to text)
→ intent + safety filter
→ authored beat match? → play authored clip (no generation)
→ else retrieval: Corpus A → B → C (hybrid keyword + vector, citation IDs)
→ LLM answer constrained to retrieved passages, in persona
→ policy layer: cutoff, refusal list, trauma protocol, companionship ban
→ streaming TTS (locked actor voice)
→ audio-driven face (MetaHuman real-time audio / Audio2Face-3D) + gesture tag
→ captions + log (question, corpus hits, refusal flags; no faces)
Components (choose per venue)
| Component | Options (2026) | Notes |
|---|---|---|
| ASR | Cloud streaming ASR or on-device model | Museum noise: use a mic array + push-to-talk |
| Retrieval store | Postgres + pgvector, or a managed vector DB | One index per figure; never a blended "Nantucket brain" |
| LLM | Hosted frontier model API; small open-weights model on the kiosk PC for offline/school venues | Keep model choice behind an interface so it can be swapped |
| Policy layer | Rules engine + classifier, separate from the persona prompt | Cutoff is enforced in code, not asked for in the prompt |
| TTS | Low-latency streaming TTS with a contracted actor voice (e.g. ElevenLabs class) | Lock the voice; log the version |
| Face | MetaHuman real-time audio-driven animation; NVIDIA Audio2Face-3D (open source) | Choose by quality on the target device |
| All-in-one character platforms | Convai (Unreal/MetaHuman plugin) and similar | Fast prototypes; check data retention and cutoff control before production |
Latency budget (target under ~1.5 s to first audio)
ASR finalization ~300 ms · retrieval ~100 ms · LLM first tokens ~400–600 ms · TTS first audio ~150–300 ms. Cover gaps with authored "thinking" idles, never filler speech.
Build steps
- Corpus ingestion with shelf marks and dated chunks (see Mitchell Corpus Inventory).
- Golden question set (400–800) with expected sources and refusal flags.
- Persona prompt from the voice dossier; policy rules from D-notes.
- Evaluation harness: run golden set on every corpus or model change; blind historian review of 50 answers.
- Staff dashboard: pin a canned answer overnight without retraining.
- Offline fallback: authored monologues only.
Owners: AI/conversation engineer, conversation designer, historian. Tutorials: AI Mind — Tutorials · start here: Tutorial — How the Mind Works · model choice: Tutorial — Choosing Cloud or Local Models · Unreal side: Tutorial — Authoring a Q&A Experience with Ambient Idle