AI Mind — Tutorials
Process note: AI Mind — Process · Unreal side: Tutorial — Conversation Client Plugin · Tutorial — Authoring a Q&A Experience with Ambient Idle
Collected 20 September 2026. Items marked ⚠️ are recent or come from a single source. Check them before relying on them.
1. Pipeline building blocks
| Part | Resource | Link | Licence / notes |
|---|---|---|---|
| Retrieval (RAG) framework | LlamaIndex | https://docs.llamaindex.ai | Open source |
| LangChain RAG tutorial | https://python.langchain.com/docs/tutorials/rag/ | Open source | |
| Haystack | https://docs.haystack.deepset.ai | Open source | |
| Vector store | pgvector (Postgres) | https://github.com/pgvector/pgvector | Open source |
| Evaluation | Ragas | https://docs.ragas.io | Open source |
| DeepEval | https://deepeval.com/docs | Open source | |
| promptfoo | https://www.promptfoo.dev/blog/promptfoo-joining-openai/ | ⚠️ Acquired by OpenAI (March 2026); watch its open-source status | |
| Guardrails / policy | NVIDIA NeMo Guardrails (Colang) | https://github.com/NVIDIA-NeMo/Guardrails · https://docs.nvidia.com/nemo/guardrails/configure-guardrails/colang | Open source |
| Llama Guard 4 (safety classifier) | https://huggingface.co/meta-llama/Llama-Guard-4-12B | Llama licence | |
| Speech recognition | faster-whisper | https://github.com/SYSTRAN/faster-whisper | MIT; local |
| whisper.cpp | https://github.com/ggml-org/whisper.cpp | MIT; local | |
| Deepgram live streaming | https://developers.deepgram.com/docs/live-streaming-audio | Cloud | |
| Text-to-speech | ElevenLabs | https://elevenlabs.io/docs | Cloud; the contracted actor voice |
| Kokoro-82M | https://huggingface.co/hexgrad/Kokoro-82M | Apache 2.0; local | |
| Piper | https://github.com/OHF-Voice/piper1-gpl | GPL-3.0; per-voice licences vary | |
| XTTS-v2 | https://github.com/coqui-ai/TTS/issues/3490 | Avoid: non-commercial model licence, and no one left to sell a commercial one | |
| Face from audio | NVIDIA Audio2Face-3D | https://github.com/NVIDIA/Audio2Face-3D · https://docs.nvidia.com/ace/ace-unreal-plugin/2.5/ace-unreal-plugin-audio2face.html | SDK and plugins MIT; models under NVIDIA Open Model licence |
2. Local inference (open-weight models on the kiosk PC)
Why local: offline venues and schools, no per-question cost, visitor audio stays on site, predictable latency. Trade-off: smaller models are weaker. Our design already constrains answers to retrieved sources, which helps a smaller model a lot.
| Runtime | Link | Notes |
|---|---|---|
| llama.cpp | https://github.com/ggml-org/llama.cpp | MIT; GGUF quantised models; the base of many tools |
| Ollama | https://ollama.com | MIT; easiest to run; OpenAI-compatible API |
| LM Studio | https://lmstudio.ai | Free, proprietary; good for trying models on a desktop |
| vLLM | https://docs.vllm.ai | Apache 2.0; best when many requests run at once (streaming server) |
| TensorRT-LLM · NVIDIA NIM | https://github.com/NVIDIA/TensorRT-LLM · https://developer.nvidia.com/nim | Fastest on NVIDIA; more set-up |
Current open-weight families (September 2026; check each model card):
| Family | Sizes that fit one RTX 5090 (32 GB) | Licence |
|---|---|---|
| Qwen 3.6 / 3.8 ⚠️ | 27B dense; 35B-A3B MoE (3B active) | Apache 2.0 |
| Gemma 4 | 12B, 26B-A4B MoE, 31B dense | Apache 2.0 |
| gpt-oss | 20B (about 16 GB) | Apache 2.0 |
| Mistral Small 4 | 119B MoE; only fits at about 3-bit | Apache 2.0 |
| Meta Muse Glimmer ⚠️ | 30B | Apache 2.0 (reported August 2026) |
| Llama 4 | — | Llama Community Licence (conditions apply) |
Rules of thumb:
- VRAM for weights is about parameters × bits ÷ 8. Q4_K_M uses about 0.6 GB per billion parameters and Q8 about 1.05 GB. Add 2–6 GB for context.
- Leave room on the same GPU for speech recognition, TTS, embeddings, lip sync and Unreal itself. That argues for a 20–35B model at Q4–Q5, or a separate inference PC in the kiosk cabinet.
- MoE models with about 3B active parameters give the fastest first token.
- ⚠️ Published speeds on a 5090 are 40–90 tokens/s depending on the model. They come from one blog; benchmark on our hardware.
3. SillyTavern and similar tools
| Tool | What it is | Licence | Our use |
|---|---|---|---|
| SillyTavern | Locally installed front end for character and role-play chat. Character cards, World Info / lorebooks (keyword-triggered context), connects to KoboldCpp, Ollama, OpenAI-compatible and cloud APIs | AGPL-3.0 | Prototyping and persona authoring sandbox only |
| KoboldCpp | llama.cpp-based local server with its own UI | AGPL-3.0 | Local back end for tests |
| text-generation-webui | Local model UI | AGPL-3.0 | Tests |
| Open WebUI | ChatGPT-style web UI for local models | BSD-3 + branding clause | Internal staff tool at most |
Why not SillyTavern in the kiosk:
- Security: its own docs warn never to expose it on the open internet without proper security. Basic auth has no rate limiting.
- Accuracy: World Info injects text when keywords match. That is good for voice and persona, but it doesn't give historian-controlled retrieval with citation IDs, refusal rules or an evaluation harness, which our Ground Rules require (AI Mind — Process).
- Licence: under AGPL, running a modified copy as a network service means publishing our changes.
Where it helps: the conversation designer and historian can quickly try a persona card, tone and period vocabulary with a local model, and collect examples for the golden question set.
Local tutorials (Augmented Traveler)
| Tutorial | Covers | Owner |
|---|---|---|
| Tutorial — How the Mind Works | Architecture for new staff: authored-first, retrieval fence, policy in code, logging, owners | Everyone on the mind |
| Tutorial — Building a Figure's Corpus | Sources, rights, transcription, shelf marks, A/B/C stores and D file, metadata, versioning, the monograph test | Corpus editor, historian |
| Tutorial — Retrieval with Citations | Postgres + pgvector schema, chunking by dated entry, hybrid search with RRF, cutoff filter, citation IDs | AI engineer |
| Tutorial — Writing a Persona for a Historical Figure | Voice dossier, persona prompt structure, A vs B first-person rules, plain speech, examples | Conversation designer, historian |
| Tutorial — Policy Layer with Guardrails | Cutoff in code, refusals, trauma protocol, companionship ban, children, prompt injection; NeMo Guardrails / Llama Guard | AI engineer, historian |
| Tutorial — Evaluating Answers | Golden question set, metrics (Ragas/DeepEval), regression, blind historian review, red-teaming, ship gates | AI engineer, historians, QA |
| Tutorial — Authored Answers for Top Questions | Choosing top questions, recording and animating, matching rules, answer IDs, updating from logs | Conversation designer, Unreal engineer |
| Tutorial — Speech In | Push-to-talk, VAD, streaming ASR local vs cloud, Nantucket vocabulary, noise, children's voices, testing | AI engineer, accessibility designer |
| Tutorial — Speech Out | Contracted actor voice and consent, streaming TTS, pronunciation lexicon, latency, QA | AI engineer, voice director |
| Tutorial — Choosing Cloud or Local Models | Decision note: criteria, options, data-retention terms, minors, suggested start, questions for the founders | CEO, CTO |
| Tutorial — Local Inference for Offline Venues | Ollama / llama.cpp on the kiosk, model and quantisation choice, VRAM budget, structured output, benchmarks | AI engineer, DevOps |
| Tutorial — Persona Sandbox with SillyTavern | Sandbox only: local install, character card and World Info from Corpus A, exporting to the golden set; never public | Conversation designer, historian |
| Tutorial — Staff Dashboard | Logs, flags, pinning canned answers, re-evaluation, corpus version, access control, review cadence | DevOps, historian |
| Tutorial — Privacy and Logging | What we log and never log, retention, notices, children (COPPA), schools (FERPA), 201 CMR 17.00, vendors; not legal advice | CTO, counsel |
4. Every AI-mind tutorial the project needs
Status: ✅ written · 🔗 an external tutorial is enough · ✏️ local tutorial to write · ❓ decision needed first.
| Need | Status | Where / plan |
|---|---|---|
| Architecture overview for new staff (retrieval first, policy layer, authored answers) | ✅ | Tutorial — How the Mind Works |
| Corpus ingestion: transcription, shelf marks, dated chunks, A/B/C separation | ✅ | Tutorial — Building a Figure's Corpus |
| Retrieval (RAG) with citations: hybrid search, pgvector, chunking | ✅ | Tutorial — Retrieval with Citations |
| Persona prompt and voice dossier | ✅ | Tutorial — Writing a Persona for a Historical Figure |
| Policy layer: cutoff date, refusals, trauma protocol, companionship ban, in code not prompt | ✅ | Tutorial — Policy Layer with Guardrails |
| Golden question set and evaluation harness (Ragas/DeepEval; blind historian review) | ✅ | Tutorial — Evaluating Answers |
| Authored answers for top questions (pre-recorded, animated) | ✅ | Tutorial — Authored Answers for Top Questions |
| Streaming speech recognition in a noisy museum (push-to-talk, mic arrays) | ✅ | Tutorial — Speech In |
| Streaming TTS with the contracted actor voice (consent, version lock) | ✅ | Tutorial — Speech Out |
| Lip sync choice (MetaHuman audio source vs Audio2Face-3D) | ✅ | Tutorial — Authoring a Q&A Experience with Ambient Idle section 4 |
| Unreal client | ✅ | Tutorial — Conversation Client Plugin |
| Local inference on the kiosk (Ollama/llama.cpp/vLLM, model choice, quantisation, VRAM sharing) | ✅ (❓ model choice) | Tutorial — Local Inference for Offline Venues |
| Cloud vs local model decision (quality, cost, privacy, offline) | ✅ (❓ founders decide) | Tutorial — Choosing Cloud or Local Models |
| Persona prototyping in SillyTavern (sandbox only) | ✅ (❓ adopt or not) | Tutorial — Persona Sandbox with SillyTavern |
| Staff dashboard: pin canned answers, review logs | ✅ | Tutorial — Staff Dashboard |
| Privacy and logging (no audio, no faces; retention) | ✅ (counsel review) | Tutorial — Privacy and Logging |
| Red-teaming (attempts to break character or policy) | ✅ | Section in Tutorial — Evaluating Answers |