Tutorial — Local Inference for Offline Venues

Augmented Traveler local tutorial. Back to AI Mind — Tutorials · the decision: Tutorial — Choosing Cloud or Local Models · GPU profiling: Tutorial — Performance Profiling · hardware: Hardware — Deployment Specifications

How to run an open-weight language model on the kiosk PC (or a small second PC in the kiosk cabinet) for offline venues and schools. It covers the runtime (Ollama or llama.cpp server), the OpenAI-compatible endpoint, model choice, quantisation, the VRAM budget next to Unreal, local embeddings, structured output for answer_meta, benchmarks, updates and licences.

Draft. Written from the runtime docs listed under Sources and the model table in AI Mind — Tutorials section 2. The AI conversation engineer and DevOps engineer validate it on the kiosk PC. Correct flags and setting names here after the first install.

1. Where it runs

Decide after section 7. Start on the same PC; move out if the frame rate or first-token time suffers.

2. Runtime: Ollama or llama.cpp server

Both run GGUF models and expose an OpenAI-compatible API, so the conversation service only changes its base URL.

vLLM (Apache 2.0) suits many requests at once. It reserves 90% of GPU memory by default (--gpu-memory-utilization 0.9, in older docs; check current), which does not suit sharing a GPU with Unreal. LM Studio is fine for trying models on a desktop; it is proprietary.

Suggested: Ollama for the first venue (simplest to operate); llama.cpp server if we need finer control.

Install on the kiosk (Ollama):

1. Install Ollama as a service (Windows installer).
2. Keep it on localhost. Ollama binds to 127.0.0.1:11434 by default. Do not set OLLAMA_HOST to expose it unless the model runs on the second PC; then allow only the kiosk PC's address through the firewall.
3. Set OLLAMA_KEEP_ALIVE=-1 so the model stays loaded (a cold load in front of a visitor is too slow).
4. Set OLLAMA_CONTEXT_LENGTH to what the persona and retrieved passages need (measure; start at 8192).
5. Pull the chosen model by exact tag. Record the tag and digest in the venue config.
6. Run ollama ps; it must say 100% GPU.

3. Choosing the model

From AI Mind — Tutorials section 2 (September 2026; check each model card):

Models marked ⚠️ come from one source. Do not ship one until the model card and licence have been read by the CTO.

4. Quantisation

• Q4_K_M: about 0.6 GB per billion parameters. The usual default.
• Q5_K_M: a little larger, a little closer to full quality.
• Q8_0: about 1.05 GB per billion; mostly a reference.
• A 2026 study on Llama-3.1-8B found Q5_K_M near baseline and Q4_K_M with minimal loss at about 69% smaller. Results vary by task, so test on our golden set, not on general benchmarks.
• AWQ is a GPU quantisation format used by vLLM, not by Ollama or llama.cpp.
• KV cache can also be quantised (OLLAMA_KV_CACHE_TYPE=q8_0) to save context memory. Test quality after.

5. VRAM budget on one RTX 5090 (32 GB)

Estimates to replace with measurements from Tutorial — Performance Profiling:

Rule: if Unreal plus speech needs more than about 14 GB, move the model to a second PC or drop to a 12B model.

6. Local embeddings and structured output

Embeddings: Ollama serves them at /api/embed (and /v1/embeddings). Its docs name embeddinggemma, qwen3-embedding and all-minilm. Use the same embedding model for indexing and querying. If the offline index was built with a cloud embedding model, rebuild it locally (Tutorial — Retrieval with Citations).

Structured output for answer_meta: the service needs answer ID, gesture tag, refusal flag and citation IDs before the audio starts (Tutorial — Conversation Client Plugin). Two ways:

1. Two calls: a short constrained call returns JSON (gesture, citation_ids, confidence) using a JSON schema; then the answer text streams. Adds a little latency; simplest to validate.
2. One streamed call: the answer streams as plain text with inline citation markers (e.g. [A-0142]); the service strips the markers before TTS and builds answer_meta from them. Faster; needs careful parsing.

Try option 2 first; fall back to option 1 if parsing proves unreliable. The policy layer checks the result either way (Tutorial — Policy Layer with Guardrails).

7. Benchmarks to run

Run with Unreal running the figure in a Shipping build, at kiosk settings.

1. Time to first token (TTFT): 50 golden questions, median and 95th percentile. Target: under 600 ms (AI Mind — Process).
2. Tokens per second: published 5090 figures (40–90 tokens/s) come from one blog; measure ours.
3. Frame rate of the figure while the model generates. It must stay at target.
4. Golden-set quality and a blind historian review of 50 answers (Tutorial — Evaluating Answers).
5. Cold start: time from boot to first answer.

# skeleton — test before use. Needs: pip install openai
import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="local")

def bench(model, messages):
    t0 = time.perf_counter()
    first, chunks = None, 0
    stream = client.chat.completions.create(model=model, messages=messages, stream=True)
    for part in stream:
        if part.choices and part.choices[0].delta.content:
            chunks += 1
            if first is None:
                first = time.perf_counter() - t0
    total = time.perf_counter() - t0
    return {"ttft_s": first, "chunks_per_s": chunks / max(total - (first or 0), 1e-6)}

Chunks are close to tokens for most runtimes; for exact counts use the timings llama.cpp returns.

8. Update procedure

1. Never update at a venue without a tested build.
2. New model or runtime version → install on the dev workstation → run section 7 → historian review.
3. Record the model tag, digest, quantisation, runtime version and settings in the venue config.
4. Turn off auto-update on kiosk PCs (Ollama auto-updates on Windows; check how to disable it). Update in a maintenance window.
5. Keep the previous model on disk for one week to roll back.

9. Licences

• Runtimes: llama.cpp MIT; Ollama MIT; vLLM Apache 2.0; LM Studio proprietary (read its terms before commercial use).
• Models: each model card's licence governs. Apache 2.0 families are simplest. Llama licence conditions need counsel.
• Keep a copy of each model's licence with the build.

10. Practice exercise

On a dev workstation with the Mitchell figure running in Unreal, load a Gemma 4 12B Q4_K_M in Ollama. Run the benchmark on 20 golden questions. Then load a 27B-class model and repeat. Record TTFT, tokens/s, frame rate and VRAM for both on one page.

11. Check before an offline venue opens

• Model and runtime versions recorded; licences filed
• ollama ps shows 100% GPU; model kept loaded
• Endpoint reachable only from the kiosk
• TTFT and frame rate targets met with Unreal running
• Offline index built with the local embedding model
• Golden set and historian review passed
• Auto-update off; rollback copy present

Sources

• Ollama, OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
• Ollama, Structured outputs: https://docs.ollama.com/capabilities/structured-outputs
• Ollama, Embeddings: https://docs.ollama.com/capabilities/embeddings
• Ollama, FAQ: https://docs.ollama.com/faq
• ggml-org, llama.cpp server README: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• vLLM, Engine arguments (v0.6.4): https://docs.vllm.ai/en/v0.6.4/models/engine_args.html
• LM Studio, OpenAI compatibility: https://lmstudio.ai/docs/developer/openai-compat
• arXiv, Which Quantization Should I Use? (2601.14277): https://arxiv.org/html/2601.14277v1
• ggml-org, whisper.cpp (model memory): https://github.com/ggml-org/whisper.cpp
• Full list: Sources — 2026-09-20