Tutorial — Speech In

Augmented Traveler local tutorial. Back to AI Mind — Tutorials · the runtime loop: AI Mind — Process · Unreal side: Tutorial — Conversation Client Plugin · captions: Tutorial — Captions UI

How the visitor's spoken question becomes text. It covers push-to-talk, voice activity detection (VAD), the choice of streaming speech recognition (ASR), custom vocabulary for Nantucket names, noise, children's voices and testing. A wrong transcript produces a confident wrong answer, so this stage deserves real testing.

Draft. Written from the vendor and project docs listed under Sources. The AI conversation engineer owns it; the UX and accessibility designer checks sections 6–7. Correct parameter names here after the first build.

1. What arrives, and what must leave

2. Push-to-talk first, VAD second

Museum rooms are loud: school groups, other exhibits, the figure's own voice. Always-listening is unreliable there.

  1. Push-to-talk is the default on the kiosk (a large lit button) and on headsets (trigger or pinch).
  2. Release of the button is end_of_speech. No guessing when the visitor has finished.
  3. VAD still helps:
    • trims silence before and after the question;
    • ends the turn if the button is held but nobody speaks for about 2 s;
    • rejects a press with no speech in it (a child pressing for fun).
  4. Silero VAD is the usual choice: MIT licence, about 2 MB, supports 8 kHz and 16 kHz, and a 30 ms chunk takes under 1 ms on one CPU thread.
  5. Mute the microphone while the figure speaks, unless we build barge-in (the visitor presses to interrupt). Decide this with the UX designer. If barge-in is wanted, the mic array must do echo cancellation (section 5).

3. Streaming ASR: local or cloud

Option Type What the docs say Our notes
faster-whisper Local, Python MIT; Whisper on CTranslate2; fp16 and int8; built-in Silero VAD filter; transcribe() takes hotwords and initial_prompt Not streaming by itself; batch the utterance at release, or use a streaming wrapper
WhisperLive Local server MIT; near-live Whisper server; backends faster_whisper, TensorRT, OpenVINO; takes 16 kHz mono int16 or float32; use_vad, hotwords, initial_prompt Closest to our protocol; run on the kiosk PC
whisper.cpp Local, C/C++ MIT; CUDA build; whisper-stream and whisper-server examples; Silero VAD via --vad; needs 16 kHz input; model sizes about 273 MB (tiny) to 3.9 GB (large) Good if we want no Python on the kiosk
Deepgram (Nova-3) Cloud streaming Keyterm prompting on Nova-3 (multi-word phrases); multilingual keyterms up to 500 tokens (about 100 words) per request Audio leaves the building; see Tutorial — Privacy and Logging
AssemblyAI Universal-Streaming Cloud streaming Keyterms prompting: up to 100 keyterms per session, 5–50 characters each; 16 kHz mono; applies to final transcripts only; free in beta at time of reading Same privacy question

Suggested default: local Whisper (WhisperLive or faster-whisper) on the kiosk GPU for every venue. Audio then never leaves the site, and offline venues work the same way. Try one cloud service as a quality comparison in the test in section 8. Record the model name and size in the venue config.

Speed check (from the faster-whisper README): large-v2 transcribed 13 minutes of audio in about 1 minute on an RTX 3070 Ti (fp16, beam 5). A 6-second question should therefore finish well inside 300 ms on a 5090. Measure it; don't assume it.

4. Custom vocabulary for Nantucket names

Whisper and the cloud services all mishear local names. Keep one vocabulary list per figure, versioned with the corpus (Tutorial — Building a Figure's Corpus).

Term Why it matters
Nantucket, Siasconset ('Sconset), Madaket, Polpis, Quidnet, Coatue Place names visitors will say
Wampanoag Must be recognised correctly; never guessed
Maria Mitchell, Lucretia Mott, Absalom Boston, Frederick Douglass Figure names
Atheneum, Pacific Bank, Main Street, New Guinea, African Meeting House Places in the answers
Essex, Veeder, Brock Whaling names
comet, telescope, transit, sextant, meridian Mitchell's science words
Quaker, Friends, meeting house, abolition Period and faith words

How to apply it:

  1. Local Whisper: pass the list through hotwords (faster-whisper, WhisperLive), or a short initial_prompt such as "Questions for Maria Mitchell about Nantucket, Siasconset, the Atheneum and the comet."
  2. Deepgram: send the terms as keyterms (Nova-3).
  3. AssemblyAI: send up to 100 keyterms; drop any term under 5 characters or it is rejected.
  4. After ASR: a small correction table for known mishearings ("sconce it" → "Siasconset"). Log each correction so the list improves.
  5. Pronunciations for the voice are a separate list (Tutorial — Speech Out).

5. Noise and microphones

6. Children's voices

Children are a large share of museum visitors, and ASR is much worse for them.

7. Showing the transcript, and a touch alternative

  1. When transcript arrives, show "You asked: …" briefly (Tutorial — Captions UI section 4).
  2. If the transcript is empty or low-confidence, the figure does not guess. It plays an authored line ("I did not quite catch that. Ask me again?"; final wording from the historian) and shows question buttons.
  3. Touch alternative: 6–8 suggested questions on screen, drawn from the authored answers (Tutorial — Authored Answers for Top Questions). Visitors who cannot or will not speak get the same experience.

8. Testing with recorded museum noise

  1. Record 30 minutes of room noise at each venue, with no recognisable speech kept (walk-through before opening, or crowd noise with no one close to the mic). Delete anything with intelligible conversation.
  2. Record 200 test questions from the golden set (Tutorial — Evaluating Answers): staff, volunteers and, with written parental consent, children. Store under the test folder only; never in production logs.
  3. Mix each question with venue noise at three levels (quiet, normal, school group). The MUSAN corpus (CC BY 4.0) adds extra noise types.
  4. Run every ASR option and compute word error rate and name accuracy (did "Siasconset" survive?).
# skeleton — test before use
import jiwer

def score(pairs):
    """pairs: list of (reference_text, asr_text)."""
    refs = [r.lower() for r, _ in pairs]
    hyps = [h.lower() for _, h in pairs]
    names = ["nantucket", "siasconset", "wampanoag", "atheneum", "mitchell"]
    hits = sum(1 for r, h in zip(refs, hyps) for n in names if n in r and n in h)
    total = sum(1 for r in refs for n in names if n in r)
    return {"wer": jiwer.wer(refs, hyps), "name_recall": hits / max(total, 1)}

Targets to agree with the CTO: word error under 10% for adults at normal noise; name recall over 95%; final transcript under 300 ms after release.

9. Practice exercise

Record yourself asking "Did you see the comet from the Pacific Bank roof?" five times: close, at 1 m, with a noise track, with the mic muted halfway, and as fast as you can. Run faster-whisper with and without hotwords. Write down the errors and add any new mishearing to the correction table.

10. Check before a venue build

Sources