Tutorial — Staff Dashboard

Augmented Traveler local tutorial. Back to AI Mind — Tutorials · what is logged: Tutorial — Privacy and Logging · evaluation: Tutorial — Evaluating Answers · owner: Position — DevOps Engineer

The staff dashboard is where the historian, conversation designer and venue staff see what visitors asked and fix problems without retraining anything. The key action is to pin a canned answer overnight (AI Mind — Process). This page lists the features, access control, build options and the review cadence from the Ground Rules — Historical Virtual Humans.

Draft. Written from the project process notes and the tool docs listed under Sources. The DevOps engineer builds it; the historian and conversation designer validate it in the first two weeks after opening. Screen and button names are placeholders.

1. What it shows, and what it never shows

Shows Never shows
Question text (after redaction) Audio
Answer text, answer ID, authored-or-generated flag Faces, camera images
Corpus hits (citation IDs) Names or contact details volunteered by visitors (redacted before storage)
Refusal flags and which rule fired Anything that identifies a person
Confidence / retrieval score Device IDs beyond the kiosk's own
Timings (ASR, retrieval, first token, first audio)
Venue, figure, anonymous session ID, corpus version

The fields come from Tutorial — Privacy and Logging. If a field isn't logged, the dashboard can't show it; that is the point.

2. Features

  1. Question log. Filter by venue, figure, date, answer type. Full text search over questions.
  2. Refusal flags. Every refused or redirected question, with the rule that fired. Staff can mark "correct refusal" or "should have answered".
  3. Low-confidence answers. Generated answers with weak retrieval scores or no citations. Sorted worst first.
  4. Pin a canned answer.
    • Pick a question (or a cluster of similar ones).
    • Choose an existing authored answer, or write a short reviewed reply.
    • It needs historian approval before it goes live.
    • It takes effect at the next nightly sync (or immediately, for a refusal).
    • Unpin the same way. Every pin and unpin is logged with who and why.
  5. Trigger re-evaluation. Run the golden set against the current corpus and model; show the score beside the last run (Tutorial — Evaluating Answers).
  6. Corpus version. Which corpus version each venue is running; when it changed; the correction note (Tutorial — Building a Figure's Corpus).
  7. Per-venue settings. Model route (cloud or local; Tutorial — Choosing Cloud or Local Models), offline mode, touch-question list, captions default, open hours. Changes need a second person to approve.
  8. Flag for the golden set. One click adds a real question (redacted) to the golden-set candidates.
  9. Export. CSV of questions for a date range, for the monthly review. No other export.

3. Access control

Role Can see Can change
Historian of record Everything in section 1 Approve pins; veto answers
Conversation designer Everything in section 1 Draft pins; flag for golden set
Second historian Blind review screen only Scores
Venue staff Their venue's log, last 7 days Offline switch; report a problem
DevOps / AI engineer Everything, plus timings and errors Settings (with a second approver), re-evaluation
Anyone else Nothing Nothing

4. Build approach options

Option What it is Licence For Against
A. Small custom app: FastAPI + a simple web UI Python API over the log database (Postgres); a few pages FastAPI MIT (check); our code Exactly our features, including pin approval; same language as the AI service We build and maintain it
B. Metabase for reading logs Self-hosted business-intelligence tool over Postgres Open-source edition AGPL Filters and charts with no code Read-only views; pins still need option A
C. Grafana for timings Dashboards over Postgres, Prometheus, Loki AGPL-3.0 (with Apache-2.0 exceptions) Good for latency and error monitoring Not a review tool for text
D. Open WebUI or similar Chat UI BSD-3 + branding clause — Wrong tool; not for logs

Suggested: A for review and pins, plus C for the DevOps engineer's latency and error panels. Use B only if the historian wants self-serve charts. Running Metabase or Grafana unmodified inside the studio avoids AGPL network-service questions; confirm with counsel if we modify them.

FastAPI supports OAuth2, OpenID Connect, API keys and HTTP Basic in its security module. Use OpenID Connect with the studio's identity provider.

5. Data model (sketch)

Keep the log database separate from the corpus database.

6. Review cadence

From the Ground Rules — Historical Virtual Humans:

Period Who What
Weeks 1–12 after opening Historian + conversation designer Read all logs. Daily in week 1, then at least twice a week
After week 12 Same Monthly review of all refusals, low-confidence answers and a sample of the rest
Press spikes (article, social media, school holiday) Same, within 48 hours Read everything from the spike period
Every corpus correction Corpus editor + AI engineer New corpus version; re-evaluation run; note in the dashboard

Each review ends with: pins added or retired, golden-set additions, corpus corrections logged, and a short note filed in the dashboard.

7. Practice exercise

Using ten made-up log rows for Maria Mitchell (write them yourself; no real data), find one refusal that should have been an answer, one low-confidence answer, and one question worth pinning. Draft the pin, send it for historian approval in the test system, and check it appears the next morning.

8. Check before opening

Sources