Tutorial — Choosing Cloud or Local Models

Augmented Traveler local tutorial. Back to AI Mind — Tutorials · the loop: AI Mind — Process · local set-up: Tutorial — Local Inference for Offline Venues · privacy: Tutorial — Privacy and Logging

A decision note written as a tutorial. It sets out how to choose where the language model runs: a hosted frontier API, a local open-weight model on the kiosk, or a hybrid. It gives the criteria, the options, a suggested starting point, and the questions the founders must settle. The decision belongs to the CEO and CTO, not to this page.

Draft. Written from provider data-usage pages read on 20 September 2026 (see Sources). Provider terms change; re-read them before signing. Not legal advice. The CTO validates the benchmark; counsel reviews section 3.

1. What the model actually does here

Our design limits what the model has to do (Tutorial — How the Mind Works):

So we need a model that follows instructions and sources well in a period voice. We don't need the strongest model for open-ended reasoning. That helps smaller local models.

2. Criteria

Criterion What to measure Cloud frontier API Local open-weight on kiosk
Quality Golden-set score and blind historian review (Tutorial — Evaluating Answers) Usually higher Lower; retrieval narrows the gap. Measure it
Latency Time to first token, 50 runs, at the venue Depends on venue internet and region Predictable; no network
Cost per 1,000 questions Method in section 4 Per-token fees Hardware already bought; power and upkeep
Privacy and retention Section 3 Text leaves the site; needs a zero-retention agreement Stays on site
Offline venues Works with no internet? No Yes
Schools and under-13s Provider terms plus COPPA/FERPA (Tutorial — Privacy and Logging) Some providers restrict or add conditions Fewest third parties
Maintenance Who updates, re-tests, patches Vendor updates the model; we re-test when models retire We update runtime and model; we re-test
Licence Terms of use; open-weight licence Provider terms (incl. age rules) Apache 2.0 for most families in AI Mind — Tutorials section 2; Llama licence has conditions
VRAM Room left for Unreal, ASR, TTS, lip sync None used 10–20 GB used, or a second PC

3. What the providers say about data (read 20 September 2026)

Provider Default Zero retention Minors
Anthropic API Inputs and outputs deleted within 30 days; not used for training without express permission ZDR by arrangement with sales, per organisation; covers the Messages API; some features excluded (Files, Batch, code execution). Flagged content may be kept up to 2 years even under ZDR. Some newest models ("Covered Models") need 30-day retention Allowed with safeguards: age-appropriate protections, COPPA compliance stated publicly, disclosure that it is AI
OpenAI API Not used for training unless opted in; abuse-monitoring logs up to 30 days ZDR or Modified Abuse Monitoring by approval; about 15 endpoints eligible Under-18 allowed with safeguards; personal data of under-13s only with zero data retention in place
Google Gemini Developer API (paid) Not used to improve products; logged for a limited time for abuse monitoring Abuse-monitoring logging can't be turned off on the Developer API; Google points to Vertex AI for guaranteed ZDR. Avoid Search/Maps grounding and caching Terms forbid use in a service "directed towards or likely to be accessed by" under-18s. That rules out the Gemini Developer API for our kiosks unless counsel reads it differently. Vertex terms not checked (unconfirmed)

What this means for us:

4. Estimating cost per 1,000 questions

Work it out; don't guess.

  1. Count tokens per question from 50 real logged turns:
    • input = persona prompt + policy text + retrieved passages + question + short history (typically a few thousand tokens);
    • output = the answer (short: 60–150 words).
  2. Subtract authored hits. If authored answers catch 40% of questions, only 600 of 1,000 reach the model.
  3. Cloud cost = generated questions × (input tokens × input price + output tokens × output price) ÷ 1,000,000. Take prices from the provider's pricing page on the day. Check if prompt caching lowers the input price for the fixed persona text.
  4. Add ASR minutes and TTS characters if those are cloud services.
  5. Local cost = (share of kiosk PC, or a second inference PC) ÷ expected questions over its life + power + staff hours for updates.
  6. Put both in Year One Budget — Draft as a line with the method written next to it.

5. The options

Option How it works For Against
A. Cloud default Hosted API with ZDR for all generated answers; authored-only when offline Best quality; least upkeep Per-question cost; depends on venue internet; minors terms; data leaves site
B. Local default Open-weight model on the kiosk (or cabinet PC) everywhere Offline venues and schools work; no third party sees questions; predictable cost Lower quality; we maintain it; VRAM pressure
C. Local default, cloud for hard questions Local answers first; low-confidence questions go to cloud Most questions stay local Two models to evaluate; "hard" is hard to define; privacy only partly improved
D. Cloud default, local fallback Cloud with ZDR; local model when the network drops or at school/offline venues Best quality when online; still works offline Two models to evaluate and keep in persona

Benchmark before choosing: run the same golden set through one cloud model and two local models (from Tutorial — Local Inference for Offline Venues). Blind historian review of 50 answers from each. Record latency at a real venue.

6. Suggested starting point (for the founders to accept or reject)

Option D, with schools local-only, because:

If the benchmark shows a local model within a small margin of the cloud model on our golden set, choose B instead: it is simpler and more private.

7. Questions the founders must settle

  1. Is a signed ZDR agreement a hard gate before the first public opening? Which provider?
  2. Do we accept any venue data leaving the island? Do venue contracts allow it?
  3. What quality margin makes local "good enough" (e.g. historian scores within 5%)?
  4. Do school visits run local-only, always?
  5. Budget: second inference PC per kiosk, or share the 5090?
  6. Who re-runs the evaluation when a cloud model is retired or a new local model appears, and how fast?
  7. Do we publish which model we use, in the out-of-character segment and on the website?

8. Practice exercise

Take 20 logged Mitchell questions. Estimate cost per 1,000 questions for one cloud model and for a local model on a second PC, using section 4. Write both numbers and the assumptions on one page for the founders.

9. Check before the decision meeting

Sources