Tutorial — Building a Figure's Corpus

Augmented Traveler local tutorial. Back to AI Mind — Tutorials · overview: Tutorial — How the Mind Works · Mitchell ingest map: Mitchell Corpus Inventory · role: Position — Corpus Editor

How to turn archive material into a corpus the mind can search: sources and rights, transcription conventions, shelf marks, dated entries, the A/B/C stores and the D file, metadata, and versioning. It ends with the test for whether a figure should talk at all.

Draft. Written from the Ground Rules, the Mitchell inventory and the TEI Guidelines. The corpus editor and historian of record test it on Kendall chapters I–II. Correct any convention here before a second figure starts.

1. Sources and rights first

List every source before transcribing a line.

Source type Mitchell example Rights Store
Public-domain edition Kendall, Maria Mitchell: Life, Letters, and Journals (Boston: Lee and Shepard, 1896); Project Gutenberg #10202 Public domain in the USA A (her words) + B (Kendall's connecting text)
Modern edition Albers, Maria Mitchell: A Life in Journals and Letters (2001) In copyright — licence needed A, where licensed
Archive manuscripts MMA, Vassar, NHA MS5 (Mitchell Corpus Inventory) Consent ≠ public domain. Licence and reproduction terms per holder A or B
Period reference Almanacs, schoolbooks, newspapers to the cutoff year Usually public domain; check the scan's terms C
Modern scholarship Historian notes, contested facts Not for ingest D only

2. Transcription conventions

Use semi-diplomatic transcription. Keep her spelling, capitals and punctuation. Expand nothing silently. Write the conventions down once and apply them the same way everywhere.

Feature Plain-text convention TEI equivalent
Illegible word [illegible] <unclear> or <gap reason="illegible"/>
Uncertain reading [word?] <unclear>word</unclear>
Struck-through text [deleted: word] <del>
Insertion above the line [inserted: word] <add>
Error kept as written word [sic] <sic> (with <corr> inside <choice>)
Abbreviation keep it; expansion in a note <choice><abbr>…</abbr><expan>…</expan></choice>
Editor's addition [text] in square brackets <supplied>
Omitted passage (by an earlier editor) […] <gap/>

3. Shelf marks and citations

Every chunk must lead back to a page or a folder.

4. Chunk by dated entry

5. Metadata schema

Each chunk is one Markdown file with YAML front matter:

---
id: MM-A-1847-10-01-a
figure: maria-mitchell
corpus: A            # A | B | C   (D never gets an id here)
date: 1847-10-01
date_certainty: exact  # exact | month | year | range
author: Maria Mitchell
recipient: null
place: Nantucket
source_edition: "Kendall 1896, Maria Mitchell: Life, Letters, and Journals"
page_or_folio: "p. 23 (check)"
shelf_mark: null     # manuscript idno when from an archive
rights: public-domain-us
transcriber: initials
transcription: semi-diplomatic
historian_status: pending  # pending | approved | contested | withdrawn
tags: [comet-1847, observing]
corpus_version: mm-corpus-v0.3
---

6. Four stores, one repository

corpora/maria-mitchell/
  A/   own words (diaries, letters, lectures)
  B/   contemporary accounts (Kendall's text, newspapers, letters about her)
  C/   period knowledge to the cutoff year
  D/   scholar notes, contested facts, trigger topics  ← never ingested
  conventions.md
  CHANGELOG.md

7. Versioning in git

  1. Every correction is a commit with the chunk ID and reason: MM-A-1847-10-01-a: date corrected per Albers p. ….
  2. Tag each release: git tag -a mm-corpus-v0.4 -m "Kendall ch. I–III approved" and push tags.
  3. The index stores corpus_version. Every log line records it.
  4. A corrected fact means a new version, a re-index and a golden-set run (Tutorial — Evaluating Answers).

8. The "thinner than a short monograph" test

The Ground Rules say: if Corpus A is thinner than a short monograph, do not build a conversational human.

Figure Corpus A Decision
Maria Mitchell Diaries and letters across decades; Kendall 1896 in the public domain; Albers 2001 and archives under licence Passes. May generate inside the fence.
Absalom Boston (1785–1855) We have found no writing in his own words so far. Known facts come from others: the Industry voyage (1822), the African Meeting House, the 1845–46 school petition for his daughter Phebe Ann Fails. Retrieval + canned bridges only.

Boston in practice:

9. Practice exercise

  1. Take Kendall chapter II from Project Gutenberg.
  2. Split the diary passages of October 1847 into dated chunks.
  3. Fill the front matter for each. Put Kendall's connecting sentences in B.
  4. Ask the historian to set historian_status on each chunk.

10. Check before first ingest

Sources