#internal/spelling

#internal/spelling

Package spelling is the spelling engine: one word checker serving every surface that needs one.

The check command runs it over a project's docs and posts (SPELL001), and the spell-corpus command runs the same engine over every sibling project on the machine. There is one engine and one accept list; a surface that wants spelling asks this package, and no surface carries a second opinion about what a word is.

#Vendored word list and its licence obligations

The acceptance oracle is a pinned snapshot of the English Speller Database (ESDB, formerly SCOWL), generated from https://app.aspell.net/create at size 70 with US, British (-ise and -ize) and Canadian spellings. It is embedded from wordlist/:

- words.txt -- the words. - COPYRIGHT.txt -- upstream's copyright file, verbatim. - SOURCE.json -- the retrieval record (URL, every parameter, upstream git revisions, SHA-256, word count), regenerated by the wordlist regeneration script.

Upstream's grant permits use, copying, modification, distribution and sale of word lists created from the database "provided that the above copyright notice appears in all copies and that both the above copyright notice and this notice appear in supporting documentation". Shipping COPYRIGHT.txt verbatim beside the words is what satisfies that, and it carries the sub-source notices upstream attaches -- the UK Advanced Cryptics Dictionary notice in particular, whose own terms require that it "be prominently displayed and the text of this document must be included verbatim". Never ship words.txt without COPYRIGHT.txt; never edit either by hand.

Two candidates were rejected before this one. A widely-used word-frequency corpus carries a share-alike condition on its data, which would reach the package that redistributes it. The largest public-domain English list is an unfiltered scrape: it accepts enough garbage to be worthless as an acceptance oracle, which is the one job here.

#The accept list

Genuine terms the general English list cannot know -- project names, tool names, technical vocabulary -- live in a single plain-text file at [AcceptListPath] (~/Projects/ark/spelling-accept.txt). One word per line, "#" starts a comment, and a word on the list is accepted everywhere, permanently.

That location is outside every repository, so the list has no version history and no backup. This was chosen deliberately: the list is cross-project by nature and belongs to the machine rather than to any one repo, and losing it costs a re-triage, not correctness.

A missing file means an empty list, not an error. A fresh machine has simply never accepted anything yet; that is genuine absence, not a degraded mode, and the check behaves identically before and after the file appears -- it just has fewer accepted terms. A file that exists and is malformed is a hard error: it was written by someone who meant something by it, and guessing at their intent would silently drop accepted terms.

#The renderer vocabulary

The accept list cannot carry a word selfdoc itself writes. selfdoc renders reference pages into a consumer's docs tree and then spell-checks them, so a heading a renderer invents becomes an error-severity lint in that consumer's project -- text they did not write, on a page they cannot edit, against a machine list they should not have to populate. Fixed renderer vocabulary is therefore carried by the engine, in [RendererVocabulary], and consulted on every run regardless of what the machine has accepted.

#RendererVocabulary

var RendererVocabulary = Vocab{

RendererVocabulary is the fixed vocabulary selfdoc's own renderers write into generated pages.

A generated page is checked by the same SPELL001 that checks an authored one, so a heading, a column header or a type word a renderer invents is judged against this engine in every project that renders one. The consumer cannot fix such a finding: the text is not theirs, and the accept list that could silence it belongs to one machine while the rendered page ships everywhere. Fixed renderer vocabulary is therefore carried by the engine itself and accepted unconditionally, on every machine, with or without an accept list.

Seeded from the strictcli reference renderer: the "Clearable" heading a sparse update publishes its unset tokens under, the "Env" column of the flag table, the "Config" rows of the configuration table, the type words the Type column renders, and the two tool names its prose names. A renderer that adds a word adds it here; the strictcli-support tests render a schema exercising every construct and fail when one is missing.

#Lookup

type Lookup interface

Lookup answers whether a vocabulary accepts a word as written.

It exists so a check run can consult the vendored list, the renderer vocabulary and the accept list as one vocabulary without copying 170,000 entries into a fresh map per document. [Vocab] satisfies it, so every call site that holds one set passes it directly.

#Vocab

type Vocab map[string]struct{}

Vocab is a set of words, the Go counterpart of the frozensets the Python surface passed around.

#AcceptListError

type AcceptListError struct

AcceptListError reports that the accept list exists but cannot be read as written.

It is never returned for a missing file -- absence is an empty list. It is returned for a line the format does not admit, naming the file, the line number and what was wrong with it.

#Misspelling

type Misspelling struct

Misspelling is one unrecognized word, located precisely enough to open and fix.

#UnknownWord

type UnknownWord struct

UnknownWord is one unrecognized word located in a single source line.

#LoadWordlist

func LoadWordlist() Vocab { return loadWordlist() }

LoadWordlist returns the vendored ESDB word list, exactly as vendored.

The list is read and indexed once per process; every later call returns the same set.

#WordlistBytes

func WordlistBytes() []byte

WordlistBytes returns the vendored word list's raw bytes, as vendored.

The digest and word count SOURCE.json records describe these bytes, so a caller verifying the retrieval record measures what actually ships.

#WordlistSource

func WordlistSource() (map[string]any, error) { return wordlistSource() }

WordlistSource returns the vendored word list's retrieval record: the URL, every generator parameter, the upstream git revisions, the digest and the word count.

#WordlistCopyright

func WordlistCopyright() string

WordlistCopyright returns upstream's copyright notice, verbatim, for display. Shipping it beside the words is what the licence conditions redistribution on.

#AcceptListPath

func AcceptListPath() string

AcceptListPath returns the one accept list's path, ~/Projects/ark/spelling-accept.txt.

The home directory is resolved per call rather than once at load, so a process that changes HOME -- an isolated test run, a sandboxed corpus sweep -- reads the list its own environment names.

#LoadAcceptList

func LoadAcceptList(path string) (Vocab, error)

LoadAcceptList reads the accept list at path, or the one [AcceptListPath] names when path is empty.

A file that does not exist is an empty list, not an error: a fresh machine has simply accepted nothing yet. A file that exists and carries a line the format does not admit is an [AcceptListError] naming the line number and the reason.

#SuggestionsFor

func SuggestionsFor(word string, vocab Lookup, limit int) []string

SuggestionsFor returns up to limit edit-distance-1 neighbours of word in vocab, sorted.

Deliberately cheap: one edit, generated and looked up, no scoring model and no distance-2 search. A word with no neighbour gets no suggestion, which is a better answer than an expensive wrong one. A word over sixteen characters gets none either -- the candidate set grows with its length and the answer stops being worth generating.

#IterUnknownWords

func IterUnknownWords(line string, vocab Lookup) []UnknownWord

IterUnknownWords returns every unrecognized word in one raw source line, each with its 0-based character offset into that line.

Markers, code spans, URLs and directive syntax are blanked before scanning, and machine tokens are skipped whole.

#CheckText

func CheckText(

CheckText spell-checks Markdown content, reporting file, line and column.

Structure comes from the block tokenizer: only text-bearing tokens are scanned, so fenced code blocks and directive blocks are excluded by construction rather than by pattern-matching them back out. Within a token the RAW source lines are scanned, not the token's parsed text, so every reported column is a real column in the file.

content is a Markdown body with its frontmatter already removed, and file is the path to name in each diagnostic. A nil vocab means the vendored word list; a nil accepted means the list [AcceptListPath] names, which is where the only error this can return comes from. lineOffset is added to every reported line, for a body whose frontmatter was stripped. suggest decides whether edit-distance-1 suggestions are computed.

The renderer vocabulary is not optional and not machine-local: a page selfdoc rendered is checked on machines that have accepted nothing, so it is consulted on every run.

#Vocab.Has

func (v Vocab) Has(word string) bool

Has reports whether the set carries word, spelled as given.

#layered.Has

func (l layered) Has(word string) bool

Has reports whether any layer carries word.

#AcceptListError.Error

func (e *AcceptListError) Error() string { return e.Message }

Error renders the diagnostic.

#Misspelling.Describe

func (m Misspelling) Describe() string

Describe renders the diagnostic message a lint carries.