#internal/extractors

#internal/extractors

Package extractors defines the language-extractor protocol, the shared behavior every extractor embeds, and the registry that resolves a language name to its extractor.

An extractor answers directives out of source code. The build pipeline hands it a directive name, the directive's attributes, its body lines, the project's declared source paths and the project's base directory; the extractor returns the Markdown that replaces the directive. It also answers the three discovery questions the coverage and quality measurements ask of a file: which symbols it exports, what one symbol's parameters and return value are, and what the module's own documentation says.

#The base

Base carries everything the language extractors share: the directive dispatch table, the no-op defaults for the optional protocol methods, and the formatting helpers that turn source-derived text into Markdown (symbol headings and spans, doc-comment renesting, Google-style docstring sections, the config-file tables, brace-block and comment-block scanning). A language package embeds Base, implements Detect, and populates the dispatch table.

#The registry

Each language package registers its own factory from an init function, so a consumer links a language in by importing its package. KnownLanguages is the authority on which names selfdoc ships an extractor for: a name on that list with no registered factory is a wiring mistake and is reported as an error rather than silently answered with a stub. A name that is not on the list is a language selfdoc has no extractor for, and NewStub answers it in band -- an unsupported-language directive renders an error marker on the page instead of aborting the build, which is what the LANG001 lint reports at check time.

#PySpaceClass

const PySpaceClass = util.PythonSpaceClass

PySpaceClass is Python's \s: the ASCII whitespace characters, the four ASCII separator controls, and the Unicode whitespace code points.

#PyNonSpaceClass

const PyNonSpaceClass = util.PythonNonSpaceClass

PyNonSpaceClass is Python's \S, the complement of PySpaceClass.

#PyWordClass

const PyWordClass = util.PythonWordClass

PyWordClass is Python's \w where it matches an identifier character: a letter, a digit or an underscore, in any script.

#Base

type Base struct

Base carries the behavior every language extractor shares: the directive dispatch table and the no-op answers to the optional protocol methods.

A language package embeds it, builds it with NewBase, and overrides whichever of the optional methods its language can answer. Name comes from the language it was built with, so the unknown-directive message names the right extractor without the language package restating it.

#JSONObject

type JSONObject struct

JSONObject is a decoded JSON or TOML table that remembers the order its keys appeared in.

Order is not a nicety here: every config table selfdoc renders lists its rows in document order, which is what a reader comparing the page against the file expects. Go's map type has no order, so the decoders in this file build this instead.

#DocSections

type DocSections struct

DocSections is a Google-style docstring parsed into its parts.

#DocParam

type DocParam struct

DocParam is one documented parameter.

#DocRaise

type DocRaise struct

DocRaise is one documented exception.

#Extractor

type Extractor interface

Extractor is the protocol every language extractor implements.

The methods split into three groups. Name and Detect identify the language. Extract resolves one directive into Markdown. The rest answer the discovery questions the coverage, quality and staleness measurements ask.

Three methods return an error where the Python protocol they replace returned an empty result: a parser that cannot run at all is a broken installation, not a file with no symbols in it. A file that cannot be read, or whose contents do not parse, still answers empty -- that is a property of the file and every extractor reports it that way.

#Handler

type Handler func(

Handler resolves one directive for one language. It is the value type of an extractor's dispatch table.

target is nil when the directive declared no target attribute, which is a different question from an empty target: a code-test directive with no target renders the whole file, while an empty one looks for a symbol with no name.

The error return is for a broken toolchain, never for an unresolvable directive -- see Extractor.Extract.

#SymbolDetails

type SymbolDetails struct

SymbolDetails is what an extractor reads out of source about one symbol's parameters and return value, together with whether the symbol's own documentation covers them. The quality measurement scores a symbol from it.

#SymbolParam

type SymbolParam struct

SymbolParam is one parameter of a symbol, as SymbolDetails reports it.

#Factory

type Factory func() Extractor

Factory builds an extractor.

It takes nothing: every extractor answers by reading and parsing files, and none of them needs an effects handle. The Python one used to run an interpreter through one and now parses in-process.

#DetectedLanguage

type DetectedLanguage struct

DetectedLanguage is one language auto-detection found in a directory.

#SourceEntry

type SourceEntry struct

SourceEntry is a declared source path with its language and the extractor that reads it.

#NewBase

func NewBase(language string, handlers map[string]Handler) Base

NewBase builds the shared part of a language extractor from the language's registry name and its directive dispatch table.

The handlers usually close over the extractor being constructed, so the normal shape is to allocate the extractor first and assign its Base second.

#RenderTable

func RenderTable(headers []string, rows [][]string) (string, error)

RenderTable renders a Markdown table in the one form the extractors emit: no per-column alignment and no padding. It exists so the nine extractors share a single call into the table renderer.

#NewJSONObject

func NewJSONObject() *JSONObject

NewJSONObject builds an empty object.

#DecodeJSON

func DecodeJSON(data []byte) (any, error)

DecodeJSON decodes JSON text into the value model this package renders: nil, bool, int64, float64, string, []any and *JSONObject.

Object key order is kept, and an integer literal stays an integer rather than becoming a float, because the config tables print those as different types.

#JSONTypeName

func JSONTypeName(value any) string

JSONTypeName names a decoded value's type as the config tables print it.

#JSONValueRepr

func JSONValueRepr(value any) string

JSONValueRepr renders a decoded value compactly enough for a table cell: a scalar verbatim, a long string truncated, and a collection as its size.

#RenderJSONIndent2

func RenderJSONIndent2(value any) string

RenderJSONIndent2 renders a decoded value the way Python's json.dumps(value, indent=2) does, byte for byte: two-space indentation, ": " between a key and its value, "{}" and "[]" for the empty collections, and every non-ASCII character escaped.

Keys are NOT sorted -- the call site this serves prints a JSON document whose top level is not an object, and reordering what the file said would misreport it.

#ConfigTableFromJSON

func ConfigTableFromJSON(value any, displayPath string, excludeKeys []string) (string, error)

ConfigTableFromJSON renders decoded JSON as the Key/Type/Value table.

A document whose top level is not an object has no keys to tabulate, so it is rendered as a fenced, indented JSON block instead.

#ConfigTableFromJSONText

func ConfigTableFromJSONText(text []byte, displayPath string, excludeKeys []string) (string, error)

ConfigTableFromJSONText parses JSON text and renders it as the Key/Type/Value table, or the error marker when it does not parse.

#ConfigFromJSON

func ConfigFromJSON(fullPath, displayPath string, excludeKeys []string) (string, error)

ConfigFromJSON reads a JSON config file and renders it as the Key/Type/Value table.

#ConfigFromTOML

func ConfigFromTOML(fullPath, displayPath string, excludeKeys []string) (string, error)

ConfigFromTOML reads a TOML config file and renders its leaf keys as the Key/Type/Value table, with nested tables flattened into dotted keys.

#HandleTableConfig

func HandleTableConfig(

HandleTableConfig is the shared table-config handler. It reads a JSON or TOML config file and renders it as a key/type/value table; any other extension is shown verbatim in a fenced block, since there is nothing to tabulate.

A language that recognizes a further config format maps table-config to its own handler and delegates here for everything it does not handle -- what the TypeScript extractor does for JSONC.

#IsFile

func IsFile(path string) bool

IsFile reports whether path is an existing regular file, the question Python's os.path.isfile answers -- which every extractor asks of a candidate it is resolving.

#IsDir

func IsDir(path string) bool

IsDir reports whether path is an existing directory, the question Python's os.path.isdir answers.

#ParseDocstringSections

func ParseDocstringSections(text string) DocSections

ParseDocstringSections parses a Google-style docstring into its structured sections. It is how the quality measurement learns which parameters and return values a symbol's own documentation covers.

#FormatDocstring

func FormatDocstring(docstring string, baseLevel int) string

FormatDocstring transforms Google-style docstring sections into markdown.

Section headers like "Args:", "Returns:" and "Raises:" followed by indented "name: description" lines become bold headers with bullet lists, so the markdown converter renders them as structured HTML instead of collapsing the whitespace.

Source-wrapped prose (Go, JSDoc and KDoc doc comments wrap at around 75 columns) is first normalized with prose.JoinWrappedLines so a soft-wrapped sentence becomes one line; blank-line paragraph breaks, indented preformatted blocks, fenced code, list items and doctest lines are left verbatim.

baseLevel is the level of the heading this text is emitted under, or 1 for the page title when it is emitted alone. Headings the doc comment wrote are renested beneath it; see DemoteDocHeadings. It has no default, because every caller knows where it is putting the text and a wrong guess puts a second H1 on the page.

#FormatError

func FormatError(message string) string

FormatError renders a message as the block-quoted marker selfdoc leaves in place of a directive it could not resolve.

#SymbolHeading

func SymbolHeading(level int, name string) string

SymbolHeading is a heading naming something the extractor read out of source.

A class, a function, a struct field, a table, a module path -- every one of them is a token the generator copied from code, and it is emitted as a code span so the page says so.

That is a claim about what the text is, and two readers act on it. A human sees a symbol set in the page's code face rather than a word in its prose face. The spell checker sees a code span and does not read it: it masks inline code, so an identifier it cannot recognize as English -- JSONResponse, returncode -- stops being a misspelling the moment the generator marks it as what it is. What remains flagged is docstring prose, which is the author's writing and the author's to fix; an identifier written into a sentence is backticked by whoever wrote the sentence.

The heading's anchor is unaffected: anchors are slugified from the rendered inline form, and rendering a code span leaves the same text, so #jsonresponse still addresses the same heading.

#SymbolSpan

func SymbolSpan(name string) string

SymbolSpan is a token the extractor read out of source, inline in generated prose.

The heading form of the same claim is SymbolHeading; this is what a generated list item, a table cell or a rendered label uses. A name that came from code is set in the code face and skipped by the spell checker, whichever generated structure it appears in.

#SymbolHeadingPattern

func SymbolHeadingPattern(name string) *regexp.Regexp

SymbolHeadingPattern matches a heading that names name, wherever the heading came from.

The coverage measurement asks one question of a page -- does any heading on it name this symbol -- and pages come from two writers. SymbolHeading writes the code-span form; an author writing a reference page by hand writes whichever form reads well to them. The optional qualifier admits a method named by its owner (Pipeline.Execute).

#DemoteDocHeadings

func DemoteDocHeadings(text string, baseLevel int) string

DemoteDocHeadings renests the headings a doc comment wrote, under the one above them.

A doc comment is written as if it owned a document -- Go's own convention is "# Usage", and a KDoc, docstring or JSDoc block can carry any markdown -- but on a generated reference page it is a subsection of a symbol, and the page already has exactly one H1: its title. Emitted verbatim, a package doc with headings puts a second H1 on the page, which is a hard error, and the symbol the doc belongs to stops being its parent in the outline.

baseLevel is the level of the heading the text sits under: the symbol's own heading, or 1 -- the page title -- for a directive that emits the doc alone. Every heading shifts by the same amount, so the doc's internal structure is kept; the shallowest one becomes the direct child of baseLevel. Nothing shifts when the doc is already nested deeply enough, and nothing goes past H6, markdown's floor.

#ParseCommaSet

func ParseCommaSet(value string) []string

ParseCommaSet splits a comma-separated attribute value into its distinct, trimmed, non-empty parts.

The result is sorted, where the Python set it replaces iterated in an arbitrary order. Only one caller iterates it -- ApplyExcludeKeys, to name the first excluded key a document does not carry -- so sorting turns an arbitrary choice among several missing keys into a stable one.

#ExcludeKeysFromAttrs

func ExcludeKeysFromAttrs(attrs map[string]string) []string

ExcludeKeysFromAttrs reads a directive's exclude attribute as a key set. An absent or empty attribute excludes nothing.

#ApplyExcludeKeys

func ApplyExcludeKeys(data *JSONObject, excludeKeys []string, displayPath string) (*JSONObject, string)

ApplyExcludeKeys drops excludeKeys from data.

It returns the filtered object, or -- when data does not carry one of the keys -- a nil object and the error marker naming it. Excluding a key that is not there is a mistake in the directive rather than a no-op: the author believes they are hiding something, and silently rendering it would publish the very value they meant to withhold.

#ReadSource

func ReadSource(filepath string) (string, error)

ReadSource reads a source file as text.

#ExtractBraceBlock

func ExtractBraceBlock(source string, openBracePos int) (string, bool)

ExtractBraceBlock returns the content between the brace at openBracePos and its match, exclusive of both, and reports whether the braces matched.

String literals are skipped, so a brace inside one does not change the depth, and a backslash escapes the next byte. openBracePos is a byte offset into source, which is what a regexp match index and strings.Index both report.

#CollectCommentLinesAbove

func CollectCommentLinesAbove(lines []string, startLine int, prefix string, skipBlankLines bool) string

CollectCommentLinesAbove collects the contiguous comment lines above startLine whose trimmed form begins with prefix, walking upward and stopping at the first line that does not.

The prefix and one optional following space are removed from each line. With skipBlankLines the walk crosses blank lines between the declaration and the comment block; without it, a blank line means the declaration has no comment. startLine is a zero-based index into lines.

#Dedent

func Dedent(text string) string

Dedent removes the longest common leading whitespace from every line of text, reproducing Python's textwrap.dedent.

Lines made only of whitespace are emptied first and then ignored when the common margin is measured, so a blank line indented less than the block does not defeat the dedent.

#KnownLanguages

func KnownLanguages() []string

KnownLanguages lists every language selfdoc ships an extractor for.

#DetectionOrder

func DetectionOrder() []string

DetectionOrder lists the languages auto-detection tries, in priority order.

#IsKnownLanguage

func IsKnownLanguage(name string) bool

IsKnownLanguage reports whether selfdoc ships an extractor for name.

#Register

func Register(name string, factory Factory)

Register records the factory for a language. Each language package calls it from an init function, so linking the package in is what makes the language available.

It panics on a name that is not a known language and on a second registration of the same name: both are mistakes in the language package itself, visible the moment the binary starts.

#Registered

func Registered() []string

Registered lists the languages whose packages are linked into this binary, sorted.

#Lookup

func Lookup(name string) (Extractor, bool, error)

Lookup builds the extractor registered for a language.

It reports an error for a known language whose package is not linked in, rather than answering with a stub: the project declared a language selfdoc supports, and quietly rendering "no extractor for 'python'" across its pages would blame the project for a wiring mistake in selfdoc. A language selfdoc does not support is not an error -- it returns false, and the caller substitutes NewStub.

#DetectLanguage

func DetectLanguage(dir string) (string, error)

DetectLanguage auto-detects a project's language from the marker files in dir, returning the empty string when none is detected.

#DetectLanguages

func DetectLanguages(dir string) ([]DetectedLanguage, error)

DetectLanguages reports every language detected in dir, not just the first. A polyglot repository answers with several, in detection-priority order.

#ResolveSourceEntries

func ResolveSourceEntries(config map[string]any) ([]SourceEntry, error)

ResolveSourceEntries resolves a config's source declarations into entries carrying their extractors.

A project that publishes no code declares no source entries, so an absent or empty source key yields none.

#SourcePaths

func SourcePaths(config map[string]any) ([]string, error)

SourcePaths lists just the declared source paths. It is empty for a project that publishes no code.

#NewStub

func NewStub(language string) Extractor

NewStub builds the extractor for a language selfdoc has none for.

#Base.Name

func (b Base) Name() string { return b.language }

Name is the language's registry name.

#Base.Extract

func (b Base) Extract(

Extract dispatches a directive to the handler registered for it, passing the path and target attributes out as their own arguments because every handler reads them.

A directive name with no handler renders an error marker naming both the directive and the language, since the same name can be valid for another language.

#Base.FileExtensions

func (b Base) FileExtensions() []string { return nil }

FileExtensions reports that the language claims no file extensions.

#Base.PublicSymbols

func (b Base) PublicSymbols(string) ([]string, error) { return nil, nil }

PublicSymbols reports no symbols, for a language whose extractor cannot read them.

#Base.ResolvePath

func (b Base) ResolvePath(string, []string, string) string { return "" }

ResolvePath resolves nothing, for a language whose extractor has no path convention.

#Base.SymbolDetails

func (b Base) SymbolDetails(string, string) (*SymbolDetails, error) { return nil, nil }

SymbolDetails reports nothing, for a language whose extractor cannot read signatures.

#Base.ModuleDocstring

func (b Base) ModuleDocstring(string) (string, error) { return "", nil }

ModuleDocstring reports no module documentation, for a language whose extractor cannot read it.

#JSONObject.Keys

func (o *JSONObject) Keys() []string

Keys lists the object's keys in the order they appeared.

A nil object has no keys, which is what a decoder that answered "no object here" means. Has, Get, Keys and Len all read a nil receiver as the empty object, so a caller that took an object out of a document does not have to know whether the document carried one.

#JSONObject.Len

func (o *JSONObject) Len() int

Len is the number of keys. A nil object has none.

#JSONObject.Has

func (o *JSONObject) Has(key string) bool

Has reports whether the object carries key. A nil object carries nothing.

#JSONObject.Get

func (o *JSONObject) Get(key string) (any, bool)

Get reports the value at key. A nil object reports nothing.

#JSONObject.Set

func (o *JSONObject) Set(key string, value any)

Set records key's value, appending the key when it is new and leaving its position alone when it is not -- the behavior of assigning into a Python dict, which is what the decoders this replaces did.

#stubExtractor.Detect

func (s *stubExtractor) Detect(string) bool { return false }

Detect reports that no directory is a project in an unsupported language: detection is by marker file, and selfdoc knows none for this language.

#stubExtractor.Extract

func (s *stubExtractor) Extract(

Extract renders the marker naming the language selfdoc cannot read.