Skip to content
internal/extractors/python
Updated
On this page

Resolving selfdoc's directives against Python source with a pure-Go parse of the file, reproducing what the standard library's ast module reports.

#internal/extractors/python

#internal/extractors/python

Package python resolves selfdoc's directives against Python source.

#What the pages are built from

Every Python reference page selfdoc has ever produced was rendered from the stdlib ast module: ast.unparse decides how an annotation, a default value and a base class read, and the tree's child order decides what order the page lists symbols in. Both are reproduced here over a tree-sitter parse -- a pure-Go one, with the Python grammar's tables read from the parser library's embedded blob -- so documenting a Python project needs no interpreter on the machine and no cgo in the build.

The split inside the package follows that: parse.go reads the tree into the shape ast has (docstrings, signatures, line spans, __all__, re-export statements, the dataclass and pydantic predicates), unparse.go reproduces ast.unparse's rendering of an expression, and handlers.go does everything a reader sees -- which symbols are skipped, how the Markdown is assembled, how docstring sections are formatted, which parameters count as documented.

A parse is cached per file for the life of the extractor, because one page asks about the same module several times. A file that cannot be read or does not parse answers empty or renders an error marker, as it always has: that is a property of the file, not of the machine.

#Extractor

Go go
type Extractor struct

Extractor reads Python source through an in-process tree-sitter parse.

#FirstSyntaxErrorLine

Go go
func FirstSyntaxErrorLine(source []byte) (line int, failed bool, err error)

FirstSyntaxErrorLine reports whether source fails to parse, and the one-based line the first failure sits on.

It is the syntax question on its own, for a caller that wants nothing else out of the file -- the documentation-example check, which asks it of a fenced code block rather than of a module.

Indentation is not a failure here. The grammar admits a fragment lifted out of a function, an unexpected indent, a dedent matching no outer level and a block that was never indented at all, where CPython raises IndentationError for each. The example check exempted those anyway, so the exemption is now structural rather than a case it has to recognize.

#New

Go go
func New() extractors.Extractor

New builds the Python extractor. Reading a Python file is a file read and a parse, and neither spawns anything.

#Extractor.Detect

Go go
func (e *Extractor) Detect(dir string) bool

Detect reports whether dir carries a Python project's marker files.

#Extractor.FileExtensions

Go go
func (e *Extractor) FileExtensions() []string { return []string{".py"} }

FileExtensions is the single extension Python owns.

#Extractor.ResolvePath

Go go
func (e *Extractor) ResolvePath(pathArg string, sourcePaths []string, baseDir string) string

ResolvePath resolves a dotted module path, a package path or a file path to a .py file.

#Extractor.PublicSymbols

Go go
func (e *Extractor) PublicSymbols(file string) ([]string, error)

PublicSymbols lists the symbols a module exports.

A module that defines __all__ as a literal list or tuple of strings is taken at its word -- those names ARE its public API, underscore-prefixed ones included. Otherwise the heuristic applies: top-level functions and classes whose name does not begin with an underscore.

#Extractor.ModuleDocstring

Go go
func (e *Extractor) ModuleDocstring(path string) (string, error)

ModuleDocstring is a module's own docstring, with soft-wrapped prose joined.

#Extractor.SymbolDetails

Go go
func (e *Extractor) SymbolDetails(file, symbol string) (*extractors.SymbolDetails, error)

SymbolDetails reports one symbol's parameters and return value.

A dotted name selects a member of a class (MyClass.my_method); a plain name is looked for among the top-level functions and classes first, and then among each class's methods, in the order the file declares them.

Search