Skip to content
internal/tokenizer
Updated
On this page

A standalone Markdown block tokenizer that splits source into typed tokens carrying line numbers, covering every line of the input once with no overlaps.

#internal/tokenizer

#internal/tokenizer

Package tokenizer is a standalone Markdown block tokenizer.

It splits Markdown source into a flat slice of typed block tokens. Each token carries 1-based start and end line numbers, so a caller knows which source lines produced it; the tokens cover every line of the input once, with no gaps and no overlaps.

The package imports nothing of selfdoc's -- it is designed for reuse outside the project.

#Span

Go go
type Span struct

Span is the 1-based, inclusive range of source lines a token covers. Every token type embeds it, which is how each satisfies [Token].

#Token

Go go
type Token interface

Token is one block-level element of a tokenized document. Every token reports the source lines it came from.

#TextBearingToken

Go go
type TextBearingToken interface

TextBearingToken is implemented by every token type that carries prose a reader sees on the page.

Code blocks, directives, thematic breaks and blank lines are excluded structurally: a rule that reads prose runs over exactly the types implementing this interface, and can never reach a fenced example or an unresolved directive marker.

Headings and tables are members. They were left out originally because their text is not stored line-per-line the way a paragraph's is, which meant every prose rule -- empty alt text, meaningless alt text, generic anchor text, broken cross-references, spelling -- silently skipped page titles and every table cell. [TokenTextLines] handles their shapes, so the exclusion is gone.

#Annotation

Go go
type Annotation struct

Annotation is one trailing "[N]: note" line attached to a fenced code block.

#CodeBlock

Go go
type CodeBlock struct

CodeBlock is a fenced code block, plus the annotation lines that follow its closing fence.

#Heading

Go go
type Heading struct

Heading is an ATX heading.

#Table

Go go
type Table struct

Table is a run of pipe-delimited rows, each stripped of surrounding whitespace. Separator rows are kept: parsing the table's structure is the renderer's job, not the tokenizer's.

#UnorderedList

Go go
type UnorderedList struct

UnorderedList is a run of "-" or "*" list items with their markers stripped.

#OrderedList

Go go
type OrderedList struct

OrderedList is a run of "N." list items with their markers stripped.

#Blockquote

Go go
type Blockquote struct

Blockquote is a run of ">"-prefixed lines, each with the marker and one following space removed.

#DefinitionEntry

Go go
type DefinitionEntry struct

DefinitionEntry is one term with its definitions.

#DefinitionList

Go go
type DefinitionList struct

DefinitionList is a run of term-and-definition pairs, possibly separated by blank lines the token also covers.

#ThematicBreak

Go go
type ThematicBreak struct

ThematicBreak is a horizontal rule: a line of only dashes, asterisks or underscores.

#BlankLine

Go go
type BlankLine struct

BlankLine is a line that is empty or only whitespace.

#Directive

Go go
type Directive struct

Directive is the legacy ":::name arg ... :::" block form. The newer marker syntax is parsed elsewhere; this token exists because documents still carry the old form.

#Paragraph

Go go
type Paragraph struct

Paragraph is a run of lines that no other block claimed.

#Tokenize

Go go
func Tokenize(content string) []Token

Tokenize splits Markdown content into a slice of block tokens.

Line numbers on every token are 1-based, counting the first line of the input as 1. The tokens cover every line once -- no gaps and no overlaps -- which is what lets a caller map any diagnostic back to a source line.

The dispatch order is fixed and each step below is tried in turn: fenced code block, thematic break, heading, ":::" directive, table, unordered list, ordered list, blockquote, blank line, definition list, and finally a paragraph, which absorbs everything no earlier step claimed. The order is what makes "---" a thematic break rather than a heading, and what keeps a "#" line inside a fence out of the heading branch.

#IsTextBearing

Go go
func IsTextBearing(tok Token) bool

IsTextBearing reports whether tok carries prose a reader sees on the page -- whether it implements [TextBearingToken].

#TokenTextLines

Go go
func TokenTextLines(tok Token) []string

TokenTextLines returns the text a content-bearing token contributes, one entry per line.

The i-th entry corresponds to source line tok.Start()+i, so a caller holding a token and an index knows the real line a diagnostic belongs on. A token that bears no text returns nil.

The text is the token's parsed text: list markers, blockquote markers and heading hashes are already stripped, so a column measured against these strings is not a column in the source line. A caller that needs exact columns should read the raw source lines the token spans instead (tok.Start() through tok.End()).

#Span.Start

Go go
func (s Span) Start() int { return s.StartLine }

Start returns the first source line the token covers.

#Span.End

Go go
func (s Span) End() int { return s.EndLine }

End returns the last source line the token covers, inclusive.

#Heading.TextLines

Go go
func (h Heading) TextLines() []string { return []string{h.Text} }

TextLines returns the heading's text as a single entry.

#Table.TextLines

Go go
func (t Table) TextLines() []string { return t.Rows }

TextLines returns the table's rows, one entry per source line.

#UnorderedList.TextLines

Go go
func (l UnorderedList) TextLines() []string { return l.Items }

TextLines returns the list's items, one entry per source line.

#OrderedList.TextLines

Go go
func (l OrderedList) TextLines() []string { return l.Items }

TextLines returns the list's items, one entry per source line.

#Blockquote.TextLines

Go go
func (b Blockquote) TextLines() []string { return b.Lines }

TextLines returns the quoted lines, one entry per source line.

#DefinitionList.TextLines

Go go
func (d DefinitionList) TextLines() []string

TextLines returns each term followed by its definitions, flattened.

#Paragraph.TextLines

Go go
func (p Paragraph) TextLines() []string { return p.Lines }

TextLines returns the paragraph's lines, one entry per source line.

Search