Skip to content
internal/tokenizer
Updated
On this page

A standalone Markdown block tokenizer that splits source into typed tokens carrying line numbers, covering every line of the input once with no overlaps.

#internal/tokenizer

#internal/tokenizer

Package tokenizer is a standalone Markdown block tokenizer.

It splits Markdown source into a flat slice of typed block tokens. Each token carries 1-based start and end line numbers, so a caller knows which source lines produced it; the tokens cover every line of the input once, with no gaps and no overlaps.

The package imports nothing of selfdoc's -- it is designed for reuse outside the project.

#Span

Go go
type Span struct

Span is the 1-based, inclusive range of source lines a token covers. Every token type embeds it, which is how each satisfies [Token].

#Token

Go go
type Token interface

Token is one block-level element of a tokenized document. Every token reports the source lines it came from.

#TextBearingToken

Go go
type TextBearingToken interface

TextBearingToken is implemented by every token type that carries prose a reader sees on the page.

Code blocks, directives, thematic breaks and blank lines are excluded structurally: a rule that reads prose runs over exactly the types implementing this interface, and can never reach a fenced example or an unresolved directive marker.

Headings and tables are members. They were left out originally because their text is not stored line-per-line the way a paragraph's is, which meant every prose rule -- empty alt text, meaningless alt text, generic anchor text, broken cross-references, spelling -- silently skipped page titles and every table cell. [TokenTextLines] handles their shapes, so the exclusion is gone.

#Annotation

Go go
type Annotation struct

Annotation is one trailing "[N]: note" line attached to a fenced code block.

#CodeBlock

Go go
type CodeBlock struct

CodeBlock is a fenced code block, plus the annotation lines that follow its closing fence.

#Heading

Go go
type Heading struct

Heading is an ATX heading.

#Table

Go go
type Table struct

Table is a run of pipe-delimited rows, each stripped of surrounding whitespace. Separator rows are kept: parsing the table's structure is the renderer's job, not the tokenizer's.

#UnorderedList

Go go
type UnorderedList struct

UnorderedList is a run of "-" or "*" list items with their markers stripped.

#OrderedList

Go go
type OrderedList struct

OrderedList is a run of "N." list items with their markers stripped.

#Blockquote

Go go
type Blockquote struct

Blockquote is a run of ">"-prefixed lines, each with the marker and one following space removed.

#DefinitionEntry

Go go
type DefinitionEntry struct

DefinitionEntry is one term with its definitions.

#DefinitionList

Go go
type DefinitionList struct

DefinitionList is a run of term-and-definition pairs, possibly separated by blank lines the token also covers.

#ThematicBreak

Go go
type ThematicBreak struct

ThematicBreak is a horizontal rule: a line of only dashes, asterisks or underscores.

#BlankLine

Go go
type BlankLine struct

BlankLine is a line that is empty or only whitespace.

#Directive

Go go
type Directive struct

Directive is the legacy ":::name arg ... :::" block form. The newer marker syntax is parsed elsewhere; this token exists because documents still carry the old form.

#Paragraph

Go go
type Paragraph struct

Paragraph is a run of lines that no other block claimed.

#Tokenize

Go go
func Tokenize(content string) []Token

Tokenize splits Markdown content into a slice of block tokens.

Line numbers on every token are 1-based, counting the first line of the input as 1. The tokens cover every line once -- no gaps and no overlaps -- which is what lets a caller map any diagnostic back to a source line.

The dispatch order is fixed and each step below is tried in turn: fenced code block, thematic break, heading, ":::" directive, table, unordered list, ordered list, blockquote, blank line, definition list, and finally a paragraph, which absorbs everything no earlier step claimed. The order is what makes "---" a thematic break rather than a heading, and what keeps a "#" line inside a fence out of the heading branch.

#IsTextBearing

Go go
func IsTextBearing(tok Token) bool

IsTextBearing reports whether tok carries prose a reader sees on the page -- whether it implements [TextBearingToken].

#TokenTextLines

Go go
func TokenTextLines(tok Token) []string

TokenTextLines returns the text a content-bearing token contributes, one entry per line.

The i-th entry corresponds to source line tok.Start()+i, so a caller holding a token and an index knows the real line a diagnostic belongs on. A token that bears no text returns nil.

The text is the token's parsed text: list markers, blockquote markers and heading hashes are already stripped, so a column measured against these strings is not a column in the source line. A caller that needs exact columns should read the raw source lines the token spans instead (tok.Start() through tok.End()).

#Span.Start

Go go
func (s Span) Start() int { return s.StartLine }

Start returns the first source line the token covers.

#Span.End

Go go
func (s Span) End() int { return s.EndLine }

End returns the last source line the token covers, inclusive.

#Heading.TextLines

Go go
func (h Heading) TextLines() []string { return []string{h.Text} }

TextLines returns the heading's text as a single entry.

#Table.TextLines

Go go
func (t Table) TextLines() []string { return t.Rows }

TextLines returns the table's rows, one entry per source line.

#UnorderedList.TextLines

Go go
func (l UnorderedList) TextLines() []string { return l.Items }

TextLines returns the list's items, one entry per source line.

#OrderedList.TextLines

Go go
func (l OrderedList) TextLines() []string { return l.Items }

TextLines returns the list's items, one entry per source line.

#Blockquote.TextLines

Go go
func (b Blockquote) TextLines() []string { return b.Lines }

TextLines returns the quoted lines, one entry per source line.

#DefinitionList.TextLines

Go go
func (d DefinitionList) TextLines() []string

TextLines returns each term followed by its definitions, flattened.

#Paragraph.TextLines

Go go
func (p Paragraph) TextLines() []string { return p.Lines }

TextLines returns the paragraph's lines, one entry per source line.

More tools from this site

  • claudestream Drive Claude Code from Python: run it as a subprocess and read its output as typed events, with async and sync sessions, sandbox policies, and tools you define in Python
  • claudewheel A TUI Claude Code Launcher that lets you have more than one profile, manage sessions lifecycle, pick the exact CC version, model to use (even older unlisted ones), pick which GitHub account to use, etc.
  • dirstat Fast, single-binary directory statistics CLI: every file under a tree grouped by format, with counts, sizes, and lines of code, as a colored terminal table or as JSON
  • fastware A batteries-included ASGI framework: msgspec JSON, a managed Granian server, dependency injection, SSE, WebSockets, auth, and a test client
  • go-toml-edit Zero-dep TOML editing library for Go with comment preservation
  • howmuchleft The fastest Claude Code statusline: context window, 5-hour, and weekly limit usage as three customizable gradient bars, rendering in about 6 ms
  • orxtra
  • pgdesign
  • predraw Declarative rendering pipeline: describe a scene in JSON and get SVG, PNG and WebP out, with light and dark style tokens, reusable components and text converted to path outlines
  • reposummary Turn a git repository's history into a Markdown journal: pick a time window or revision range and get a readable digest of what changed, optionally narrated by an LLM
  • rlsbl Release orchestration and project scaffolding CLI that bumps versions, validates a structured JSONL changelog, tags only the commit CI verified, and publishes to npm, PyPI, Go and more
  • safegit git wrapper CLI that gives each commit its own temporary index and retries ref updates on conflict, so concurrent agents share one repository
  • saferm Command-line replacement for rm that archives every deletion with a mandatory reason and the context it ran in, so deleted files can be listed, inspected and restored
  • strictcli
  • stricttest An always-on test-isolation floor: a pytest plugin and a Go env-hygiene module that make a test suite structurally unable to reach real credentials, the real HOME, the network, or the development repository.
  • wesktop A Python framework that turns an ASGI web app into a desktop application, serving it from a local Granian server and displaying it in a native OS window via pywebview
Search