What we learned

Why the server is built the way it is, what we measured, and what we need upstream. See How it works for the short version.

A build and a live parse¶

The server reads a project two ways, because neither covers everything.

Closed files’ targets, titles and enumerators (the 1 in Figure 1) come from the build, read page by page from myst start’s content server. We can’t use myst.xref.json instead, because it isn’t rewritten on edits and has no title or enumerator.

Open documents get a live parse with mystmd’s parser and its per-page transforms, on that page alone. So an open page’s enumerators can differ from the build’s when frontmatter sets numbering.

Directive transforms drop position on the nodes they make (see MyST Author’s data and packaging gaps). mystmd-client’s parseMyst copies the directive’s position onto its children, and src/index-targets.ts falls back to the first positioned descendant in built page JSON.

References come from regexes¶

mystmd’s parser finds targets and warnings, but regexes in src/syntax.ts find where each reference is. The parse tree doesn’t give us usable positions:

• markdown-it records line numbers for blocks, but no columns inside a line.

• In mystmd’s parse of one sample, roles had no position after transforms, and before transforms their columns ran past the end of the line.

• [](#x) links got a placeholder position (column 1 to 1).

• <#x> wasn’t parsed as a reference.

• @key was reported one line early.

Replacing the reference regexes¶

A parser such as myst-parser-unified could replace the reference regexes if it provides:

• Pre-transform myst-spec mdast, so myst-transforms still run on it. mystmd’s transforms use older majors of unified and vfile, so the parser’s trees must still work with them.

• Positions for the parts inside nodes, such as the target in {ref}`text <target>`, the label in (x)=, and directive arguments and options.

• Tolerance of half-typed input, since most documents are mid-edit.

• Enough speed to parse every page in the project for find references.

The swap point is refsInText and labelDefinition in src/syntax.ts, not mystmd-client/parse. refAt and optionAt read the cursor’s context in half-typed text, so they stay regex-based.

labelChars leaves out <> on purpose, so a rename doesn’t end {ref}`text <label>` early. labelDefinition adds them back when matching (x)=, since mystmd allows them there.

Conformance¶

npm run conformance compares the regexes with mystmd’s parse. See Conformance for what it checks, and run it for the current numbers.

We fixed known differences with general rules rather than single cases: trimming padded role targets, accepting link titles, and counting Markdown images as path links. :label: and :name: values with spaces don’t count, because on real projects accepting them read frontmatter name: fields, YAML in code blocks, and option text like a button’s :label: as label definitions. The differences left are in known at the top of scripts/conformance.ts, with the reason for each.

We also ran MYST_CORPUS=<path> npm run conformance by hand on 2026-10-13, on a corpus of public MyST projects:

Most false positives are noise: {myst} example bodies that mystmd keeps as source, code blocks, and frontmatter like a kernel’s name: python3. The one real gap is a link written as [](<url>) with parentheses in the URL, which the regexes read as a path link.

The parse-first experiment¶

The experiment/parse-first branch indexes closed files with the live parser instead of the build. It was never merged. On seven public projects the parse and the build agreed on nearly every target, and the first project-wide completion took about a second with the parse against over a minute for a cold build. The gaps are frontmatter numbering, which the live parse ignores, and mystmd’s own warnings for closed files.

We keep the build because it’s the source of truth for what mystmd renders, including what plugins add.

What we need upstream¶

This server answers mystmd’s LSP and editor support issue. It would be simpler with:

• Column positions in mystmd’s parser, so references can come from the tree instead of regexes.

• A parser switch in mystmd, or a parse-first index, so the build path can go.

• enumerator and title in myst.xref.json, so the server stops reading pages from the content server.

• Structured diagnostics from myst start, so closed files get mystmd’s own warnings at their file and line.

• myst start listening on IPv4 without HOST, so mystmd-client’s startMyst and the configuration page can stop setting it.

• mystmd exporting the helpers in mystmd-client, so this repository can drop it.