What we learned

This page is for mystmd maintainers and people building MyST tools. It says what this parser found, what an adapter from its tree to mystmd’s covers, and what blocks mystmd from switching.

Why exact positions matter¶

Editors, language servers and source maps need the line and column of every node. mystmd’s markdown-it parser 🟢 records line numbers for blocks, but no columns inside a line. mystmd-lsp’s design doc lists where that leaves positions missing or wrong, like on roles. Directive transforms also drop position on the figure and table containers they create.

Tools work around this. mystmd-client’s parseMyst, which mystmd-lsp calls, copies a directive’s position onto its children before transforms run. mystmd-lsp’s index falls back to the first positioned descendant of a node.

Here every node has an exact line, column and offset, including in role bodies and directive arguments. The mystParse section has a case where mystmd reports column 1 and this parser reports the real column.

Measured results¶

npm run score scores the myst-spec examples five ways, and Score per file has the numbers and what each one counts. The ceiling score is a ceiling for transformed, because where myst-parser itself disagrees with the spec we can’t do better. Most of the MyST examples this parser misses after directives and roles are ones myst-parser misses too. Comparing the transformed and ceiling lists in scoreboard.json shows which. Comparing the transformed and ceiling lists in scoreboard.json shows which.

MYST_CORPUS=<folder> npm run corpus compares both parsers on every .md file in a local folder, skipping dot-folders and _build. Our folder held the docs of mystmd, jupyter-book, 2i2c and example-outputs, plus the docs of mystmd-lsp, myst-syntax and this repo.

markdown-it gives inline nodes on later lines of a paragraph the paragraph’s line, so start lines only agree with mystmd’s on fewer than half the files.

What an adapter covers, and what it can’t¶

toMystmd handles the differences in Differences from mystmd. Two are left:

• Smartquotes need the source, to see which quotes were escaped, and a port of markdown-it’s smartquotes rule.

• Linkify needs markdown-it’s linkify-it.

The same page lists two other differences that no adapter can fix: how CommonMark 0.30 and 0.31.2 end HTML comments, and the last line ending of an HTML block, which markdown-it drops.

What blocks a switch in mystmd¶

mystmd-lsp’s design doc (“Replacing the reference regexes”) lists four requirements for a parser that replaces its reference regexes.

Pre-transform myst-spec mdast, so myst-transforms still run

Partly met. The drop-in mystParse gives myst-parser’s tree and messages on nearly every MyST example. It accepts a vfile@5 or vfile@6. Smartquotes and linkify are the gap.

Positions for the parts inside nodes

Partly met. Directive arguments and options, including each mystOption node, have exact positions. The label in (x)= has no node of its own, only the mystTarget around it. The <target> in {ref}`text <target>` is a positioned node in the role body, which the ref role drops.

Tolerance of half-typed input

Met, as far as mystmd goes. An unclosed fence runs to the end of the document and an unclosed role stays text, like in mystmd. No corpus file threw.

Speed enough to parse every project page

Met, as the corpus table above shows.

What it needs upstream¶

• 🟢 Column positions in mystmd’s parser.

• 🟢 A unified-native parser for mystmd.

• 🟢 myst-spec as a conformance suite that other parsers reuse.

• An mdast adapter in mystmd, which myst-author proposes.

• myst-parser exporting applyDirectives and applyRoles, or accepting a custom tokenizer, so mystParse doesn’t need an exact pin.

Where it could go next¶

• Smartquotes and linkify in toMystmd.

• \r\n line endings: this parser keeps them, and mystmd turns them into \n.

• Typed YAML scalars in option blocks, which need a YAML parser (YAML options).

• mystmd’s error nodes for malformed directive and role headers (inline options).