Languages

A language in Lumis is a Tree-sitter parser: a WebAssembly module that Lumis compiles and loads. You have two ways to get one, and you can use both.

On demand is the default and needs no setup. Highlight a document and Lumis loads whatever that document turns out to name. Preloading is the same work, moved off your users' first request — it is the "pick a language" step done ahead of time instead of during a request.

Both end in the same place. Preloading only decides when you pay.

On demand

Nothing to declare. Lumis resolves and loads a language the first time a document names it, then reuses it for every later document.

Where the bytes come from depends on the runtime. In Elixir and Node a parser is an ordinary dependency: mix deps.get and npm install deliver it, and highlighting reads it off the code path without touching the network. The CLI has no manifest to read, so it downloads and verifies a parser the first time it needs one. A browser loads the parser packages the application imported, and downloads nothing unless a resolver is configured.

:::note A package is named after the WebAssembly parser it carries, not after a language, because the parser is what Lumis loads. Most parsers serve one language, and the package's default export is that language. A few serve more than one, and export each by name: @lumis-sh/wasm-embedded-template has ejs and erb, and @lumis-sh/wasm-markdown has markdown and mdx. Language catalog lists the package for every language.

A grammar split in two is a dependency of the package that needs it, on npm and on Hex alike: installing @lumis-sh/wasm-markdown or lumis_wasm_markdown brings markdown_inline along. :::

That includes languages injected inside another one, in the same pass: a Markdown file with a fenced Rust block highlights that block, and an HTML page highlights its <style> and <script> contents, without any of them appearing in your code.

A language that cannot be loaded costs its own block, not the document. A thousand-line file still highlights when one fenced block names something the project never installed; that block stays plain. A parser package installed after the process started is picked up when the process restarts.

The cost is a Wasmtime compile: seven parsers take about 8 s to compile, against 1.3 s once their compiled forms are stored. Registering an already-compiled parser is 3-15 ms, and about 0.3 ms after that. So the penalty is real but it is paid once per language, by whichever request arrives first.

That is fine for a CLI, a script, a build step, or anything where one slow first render does not matter. Preload when it does.

:::note Browsers are the exception. A browser cannot download a parser in the middle of highlighting, so a language has to be loaded somewhere on the page before a document injects it, including one from a registered bundle. It does not have to be loaded into the highlighter doing the work: once any highlighter on the page has loaded a language from its package, every other highlighter without a resolver uses it for injected blocks too. A global resolver set with languages.configureWasmResolver() or languages.configureLanguagePackageResolver() counts as a resolver for every highlighter, so none of them borrow. Node does not have this limit: it highlights through a native addon. :::

Preloaded

Name the languages you want. Which call depends on whether the process doing the work is the one that will serve:

Loading compiles the language and keeps it in this runtime, so nothing compiles it again. It also writes the compiled module to the data directory, where it survives restarts. A later process that loads the same parser from that directory reuses it instead of compiling.

A process that will serve loads in the background. A process that is not serving, such as an image build or a release task, runs the same load, waits for it, and exits. What it leaves behind is the compiled modules, which the serving process picks up. Compiled modules only carry over to the same platform and Lumis version, so run that step where the server's image is built, with the same parser packages installed.

The CLI's lumis languages download fills the CLI's own store. Node and Elixir load parsers from the packages a project installed, not from that store, so for them the CLI saves only the compile, and only when it fetched the same parser version the project installed. Run the application's own load instead.

:::note Preloading is an optimization, never a prerequisite. An application that starts cold serves correctly the whole time it is loading, so preloading must never delay a boot or fail one — every example below keeps both true. :::

Which languages to name

Name your root languages and anything they can inject. Markdown may need the languages used in fenced code blocks, and HTML may inject CSS and JavaScript. Prefer a focused list or a small bundle. Anything you miss still loads on demand.

bundle-full is not the safe default

Every language in a process shares one 128 MB Tree-sitter Wasm store, and nothing is ever reclaimed from it. The bound is the total of everything loaded, so the published set can exhaust it, and the languages that fail are whichever were asked for last.

Preload what you serve. Installing a parser is cheap; loading one is what spends the budget.

bundle-full leaves out big parsers so it fits under 80 MB: arduino, d, fortran, fsharp, julia, lean, nim, ocaml_interface, systemverilog, and tlaplus.

Add languages a bundle leaves out

A bundle is a starting set, not a limit. Install the parser package for each language you need and pass it next to the bundle:

npm install @lumis-sh/wasm-bundle-full @lumis-sh/wasm-julia

import { createHighlighter } from "@lumis-sh/lumis";
import full from "@lumis-sh/wasm-bundle-full";
import julia from "@lumis-sh/wasm-julia";

const highlighter = await createHighlighter({ languages: [full, julia] });

Language catalog has every name, the package each one resolves to, and what each one costs: Size on disk and Memory against this budget. Neither column is what you download — a package manager fetches the parser gzipped, which is several times smaller.

All of these operations are safe to repeat. A normal run reuses valid files; the CLI's --force is an explicit update that resolves the compatible package range again.

Preloading, per runtime

import * as languages from "@lumis-sh/lumis/languages";

await startServer();

// Not awaited, so a slow compile delays no request. The
// `.catch()` is required: an unhandled rejection
// terminates the process, which is exactly the failure
// loading in the background is meant to avoid.
languages.load(["javascript", "html", "css"]).catch(
  (error) => {
    logger.warn(
      { error },
      "Lumis preload failed; languages load on demand",
    );
  },
);

LUMIS_DATA_DIR=/app/lumis node --input-type=module -e '
  import * as languages from "@lumis-sh/lumis/languages";
  await languages.load(["javascript", "html", "css"]);
'

Point every process at one directory

The CLI, Elixir NIF, and native Node addon keep compiled modules in the same directory, so a parser one of them compiles is already compiled for the others, as long as it is the same parser version. Name the directory explicitly when one process prepares it for another:

Elixir configuration takes precedence over the environment. Set LUMIS_DATA_DIR for the deployed highlighter when it must read a directory a build step prepared.

CLI behavior has the directory layout.