Custom Formatters

Build your own output format on top of Lumis tokens in Rust, JavaScript, TypeScript, and Elixir.

Write your own formatter when the built-in ones don't produce the output you need. Common reasons:

wrap output with custom HTML
emit Markdown, LaTeX, or a structured AST
overlay your own semantic ranges with annotations
add data attributes or extra classes
feed Lumis tokens into an existing rendering pipeline

Rust

Implement the Formatter trait.

Lumis highlights the source and hands your formatter the resulting event stream. A Start opens a scope, Source events carry byte ranges into the original text, and End closes it.

lumis::formatters::Formatter is a re-export of lumis_core::formatter::Formatter, so one implementation serves both crates: hand it to lumis::highlight(), which parses the source with the grammar language() names, or call render() yourself with an event stream you already have.

use lumis::{
    events::HighlightEvent,
    formatters::html::{
        closing_tags, open_code_tag, open_pre_tag,
        span_inline,
    },
    formatters::Formatter,
    languages::Language,
    themes,
};
use std::io::{self, Write};

struct MinimalHtmlFormatter {
    language: Language,
    theme: Option<themes::Theme>,
}

impl Formatter for MinimalHtmlFormatter {
    fn language(&self) -> Language {
        self.language
    }

    fn render(
        &self,
        source: &str,
        events: &[HighlightEvent<'_>],
        output: &mut dyn Write,
    ) -> io::Result<()> {
        let mut scopes: Vec<&str> = Vec::new();

        open_pre_tag(output, None, self.theme.as_ref())?;
        open_code_tag(output, &self.language)?;

        for event in events {
            match event {
                HighlightEvent::Start { .. } => {
                    scopes.push(
                        event.scope().unwrap_or_default(),
                    );
                }
                HighlightEvent::End => {
                    scopes.pop();
                }
                HighlightEvent::Source { start, end } => {
                    let text = &source[*start..*end];
                    let scope = scopes
                        .last()
                        .copied()
                        .unwrap_or_default();
                    write!(
                        output,
                        "{}",
                        span_inline(
                            text,
                            Some(self.language),
                            scope,
                            self.theme.as_ref(),
                            false
                        )
                    )?;
                }
                _ => {}
            }
        }

        closing_tags(output)?;
        Ok(())
    }
}

render writes into output and returns io::Result<()>. An Err comes back from write_highlight() and write_highlight_with_options(); highlight() panics with it. When a time or match limit bound the highlight, Lumis calls render_budgeted() instead, with the BudgetExhausted that applied. Its default calls render(), so override it only to record the fact in your output.

JavaScript / TypeScript

Pass an object with language and render(source, events). Lumis highlights the source and hands you the event stream; the synchronous free functions highlightIter() and highlightEvents() from @lumis-sh/lumis remain available if you need to highlight something else from inside render().

import {createHighlighter} from '@lumis-sh/lumis'
import type {Formatter} from '@lumis-sh/lumis/formatters'
import {
  openPreTag,
  openCodeTag,
  closingTags,
  spanInline,
} from '@lumis-sh/lumis/formatters/html'
import rust from '@lumis-sh/wasm-rust'
import frappe from '@lumis-sh/themes/catppuccin_frappe'

const hl = await createHighlighter({languages: [rust]})

const formatter: Formatter = {
  language: rust,
  render(source, events) {
    const bytes = new TextEncoder().encode(source)
    const decoder = new TextDecoder()
    const scopes: string[] = []
    const parts: string[] = []

    parts.push(
      openPreTag({preClass: 'docs-demo', theme: frappe}),
    )
    parts.push(openCodeTag(this.language))

    for (const event of events) {
      if (event.type === 'start') {
        scopes.push(event.scope)
      } else if (event.type === 'end') {
        scopes.pop()
      } else if (event.type === 'source') {
        const slice = bytes.subarray(event.start, event.end)
        const text = decoder.decode(slice)
        const scope = scopes[scopes.length - 1]
        parts.push(
          scope
            ? spanInline(text, {
                language: 'rust',
                scope,
                theme: frappe,
              })
            : text,
        )
      }
    }

    parts.push(closingTags())
    return parts.join('')
  },
}

const html = hl.highlight('fn main() {}', formatter)

render returns the document as a string. Its optional third argument, budget, is 'time' or 'matches' when a limit bound the highlight, and undefined otherwise. An error thrown inside render propagates out of highlight() unchanged.

Elixir

Implement the Lumis.Formatter behaviour and pass the module as the formatter. Lumis highlights the source and hands render/3 the event stream, with the resolved language in options.

Lumis.Formatter.HTML and Lumis.Formatter.ANSI have the same HTML and terminal pieces the built-in formatters use.

defmodule MinimalHtmlFormatter do
  @behaviour Lumis.Formatter

  alias Lumis.Formatter.HTML

  @impl true
  def render(source, events, options) do
    language = Keyword.fetch!(options, :language)
    theme = Keyword.get(options, :theme)

    # Resolving a scope against a theme is a table
    # lookup, so build the table once here rather
    # than crossing into Rust once per token.
    attrs =
      HTML.span_attrs(theme: theme, language: language)

    body =
      Enum.map(events, fn
        {:start, %{scope: scope}} ->
          HTML.open_span(attrs, scope)

        :end ->
          "</span>"

        {:source, %{start: start, end: stop}} ->
          len = stop - start
          HTML.escape(binary_part(source, start, len))

        # Lumis adds event kinds as it grows. Render
        # the ones you know and skip the rest, or a
        # newer Lumis raises FunctionClauseError here.
        _event ->
          []
      end)

    [
      HTML.open_pre_tag(theme: theme),
      HTML.open_code_tag(language),
      body,
      HTML.closing_tags()
    ]
  end
end

Lumis.highlight!("defmodule App do\nend",
  formatter:
    {MinimalHtmlFormatter,
     language: "elixir", theme: "github_light"}
)

render/3 returns iodata, so there is no need to flatten it into a binary; Lumis does that once at the end. options also carries :budget_exhausted: :time or :matches when a limit bound the highlight, and nil otherwise. To fail, raise: Lumis.highlight/2 returns {:error, %Lumis.RenderError{reason: :custom_formatter}} naming the formatter and the exception, and Lumis.highlight!/2 raises. When the result is not one string, such as a list of rendered lines, Lumis.highlight_events/3 returns the same event stream without a formatter.

For line-based output, reach for render_lines_from_events/3 rather than splitting rendered markup on newlines. The results contain no line terminators; join wrapped lines with "\n" when building a block. Use the shared line layout CSS for a custom formatter's full-width lines. A <span> that crosses a newline has to be closed and reopened for each line's tags to nest. Swapping span_attrs/1 for span_multi_themes_attrs/1 is the whole difference between one theme and a set of them, restyled later by CSS alone:

defmodule MultiThemeFormatter do
  @behaviour Lumis.Formatter

  alias Lumis.Formatter.HTML

  @themes [light: "github_light", dark: "dracula"]

  @impl true
  def render(source, events, options) do
    language = Keyword.fetch!(options, :language)

    attrs =
      HTML.span_multi_themes_attrs(
        themes: @themes,
        default_theme: "light-dark()",
        language: language
      )

    body =
      source
      |> HTML.render_lines_from_events(events, attrs)
      |> Enum.with_index(1)
      |> Enum.map(fn {line, number} ->
        HTML.wrap_line(number, line)
      end)
      |> Enum.intersperse("\n")

    [
      HTML.open_multi_themes_pre_tag(
        themes: @themes,
        default_theme: "light-dark()"
      ),
      HTML.open_code_tag(language),
      body,
      HTML.closing_tags()
    ]
  end
end

Lumis.highlight!("defmodule App do\nend",
  formatter: {MultiThemeFormatter, language: "elixir"}
)

Lumis.Formatter.ANSI is the terminal counterpart, and styles/1 is where a scope becomes a color. Reach for it rather than theme.highlights, which misses the fallbacks :terminal applies: tag.delimiter is painted by tag in a theme that styles only tag. A theme can also style a scope per language, and an injected block carries its own language, so the table is per language.

defmodule MinimalTerminalFormatter do
  @behaviour Lumis.Formatter

  alias Lumis.Formatter.ANSI

  @impl true
  def render(source, events, options) do
    theme = Keyword.get(options, :theme)

    {output, _scopes, _tables} =
      Enum.reduce(events, {[], [], %{}}, fn
        {:start, %{scope: scope, language: language}},
        {output, scopes, tables} ->
          {output, [{scope, language} | scopes], tables}

        :end, {output, [_scope | scopes], tables} ->
          {output, scopes, tables}

        {:source, %{start: start, end: stop}},
        {output, scopes, tables} ->
          text = binary_part(source, start, stop - start)

          case scopes do
            [] ->
              {[text | output], scopes, tables}

            [{scope, language} | _rest] ->
              # One table per language, built once
              # and read per token.
              tables =
                Map.put_new_lazy(tables, language, fn ->
                  ANSI.styles(
                    theme: theme,
                    language: language
                  )
                end)

              style =
                ANSI.style_for(tables[language], scope)

              painted = ANSI.paint(text, style)
              {[painted | output], scopes, tables}
          end

        # Lumis adds event kinds as it grows. Render
        # the ones you know and skip the rest, or a
        # newer Lumis raises FunctionClauseError here.
        _event, state ->
          state
      end)

    Enum.reverse(output)
  end
end

Lumis.highlight!("defmodule App do\nend",
  formatter:
    {MinimalTerminalFormatter,
     language: "elixir", theme: "dracula"}
)

ANSI.paint/2 takes the nil that style_for/2 returns for an unstyled scope and hands the text back unchanged, so there is no branch to write for it. This minimal example paints only the innermost scope; use :terminal when enclosing scopes should contribute their styles too.

Highlight options

The built-in formatters take options that enrich highlighting. Custom formatters take the same options, as a trailing argument to the token iterator. The flat iterator presents rainbow brackets as the same six scopes the built-ins render for backward compatibility.

highlightIter(
  source,
  this.language,
  frappe,
  (text, language, _range, scope) => {
    // scope is punctuation.bracket.rainbow.1
    // through .6 on bracket pairs
  },
  {rainbowBrackets: true},
)

Nested events instead of flat tokens

highlightIter / highlight_iter hand you one callback per token. Its scope and language identify the innermost open scope; its style combines all open scopes. Foreground, background, and underline style come from the innermost scope that sets each property. Bold, italic, and strikethrough accumulate; false does not clear an enclosing style. A scope with no theme entry inherits its enclosing style, so an unstyled variable inside a colored string keeps the string's color. The terminal formatter uses the same rule.

When nesting matters — a string scope wrapping injected tag scopes inside a template literal — take the events instead. A start opens a scope, source events carry byte ranges, and end closes it. Annotations add their own start/end pairs, carrying your data and a resolved UTF-8 byte range. An invalid annotation range returns an error.

Call highlightEvents inside a formatter’s render method.

import {createHighlighter, highlightEvents} from '@lumis-sh/lumis'
import type {Formatter} from '@lumis-sh/lumis/formatters'
import rust from '@lumis-sh/wasm-rust'

const hl = await createHighlighter({languages: [rust]})

const formatter: Formatter = {
  language: rust,
  render(source) {
    const events = highlightEvents(source, this.language, {
      rainbowBrackets: true,
      annotations: [{
        range: {type: 'offset', start: 4, end: 9},
        data: {id: 7},
      }],
    })

    for (const event of events) {
      if (event.type === 'annotationStart') {
        console.log(event.range, event.data.id)
      }
      if (event.type === 'start') {
        console.log(event.scope, event.language)
      }
      if (
        event.type === 'decorationStart' &&
        event.decoration.type === 'rainbowBracket'
      ) {
        console.log(
          'real zero-based depth',
          event.decoration.depth,
        )
      }
    }
    return ''
  },
}

hl.highlight('let value = (1);', formatter)

Available helpers

Every runtime offers the same set, so a formatter written against one can be ported to another without reimplementing a piece of it. Names follow each host's convention — scope_to_class in Rust and Elixir, scopeToClass in the npm package — and so do argument shapes.

Runtime	Module	Reference
JavaScript / TypeScript	`@lumis-sh/lumis/formatters/html`	source
JavaScript / TypeScript	`@lumis-sh/lumis/formatters/ansi`	source
Rust	`lumis::formatters::html`	docs.rs
Rust	`lumis::formatters::ansi`	docs.rs
Elixir	`Lumis.Formatter.HTML`	hexdocs
Elixir	`Lumis.Formatter.ANSI`	hexdocs

HTML

Capability	Helper
Escape text, an attribute value, or braces a template would read	`escape`, `escape_attr`, `escape_braces`
Resolve a scope to a CSS class, or a style to CSS declarations	`scope_to_class`, `style_to_css`, `text_decoration`
Open a `<span>` for a scope	`open_span`, `span_inline_attrs`, `span_inline`, `span_linked_attrs`, `span_linked`
Open one for a set of themes, as CSS custom properties	`span_multi_themes_attrs`, `span_multi_themes`, `sanitize_theme_name`
Open and close the block	`open_pre_tag`, `open_multi_themes_pre_tag`, `open_code_tag`, `close_pre_tag`, `close_code_tag`, `closing_tags`
Take the block attributes instead of the rendered tag, to merge your own	`pre_attrs`, `multi_themes_pre_attrs`, `code_attrs`
Render a tag from attributes, escaping the values and checking the names	`open_tag`, `valid_attr_name`
Wrap a line, and decide whether it is highlighted	`wrap_line`, `line_is_highlighted`, `highlight_line_class`
Turn the whole event stream into lines, reopening spans across newlines	`render_lines_from_events`

Rendered lines carry their exact source \n or \r\n, after any closing syntax spans. The unterminated final line carries none, so pass each result to wrap_line directly in every runtime.

ANSI

Capability	Helper
Build a color escape	`hex_to_rgb`, `rgb_to_ansi`, `style_to_ansi`
Paint text, and clear formatting	`paint`, `reset`

A few helpers are one runtime's alone, because the boundary dictates the signature rather than taste. Elixir returns whole scope tables from classes/0, span_attrs/1 and ANSI.styles/1, because crossing into Rust once per token to read a constant costs more than the highlighting; the npm package exports encodeSource and decodeSourceSlice, because its strings are UTF-16 and the event offsets are UTF-8 byte offsets. fixtures/formatter-helpers.json records the set, the exceptions, and why each one is one.

Tip

Copy the built-in formatter closest to what you want, then strip out what you don't need.

On this page