tokenize

Unofficial

Tokenizes a string of code using the given grammar.

Import:

import type { tokenize } from '@obsidian-typings/obsidian-public-latest';

Signature:

function tokenize(text: string, grammar: Grammar): (string | PrismToken)[]

Parameters:

Parameter	Type	Description
text	string	The code string to tokenize.
grammar	Grammar	The grammar to use for tokenization.

Returns: (string | PrismToken)[] — An array of strings and tokens.