PrismModule.tokenize method
PrismModule › tokenize
Unofficial
Tokenizes text using a grammar.Signature:
tokenize(text: string, grammar: Grammar): Array<PrismToken | string>Parameters:
| Parameter | Type | Description |
|---|---|---|
| text | string | The text to tokenize. |
| grammar | Grammar | The grammar to use. |
Returns: Array<PrismToken | string> — The token stream.