Module leo_parser::tokenizer
source · Expand description
The tokenizer to convert Leo code text into tokens.
This module contains the tokenize()
method which breaks down string text into tokens,
optionally separated by whitespace.
Re-exports§
pub use self::token::KEYWORD_TOKENS;
Modules§
Functions§
- tokenize 🔒Creates a new vector of spanned tokens from a given file path and source code text.
- Yields spanned tokens from the given source code text.