Interface describes tokenizer.
You can find more information in comments below. Overrided methods can be found in corresponding base class or interface.
class ITokenizer :public virtual IReleasable
{
public:
virtual bool Next(size_t &index, Token &token) = 0; // Parse token at specified index. Update "index" and store token information in "token"
virtual bool Next(size_t &index, Tokens &tokens) = 0; // Parse token at specified index, add to tokens list if needed (use grouping). If there are group operators, tokens can be used only after parsing completion
};| Namespace: | nitisa::script |
| Include: | Nitisa/Modules/Script/Interfaces/ITokenizer.h |