Structure representing token extracted from source by tokenizer.
If Type = Group only first operator in a group parameters is stored in the structure. Count member shows how many tokens were treaded as a group.
You can find more information in comments below.
struct Token
{
TokenType Type; // Token type
size_t Index; // Token start index in a source
size_t Length; // Token length
size_t ContentIndex; // Token content start index in a source
size_t ContentLength; // Token content length
String Value; // Token value
String Prefix; // Prefix
String Postfix; // Postfix
size_t Count; // Number of combined operators if it is a grouppable one
bool Finished; // Whether token is finished or ended up with source end
const Operator *Op; // Empty if token is not an operator or group. Operator if token is operator, group or string. First operator if token is a group.
byte Base; // Base for integer and float values
uint64 Integer; // Integer value if token is integer
double Float; // Float value if token is float
};| Namespace: | nitisa::script |
| Include: | Nitisa/Modules/Script/Core/Token.h |