Tokens

Token definitions categorize lexical elements in TLSQL statements. The system consists of:

• TokenType: Enumeration of all token types.

• Token: Data class representing a token with type, value, and position.

Token Categories

• Keywords: TRAIN, PREDICT, VALIDATE, WITH, FROM, WHERE.

• Operators: Comparison (>, <, >=, <=, =, !=) and logical (AND, OR, NOT).

• Literals: IDENTIFIER, STRING, NUMBER.

• Punctuation: Parentheses, commas, semicolons, dots, asterisks.

• Special: EOF marker.

class TokenType(value)

Bases: Enum

Token types for TLSQL language.

class Token(type, value, line_num, col_num)

Bases: object

Represents a single token in the input.

type

Token type.

value

String value of token.

line_num

Line number where token is located.

col_num

Column number where token is located.