Token

In the context of natural language processing (NLP), a token is the smallest unit of text that a language model processes. Tokens can be words, subwords (smaller units within words), or even characters, depending on the model's tokenization method. The number of tokens in a piece of text determines how much "context" the model can handle, and it is often used to measure the length of input sequences.

Generated by Poolnoodle-Deeptought.