Definition
In artificial intelligence, a token is the smallest unit of text that a language model processes. Before a model can read your prompt, a component called a tokenizer splits the text into tokens and converts each one into a number. The model works entirely with these numbers, and when it writes a response it generates one token at a time, which the tokenizer turns back into readable text.
A token is not the same as a word. Common short words like "the" or "and" are usually a single token. Longer or rarer words are split into pieces: "unbelievable" might become "un", "believ" and "able". Spaces, punctuation, numbers and emojis also consume tokens. As a rule of thumb for English, OpenAI estimates that one token is roughly four characters or about three-quarters of a word, so 100 tokens is about 75 words and 1,000 words is about 1,300 tokens.
How tokenization works
Most modern models use subword tokenization methods such as byte-pair encoding (BPE). During development, the tokenizer is trained on a large text sample and learns which character combinations occur most often. Frequent combinations become single tokens; rare ones are built from smaller pieces. This gives the model a fixed vocabulary — typically tens of thousands to a couple of hundred thousand tokens — that can represent any text, including new words, names and code.
Tokenization differs between languages. English is generally the most efficient because tokenizers are trained heavily on English text. Many other languages, including German, Russian, Arabic, Hindi and Japanese, often need more tokens to express the same meaning. That means the same document can be "longer" in tokens, more expensive to process and quicker to fill the context window when written in some languages.
Why tokens matter
Tokens affect almost every practical aspect of using AI:
- Context window. Model limits are measured in tokens, covering your prompt, uploaded material, chat history and the reply.
- Output length. Many tools cap the number of tokens in a single response, which is why long answers sometimes stop mid-sentence.
- Cost. AI APIs and many enterprise contracts charge per million input and output tokens. Output tokens usually cost more than input tokens.
- Speed. Models generate tokens sequentially, so longer answers take longer.
- Quirks. Because models see tokens rather than letters, they sometimes struggle with tasks like counting letters in a word or reversing a string.
Example
A customer-support team wants to use an AI model through an API to summarise incoming support emails. Each email averages 300 words, the instructions add another 150 words, and each summary is about 80 words.
Using the rule of thumb, the team estimates roughly 600 input tokens (300 + 150 words, multiplied by about 1.33) and about 110 output tokens per email. At 20,000 emails a month, that is around 12 million input tokens and 2.2 million output tokens. With published per-million-token prices, the team can calculate the monthly cost in minutes and compare models before committing.
They then optimise. Shortening the instructions from 150 to 60 words and stripping email signatures and quoted reply chains cuts input tokens by more than a third. Asking for a fixed three-bullet summary keeps output tokens predictable. The summaries are just as useful, the bill is lower and responses arrive faster. Understanding tokens turned a vague "AI will cost something" into a concrete, controllable budget line.
Tips for working with tokens
- Write lean prompts. Clear and concise instructions save tokens and often improve quality.
- Remove noise. Strip signatures, disclaimers, duplicated threads and irrelevant appendices.
- Control output length. Specify word counts, bullet counts or a format.
- Use token counters. Tools such as OpenAI's online tokenizer show exactly how text is split.
- Plan for languages. Budget extra tokens for non-English content.
- Watch long chats. Every earlier message is re-sent as context, so long conversations consume tokens quickly.
Key takeaways
- A token is a word or piece of a word; about 0.75 English words on average.
- Context limits, cost and speed are all measured in tokens.
- Non-English text often uses more tokens for the same meaning.
- Lean prompts and controlled output keep AI use fast and affordable.