What are tokens in AI? Definition and examples

Tokens are the small units of text, often a word or part of a word, that language models read and write, and the unit in which AI usage is usually priced.

How it works

Before a language model sees any text, a tokenizer splits it into pieces from a fixed vocabulary. Google's crash course puts it simply: "A token could be a word, a subword (a subset of a word), or even a single character" (Google). Common words are usually one token; rare words, names, numbers and code get split into several.

Rules of thumb vary by model and language. For Gemini models, Google says a token is about 4 characters and that 100 tokens equal about 60 to 80 English words (Gemini API docs). Other model families use different tokenizers, so the same text can count differently. Languages other than English, and text full of numbers or code, often need more tokens for the same meaning.

Images, audio and video are counted in tokens too. Gemini's documentation, for example, gives fixed token costs per image tile and per second of audio.

Why it matters for your website

Tokens drive three things you will run into with any AI tool:

• Cost. AI providers usually bill per input and output token. Vendors who sell to you often wrap that in their own unit, such as "messages", "credits", "answers" or "conversations", and the conversion is not always one-to-one.
• Limits. Everything in a request counts against the model's context window: Anthropic's documentation lists the system prompt, every message, tool definitions and the reply itself (Claude docs).
• Retrieval design. Content is split into passages of a set token size by chunking before indexing, so very long pages are never seen whole by the model.

When you compare AI chat tools, ask what one billing unit includes: does a long answer with five source passages cost the same as a one-line reply?

Example

A rough illustration of how a tokenizer might split a sentence (exact splits differ by model):

Text:    "Our Pro plan includes CRM sync."
Tokens:  "Our" | " Pro" | " plan" | " includes" | " CR" | "M" | " sync" | "."
Count:   8 tokens for 6 words

By Google's rule of thumb for Gemini, a 1,500-word help article comes to roughly 1,900 to 2,500 tokens, before the question, instructions and answer are added.

Related terms

• Large language model (LLM): the model that reads and writes tokens.
• Context window: the token budget of a single request.
• Chunking: splitting content into token-sized passages.
• Embeddings: numeric representations computed from tokens.