
Data & AI
View more blogs
From chatbots and copilots to enterprise AI tools, every prompt comes with a cost. In this blog, Senior Interaction Designer Rachel Derbyshire explores how AI actually processes language, why tokens have become the currency of AI, and what that means for performance, pricing and responsible use.
Introduction
When you type a prompt (whether that's a question or a request) into ChatGPT, Claude, Copilot or any other Large Language Model (LLM), it will generate a response which appears to be thoughtfully composed.
It’s easy to imagine an intelligent robot or computer reading your prompt, comparing it to training data or text, pondering over it and then issuing a thoughtful reply. However, LLMs do not read. They predict using mathematics. Knowing this will help you understand the cost implications.
To understand why AI costs what it does, we first need to understand how large language models process language behind the scenes.
This blog explores:
How AI models process your words
What tokens are and why they drive costs
The hidden costs behind prompts, responses and additional processing
The role of language and tokenisation in pricing and bias
Practical ways to optimise usage and reduce costs
Tokens: turning words into units
When you enter a prompt, the model doesn't see words in the same way a person does. Instead, it converts the text into tokens, which are then translated into numerical representations that the model can process. Without tokenisation, the model cannot process the text in the format it requires.
Let's take a sentence or question like:
“The cat sat on the mat”.
That's a simple example; let's expand on it.
“Why did the cat sit on the mat?”
That's better. As tokens, this might look like:
['why', 'did', 'the', 'cat', 'sit', 'on', 'the', 'mat', '?']
Most LLMs can process long text structure to preserve context when processing language (Anthropic, 2023). However, the exact way words are split into tokens depends on both the model and the language, which is why the same sentence can produce different token counts across different platforms (Petrov et al., 2023).
Note: Tokens aren't restricted to individual words; they can also be parts of words. For example, the word "unhappiness" might be split into three tokens:
[‘un’, ‘happi’, ‘ness’].
Attention and prediction: How LLMs generate responses
When humans read, we skim for key ideas, find familiar patterns, skip redundancies, and infer meaning (Rayner et al, 2012). LLMs, by contrast, examine every token within their available context window and compute mathematical relationships between them using an attention mechanism.
The model repeatedly asks a simple question: given all the tokens so far, what is the most statistically likely next token? (Bergmann and Stryker, 2026).
Models can do this in several ways.
Autoregressive models predict the next token in a sequence, add that token to the context and then repeat the process. Most major models, including GPT-4 (OpenAI), Claude (Anthropic) and Gemini (Google), are autoregressive, meaning they generate each new token based on all the tokens that came before it (Singh, 2023). As a result, every response is built one token at a time.
Because processing and generating tokens requires computational resources, most providers use token-based pricing and rate limits to manage usage. These limits are typically measured using metrics such as Tokens Per Minute (TPM), Requests Per Minute (RPM) or input and output tokens per minute. The exact pricing and rate-limiting approach varies by provider and model.
The table below shows a summary of common rate-limiting metrics used by major AI providers.
AI Company | Model | Rate Limit Metrics |
Anthropic | Claude | ITPM (Input Tokens / Min), OTPM (Output Tokens / Min), RPM (Requests / Min) |
DeepSeek | DeepSeek series | TPM (Tokens Per Minute), RPM |
Gemini series | TPM, RPM, RPD (Requests Per Day) | |
Microsoft | Azure OpenAI / Azure AI | TPM (Tokens Per Minute), RPM |
Mistral | Mistral AI | TPM, RPM |
OpenAI | ChatGPT-4o, GPT-5 series | TPM, RPM |
(Dubey, 2026)
What happens between input and output?
At this point, the LLM has not yet "read" your prompt or generated a response. It has simply converted the text into tokens. Between tokenisation and response generation, however, is where most of the real work happens.
Each token is initially mapped to a numerical embedding. These representations are then processed through the model's neural network layers, where attention mechanisms help capture relationships between tokens and their surrounding context.
For illustration, the all-MiniLM-L6-v2 model (a lightweight, open-source sentence embedding model) maps sentences and short paragraphs into 384-dimensional representations. Its internal processing includes token-level representations, while its standard sentence-transformer output combines these into a single 384-dimensional sentence embedding.

Figure 1, Visualisation of token turned into embedding vector of the sentence “Why did the cat sit on the mat?”. Image created by the author.
Using a small model like all-MiniLM-L6-v2, each token is represented by in the 384-dimensional vector. A prompt containing ten tokens therefore produces ten vectors, each with 384 dimensions. Then using pooling methods reduce it to a single vectors.
This is why many AI services use token counts as the basis for pricing, rather than attempting to charge for the individual numerical values within embedding vectors. It’s much easier to count how many tokens you used than to track and measure all the individual numbers inside each vector.
You might therefore assume that using fewer words or simpler language in your prompts will always reduce costs. However, a vague prompt can generate follow-up questions and additional reasoning, increasing total token usage.
This cost can also apply to the generated tokens in the response to your prompt. Most major providers charge for both input (your prompt) and output (the model's reply) tokens within the same conversation.
The exact token count in the output will depend on the model and method of tokenisation being used (Create a model response, no date).
Hidden Costs, Reasoning and Bias
Beyond the input and output tokens, there are additional tokens that may also be used:
Past conversation tokens: Tokens from previous prompts and responses that are included to provide context for the current interaction. As a conversation grows, the number of these tokens can increase significantly.
Reasoning tokens: Additional tokens used by some models during internal processing to work through a task before generating a response. While not always visible to users, they can contribute to overall token usage and cost.
There is also the issue of language bias.
So far, we have focused on the English language. However, as discussed, LLMs do not read words; they process tokens. A word in one language may be represented by a single token, while a word with the same meaning in another language might be split into three or four tokens.
A 2026 study of semantically identical prompts across 23 languages found that multilingual LLM evaluators assigned systematically different scores depending on the language used. Lower-resource languages often received more generous scores than higher-resource languages.
The authors described this as a structural language-level bias rather than a simple data-quality issue (Zhou et al., 2026). For example, the English word "unhappiness" may be split into three tokens, while its Spanish equivalent, "infelicidad", may be split into two or four, depending on the model and tokenisation method (Petrov et al., 2023).
The research suggests that language behaviour cannot be explained solely by model size or benchmark scores. Language-level factors, including tokenisation and the availability of training resources, also play an important role in observed biases. This means that two users asking the same question in different languages may consume different numbers of tokens, receive slightly different outputs and incur different costs.
Conclusion: Keeping AI costs under control
LLMs process language very differently from humans. They break text into tokens, convert those tokens into mathematical representations and repeatedly predict the most likely next token. Each of these steps consumes computational resources, which is why providers typically charge based on token usage.
Understanding how this process works can help organisations make more informed decisions about how they use AI. While individual interactions may appear inexpensive, costs can quickly grow when prompts become longer, conversations extend over time, reasoning capabilities are enabled or models are used at scale. Factors such as tokenisation, language and model selection can also affect both performance and cost in ways that are not always obvious.
By understanding how tokens are generated, processed and billed, teams can design more efficient prompts, choose the right model for the task and avoid unnecessary expenditure. As AI becomes increasingly embedded in products, services and day-to-day work, understanding the relationship between language, computation and cost is becoming an essential part of using these technologies effectively.
Here are some practical ways to control usage and costs:
Set token limits: Most LLM platforms let you set a maximum token count per request directly in the user interface or through API parameters (such as the max_tokens parameter in the OpenAI API or outputTokenLimit in Anthropic's Claude API). If you know a response should not exceed 250 tokens, set a limit to avoid unexpectedly long and costly outputs.
Use token counter tools: Many platforms provide built-in token counters or prompt analysis tools that allow you to estimate token usage before submitting a prompt, helping you manage costs more effectively.
Implement caching: If you regularly ask similar questions, caching can reduce the need to repeatedly process the same requests. Several providers and third-party tools offer features that store and reuse common responses.
Choose the right model for the task: Not every job requires the largest model. Tasks such as summarisation, classification and document extraction can often be handled by smaller, lower-cost models. Reserving larger models for complex reasoning tasks can lead to significant cost savings.

References
Anthropic (2023) ‘Prompt engineering for Claude's long context window ‘, https://www.anthropic.com/news/prompting-long-context (Accessed: November 10, 2026).
Bergmann, D. and Stryker, C. (2026) 'Attention mechanism,' What is an attention mechanism?, 4 August. https://www.ibm.com/think/topics/attention-mechanism (Accessed: July 17, 2026).
Dubey, K. (2026) LLM pricing comparison: Cost per token across every major model (2026). https://www.opslyft.com/blog/llm-pricing-comparison-cost-per-token (Accessed: July 28, 2026).
Create a model response (no date). https://developers.openai.com/api/reference/resources/responses/methods/create#responses-create-usage (Accessed: July 18, 2026).
Zhou, E. et al. (2026) Lower-Resource, Higher Scores: Language bias in LLM evaluators. https://arxiv.org/abs/2607.14480.
Petrov, A., Malfa, E. L., Torr, P. H. & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. arXiv preprint. https://doi.org/10.48550/arXiv.2305.15425
Rayner, K., Pollatsek, A., Ashby, J., & Clifton, C. (2012). Psychology of Reading (2nd ed.). Psychology Press.
Jha, A. K. (2026). Tokens in LLMs: What They Are and Why They Matter. MemX. https://memx.app/glossary/tokens/
This blog is the authors own work and Grammarly AI was used to improve grammar, spelling and reduce word count.

OpenPerspectives is our platform for Opencast people to share their thoughts and perspectives on modern digital delivery. It offers practical insight into user-centred design, engineering excellence, product leadership, data-driven decision making and building expert capabilities, grounded in real-world experience.
Related Content
View more blogs





