OpenPerspectives logo

How AI really reads and why it's costing you money

How AI really reads and why it's costing you money

Data & AI

View more blogs

From chatbots and copilots to enterprise AI tools, every prompt comes with a cost. In this blog, Senior Interaction Designer Rachel Derbyshire explores how AI actually processes language, why tokens have become the currency of AI, and what that means for performance, pricing and responsible use. 

Introduction

When you type a prompt (whether that's a question or a request) into ChatGPT, Claude, Copilot or any other Large Language Model (LLM), it will generate a response which appears to be thoughtfully composed. 

It’s easy to imagine an intelligent robot or computer reading your prompt, comparing it to training data or text, pondering over it and then issuing a thoughtful reply. However, LLMs do not read. They predict using mathematics. Knowing this will help you understand the cost implications.  

To understand why AI costs what it does, we first need to understand how large language models process language behind the scenes. 

This blog explores: 

  • How AI models process your words  

  • What tokens are and why they drive costs  

  • The hidden costs behind prompts, responses and additional processing  

  • The role of language and tokenisation in pricing and bias  

  • Practical ways to optimise usage and reduce costs 

Person typing on a laptop with an AI interface open on the screen, with a smartphone beside the laptop.
Person typing on a laptop with an AI interface open on the screen, with a smartphone beside the laptop.
Tokens: turning words into units

When you enter a prompt, the model doesn't see words in the same way a person does. Instead, it converts the text into tokens, which are then translated into numerical representations that the model can process. Without tokenisation, the model cannot process the text in the format it requires. 

Let's take a sentence or question like:  

“The cat sat on the mat”.  

That's a simple example; let's expand on it.  

“Why did the cat sit on the mat?” 

That's better. As tokens, this might look like: 

['why', 'did', 'the', 'cat', 'sit', 'on', 'the', 'mat', '?'] 

Most LLMs can process long text structure to preserve context when processing language (Anthropic, 2023). However, the exact way words are split into tokens depends on both the model and the language, which is why the same sentence can produce different token counts across different platforms (Petrov et al., 2023). 

Note: Tokens aren't restricted to individual words; they can also be parts of words. For example, the word "unhappiness" might be split into three tokens: 

[‘un’, ‘happi’, ‘ness’]. 

Attention and prediction: How LLMs generate responses 

When humans read, we skim for key ideas, find familiar patterns, skip redundancies, and infer meaning (Rayner et al, 2012). LLMs, by contrast, examine every token within their available context window and compute mathematical relationships between them using an attention mechanism.  

The model repeatedly asks a simple question: given all the tokens so far, what is the most statistically likely next token? (Bergmann and Stryker, 2026). 

Models can do this in several ways. 

Autoregressive models predict the next token in a sequence, add that token to the context and then repeat the process. Most major models, including GPT-4 (OpenAI), Claude (Anthropic) and Gemini (Google), are autoregressive, meaning they generate each new token based on all the tokens that came before it (Singh, 2023). As a result, every response is built one token at a time. 

Because processing and generating tokens requires computational resources, most providers use token-based pricing and rate limits to manage usage. These limits are typically measured using metrics such as Tokens Per Minute (TPM), Requests Per Minute (RPM) or input and output tokens per minute. The exact pricing and rate-limiting approach varies by provider and model. 

The table below shows a summary of common rate-limiting metrics used by major AI providers. 


AI Company  

Model  

Rate Limit Metrics  

Anthropic  

Claude  

ITPM (Input Tokens / Min), OTPM (Output Tokens / Min), RPM (Requests / Min) 

DeepSeek  

DeepSeek series   

TPM (Tokens Per Minute), RPM  

Google  

Gemini series   

TPM, RPM, RPD (Requests Per Day)  
  

Microsoft  

Azure OpenAI / Azure AI  

TPM (Tokens Per Minute), RPM  
  

Mistral  

Mistral AI  

TPM, RPM  
  

OpenAI  

ChatGPT-4o, GPT-5 series  

TPM, RPM  

(Dubey, 2026) 

What happens between input and output? 

At this point, the LLM has not yet "read" your prompt or generated a response. It has simply converted the text into tokens. Between tokenisation and response generation, however, is where most of the real work happens. 

Each token is initially mapped to a numerical embedding. These representations are then processed through the model's neural network layers, where attention mechanisms help capture relationships between tokens and their surrounding context. 

For illustration, the all-MiniLM-L6-v2 model (a lightweight, open-source sentence embedding model) maps sentences and short paragraphs into 384-dimensional representations. Its internal processing includes token-level representations, while its standard sentence-transformer output combines these into a single 384-dimensional sentence embedding.

Visualisation of token turned into embedding vector of the sentence “Why did the cat sit on the mat?”. Image created by the author.

Figure 1, Visualisation of token turned into embedding vector of the sentence “Why did the cat sit on the mat?”. Image created by the author.

Using a small model like all-MiniLM-L6-v2, each token is represented by in the 384-dimensional vector. A prompt containing ten tokens therefore produces ten vectors, each with 384 dimensions. Then using pooling methods reduce it to a single vectors. 

This is why many AI services use token counts as the basis for pricing, rather than attempting to charge for the individual numerical values within embedding vectors. It’s much easier to count how many tokens you used than to track and measure all the individual numbers inside each vector.  

You might therefore assume that using fewer words or simpler language in your prompts will always reduce costs. However, a vague prompt can generate follow-up questions and additional reasoning, increasing total token usage.   

This cost can also apply to the generated tokens in the response to your prompt. Most major providers charge for both input (your prompt) and output (the model's reply) tokens within the same conversation. 

The exact token count in the output will depend on the model and method of tokenisation being used (Create a model response, no date). 

Hidden Costs, Reasoning and Bias 

Beyond the input and output tokens, there are additional tokens that may also be used: 

  • Past conversation tokens: Tokens from previous prompts and responses that are included to provide context for the current interaction. As a conversation grows, the number of these tokens can increase significantly. 

  • Reasoning tokens: Additional tokens used by some models during internal processing to work through a task before generating a response. While not always visible to users, they can contribute to overall token usage and cost. 

There is also the issue of language bias.  

So far, we have focused on the English language. However, as discussed, LLMs do not read words; they process tokens. A word in one language may be represented by a single token, while a word with the same meaning in another language might be split into three or four tokens. 

A 2026 study of semantically identical prompts across 23 languages found that multilingual LLM evaluators assigned systematically different scores depending on the language used. Lower-resource languages often received more generous scores than higher-resource languages. 

The authors described this as a structural language-level bias rather than a simple data-quality issue (Zhou et al., 2026). For example, the English word "unhappiness" may be split into three tokens, while its Spanish equivalent, "infelicidad", may be split into two or four, depending on the model and tokenisation method (Petrov et al., 2023). 

The research suggests that language behaviour cannot be explained solely by model size or benchmark scores. Language-level factors, including tokenisation and the availability of training resources, also play an important role in observed biases. This means that two users asking the same question in different languages may consume different numbers of tokens, receive slightly different outputs and incur different costs. 

Conclusion: Keeping AI costs under control

LLMs process language very differently from humans. They break text into tokens, convert those tokens into mathematical representations and repeatedly predict the most likely next token. Each of these steps consumes computational resources, which is why providers typically charge based on token usage. 

Understanding how this process works can help organisations make more informed decisions about how they use AI. While individual interactions may appear inexpensive, costs can quickly grow when prompts become longer, conversations extend over time, reasoning capabilities are enabled or models are used at scale. Factors such as tokenisation, language and model selection can also affect both performance and cost in ways that are not always obvious. 

By understanding how tokens are generated, processed and billed, teams can design more efficient prompts, choose the right model for the task and avoid unnecessary expenditure. As AI becomes increasingly embedded in products, services and day-to-day work, understanding the relationship between language, computation and cost is becoming an essential part of using these technologies effectively. 

Here are some practical ways to control usage and costs: 

  • Set token limits: Most LLM platforms let you set a maximum token count per request directly in the user interface or through API parameters (such as the max_tokens parameter in the OpenAI API or outputTokenLimit in Anthropic's Claude API). If you know a response should not exceed 250 tokens, set a limit to avoid unexpectedly long and costly outputs. 

  • Use token counter tools: Many platforms provide built-in token counters or prompt analysis tools that allow you to estimate token usage before submitting a prompt, helping you manage costs more effectively. 

  • Implement caching: If you regularly ask similar questions, caching can reduce the need to repeatedly process the same requests. Several providers and third-party tools offer features that store and reuse common responses. 

  • Choose the right model for the task: Not every job requires the largest model. Tasks such as summarisation, classification and document extraction can often be handled by smaller, lower-cost models. Reserving larger models for complex reasoning tasks can lead to significant cost savings. 

Three people seated around a table, writing and working together during a group session.

References

Anthropic (2023) ‘Prompt engineering for Claude's long context window ‘, https://www.anthropic.com/news/prompting-long-context (Accessed: November 10, 2026). 

Bergmann, D. and Stryker, C. (2026) 'Attention mechanism,' What is an attention mechanism?, 4 August. https://www.ibm.com/think/topics/attention-mechanism (Accessed: July 17, 2026). 

Dubey, K. (2026) LLM pricing comparison: Cost per token across every major model (2026). https://www.opslyft.com/blog/llm-pricing-comparison-cost-per-token (Accessed: July 28, 2026). 

Create a model response (no date). https://developers.openai.com/api/reference/resources/responses/methods/create#responses-create-usage (Accessed: July 18, 2026). 

Zhou, E. et al. (2026) Lower-Resource, Higher Scores: Language bias in LLM evaluators. https://arxiv.org/abs/2607.14480. 

Petrov, A., Malfa, E. L., Torr, P. H. & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. arXiv preprint. https://doi.org/10.48550/arXiv.2305.15425 

Rayner, K., Pollatsek, A., Ashby, J., & Clifton, C. (2012). Psychology of Reading (2nd ed.). Psychology Press. 

Jha, A. K. (2026). Tokens in LLMs: What They Are and Why They Matter. MemX. https://memx.app/glossary/tokens/ 

This blog is the authors own work and Grammarly AI was used to improve grammar, spelling and reduce word count.  


 

OpenPerspectives logo

OpenPerspectives is our platform for Opencast people to share their thoughts and perspectives on modern digital delivery. It offers practical insight into user-centred design, engineering excellence, product leadership, data-driven decision making and building expert capabilities, grounded in real-world experience.

Related Content

Blog post

Visitors chat at the Opencast exhibition stand inside a bright conference venue. An Opencast banner displaying "Human-focused digital & tech solutions" stands behind a table with branded materials and interactive activities, while attendees gather and talk nearby.
Open Perspectives Logo

Why public sector transformation needs to shift from siloed change to ecosystem transformation

In this blog, Opencast Senior Service Designer Dr Angela Orviz reflects on her presentation at SDinGov 2026, exploring why service designers need a deeper understanding of data and analytics ecosystems. She discusses the shift from siloed transformation to ecosystem thinking, the challenges of data transformation in government, and the role service design can play in supporting more strategic decision-making.

UCD

|

Government

Read more

Blog post

People seated around a table during a collaborative workshop or meeting. One person in the foreground is out of focus, while another participant sits at the table with a takeaway coffee cup and drink can visible. The setting is an informal office or meeting space with dark walls and framed artwork in the background.
Open Perspectives Logo

Connecting policy, delivery and user needs to deliver better public services

In this blog, Juliet Kampasi, Service Design Lead at Opencast, and Ishika Mukherjee, Lead User Researcher, explore why bringing policy, delivery and user needs closer together is essential for creating better public services. Drawing on their experience across government transformation programmes, they discuss how embedding user-centred design earlier in the policy process can help organisations reduce uncertainty, make more informed decisions and deliver services that better meet the needs of citizens and businesses.

UCD

|

Government

Read more

Blog post

Side view of several conference attendees seated in a row during a presentation or workshop.

The data & analytics ecosystem behind public services

Ahead of her SDinGov session, The Data and Analytics Ecosystem: The Hidden Layer of Service Transformation (24 September, 2pm), Dr Angela Orviz explores why service designers need to understand the data and analytics ecosystems that underpin public services and previews the themes that will be discussed during the talk.

UCD

|

Government

Read more

View more blogs