About LLM Token? (It's like LLM Atom)

Imagine we are building Lego castle. We hand workes individual Lego bricks.

In the world of Large Language Models (LLMs), tokens are those Lego bricks, the fundamental atoms of language.

LLM doesn't see words the way we do. It doesn't read letter by letter, either. Instead, it chops text up into chunks called tokens. A token can be a whole word, part of a word (like a prefix or suffix), or even just a punctuation mark.

Token and Words

One token is roughly 0.75 words, or 100 tokens is about 75 words.

Real-World Example: How an LLM sees text.

Reading a book is good!

To us, that's 5 words. But to LLM, it's tokenized like this:

Token 1: Reading
Token 2: a
Token 3: book
Token 4: is
Token 5: good
Token 6: !

Total 6 tokens for 5 words.

Longer or less common words get chopped up even more. Example "metamorphic rock"

Token 1: meta (Prefix atom)
Token 2: morph (Root/subword atom)
Token 3: ic (Suffix atom)
Token 4: rock (Note the leading space is bundled into this token!)
Total Count: 4 tokens for 2 words

This matters for the cost and your AI bill.

When you use paid AI APIs or tools, companies charge you per token.

Generating text (output) is more expensive (3 to 5 times more) than reading text you send it (input).

The Technical Reason: LLMs generate text autoregressively, meaning they can only write one token at a time.

Example:

1. To write a 500-word essay, LLM has to run 500 separate times, predicting the next token, locking it in, and then using that token to predict the next one.
2. Reading an input prompt can be processed all at once in parallel (which takes way less computational power). Because writing requires more time and energy from the computer's GPUs, you pay a heavy premium for the output.

What does "150K Tokens" Mean? (Capacity)

When you hear LLM "150K token context window" (150,000 tokens), it's "size of LLM short-term working memory".

Everything on that memory counts toward the limit:

1. The prompt instructions you give it.
2. The documents you upload for it to read.
3. The conversation history, all your previous back-and-forth messages.
4. The response the AI generates back to you.

If 1,000 tokens is 750 words:

150,000 tokens is roughly 112,500 words.

That is equivalent to reading 350-to-400-page text all at once, and being able to answer questions about any page in it!

If you exceed that limit (e.g., put 500-page book), LLM starts suffering from "forgetfulness", it will literally drop the oldest parts of text to make room for new data.