Skip to content
Codextroop TechnologiesCodexLab

Text as Tokens

Language models read text as tokens, small pieces that may be whole words or parts of words. Limits count tokens, not pages.

Class 9 to 10beginner18 min3 lessons1 interactive lab
By the end you will be able to
  • Define a token as a chunk of text the model reads
  • Explain why the same sentence can use different token counts
  • Use the token split lab to see a line break apart
Topic contents

Lesson 1 of 3

Not always whole words

When you chat with an AI, it does not see letters one by one the way you might spell a word. It cuts text into tokens. A token might be a short word, a piece of a long word, or even punctuation.

Breaking text this way lets one model handle many languages and spellings with a fixed dictionary of pieces.

Open the token split lab. Type a sentence or pick a sample and watch it split. Notice how common words stay whole while rare long words break.

Interactive lab

Token split

Type a sentence or pick a sample and see how it splits into tokens. Compare short words and long rare words.

Token splitter

How chatbots chop text
Simplified split for learning. Real models use trickier rules.
1 / 7

Start

Models do not read whole words the way you do. They chop text into tokens, which can be word pieces or punctuation. This lab uses a simplified split for learning.

Continue

Lesson 2 of 3

Limits you actually hit

Apps say you reached the limit because the conversation filled the token window. That window is how many tokens the model can consider at once for memory and reply.

Long pasted essays eat tokens fast. Short clear questions leave room for useful answers.

Emoji and code also become tokens. Weird spelling can use more pieces than normal words.

Continue

Lesson 3 of 3

Why tokens matter later

The next topic shows what the model does with tokens: guess the next one, again and again, until a full reply appears.

Knowing about tokens helps you write prompts that fit. Put the important question at the end after brief context.

Tokens are not thoughts. They are packaging for text the machine can count.

Continue

Practice

Work these out yourself

No answer key here on purpose: these are the questions worth thinking through before you move on. Open one and work it out.

0/3 attempted

Assessment

Check your understanding

Answer each question, then read the explanation. That is where the learning is.

0/4
  1. Question 1: What is a token in this lesson?
    Question 1 / 4

    What is a token in this lesson?

    Select an option first
  2. Question 2: Context limits mostly count what?
    Question 2 / 4

    Context limits mostly count what?

    Select an option first
  3. Question 3: Rare long words often split into what?
    Question 3 / 4

    Rare long words often split into what?

    Select an option first
  4. Question 4: Tokens are best described as what?
    Question 4 / 4

    Tokens are best described as what?

    Select an option first
4 of 4 questions left.

Finished this topic?

0 of 3 lessons marked done. Marking the topic complete ticks the rest.