Token Vectors and Similarity

A token's vector is a record of the company it keeps. That works well for words with specific company, and fails for words that appear near everything.

Worked example — Token Vectors and Similarity

Cosine similarity, A to B	0.299
A's occurrences in the corpus	230
B's occurrences in the corpus	150
Tokens with a vector	2,742

These are the values the calculator shows when the page opens. Change any input above to recompute them.

Loading interactive simulation...

These vectors count company; a transformer's are learned 🖖

Every number here comes from counting how often two tokens appear within four positions of each other, turned into a score that is large when the pairing is more common than chance. That is a real method with a real name, and it is not what is inside a language model. A transformer's embeddings start random and are adjusted by training until they help predict the next token, so they end up encoding whatever made prediction easier — including things no counting method would find. What survives the comparison is the shape: a token becomes a direction, and closeness is measured by the angle between directions.

A close neighbour is not a synonym 🖖

␣Earth's neighbours are ␣planet, ␣Sun, ␣Jupiter and ␣Moon — none of which means Earth. ␣star's are ␣yellow, ␣white and ␣red, which are its properties rather than its synonyms. This is what distributional similarity actually measures: not what a token means, but what the corpus was talking about when it used it. Two tokens score highly when they are discussed in the same company, and being discussed together is a much broader relation than meaning the same thing.

There is no map here, and that is not an omission 🖖

Each vector has one dimension for every other token it can appear near — 2,742 of them — and nothing on this page reduces that to two. Cosine does not need it: an angle between two directions is defined in any number of dimensions, and the grid above shows every pair of angles directly. The flat scatter plots you have seen elsewhere are a projection, chosen to be readable rather than accurate, and two tokens that look adjacent on one can be far apart in the space the numbers actually live in.

Problem solved in full

Earth and Sun at 0.299 across 13 meetings 6 steps
The panel scores ␣Earth against ␣Sun at 0.299. Earth occurs 230 times in the corpus and Sun 150, across 385,078 tokens that open 2,770,646 context slots between them, and the two stand within 4 tokens of each other 13 times. Work out how much of the 0.299 those 13 meetings are worth.
1. One coordinate per context word, and the vectors are scaled to length 1 before anything is compared. That scaling is what makes cosine similarity a plain dot product: 0.299 is a sum of products, one term for every context word the two tokens have in common.
  
  $cos θ = \frac{a \cdot b}{∥ a ∥ ∥ b ∥} = \overset{a}{^} \cdot \hat{b} = c \sum \overset{a}{^}_{c} \hat{b}_{c}$
2. A coordinate is a surprise ratio, not a count — how often the pair actually turns up together against how often a shuffled corpus would put them together. Negative surprise is clipped to 0, so the vector records what a token is drawn to and has no way to record what it avoids.
  
  $a_{c} = max (0, ln \frac{n _{A c} / P}{( f _{A} / T ) ( f _{c} / T )})$
3. Chance first. Earth holds 230 of the 385,078 positions and Sun 150, so across all 2,770,646 slots you would expect the pair 0.6446 times — less than 1 occurrence in the whole corpus.
  
  $E = P \cdot \frac{f _{A}}{T} \cdot \frac{f _{B}}{T} = 2770646 \cdot \frac{230}{385 078} \cdot \frac{150}{385 078} = 0.6446$
4. They occur 13 times, 20.17 times what chance predicts, and the coordinate is the logarithm of that ratio: 3.004. Both vectors get the same value, because the count and the two frequencies read the same in either direction.
  
  $\frac{n}{E} = \frac{13}{0.6446} = 20.17 ⟹ a_{B} = b_{A} = ln 20.17 = 3.004$
5. Now scale. Earth's vector has 419 positive coordinates and a length of 30.381; Sun's has 302 and a length of 29.144. Divide 3.004 by each and the unit coordinates are 0.0989 and 0.1031.
  
  $\overset{a}{^}_{B} = \frac{3.004}{30.381} = 0.0989, \hat{b}_{A} = \frac{3.004}{29.144} = 0.1031$
6. Their product, 0.0102, is the whole contribution of those 13 meetings. Earth and Sun share 105 context words in total; add all 105 products and you get 0.2992, which the tool rounds to 0.299 — so the direct evidence supplies 3.4% of the score.
  
  $\overset{a}{^}_{B} \hat{b}_{A} = 0.0102, c \sum \overset{a}{^}_{c} \hat{b}_{c} = 0.2992, \frac{0.0102}{0.2992} = 3.4%$
Answer

The tool prints 0.299, and the times these two words actually meet account for 0.0102 of it. The rest is company: 105 shared context words out of the 2,742 tokens that have a vector at all. Two things follow. Because clipping removes every negative coordinate, each of the 105 products is positive, so the scale runs from 0 to 1 rather than from -1 to 1 — a pair with nothing in common scores 0, not the opposite of 1. Read as an angle, 0.299 is 72.6°, which puts Earth and Sun 17.4° away from sharing nothing. And the ranking above the score is finer than the evidence beneath it: ␣planet tops Earth's neighbour list at 0.307, clear of Sun by 0.008, while the single Earth-and-Sun coordinate is worth 0.0102 on its own. One context word is enough to change which token is called nearest.

$θ = arccos 0.299 = 72. 6^{\circ} (θ_{m a x} = 9 0^{\circ})$

Learning path

How an LLM picks the next word

Leads to Temperature & sampling

References (2)

The vector arithmetic this tool measures, in the paper that made it famous: T. Mikolov, K. Chen, G. Corrado and J. Dean, "Efficient Estimation of Word Representations in Vector Space." arXiv:1301.3781, 2013.
The co-occurrence counts underneath it, made explicit: J. Pennington, R. Socher and C. D. Manning, "GloVe: Global Vectors for Word Representation." Proceedings of EMNLP 2014, 1532–1543, 2014.

By the 42Math team · Updated 2026-08-10

Reviewed against published sources. Corrections welcome. Editorial policy · Corrections

All Computing lessons →

Example problems

Same subject - ␣Earth and ␣Sun score 0.299. Earth's nearest neighbour is ␣planet at 0.307 — the astronomy tokens sit together because the corpus discusses them together.
Near-synonyms - ␣speed and ␣velocity score 0.282, and ␣speed is ␣velocity's nearest neighbour of all 2742 tokens. Words used interchangeably keep interchangeable company.
Different subjects - ␣Earth and ␣velocity score 0.122 — less than half the Earth–Sun figure. Different subjects, different neighbours.
Two common words - ␣the and ␣is score 0.505, the highest pair here and higher than ␣Earth and ␣Sun at 0.299. Neither shares any meaning with the other; both simply appear near everything.
Star and light - ␣star and ␣light score 0.262, and star's own neighbours are ␣yellow, ␣white and ␣red — the corpus talks about stars by colour.