Tool 03
Turn text into vectors, compare cosine similarity, and project embeddings into 2D.
Embeddings turn discrete token IDs into continuous vectors — lists of numbers that capture meaning. Similar concepts end up near each other in vector space. This is the foundation of semantic search, RAG retrieval, and clustering.
After tokenization, each token ID is looked up in an embedding table — a large matrix where each row is a vector. The model learns these vectors during training so that useful relationships emerge.
"Cat" and "kitten" end up close together. "Cat" and "refrigerator" end up far apart. The model never sees a dictionary definition — it learns geometry from billions of examples.
The embedding matrix has shape [vocab_size, embedding_dim]. Token ID 42 simply picks row 42. That row is the vector fed into the first transformer layer.
Larger embedding_dim (e.g. 768, 4096) means more capacity to represent nuance, but also more parameters and compute.
Cosine similarity measures the angle between two vectors — 1.0 means identical direction, 0 means orthogonal, -1 means opposite. It ignores vector length, focusing on direction (meaning).
Euclidean distance measures how far apart points are in space. For normalized embeddings, cosine and distance often agree on ranking. Search systems usually rank by cosine similarity.
Real embeddings live in hundreds or thousands of dimensions. The 2D scatter plot above squashes that space down so you can see it — but neighbors can shift when projected.
Treat the plot as intuition, not ground truth. Trust the cosine similarity numbers more than visual proximity for close calls.
Enter synonym pairs (cat/kitten, happy/joyful) and antonym pairs (hot/cold) — compare cosine scores. Try domain-specific terms (Python/JavaScript) vs unrelated words.
Toggle Ollama to get real embeddings from a local model, or use synthetic vectors to learn the mechanics when Ollama is unavailable.