Embeddings

Embeddings are lists of numbers, vectors, that represent the meaning of a piece of text. The model that produces them is trained so that texts meaning similar things land near each other and unrelated texts land far apart. Vector search depends on that property: a search for meaning becomes a measurement of distance. In a RAG system the embedding model sets a ceiling on retrieval, because a passage the embedding places far from the query is a passage the model never sees.

From words to coordinates#

An embedding is dense: it spreads meaning across hundreds or thousands of dimensions, and every number carries part of the signal. A keyword vector is the opposite. A bag-of-words vector for a document has a slot for every word in the vocabulary and almost all of them are zero, while a 1536-dimension embedding uses all 1536 numbers to summarise what the text means.

"Dog" and "puppy" sit close together, "dog" and "wolf" a little further apart, and "dog" and "invoice" far apart, though none of them share letters. The model learned those positions from how words are used across a very large amount of text, so paraphrases and synonyms cluster without anyone writing rules. The same holds for sentences: "How do I get my money back?" and "What is the refund procedure?" embed to nearly the same point while sharing almost no words, a pair keyword search would miss.

Click any word below to see its nearest neighbours.

Cosine similarity, the distance that matters#

Two embeddings are usually compared by cosine similarity, the angle between the vectors rather than their length. Vectors pointing the same way score near 1, unrelated ones near 0, and opposite ones near -1. Direction tracks meaning and length mostly does not, so the angle is the useful measure. OpenAI normalises its embeddings to length 1, and at that length cosine similarity and straight-line distance rank results identically. Gemini's gemini-embedding-001 normalises only its full 3072-dimension output, so a shortened vector has to be normalised again before the two measures agree.

The scores are less intuitive than the scale suggests. Two paraphrases might score 0.9, a related pair 0.6, and two unrelated texts 0.2 rather than 0, and the values shift from one model to the next. Never copy a similarity cutoff from someone else's post; set it from labelled pairs of your own data. Retrieval is then simple to state: embed the query and return the stored vectors with the highest similarity, through an approximate nearest-neighbour index so a query does not compare itself with millions of vectors one at a time.

Where embeddings miss#

An embedding is a summary, and a summary blurs exact strings. A part number, an error code, a product name or a person's name carries most of its meaning in characters the embedding treats as roughly interchangeable with similar ones. On Prospex, Bigcircle's own prospect research engine, "How do they make money?" is a semantic question and "Are they using Snowflake?" is a keyword question, and pure vector search will return a passage about a different data warehouse for the second. We run semantic and keyword retrieval together rather than as a fallback chain, because the two fail on different queries (hybrid search). When a question turns on which record a passage belongs to, the passages need checking across the whole packet before anything is written; document intelligence for agents covers that check.

The other miss happens at write time. An embedding represents the chunk it was given, so a chunk boundary that separates a person's role from the company they hold it at produces two vectors that each mean less than the sentence did. Prospex uses a custom chunker for that reason, and we treat a bad boundary as a retrieval bug created when the index was built, not at query time (see chunking).

Choosing a model and sizing the vector#

Check the MTEB leaderboard to build a shortlist, then decide on your own evaluation set, because a model that leads a general benchmark can still lose on your contracts or your codebase. Open-weight models are a reasonable choice if you want to run embedding yourself.

Dimension count is the second decision. Larger vectors, 1536 or 3072 dimensions, hold more detail and cost more to store and search: at 4 bytes per number, ten million 3072-dimension vectors take about 123 GB before index overhead. Matryoshka representation learning softens the trade. Models trained this way, including OpenAI's text-embedding-3 family and Google's Gemini embedding models, put the most useful information in the first dimensions, so a vector can be cut short and keep most of its quality. OpenAI reports that text-embedding-3-large cut to 256 dimensions still outperforms the older ada-002 at its full 1536. Google publishes MTEB scores for gemini-embedding-001 of 68.17 at 1536 dimensions and 67.99 at 768. Cutting from 3072 to 768 dimensions removes 75% of the storage, and the quality you lose is small enough to measure on your own set before deciding.

Beyond retrieval#

The same vectors answer other "is this like that?" questions. Clustering groups support tickets or survey answers by topic with no labels. Near-duplicate detection flags pairs above a high similarity score and catches reworded copies that exact hashing misses. A small classifier trained on the vectors can replace fine-tuning a language model for a routing or tagging job, and recommendations become a nearest-neighbour lookup when items and users share one space. A corpus already embedded for retrieval can feed all of these without a second embedding pass.

Switching models means re-embedding everything#

Embeddings from different models are not comparable. Each model defines its own coordinates, so a query embedded with model B means nothing against documents embedded with model A, even when both vectors have 1536 dimensions. The same applies across major versions of one model and to a change of truncation length.

A model switch is therefore a migration, not a config change: re-embed the whole corpus, rebuild the index and set the similarity cutoffs again. The API bill is usually the small part. Ten million chunks of about 400 tokens is 4 billion tokens, around 80 dollars at text-embedding-3-small's 0.02 dollars per million, while the pipeline run, the index rebuild and the evaluation take days of work. Build the new index beside the old one, run real queries against both, compare retrieval metrics, then cut over. Price that migration when you choose the first model, because it is the reason teams stay on a weaker embedding long after a better one ships.