How Embedding Models Actually Work

When I was building the search infrastructure for a healthcare knowledge platform at HCLTech, we replaced keyword-based search with embedding-based semantic search. A query like "elevated liver enzymes" started returning results for "hepatic biomarkers" and "ALT/AST elevation" - terms that never appeared in the original query. Recall improved by 34%. This is what embeddings do: they encode meaning, not just text.

What Is an Embedding?

An embedding is a dense vector representation of text. A typical embedding model outputs a vector of 768 to 3,072 floating-point numbers. The key property: semantically similar text has geometrically similar vectors. "Dog" and "puppy" are close in vector space. "Dog" and "arbitrage" are far apart.

from openai import OpenAI
import numpy as np

client = OpenAI()

def embed(text: str) -> list:
    return client.embeddings.create(
        model="text-embedding-3-small", input=text
    ).data[0].embedding

def cosine_similarity(a: list, b: list) -> float:
    a, b = np.array(a), np.array(b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

e1 = embed("elevated liver enzymes")
e2 = embed("hepatic biomarkers")
e3 = embed("stock market volatility")

print(cosine_similarity(e1, e2))  # ~0.82
print(cosine_similarity(e1, e3))  # ~0.31

How Embedding Models Are Trained

Modern text embedding models are BERT-family transformers trained with contrastive learning. The training process uses a siamese network: two identical encoders processing two pieces of text. Training pairs include positive pairs (semantically similar text) and negative pairs (semantically different text). The loss function pushes positive pair embeddings closer together and negatives further apart. Over millions of training pairs, the model learns to encode semantic relationships in geometric form.

Matryoshka Representation Learning

OpenAI's text-embedding-3-* models use Matryoshka Representation Learning (MRL): the first N dimensions of the full embedding are themselves a valid lower-dimensional embedding. You can truncate text-embedding-3-large from 3,072 to 256 dimensions and retain ~85% of performance at 12x lower storage cost.

response = client.embeddings.create(
    model="text-embedding-3-large",
    input=text,
    dimensions=256  # Truncate from 3072 to 256
)
# Storage: 256 * 4 bytes = 1KB vs 12KB at full size
# For 10M documents: 1GB vs 12GB

Choosing the Right Embedding Model

The MTEB (Massive Text Embedding Benchmark) leaderboard is the standard reference. Top performers:

  • text-embedding-3-large (OpenAI): Strong general-purpose, $0.13/M tokens.
  • voyage-large-2 (Voyage AI): Often outperforms OpenAI on RAG tasks, $0.12/M tokens.
  • BAAI/bge-large-en-v1.5: Open-source, runs locally. Good for privacy-sensitive use.
  • E5-mistral-7b-instruct: Excellent for asymmetric retrieval (short query, long document).

Asymmetric vs Symmetric Retrieval

Many models are trained on symmetric pairs (similar articles) and perform poorly on asymmetric retrieval (short question, long document answer). For Q&A over documents, use models trained for asymmetric retrieval. Or use HyDE (Hypothetical Document Embeddings): generate a hypothetical answer with an LLM, then embed that answer instead of the raw question.

def hyde_embed(question: str) -> list:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Write a brief, factual answer to: {question}"
        }]
    )
    hypothetical_answer = response.choices[0].message.content
    return embed(hypothetical_answer)

Embeddings Beyond Search

Clustering: At Mamaearth, we clustered customer reviews by semantic theme - surfacing product feedback signals keyword search missed entirely. Anomaly detection: Flag clinical notes statistically far from a doctor's historical baseline. Zero-shot classification: Embed class descriptions, classify incoming items by nearest class.

classes = {
    "billing": embed("invoice payment charge subscription"),
    "technical": embed("bug error crash not working"),
    "account": embed("login password access account settings")
}

def classify_ticket(ticket_text: str) -> str:
    ticket_emb = embed(ticket_text)
    scores = {
        label: cosine_similarity(ticket_emb, class_emb)
        for label, class_emb in classes.items()
    }
    return max(scores, key=scores.get)

Where Embeddings Break Down

  • Negation: "Patient has no fever" and "Patient has fever" have similar embeddings.
  • Long documents: Most models have a 512-8K token limit. Truncation silently drops content. Use chunking.
  • Domain specificity: General web-trained models produce poor embeddings for specialized domains. Consider PubMedBERT for biomedical, LegalBERT for legal.
  • Cross-lingual: Multilingual models underperform monolingual models on individual languages.

Choosing the wrong embedding model early means rebuilding your entire vector index later - a painful and expensive migration. Pick deliberately, test on your domain.

More on this

Scaling and Indexing Embeddings

Once you generate millions of embeddings, storing and searching them efficiently becomes a new challenge. A 768-dimensional embedding for 100 million documents means 76.8 billion floating-point numbers. Brute-force searching every vector for the closest match isn't feasible. A single query could take minutes.

This is where vector databases come in. Tools like Pinecone, Weaviate, Qdrant, and Milvus are built specifically for this. They use Approximate Nearest Neighbor (ANN) algorithms. Algorithms like HNSW (Hierarchical Navigable Small World) or IVFPQ (Inverted File with Product Quantization) create an index that allows for very fast, near-exact searches. You trade a tiny bit of recall accuracy for massive speed improvements. For example, an HNSW index might find 99% of the true nearest neighbors in milliseconds, even across hundreds of millions of vectors. This is critical for real-time applications like semantic search or recommendation engines.

Fine-tuning Embeddings for Your Domain

General-purpose embedding models like those from OpenAI or Voyage AI are powerful. They're trained on vast amounts of public internet data. But sometimes, your specific domain has unique jargon, acronyms, or relationships that a general model just won't capture perfectly. I saw this in healthcare; terms like "STAT orders" or "PRN medications" have very specific meanings that a model trained on general text might not fully grasp.

This is when fine-tuning becomes valuable. You can take a pre-trained model and continue its training on your proprietary data. The process usually involves creating a small dataset of positive pairs (semantically similar sentences from your documents) and negative pairs. You might use a technique like Sentence-BERT (SBERT) for this. Even a few thousand well-curated examples can significantly boost performance for your specific use case. This helps the model learn the nuances of your particular dataset, leading to more relevant results for your users.

What Embeddings Don't Understand (Yet)

Embeddings are excellent at capturing semantic similarity based on patterns in their training data. But they don't possess true understanding or common sense. One common limitation is negation. If you embed "The cat is not on the mat" and "The cat is on the mat," their embeddings might still be quite close. The presence of "cat" and "mat" dominates the vector space, often overshadowing the "not."

They also struggle with factual accuracy or reasoning beyond direct textual relationships. An embedding for "Paris is the capital of France" is close to "The capital of France is Paris." However, it won't inherently know that Paris is a city, or that France is a country. These are higher-level concepts. When debugging, I often look at the nearest neighbors for a query. If I see irrelevant results that share keywords but miss the core intent, it often points to these kinds of limitations. Embeddings are powerful tools, but they are statistical representations, not sentient intelligence.

Further Reading