Text Embeddings in Practice

Lesson 0.1 traced the theoretical foundations from Harris (1954) through BERT (2018). This lesson picks up the practical thread: how did we get from "word vectors are interesting" to "here is a leaderboard with 200+ models ranked across 56 datasets"?

The answer is four architectural shifts, each unlocking a new class of applications. Understanding these shifts tells you why certain models are good at certain tasks and saves you from cargo-culting someone else's model choice.

# Word2Vec: one fixed vector per word
from gensim.models import Word2Vec

model = Word2Vec(sentences, vector_size=300, window=5, sg=1)  # Skip-gram
vec_king = model.wv["king"]      # Shape: (300,) — always the same vector
vec_bank = model.wv["bank"]      # Same vector for river bank and bank account

# "Sentence embedding" = crude average
import numpy as np
sentence = "the cat sat on the mat"
sent_vec = np.mean([model.wv[w] for w in sentence.split()], axis=0)

# BERT cross-encoder: accurate but O(n²)
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")

# To compare sentence A with 10,000 candidates:
# Must run model(A + candidate_i) for EACH of the 10,000 candidates
# At ~10ms per pair = 100 seconds for one query
# For all-pairs on 10,000 docs: 10000² / 2 = 50M pairs ≈ 65 hours

# Sentence-BERT bi-encoder: encode once, search fast
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

# Encode corpus ONCE — O(n)
corpus_embeddings = model.encode(corpus_of_10000_docs)  # (10000, 384)

# Search ANY query in milliseconds — just a dot product
query_vec = model.encode("machine learning frameworks")  # (384,)
scores = corpus_embeddings @ query_vec  # (10000,) — instant
top_results = scores.argsort()[-5:][::-1]  # Top 5

The Massive Text Embedding Benchmark (MTEB) is the industry standard for comparing embedding models. Created by Muennighoff et al. at Hugging Face in 2022, it evaluates models across 8 task types and 56+ datasets spanning retrieval, classification, clustering, semantic textual similarity (STS), reranking, pair classification, summarization, and BitextMining.

MTEB matters because a model that excels at retrieval may be mediocre at clustering, and vice versa. The benchmark forces you to think about which task you care about, not just the average score. When evaluating models below, pay attention to the Retrieval column if you're building search/RAG, STS if you need fine-grained similarity, and Classification if you're labeling text.

The embedding landscape is crowded. Here are the models that matter, with real MTEB scores and the trade-offs that determine which one you should use.

Every text embedding pipeline has the same three steps: choose a model, encode text into vectors, compare vectors by similarity. The code below is copy-paste ready. Run it.

from sentence_transformers import SentenceTransformer
import numpy as np

# Load model (~1.2GB download first time for bge-large)
model = SentenceTransformer("BAAI/bge-large-en-v1.5")

# Encode documents — normalize for cosine similarity via dot product
documents = [
    "Python is a programming language",
    "JavaScript runs in the browser",
    "SQL is used to query databases",
    "Redis is an in-memory data store",
    "PostgreSQL is a relational database",
]
embeddings = model.encode(documents, normalize_embeddings=True)
# Shape: (5, 1024) — 5 documents, 1024 dimensions each

# Encode a query
query = "how to store data persistently"
query_vec = model.encode(query, normalize_embeddings=True)

# Similarity = dot product (vectors are normalized)
scores = embeddings @ query_vec
ranked = np.argsort(scores)[::-1]

print("Results:")
for idx in ranked:
    print(f"  {scores[idx]:.3f}: {documents[idx]}")

Results:
  0.701: PostgreSQL is a relational database
  0.654: Redis is an in-memory data store
  0.598: SQL is used to query databases
  0.412: Python is a programming language
  0.389: JavaScript runs in the browser

from openai import OpenAI
import numpy as np

client = OpenAI()  # Uses OPENAI_API_KEY env variable

def embed_texts(texts: list[str], model="text-embedding-3-large", dims=1024):
    """Embed texts with OpenAI. Supports Matryoshka truncation via dims."""
    response = client.embeddings.create(
        model=model,
        input=texts,
        dimensions=dims,  # Matryoshka: 256, 512, 1024, or 3072
    )
    return np.array([d.embedding for d in response.data])

# Embed documents and query
doc_embeddings = embed_texts([
    "Python is a programming language",
    "Redis is an in-memory data store",
    "PostgreSQL is a relational database",
])
query_embedding = embed_texts(["how to store data persistently"])

# Compare
scores = doc_embeddings @ query_embedding.T
print(scores.squeeze())  # [0.41, 0.65, 0.71]

import cohere
import numpy as np

co = cohere.ClientV2("your-api-key")

# Cohere distinguishes document vs query embeddings
# This matters: query embeddings are optimized for retrieval
doc_response = co.embed(
    texts=["Python is a programming language",
           "Redis is an in-memory data store",
           "PostgreSQL is a relational database"],
    model="embed-english-v3.0",
    input_type="search_document",   # For corpus documents
    embedding_types=["float"],
)
doc_embeddings = np.array(doc_response.embeddings.float_)

query_response = co.embed(
    texts=["how to store data persistently"],
    model="embed-english-v3.0",
    input_type="search_query",      # For queries — different optimization
    embedding_types=["float"],
)
query_embedding = np.array(query_response.embeddings.float_)

scores = doc_embeddings @ query_embedding.T
print(scores.squeeze())

The dot-product approach above works for thousands of documents. For millions or billions, you need an approximate nearest neighbor (ANN) index. FAISS (Facebook AI Similarity Search) is the most widely used library.

from sentence_transformers import SentenceTransformer
import faiss
import numpy as np

model = SentenceTransformer("BAAI/bge-large-en-v1.5")

# Simulate a larger corpus
documents = [
    "Python is a high-level programming language",
    "JavaScript enables interactive web pages",
    "SQL queries relational databases",
    "Redis provides in-memory caching",
    "PostgreSQL is an advanced relational database",
    "Docker containerizes applications for deployment",
    "Kubernetes orchestrates container workloads",
    "TensorFlow is a machine learning framework",
    "React is a JavaScript UI library",
    "FastAPI is a modern Python web framework",
]

# Encode and normalize
embeddings = model.encode(documents, normalize_embeddings=True).astype("float32")
dim = embeddings.shape[1]  # 1024

# Build FAISS index — IndexFlatIP = exact inner product (cosine for normalized vecs)
index = faiss.IndexFlatIP(dim)
index.add(embeddings)
print(f"Index contains {index.ntotal} vectors of dim {dim}")

# Search
query = "tools for deploying web applications"
query_vec = model.encode([query], normalize_embeddings=True).astype("float32")

# D = distances (similarities), I = indices
D, I = index.search(query_vec, k=3)

print("\nTop 3 results:")
for score, idx in zip(D[0], I[0]):
    print(f"  {score:.3f}: {documents[idx]}")

Index contains 10 vectors of dim 1024

Top 3 results:
  0.712: Docker containerizes applications for deployment
  0.653: Kubernetes orchestrates container workloads
  0.584: FastAPI is a modern Python web framework

Embedding dimension is the single most impactful architecture decision for production systems. It determines storage cost, search latency, and (to a point) embedding quality.

Model choice is a function of four constraints: quality requirements, latency budget, infrastructure, and cost tolerance. Here is the decision tree.