MTEB — measuring the quality
of text understanding.

Given a query, find the most relevant documents from a corpus.

Classify text into categories using embeddings as features.

Group semantically similar texts into clusters without labels.

Given a query and candidate documents, reorder by relevance.

Predict the degree of semantic equivalence between sentence pairs.

Determine the relationship between two texts (duplicate, paraphrase, entailment).

Evaluate how well a summary captures the meaning of a source document.

Find translation pairs between two sets of sentences in different languages.

# Install
pip install mteb sentence-transformers

# Run full English benchmark
import mteb
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")

# Run all English tasks (56+ datasets)
evaluation = mteb.MTEB(tasks=mteb.get_tasks(languages=["eng"]))
results = evaluation.run(model, output_folder="results/qwen3-0.6b")

# Or run specific task types
retrieval_tasks = mteb.get_tasks(
    languages=["eng"],
    task_types=["Retrieval"]
)
evaluation = mteb.MTEB(tasks=retrieval_tasks)
results = evaluation.run(model, output_folder="results/retrieval-only")

# Option 1: SOTA (KaLM-Gemma3-12B — needs ~24GB VRAM)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("KaLM-ai/KaLM-Embedding-Gemma3-12B-2511")
embeddings = model.encode(["What is machine learning?", "ML is a subset of AI."])
print(f"Similarity: {embeddings[0] @ embeddings[1]:.4f}")

# Option 2: Best bang-for-buck (Qwen3-0.6B — runs on CPU!)
model = SentenceTransformer("Qwen/Qwen3-Embedding-0.6B")
embeddings = model.encode(["Hello world", "Bonjour le monde"])

# Option 3: Production serving with HuggingFace TEI
# docker run --gpus all -p 8080:80 \
#   ghcr.io/huggingface/text-embeddings-inference:latest \
#   --model-id Qwen/Qwen3-Embedding-0.6B

Normalised Discounted Cumulative Gain at rank 10. Measures how well the model ranks relevant documents in the top 10 results, with higher positions weighted more heavily.

NDCG@10 = DCG@10 / IDCG@10
DCG@10 = Σ(rel_i / log₂(i+1))

A score of 1.0 means all relevant documents appear at the top. Most models score 0.4–0.7, reflecting the difficulty of zero-shot retrieval.

Spearman rank correlation between model cosine similarities and human similarity judgments. Measures whether the model's relative ordering of sentence pairs matches human intuition.

ρ = 1 − (6 × Σd_i²) / (n(n²−1))
where d_i = rank difference for pair i

Spearman correlation of 0.80+ indicates strong alignment with human judgment. Top models now exceed 0.81.

Find relevant documents by meaning, not just keyword overlap. Embeddings enable natural-language queries over large corpora.

Retrieve context chunks for LLM generation. Embedding quality directly determines answer accuracy in retrieval-augmented pipelines.

Identify near-duplicate content, support tickets, or records using cosine similarity between embedding pairs.

Group documents by semantic similarity. Embeddings provide dense features for k-means, HDBSCAN, or topic extraction.

Sentence transformers

Models trained specifically for sentence and paragraph embedding. Run locally with full control.

• Optimised for retrieval, fast inference
• Many specialised variants available

• Fixed context length
• May need domain fine-tuning

LLM embeddings via API

Embedding endpoints from OpenAI, Cohere, Voyage, and others. Zero infrastructure to manage.

• High quality, long context
• No infrastructure to maintain

• Cost per token
• Data leaves your system

Sparse + dense hybrid

Combine BM25 with dense embeddings for better recall. Best of both worlds for production search.

• Handles exact matches well
• More robust for rare terms

• More complex pipeline
• Two indices to maintain

from openai import OpenAI

client = OpenAI()

def get_embedding(text: str, model='text-embedding-3-large'):
    response = client.embeddings.create(input=text, model=model)
    return response.data[0].embedding

documents = [
    'The cat sat on the mat',
    'A dog played in the park',
    'Machine learning is fascinating'
]

embeddings = [get_embedding(doc) for doc in documents]
print(f'Embedding dimension: {len(embeddings[0])}')

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('BAAI/bge-large-en-v1.5')

documents = [
    'The cat sat on the mat',
    'A dog played in the park',
    'Machine learning is fascinating'
]

embeddings = model.encode(documents, normalize_embeddings=True)

query = 'pets resting at home'
query_embedding = model.encode(query, normalize_embeddings=True)

similarities = np.dot(embeddings, query_embedding)
for doc, sim in zip(documents, similarities):
    print(f'{sim:.3f}: {doc}')