Text Classification

Text classification assigns predefined categories to text. Given an input document, email, tweet, or sentence, the model outputs one or more labels with confidence scores. It is the single most deployed NLP task in production — powering spam filters, customer support routing, content moderation, medical coding, legal document triage, and sentiment dashboards at every major technology company.

The task sounds simple, but the design space is enormous. You can classify with a regex, a Naive Bayes model, a fine-tuned BERT, or a prompted GPT-4. The right choice depends on your accuracy requirements, latency budget, label set stability, and how much labeled data you have. This lesson covers all of them.

Text classification has been reinvented at least five times. Each generation solved a real limitation of the last, and each — including the current LLM era — introduced new trade-offs that the next generation will have to address. Understanding this progression is the fastest way to build intuition for which tool to reach for.

The history matters because every approach is still in active production use somewhere. Naive Bayes still powers spam filters at ISPs. SVMs still classify medical records. BERT fine-tunes still handle the majority of high-throughput classification. Knowing the lineage helps you pick the right tool, not just the newest one.

# Naive Bayes: P(class | document) ∝ P(class) × ∏ P(word_i | class)
# "naive" = assume each word is independent given the class

P(spam | "free money click now") ∝
    P(spam) × P("free"|spam) × P("money"|spam) × P("click"|spam) × P("now"|spam)
#   0.3    ×    0.08         ×    0.05          ×    0.04          ×    0.02

P(ham | "free money click now") ∝
    P(ham)  × P("free"|ham)  × P("money"|ham)  × P("click"|ham)  × P("now"|ham)
#   0.7    ×    0.01         ×    0.02          ×    0.005         ×    0.03

# Spam wins by a large margin despite the independence assumption being wrong

# SVM with tf-idf features — dominated text classification for a decade
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline

classifier = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=50000, ngram_range=(1, 2))),
    ('svm', LinearSVC(C=1.0))
])

classifier.fit(train_texts, train_labels)
predictions = classifier.predict(test_texts)
# Accuracy on 20 Newsgroups: ~85-88% — still a strong baseline

Text classification progress is measured primarily against two benchmark suites created at NYU, the University of Washington, and DeepMind. Understanding them is essential because they define what "state of the art" means in this space.

Zero-shot classification uses a model trained on natural language inference (NLI) to classify text into any categories you provide — no training required. The model was never taught your specific labels; it reasons about whether a hypothesis ("This text is about technology") is entailed by the input text.

from transformers import pipeline

# BART trained on Multi-Genre NLI (Williams et al., 2018)
classifier = pipeline(
    "zero-shot-classification",
    model="facebook/bart-large-mnli"
)

# Single-label classification — labels are mutually exclusive
result = classifier(
    "The Federal Reserve raised interest rates by 25 basis points",
    candidate_labels=["finance", "politics", "technology", "sports"]
)

print(result['labels'][0], f"({result['scores'][0]:.3f})")
# finance (0.962)

# Multi-label classification — each label scored independently
result = classifier(
    "Apple announces AI chip for data centers, stock surges 5%",
    candidate_labels=["technology", "business", "science", "politics"],
    multi_label=True  # Key parameter: sigmoid instead of softmax
)

for label, score in zip(result['labels'], result['scores']):
    print(f"  {label}: {score:.3f}")
# technology: 0.953
# business: 0.821
# science: 0.231
# politics: 0.018

# Internally, for each label:
premise  = "The Federal Reserve raised interest rates by 25 basis points"
hypothesis = "This example is finance."   # Constructed from label

# NLI model outputs: P(entailment), P(neutral), P(contradiction)
# P(entailment) becomes the classification score for "finance"
# Repeat for all labels, normalize with softmax (or sigmoid for multi-label)

from openai import OpenAI
import json

client = OpenAI()

def classify_text(text: str, labels: list[str]) -> dict:
    """Zero-shot classification via structured LLM output."""
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={"type": "json_object"},
        messages=[{
            "role": "system",
            "content": f"""Classify the text into one of these categories: {labels}.
Return JSON: {{"label": "...", "confidence": 0.0-1.0, "reasoning": "..."}}"""
        }, {
            "role": "user",
            "content": text
        }]
    )
    return json.loads(response.choices[0].message.content)

result = classify_text(
    "The service was okay, nothing special but not terrible either.",
    ["positive", "negative", "neutral"]
)
print(result)
# {"label": "neutral", "confidence": 0.85,
#  "reasoning": "Mixed signals — acknowledges it wasn't bad but..."}

For production systems with fixed categories and high throughput requirements, fine-tuning a pre-trained transformer on your labeled data delivers the best accuracy-to-latency ratio. The model learns your specific label semantics and domain vocabulary.

from transformers import pipeline

# Load DistilBERT fine-tuned on Stanford Sentiment Treebank
sentiment = pipeline(
    "sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english"
)

# Single prediction
result = sentiment("This movie was absolutely terrible and a waste of time.")
print(result)
# [{'label': 'NEGATIVE', 'score': 0.9998}]

# Batch processing — 10x faster than individual calls
texts = [
    "I love this product, best purchase ever!",
    "The service was awful, waited 3 hours.",
    "It's okay I guess, nothing remarkable.",
    "Incredible experience, exceeded all expectations!",
]
results = sentiment(texts, batch_size=32)

for text, res in zip(texts, results):
    print(f"  {res['label']:8s} ({res['score']:.3f}) | {text[:50]}")
# POSITIVE (0.999) | I love this product, best purchase ever!
# NEGATIVE (0.999) | The service was awful, waited 3 hours.
# POSITIVE (0.724) | It's okay I guess, nothing remarkable.
# POSITIVE (0.999) | Incredible experience, exceeded all expectations!

from transformers import (
    AutoTokenizer, AutoModelForSequenceClassification,
    TrainingArguments, Trainer
)
from datasets import load_dataset
import numpy as np

# 1. Load pre-trained model + tokenizer
model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name, num_labels=3  # e.g., positive/negative/neutral
)

# 2. Prepare your dataset
dataset = load_dataset("csv", data_files={
    "train": "train.csv",  # columns: text, label
    "test": "test.csv"
})

def tokenize(batch):
    return tokenizer(batch["text"], padding=True, truncation=True, max_length=128)

tokenized = dataset.map(tokenize, batched=True)

# 3. Train — typically 2-5 epochs is enough
trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="./results",
        num_train_epochs=3,
        per_device_train_batch_size=16,
        learning_rate=2e-5,       # Standard for BERT fine-tuning
        weight_decay=0.01,
        eval_strategy="epoch",
    ),
    train_dataset=tokenized["train"],
    eval_dataset=tokenized["test"],
)

trainer.train()
# With 5,000 examples, expect ~90-95% accuracy on most binary tasks
# With 500 examples, expect ~80-85% — still often beats zero-shot

A subtle but critical distinction that determines your loss function, evaluation metrics, and inference logic.

# Softmax: probabilities sum to 1.0
logits = model(text)  # [2.1, -0.5, 0.3]
probs = softmax(logits)  # [0.82, 0.06, 0.12]
label = argmax(probs)  # "positive"

# Sigmoid: each label scored independently
logits = model(text)  # [2.1, 1.8, -2.0, 0.1]
probs = sigmoid(logits)  # [0.89, 0.86, 0.12, 0.52]
labels = [l for l, p in zip(labels, probs) if p > 0.5]
# ["tech", "finance"]

Every classification model outputs probability scores. Turning those scores into production decisions requires thresholding — and the right threshold depends entirely on the cost of mistakes.

from transformers import pipeline

classifier = pipeline(
    "sentiment-analysis",
    model="distilbert-base-uncased-finetuned-sst-2-english"
)

def classify_with_routing(text: str, auto_threshold=0.92, flag_threshold=0.65):
    """Three-tier classification with human-in-the-loop fallback.

    >= auto_threshold: Act automatically (high precision)
    >= flag_threshold: Flag for human review (balanced)
    <  flag_threshold: Reject / mark as uncertain
    """
    result = classifier(text)[0]
    score = result['score']
    label = result['label']

    if score >= auto_threshold:
        return {"label": label, "score": score, "action": "auto"}
    elif score >= flag_threshold:
        return {"label": label, "score": score, "action": "review"}
    else:
        return {"label": "UNCERTAIN", "score": score, "action": "skip"}

# High confidence — auto-process
print(classify_with_routing("This product is absolutely amazing!"))
# {'label': 'POSITIVE', 'score': 0.9998, 'action': 'auto'}

# Medium confidence — needs human review
print(classify_with_routing("It was fine I guess, met expectations."))
# {'label': 'POSITIVE', 'score': 0.7834, 'action': 'review'}

# Low confidence — skip or escalate
print(classify_with_routing("The product arrived."))
# {'label': 'UNCERTAIN', 'score': 0.5612, 'action': 'skip'}

The right approach depends on four variables: how much labeled data you have, how fast you need inference, whether your labels change, and your accuracy requirements.