Methodology
Improving learning efficiency? Test self-supervised, few-shot, transfer, and continual learning approaches.
ML methodology research develops the meta-techniques that improve how models learn: few-shot and meta-learning, transfer learning, neural architecture search, and hyperparameter optimization. These methods determine how efficiently practitioners can adapt models to new tasks and domains.
State of the Field (2025)
- In-context learning in LLMs has largely replaced classical meta-learning (MAML, Prototypical Networks) for few-shot classification, achieving competitive accuracy without task-specific fine-tuning
- Neural architecture search (NAS) shifted from full search to efficient one-shot methods (DARTS, OFA) and hardware-aware search, with foundation model fine-tuning reducing the need for custom architectures
- Parameter-efficient fine-tuning (LoRA, QLoRA, adapters) dominates transfer learning, enabling adaptation of billion-parameter models with <1% trainable parameters and minimal compute
- Automated ML (AutoML) platforms reached production maturity: AutoGluon, H2O, and Vertex AI AutoML handle end-to-end pipelines, but expert practitioners still outperform on complex, domain-specific problems
Quick Recommendations
Few-shot learning on new tasks
In-context learning with frontier LLMs, or LoRA fine-tuning for specialized domains
ICL requires zero training and works across modalities. LoRA fine-tuning with 50-500 examples often matches full fine-tuning at 1% of the compute cost.
Hyperparameter optimization
Optuna or Ray Tune
Optuna provides Bayesian optimization with pruning for efficient search. Ray Tune scales to distributed clusters. Both integrate with major ML frameworks out of the box.
Neural architecture search (custom model design)
OFA (Once-for-All) or hardware-aware NAS
OFA trains a supernet once and extracts specialized subnets for target hardware. Avoids the prohibitive cost of full NAS. Most practical when deploying across diverse hardware profiles.
Transfer learning for domain adaptation
Foundation model + LoRA/QLoRA fine-tuning
Start from the largest pre-trained model you can serve, fine-tune with LoRA on domain data. QLoRA enables 65B model fine-tuning on a single 48GB GPU. This beats training from scratch in virtually every scenario.
Tasks & Benchmarks
Continual Learning
Learning new tasks without forgetting old ones.
Few-Shot Learning
Learning from very few examples.
Self-Supervised Learning
Learning representations without labeled data.
Transfer Learning
Transferring knowledge between tasks and domains.
Show all datasets and SOTA results
Continual Learning
No datasets indexed yet. Contribute on GitHub
Few-Shot Learning
No datasets indexed yet. Contribute on GitHub
Self-Supervised Learning
No datasets indexed yet. Contribute on GitHub
Transfer Learning
No datasets indexed yet. Contribute on GitHub
Honest Takes
Classical meta-learning is mostly obsolete
MAML, Prototypical Networks, and Matching Networks were elegant solutions to few-shot learning. But in-context learning with LLMs achieves comparable or better few-shot accuracy with zero task-specific training. The only remaining niche is ultra-low-resource edge deployment where LLMs can't run.
NAS lost its reason to exist
When the answer to most ML problems is 'fine-tune a foundation model,' searching for novel architectures becomes academic. NAS still matters for edge deployment and specialized hardware, but for 90% of practitioners, architecture search is a solved problem: use the best pre-trained model available.
AutoML is great for baselines, dangerous for production
AutoML platforms produce surprisingly strong models in hours. But they optimize for benchmark metrics, not production requirements like latency, interpretability, data drift handling, and failure modes. Use AutoML to establish baselines, then build production systems with engineering discipline.