Foundation Models
Large pretrained VLMs/LLMs as the robot's semantic brain.
Prerequisites
Why it matters in robotics
Foundation models are reshaping robotics: vision-language-action (VLA) models such as RT-2, OpenVLA, and pi-0 turn internet-pretrained large models into generalist robot policies. Large-scale pretraining on huge amounts of text, images, and video produces abilities that only show up once a model is big enough, and pretraining that links images with language gives robots open-vocabulary understanding of what they see; adapting these models to a robot — training them further at varying cost and depth, and combining web data with actual robot data — is central to using them. Their limitations matter just as much: making things up, weak grounding in real physics, inference speed versus how fast the robot needs to react, and the chronic scarcity of robot data.
What to study
- ✓Transformer pretraining and scaling: self-attention, next-token objective, scaling laws (loss as a power law in params/data/compute), and emergent abilities that appear only at scale.
- ✓Vision-language grounding: contrastive image-text pretraining (CLIP), VLM architectures, and how open-vocabulary visual features enable semantic generalization in robots.
- ✓Adapting pretrained models to robots: building VLAs by co-training on web + robot data, discrete action tokenization (RT-2, OpenVLA) vs. flow/diffusion action experts (pi-0), and parameter-efficient fine-tuning with LoRA/adapters vs. full fine-tuning vs. prompting.
- ✓Deployment limitations: inference latency vs. control frequency, hallucination and weak physical grounding, robot data scarcity, and safety/uncertainty considerations for closed-loop control.
Where to study
Cheatsheet
Symbols used below
Scaling laws
Loss falls off as a power law in each resource held alone: , .
For a fixed compute budget , there's an optimal split between growing and growing — over- or under-sizing the model relative to data wastes compute.
Emergent abilities
Some capabilities (multi-step reasoning, in-context learning, instruction following) stay near zero across small models and then appear abruptly past a scale threshold — they aren't a smooth extrapolation of the pretraining loss curve, which makes them hard to predict in advance.
Vision-language grounding
CLIP-style contrastive pretraining pulls matching image-text pairs together and pushes mismatched pairs apart in a shared embedding space, using a large batch of in-batch negatives.
The result is open-vocabulary recognition: the model can score a novel text label against an image without ever being trained on that class.
Adapting to robots
Three levels of adaptation, cheapest to most expensive: prompting (no weight updates, relies entirely on pretrained knowledge), parameter-efficient fine-tuning (LoRA/adapters — a small number of new weights are trained, backbone frozen), and full fine-tuning (every weight updated, most capacity but most compute and most prone to forgetting pretrained knowledge).
VLAs typically co-train on web-scale image-text data plus a much smaller robot-action dataset so the policy keeps its semantic generalization while learning to act.
Deployment limits
Hallucination and weak physical grounding mean a model can describe a plausible-sounding but physically wrong action.
Inference latency competes directly with control frequency: a large backbone that takes 200 ms to produce an action caps the control loop at 5 Hz, so systems lean on action chunking or a separate fast low-level controller to bridge the gap.
| Prompting | Adapters / LoRA | Full fine-tuning | |
|---|---|---|---|
| Weights touched | None | Small added subset | All |
| Data needed | A few examples in-context | Hundreds–thousands | Large curated set |
| Compute cost | ~None | Low | High |
| Forgetting risk | None | Low | Higher |