Sensors & How They Work
IMUs, LiDAR, cameras, GPS — how robots perceive.
Prerequisites
Why it matters in robotics
Sensors are the front end of every autonomy stack, so what their data physically represents matters before it ever reaches a filter or planner. IMUs drift over time and accumulate heading and position error if their readings are simply added up on their own; cameras and LiDAR need careful calibration to know exactly where they're pointed and how their raw readings map to real-world distances; and GPS, encoders, and range sensors fail through multipath, wheel slip, dropout, and specular surfaces. A sensor's error characteristics drive downstream consequences — why no single sensor can be trusted alone, and why several complementary sensors must be combined — which makes this foundational for state estimation, SLAM, and perception.
What to study
- ✓IMU error model: bias (constant plus bias instability), scale-factor error, white noise vs. random walk (angular/velocity random walk), and why integration turns small rate errors into unbounded position/heading drift.
- ✓Camera and LiDAR operating principles plus calibration: pinhole intrinsics and lens distortion, time-of-flight/structured ranging, and intrinsic, extrinsic, and temporal (time-sync) calibration via checkerboards and tools like Kalibr.
- ✓Other sensors: wheel encoders (resolution, slip, dead-reckoning error growth), GPS/GNSS (multipath, urban-canyon dropout, ~meter-level absolute accuracy, RTK), and range sensors (ultrasonic/ToF range and specular-surface limits).
- ✓Noise models and failure modes: zero-mean Gaussian measurement noise, reading a datasheet and Allan deviation plot, and the characteristic failure mode of each sensor that motivates multi-sensor fusion.
Where to study
Cheatsheet
Symbols used below
IMU error model
A gyro/accelerometer reading is the true value plus a slowly-varying bias (itself a random walk, i.e. bias instability), a scale-factor error, and white noise.
Integrating rate to get angle (or acceleration to get position) integrates the bias too, so a constant bias becomes a linear drift in angle and a quadratic drift in position — this is why raw IMU dead-reckoning is unbounded without an external reference.
Camera model
Pinhole projection: , — a 3D point projects to the image plane scaled by focal length and inversely by depth.
Lens distortion (radial/tangential) bends this ideal mapping near the edges of the frame; intrinsic calibration (focal length, principal point, distortion coefficients) corrects it, typically fit from checkerboard images.
Stereo depth
, where is the disparity (pixel shift of the same point between the two cameras).
Depth resolution degrades quadratically with distance (), so stereo is precise up close and gets rapidly noisier far away.
Calibration types
Intrinsic: a sensor's internal geometry (camera focal length/distortion, LiDAR beam angles). Extrinsic: the rigid transform between sensors' frames (or sensor to robot body). Temporal: aligning timestamps across sensors that sample at different rates or with different latency — an uncorrected time offset shows up as apparent extrinsic error at any speed.
GPS/GNSS and encoders
GPS gives roughly meter-level absolute position from satellite ranging but degrades badly with multipath (reflections off buildings) and urban-canyon signal blockage; RTK correction data can push this to centimeter level.
Wheel encoders give cheap, high-rate relative motion via dead reckoning, but wheel slip means the wheel can rotate without the robot actually translating that far, so position error still grows unbounded over distance without correction.
Reading noise specs
Datasheets report noise as zero-mean Gaussian with a stated (or an Allan deviation plot for IMUs), which separates the noise into regions dominated by white noise (short averaging times, deviation falls as ) versus bias instability (flattens out) versus rate random walk (rises again at long ) — the minimum of the curve is the sensor's best achievable stability.
| Camera | LiDAR | Radar | IMU | GPS | |
|---|---|---|---|---|---|
| Direct measurement | Bearing + appearance | Precise range + bearing | Range + range-rate | Angular rate + acceleration | Absolute position |
| Strength | Rich semantics | Accurate geometry | Works in fog/rain, sees velocity | High-rate, self-contained | Global, drift-free |
| Weak point | No direct depth, lighting-sensitive | Sparse at range, costly | Poor angular resolution (cross-range) | Drifts without a reference | Multipath, dropout indoors/urban canyon |
Application focus
Practice questions (25)
- Why an IMU alone cannot be trusted for longmedium
- Heading error from a constant gyro biaseasy
- Explain the pinhole camera modelInterviewmedium
- Main parameters in camera calibrationInterviewmedium
- Major specifications when selecting a cameraInterviewmedium
- Lidar–camera calibrationInterviewhard
- Calibrating IMU, lidar, and camera togetherInterviewhard
- Calibrating an IMU (intrinsics)Interviewhard
- Lidar vs camera — advantages and disadvantagesInterviewmedium
- The blooming effect in lidarInterviewmedium
- Impact of retroreflectors on lidar dataInterviewmedium
- Common lidar point-cloud issuesInterviewmedium
- Retroreflector returns vs sensor-blockage returnsInterviewhard
- 1550 nm vs 905 nm lidar (and their detectors)Interviewhard
- Detecting blockage/occlusion on a cameraInterviewmedium
- Detecting blockage on a lidarInterviewmedium
- False positives and false negatives in blockage detectionInterviewmedium
- Choosing lidar specs and the region of interestInterviewhard
- Scanning-mirror lidar vs mechanical spinning lidarInterviewhard
- Radar's poor cross-range (perpendicular) accuracyInterviewhard
- Modeling GPS noise and computing DOPInterviewhard
- GPS velocity noise, and deriving velocity covariance from positionInterviewhard
- Modeling noise for a sensor that doesn't report its uncertaintyInterviewmedium
- Simulate a lidar: range to a wall, and 3D distanceInterviewmedium
- Design a perception system for a truck operating at nightInterviewhard