Core Goal
Understand what is happening (scene interpretation)
Predict what others will do
Plan a safe, comfortable route and maneuvers
Control the vehicle accurately in real time
Monitor safety and handle failures
System Architecture (Sense → Think → Act)
Sense (Sensors)
Cameras
What they measure
Color/texture, lane markings, traffic lights/signs, object appearance
Strengths
High semantic detail, low cost
Limitations
Sensitive to glare, low light, fog/rain; depth is indirect
Typical placement
Front-facing, surround-view (side/rear), interior driver monitoring (if present)
Radar (Radio Detection and Ranging)
What it measures
Object distance and relative speed (Doppler)
Strengths
Works well in rain/fog/night; strong velocity estimation
Limitations
Lower angular resolution; harder classification
Typical use
Long-range front radar for highway; corner radars for cross-traffic
LiDAR (Light Detection and Ranging)
What it measures
3D point cloud of distances to surfaces
Strengths
Accurate geometry and depth; good for mapping/localization
Limitations
Cost, moving parts (for some types), degraded by heavy rain/snow; reflective edge cases
Variants
Spinning (mechanical), solid-state, flash
Ultrasonic Sensors
What they measure
Very short-range distance (parking maneuvers)
Strengths
Cheap, effective within a few meters
Limitations
Limited range/resolution; unreliable at speed
GNSS/GPS + RTK (if available)
What it provides
Global position estimate, sometimes centimeter-level with RTK
Limitations
Urban canyons, tunnels, multipath errors
IMU (Inertial Measurement Unit)
What it measures
Acceleration and angular rates
Role
Dead-reckoning between GPS updates; stabilizing localization
Wheel Odometry / Vehicle Sensors
What it measures
Wheel speed, steering angle, yaw rate, throttle/brake status
Role
State estimation and control feedback
V2X (Vehicle-to-Everything, optional)
What it adds
Signal phase and timing, hazard messages, cooperative awareness
Limitations
Infrastructure and adoption dependency; security considerations
Sensors trade off semantics, geometry, and robustness; redundancy across modalities improves reliability.
Think (Onboard Compute + AI)
Compute Hardware
GPU/AI accelerators (NPUs/TPUs), CPUs, safety microcontrollers
Real-time constraints
Low latency pipelines; deterministic scheduling for critical tasks
Redundancy
Dual compute paths; watchdogs and safety supervisors
Software Stack Overview
Sensor calibration
Intrinsic/extrinsic calibration for camera/LiDAR/radar alignment
Sensor fusion
Combining modalities to improve accuracy and robustness
Perception
Detect, classify, and track objects; recognize lanes and signs
Localization & mapping
Estimate precise vehicle pose in a map or relative frame
Prediction
Forecast trajectories and intent of other agents
Planning
Decide maneuvers and generate a drivable path/speed profile
Control
Convert plan into steering/throttle/brake commands
Safety layer
Fault detection, minimal risk maneuver, operational constraints
Act (Vehicle Control)
Drive-by-wire interfaces
Brake-by-wire or electro-hydraulic braking
Throttle/torque control (ICE/EV)
Control loops
Lateral control
Lane keeping, curvature tracking, stability constraints
Longitudinal control
Adaptive cruise behavior, gap keeping, smooth braking
Controllers
PID, LQR, MPC (Model Predictive Control), robust control variants
Comfort and drivability
Jerk limits, smooth steering rate, passenger comfort constraints
Perception: Understanding the Scene
Object Detection and Classification
Vehicles, pedestrians, cyclists, animals
Static obstacles
Cones, debris, stalled vehicles, construction barriers
Traffic infrastructure
Traffic lights, signs, crosswalks, lane boundaries, road edges
AI methods commonly used
Convolutional neural networks (CNNs) / Vision Transformers (ViT) for vision
3D detection networks for LiDAR point clouds
Radar-based detection with learned or classical approaches
Tracking (Temporal Consistency)
Multi-object tracking
Assign detections across frames to maintain identities
State estimation for each object
Position, velocity, acceleration, heading
Common techniques
Kalman filters, particle filters, joint probabilistic data association
Sensor Fusion
Why fuse
Complementary strengths (camera semantics + radar velocity + LiDAR geometry)
Fusion levels
Early fusion (raw/feature-level)
Mid fusion (network feature fusion)
Late fusion (decision-level)
Handling sensor uncertainty
Confidence scores, covariance, outlier rejection
Road and Lane Understanding
Lane line detection
Painted lines, curbs, implied lanes in unmarked roads
Drivable space segmentation
Traffic light state recognition
Red/yellow/green, arrows, flashing states
Sign recognition
Speed limits, stop/yield, no turn, school zones
Localization and Maps
Localization Approaches
Map-based localization
Match sensor observations to a pre-built HD map
Mapless / online localization
Rely on onboard perception and relative positioning
Combined methods
GPS/RTK + IMU + vision/LiDAR/radar odometry
HD Maps (when used)
Contents
Lane geometry, boundaries, traffic control elements, 3D landmarks
Benefits
Better anticipation and localization accuracy
Challenges
Cost to build/maintain; changes due to construction/weather
SLAM (Simultaneous Localization and Mapping)
Purpose
Build/update a map while estimating vehicle pose
Modalities
Visual SLAM, LiDAR SLAM, radar SLAM, multi-sensor SLAM
Prediction: Forecasting Other Road Users
What is predicted
Future trajectories of vehicles, pedestrians, cyclists
Intent
Lane changes, turns, yielding, crossing decisions
Inputs to prediction
Current motion (velocity, acceleration)
Context
Lanes, traffic rules, signals, crosswalks
Interaction
Social driving cues, right-of-way negotiation
Methods
Physics-based models
Constant velocity/acceleration, kinematic constraints
Learning-based models
Recurrent/transformer trajectory models, graph neural networks
Uncertainty modeling
Multiple hypotheses; probability distributions over futures
Planning: Deciding What to Do
Planning Layers
Route planning (global)
From start to destination via road network
Behavior planning (tactical)
Decide maneuvers: follow, yield, overtake, merge, unprotected turn
Motion planning (local/trajectory)
Generate a feasible path and speed profile in seconds horizon
Constraints considered
Safety
Collision avoidance, safe stopping distance, visibility limits
Legality
Traffic rules, right-of-way, speed limits
Vehicle dynamics
Turning radius, tire friction, stability limits
Comfort
Smooth acceleration and steering; low jerk
Efficiency
Progress, energy use, minimizing unnecessary braking
Common algorithms
Optimization-based
MPC, quadratic programming, constrained nonlinear optimization
Sampling-based
RRT/RRT*, trajectory rollouts
Risk-aware planning
Cost functions balancing safety, legality, comfort, progress
Control: Executing the Plan
Feedback and feedforward control
Correct deviations due to wind, road slope, sensor noise
Actuator limits and delays
Saturation handling, latency compensation
Stability and safety constraints
Traction limits, ABS/ESC coordination, emergency braking logic
Safety, Redundancy, and Fail-Operational Design
Operational Design Domain (ODD)
Definition
Conditions where the system is intended to operate
Dimensions
Road type, speed range, weather, lighting, traffic density, map coverage
ODD monitoring
Detecting when conditions exceed capability
Functional safety and standards
ISO 26262 (functional safety)
SOTIF (safety of intended functionality)
Redundancy strategies
Sensor redundancy
Multiple modalities; overlapping coverage
Compute redundancy
Primary + backup compute; safety microcontroller
Power and actuation redundancy
Backup power, redundant braking/steering paths (platform-dependent)
Fault detection and handling
Self-checks
Sensor health monitoring, calibration drift detection
Degradation modes
Reduced-speed mode, handover request, minimal risk maneuver
Minimal Risk Maneuver (MRM)
Controlled pull-over/stop with hazard lights when needed
Cybersecurity
Secure boot, signed updates, network segmentation
Intrusion detection, V2X security (if used)
Data, Training, and Validation (AI Lifecycle)
Data collection
Fleet logging of edge cases, rare events, near misses
Labeling and ground truth
Human labeling, auto-labeling, simulation-assisted labeling
Model training
Supervised learning for detection/segmentation
Self-supervised learning for representation
Reinforcement learning (limited/targeted use in some components)
Simulation
Large-scale scenario generation
Sensor simulation (camera/radar/LiDAR)
Closed-loop testing for planning/control
On-road testing and validation
Scenario-based testing, regression suites, coverage metrics
Continuous improvement
Over-the-air updates, monitoring performance, rollback mechanisms
Typical Driving Scenario Flow (End-to-End Loop)
1) Sensors capture environment
2) Preprocessing and calibration align data in time and space
3) Perception detects lanes, signals, and objects; tracking estimates motion
4) Localization estimates precise vehicle pose
5) Prediction forecasts other agents’ possible futures
6) Planning chooses behavior and generates a safe trajectory
7) Control executes steering/brake/throttle to follow trajectory
8) Safety monitor checks constraints and triggers fallback if needed
9) Loop repeats multiple times per second
Key Challenges and Edge Cases
Adverse weather and visibility
Heavy rain, snow, fog, glare, low sun angles
Rare/unexpected scenarios
Unusual construction, debris, emergency vehicles, hand signals from police
Human behavior and negotiation
Aggressive merges, ambiguous yielding, jaywalking
Sensor artifacts
Lens occlusion, LiDAR dropouts, radar multipath, calibration drift
Map staleness (if map-dependent)
Changed lanes, temporary signage, new road layouts
Levels of Driving Automation (Context)
SAE Levels
L0–L2: driver responsible (assist features)
L3: system drives within ODD; expects takeover when requested
L4: system drives within ODD without human fallback
L5: drives everywhere (not currently deployed broadly)
Wondershare EdrawMind
