Lower layers learn simple patterns (e.g., edges, tones)

Higher layers learn complex concepts (e.g., objects, words, intents)

Takes inputs, applies weights, adds bias, produces an activation

Activation functions introduce nonlinearity

Input layer

Hidden layers (multiple in “deep” models)

Output layer

Computes predictions from inputs

Measures prediction error

Computes gradients of loss w.r.t. parameters using the chain rule

Updates parameters to reduce loss

Learning rate schedules

Many parameters allow modeling complex functions

Learns features automatically rather than relying on handcrafted features

Function

Common uses

Key idea

Function

Common uses

Typical components

Key idea

Function

Common uses

Limitations

Key idea

Function

Common uses

Core parts

Function

Use cases

Key idea

Function

Use cases

Challenges

Key idea

Function

Use cases

Uses labeled data (input, target)

Tasks

Learns structure without explicit labels

Tasks

Creates training signals from data itself

Examples

Benefit

Combines small labeled with large unlabeled datasets

Learns policies from rewards and interactions

Uses

Cleaning, normalization/standardization

Handling missing values (for structured data)

Labeling and quality checks (for supervised tasks)

Data augmentation

Choose architecture family based on data modality

Define input/output formats

Select loss function and metrics

Forward pass → compute loss

Backward pass → compute gradients

Optimizer step → update weights

Regularization

Normalization techniques

Hold-out validation and test sets

Metrics

Error analysis

Latency and throughput considerations

Model compression

Monitoring

Parameters: learned weights and biases

Hyperparameters: learning rate, batch size, depth, width, dropout rate

Underfitting vs. overfitting

Ability to perform well on unseen data

CNNs: locality and translation invariance

Transformers: flexible attention-based dependency modeling

More data + larger models + more compute often improves performance up to limits

Images, audio, text, video

Pretrain on large corpora, fine-tune on specific tasks

Large training sets and GPUs/TPUs often required

Hard to explain decisions; requires interpretability tools

Sensitive to distribution shift and adversarial perturbations

Models can learn societal or dataset biases

GANs, very deep networks without careful design

Recognition, detection, segmentation, medical imaging

Search, chatbots, summarization, sentiment, translation

ASR, TTS, speaker identification, music generation

Personalized feeds, ads, retrieval

Protein structure, materials discovery, weather modeling

Fraud detection, risk modeling, anomaly detection

Large datasets available or pretraining can be leveraged

Problem involves images/text/audio or complex patterns

High accuracy is critical and compute budget exists

Data is small and structured

Interpretability and simplicity are primary requirements

Resource constraints are strict

One full pass through the training dataset

Subset of data used for one optimization step

Direction and magnitude for parameter updates

Performs well on training data but poorly on new data

Reusing a pretrained model for a new task

Dense vector representation of discrete items (words, users, products)

Mechanism to weight relevant parts of input when producing outputs