ML System Design

Machine learning infrastructure: serving, training, monitoring, and production systems

100 minutes

9Detailed Sections

Senior Level

Online inference: low-latency response to individual requests. REST APIs (slower, high latency), gRPC (faster, binary, HTTP/2).

Batch inference: process large volumes asynchronously (daily model updates). Constraints: model size (gigabytes), latency requirement (sub-100ms for real-time).

Solutions: quantization (reduce precision for speed), distillation (train smaller model from larger), caching predictions. Tools: TensorFlow Serving, KServe, Ray Serve, Seldon.

Deployment: A/B test on subset, canary rollout. Monitoring: latency, error rate, throughput.

Real-world: Netflix recommends movies via batch; Uber estimates rides via online inference; Amazon personalizes via both.

Key Takeaways

Online Inference: Per-request model execution; REST or gRPC; sub-100ms latency required

Batch Inference: Precompute predictions for many items; daily/hourly updates

Optimization: Quantization (reduce precision), Distillation (smaller model)

Latency Challenge: Model size and computation limits real-time speed

Deployment Strategy: A/B test subset, canary rollout, monitor inference quality

Monitoring: Latency (p50/p99), throughput, error rate, model staleness

Visual Diagram

Request -> Model Server (inference) -> Response (online) vs Batch job -> Precompute predictions

All Tutorials Practice Questions

ML System Design

Table of Contents

Model Serving: REST, gRPC, Batch vs Online Inference

Key Takeaways

Visual Diagram

Feature Stores: Centralized Feature Management

Training Pipelines: Orchestration and Data Handling

Model Versioning and Registry

A/B Testing and Experimentation for ML

Model Drift: Monitoring and Retraining Strategies

Embeddings and Vector Databases for Similarity

End-to-End ML System: Architecture and Lessons

Case Studies: ML Systems That Power Billions of Decisions