Covers designing and operating machine learning systems to handle growth in data volume, model complexity, and traffic. Topics include distributed training strategies such as data parallelism, model parallelism, and pipeline parallelism; coordination and orchestration approaches like parameter servers, gradient aggregation, and framework tools such as PyTorch distributed, Horovod, and TensorFlow strategies; data pipeline and I O considerations including sharding, efficient formats, preprocessing bottlenecks, streaming and batch ingestion; serving and inference scaling including model sharding, batching for throughput, autoscaling, request routing, caching, and latency versus throughput tradeoffs. Also includes monitoring, profiling, checkpointing and recovery, reproducibility, cost and resource optimization, and common bottleneck analysis across network, storage, CPU preprocessing, and accelerator utilization.
MediumTechnical
36 practiced
Compare Horovod, PyTorch Distributed Data Parallel (DDP), and TensorFlow's MirroredStrategy in terms of ease of integration into existing training code, performance at scale, fault tolerance, and ecosystem/tooling support. Which would you recommend for fast-prototyping vs large-scale training?
EasyTechnical
32 practiced
Compare common serialization and storage formats used in ML pipelines (e.g., Parquet, TFRecord, Avro, ORC). For each format explain: optimal usage scenarios, read/write performance characteristics, compression trade-offs, and how they affect parallel ingestion and shuffling for large-scale training.
EasyTechnical
26 practiced
Explain the latency versus throughput trade-offs in inference serving. Provide concrete examples of two techniques (e.g., request batching and model quantization) and explain how they shift the latency/throughput curve. As a data engineer, what telemetry would you collect to decide which approach to apply?
HardSystem Design
25 practiced
Design monitoring, SLOs, and alerting for an ML platform that serves both training and inference workloads. Define 5 SLOs (with targets) for training pipelines and 5 SLOs for inference serving, explain the metrics to measure them, and describe escalation/runbook steps when an SLO is breached.
MediumTechnical
28 practiced
Explain model parallelism strategies for extremely large models that don't fit on one GPU: tensor (operator) parallelism, pipeline (layer) parallelism, and expert-based (Mixture-of-Experts) parallelism. For each, explain how activations and gradients move across devices and one engineering challenge for data engineers to support training pipelines.
Unlock Full Question Bank
Get access to hundreds of AI System Scalability interview questions and detailed answers.
