Design end to end data pipeline solutions from problem statement through implementation and operations, integrating ingestion transformation storage serving and consumption layers. Topics include source selection and connectors, ingestion patterns including batch streaming and micro batch, transformation steps such as cleaning enrichment aggregation and filtering, and loading targets such as analytic databases data warehouses data lakes or operational stores. Cover architecture patterns and trade offs including lambda kappa and micro batch, delivery semantics and fault tolerance, partitioning and scaling strategies, schema evolution and data modeling for analytic and operational consumers, and choices driven by freshness latency throughput cost and operational complexity. Operational concerns include orchestration and scheduling, reliability considerations such as error handling retries idempotence and backpressure, monitoring and alerting, deployment and runbook planning, and how components work together as a coherent maintainable system. Interview focus is on turning requirements into concrete architectures, technology selection, and trade off reasoning.
HardTechnical
0 practiced
A terabyte-scale feature table consists of millions of small Parquet files causing extremely slow Spark jobs and high metadata overhead. Propose a step-by-step plan to compact files without interrupting serving, including strategies for progressive compaction, atomic dataset swapping, validation of results, throttling compaction jobs, and rollback in case of errors.
MediumSystem Design
0 practiced
Design a metadata catalog and data lineage system for ML pipelines that helps data scientists debug model inaccuracies. Specify what metadata to capture (source datasets, transform code versions, schema versions, feature versions, job/run IDs), how to store lineage (graph store), how to query lineage during incidents, and retention considerations.
MediumSystem Design
0 practiced
Design an approach to enrich streaming user events with a slowly changing user profile table (SCD) to compute features in real-time. Describe how you'd keep profile state up-to-date (CDC, streaming join, or external KV-store), caching policies, consistent snapshot semantics, and how to handle late updates to profiles that arrive after events.
HardTechnical
0 practiced
A streaming aggregation job's state is exploding due to very high-cardinality keys (user_id). Provide strategies to reduce state size: approximate algorithms (HyperLogLog for distinct counts), bloom filters, TTLs, pre-aggregation, partitioning/hot-key handling, and offloading cold keys to external stores. Propose a concrete plan to reduce state by 10x while keeping error under 1%.
EasyTechnical
0 practiced
List and compare common storage formats used in analytics pipelines: CSV, JSON, Avro, Parquet, ORC, and table formats like Delta Lake / Apache Iceberg. For an ML offline feature store that supports large-scale training, which format(s) would you choose and why? Discuss implications for schema evolution, compression, read performance, and random access.
Unlock Full Question Bank
Get access to hundreds of Data Pipeline Architecture interview questions and detailed answers.
