Stream Processing and Event Streaming Questions

Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.

HardSystem Design

0 practiced

Design an automated integration testing environment that can run deterministic end-to-end tests for streaming pipelines at realistic scale. Describe how to provision test clusters, seed synthetic or replayed data, manage state storage (e.g., ephemeral S3), and validate correctness and performance without affecting production.

EasyTechnical

0 practiced

Write a small Python function that computes consumer lag per partition given two dictionaries: end_offsets and committed_offsets, both keyed by partition number. Return a dict of partition->lag and a total lag. Include handling for partitions with missing committed offsets (assume start at earliest). Example: end_offsets={0:100,1:200}, committed_offsets={0:90,1:190} -> partition lags {0:10,1:10}, total 20.

MediumSystem Design

0 practiced

Design a fault-tolerant streaming ingestion and processing pipeline for a web clickstream system that must handle 100k events/sec, provide p99 processing latency under 50ms, retain raw events for 7 days, and maintain an SLO of 99.9% successful processing. Specify choice of message broker, partitioning strategy, processing engine, state backend, checkpointing, scaling tactics, and monitoring/alerting approach. Outline trade-offs.

HardTechnical

0 practiced

Design a robust dead-letter handling strategy for a streaming pipeline that processes heterogeneous JSON events. How do you detect poison messages, route them to a DLQ, provide observability, and allow safe manual/automated replay after fixes? Consider at-least-once semantics, performance impact, and data retention policies.

MediumTechnical

0 practiced

Explain how to achieve end-to-end exactly-once semantics in a pipeline using Kafka and Flink. Cover Kafka producer idempotence and transactions, Flink checkpointing and two-phase commit sinks, and idempotent or transactional external systems. Discuss performance and operational trade-offs.

Unlock Full Question Bank

Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.

Join thousands of developers preparing for their dream job.