Stream Processing and Event Streaming Questions

Designing and operating systems that ingest, process, and serve continuous event streams with low latency and high throughput. Core areas include architecture patterns for stream native and event driven systems, trade offs between batch and streaming models, and event sourcing concepts. Candidates should demonstrate knowledge of messaging and ingestion layers, message brokers and commit log systems, partitioning and consumer group patterns, partition key selection, ordering guarantees, retention and compaction strategies, and deduplication techniques. Processing concerns include stream processing engines, state stores, stateful processing, checkpointing and fault recovery, processing guarantees such as at least once and exactly once semantics, idempotence, and time semantics including event time versus processing time, watermarks, windowing strategies, late and out of order event handling, and stream to stream and stream to table joins and aggregations over windows. Performance and operational topics cover partitioning and scaling strategies, backpressure and flow control, latency versus throughput trade offs, resource isolation, monitoring and alerting, testing strategies for streaming pipelines, schema evolution and compatibility, idempotent sinks, persistent storage choices for state and checkpoints, and operational metrics such as stream lag. Familiarity with concrete technologies and frameworks is expected when discussing designs and trade offs, for example Apache Kafka, Kafka Streams, Apache Flink, Spark Structured Streaming, Amazon Kinesis, and common serialization formats such as Avro, Protocol Buffers, and JSON.

HardTechnical

0 practiced

Events may arrive up to 24 hours late and out-of-order for a sessionization job that must produce daily aggregations. Propose a watermarking and allowed-lateness strategy, describe how to implement corrections/updates to previously emitted aggregates (retractions or upserts), and explain the operational implications for state retention and storage.

HardTechnical

0 practiced

A production pipeline is producing inconsistent states: duplicates, high consumer lag, and occasional data loss. Outline a step-by-step debugging and mitigation plan. What metrics and tools would you inspect (e.g., kafka-consumer-groups, broker logs, consumer GC), and how would you perform a safe replay of a topic to fix historical data without violating downstream constraints?

EasyTechnical

0 practiced

Implement a simple in-memory deduplication layer in Python for a streaming consumer that may receive duplicate events. Each event is a JSON object with an 'event_id' and 'timestamp'. Requirements: deduplicate within a sliding time window of 10 minutes, bound memory usage, and explain time/space complexity. You may use standard Python libraries.

EasyTechnical

0 practiced

You're building an ML inference pipeline that ingests telemetry events in real time. Explain the key architectural and semantic differences between a traditional message-queue (e.g., RabbitMQ) and a commit-log system (e.g., Apache Kafka). Discuss ordering guarantees, retention and replay capabilities, consumer semantics, and typical failure/recovery trade-offs. When would you choose one over the other for model feature ingestion and why?

EasyTechnical

0 practiced

Explain consumer-group semantics in systems like Kafka: how parallelism is achieved, what happens during rebalancing, and the differences between auto-commit and manual offset commit. How do these behaviors affect at-least-once and exactly-once processing guarantees for downstream ML pipelines?

Unlock Full Question Bank

Get access to hundreds of Stream Processing and Event Streaming interview questions and detailed answers.

Join thousands of developers preparing for their dream job.