Scalability and Code Organization Questions

Focuses on designing software and codebases that remain maintainable and performant as features and user load grow. Areas include modularity and separation of concerns, component and API boundaries, when and how to refactor, trade offs between monolith and service oriented architectures, data partitioning and caching strategies, performance optimization, testing strategies, dependency management, code review practices, and patterns for maintainability and evolvability. Interview questions may ask candidates to reason about design choices, identify coupling and cohesion issues, and propose practical steps to evolve an existing codebase safely.

MediumSystem Design

37 practiced

You need to expose a GPU-based model behind a REST API but only have a small pool of GPUs. Propose a GPU pooling and request routing design that maximizes throughput while meeting per-request latency SLOs. Discuss queueing, batching, scheduling policies, and how to pre-warm or evict models on GPUs.

EasyTechnical

29 practiced

Describe three caching strategies applicable to model-serving: prediction result caching, feature value caching, and model artifact caching. For each strategy explain a typical cache key design, expected hit-rate influences, freshness concerns, and how staleness can affect model quality or business metrics.

MediumTechnical

29 practiced

Design a lightweight Python SDK for internal inference clients that enforces API contracts, performs retries with exponential backoff, caches recent results, and emits metrics. Sketch a minimal client interface and explain how you would distribute, version, and test the SDK so teams can adopt it safely.

HardTechnical

41 practiced

You have two implementations of feature transformations: an offline library used in training and a separate online library used in inference. Propose a code organization and build/packaging approach to ensure parity, minimize duplication, and allow hotfixes to propagate safely into production. Include testing patterns to validate parity across versions.

MediumSystem Design

56 practiced

Design a real-time model-serving architecture for a recommendation engine that must handle 10,000 requests/sec with 50ms P95 latency. Describe components (API front-end, feature retrieval, caches, model-serving pods, batching layers), data flow, caching hierarchy, and how you would ensure feature freshness without touching DB internals.

Unlock Full Question Bank

Get access to hundreds of Scalability and Code Organization interview questions and detailed answers.

Join thousands of developers preparing for their dream job.