Production Readiness and Professional Standards Questions
Addresses the engineering expectations and practices that make software safe and reliable in production and reflect professional craftsmanship. Topics include writing production suitable code with robust error handling and graceful degradation, attention to performance and resource usage, secure and defensive coding practices, observability and logging strategies, release and rollback procedures, designing modular and testable components, selecting appropriate design patterns, ensuring maintainability and ease of review, deployment safety and automation, and mentoring others by modeling professional standards. At senior levels this also includes advocating for long term quality, reviewing designs, and establishing practices for low risk change in production.
HardTechnical
0 practiced
A model's calibration has slowly drifted over months. Propose a systematic diagnostic plan to determine whether the cause is data drift, label shift, model degradation, threshold misconfiguration or an evaluation-pipeline bug. Describe which experiments, metrics (reliability diagrams, calibration error, PSI, KL divergence), and controlled A/B tests you would run and the corrective actions for each root cause.
MediumSystem Design
0 practiced
Design a monitoring dashboard for a binary classification fraud model that handles 10k requests per minute and 500k predictions per day. Specify online metrics (prediction distribution, p95 latency, error rate), data metrics (feature drift scores, missing value rates), alert thresholds, sample queries or PromQL-like expressions for each metric, and an alerting policy describing severity and on-call steps.
MediumSystem Design
0 practiced
Design a CI/CD pipeline for model development and deployment that includes automated unit/integration tests, data quality checks, model validation against baseline, artifact storage and signing, canary rollout and automatic rollback. Describe pipeline triggers, stages, required artifacts at each stage, gating criteria, and how you'd secure pipeline credentials and artifacts.
HardTechnical
0 practiced
Draft a release testing matrix for a model change. List concrete tests at unit, integration, model-acceptance (offline comparison), shadow/parallel testing, canary, and full rollout. For each stage specify automation level, acceptance criteria, expected runtime, required artifacts (logs, metrics, diff reports) and who must sign off before promotion to the next stage.
MediumTechnical
0 practiced
Describe a pytest-based integration test that starts a local FastAPI model server, sends (1) a valid inference request and (2) a malformed request, asserts schema conformity and correct error fields, verifies response time is below a threshold (for example 200ms), and verifies proper status codes. Explain how you'd run this test in CI with docker-compose.
Unlock Full Question Bank
Get access to hundreds of Production Readiness and Professional Standards interview questions and detailed answers.
