Python Programming Questions
Python as an interview language: core syntax, data types and built-in collections, comprehensions, iterators and generators, idiomatic style, and the standard library, extending into data-oriented and automation use of the language and its common libraries. Covers writing correct, Pythonic code and reasoning about the language's semantics. The most heavily exercised language surface in this category across engineering and data roles.
Design a producer-consumer pipeline in Python where producers can outpace consumers. How do you apply backpressure so producers don't overwhelm consumers, using a bounded queue? Walk through both a threading-based version and an asyncio.Queue-based version, and the wake-up semantics involved (notify vs notify_all, or await put/get).
Sample Answer
Direct answer
A bounded queue (fixed maxsize) applies backpressure automatically: once it is full, put() blocks (threading) or suspends (await put, asyncio) until a consumer frees a slot, so a fast producer is throttled to the consumers' pace without any extra code. queue.Queue gives you this with an internal threading.Condition; asyncio.Queue gives you the same shape with await instead of blocking waits. The wake-up semantics differ by mechanism: a raw condition variable needs you to choose notify() (wake exactly one waiter) versus notify_all() (wake everyone, most of whom will just recheck and go back to waiting), while asyncio.Queue hides that choice entirely, await put/await get suspend and resume the right coroutine without you managing wake-ups by hand.
Structured elaboration
Threading version: queue.Queue(maxsize=N) is backed by a Lock plus two Conditions internally (not-full, not-empty). put() blocks on the not-full condition when the queue is at capacity; get() blocks on not-empty when it is empty. You do not touch the condition variables yourself, Queue already calls notify() correctly.
asyncio version: a coroutine is a function defined with async def that can pause mid-execution, at an await, and let other coroutines run, then resume later exactly where it left off; the event loop is the scheduler that decides which paused coroutine gets to run next. asyncio.Queue(maxsize=N) has the same contract as the threading version, but await q.put(item) suspends the coroutine (yielding control to the event loop, not blocking an OS thread) when full, and await q.get() suspends when empty. Concurrency comes from running many producer/consumer coroutines as tasks on one event loop instead of many OS threads.
Graceful shutdown, both versions: signal producers to stop (an Event), let running producers finish their current item, then drain the queue (q.join(), which waits until every put item has had a matching task_done()), then push one sentinel value per consumer so each one exits its loop cleanly. q.join() hanging is almost always a sign that a consumer path skipped task_done(), most often via an exception on the processing line before it reached the task_done() call.
The harder variant, without queue.Queue/asyncio.Queue: you build the bounded buffer yourself from a Lock plus two Condition objects sharing that lock, one for "not full", one for "not empty." This is exactly what queue.Queue does internally, made explicit:
import threading, collections
class BoundedBuffer:
def __init__(self, maxsize):
self.maxsize = maxsize
self.items = collections.deque()
self.lock = threading.Lock()
self.not_full = threading.Condition(self.lock)
self.not_empty = threading.Condition(self.lock)
def put(self, item):
with self.not_full:
while len(self.items) >= self.maxsize:
self.not_full.wait() # releases the lock, sleeps until notified
self.items.append(item)
self.not_empty.notify() # wake exactly one waiting consumer
def get(self):
with self.not_empty:
while not self.items:
self.not_empty.wait()
item = self.items.popleft()
self.not_full.notify() # wake exactly one waiting producer
return item
notify() (one waiter) is correct here because exactly one slot opened up, waking everyone with notify_all() would just cause the other waiters to recheck their while condition and go back to sleep, which is harmless but wasteful (a "thundering herd" of wasted wake-ups). notify_all() earns its keep when a single event can satisfy many different waiters at once, which is not the case for a single-slot state change like this.
Worked example
asyncio.Queue-based pipeline with graceful shutdown, verified on CPython 3.12 (produced and consumed counts always match, since q.join() guarantees every item was drained before sentinels are sent; the exact counts vary run to run because they depend on real-time scheduling):
import asyncio, random
SENTINEL = object()
async def producer(pid, q, stop_event, rng):
i = 0
while not stop_event.is_set():
await q.put((pid, i)) # blocks here once the queue is full
i += 1
await asyncio.sleep(rng.random() * 0.02)
async def consumer(cid, q, processed):
while True:
item = await q.get()
if item is SENTINEL:
q.task_done()
break
processed.append((cid, item))
q.task_done()
async def run_pipeline(n_producers=2, n_consumers=3, maxsize=5, run_time=0.3, seed=7):
rng = random.Random(seed)
q = asyncio.Queue(maxsize=maxsize)
stop_event = asyncio.Event()
processed = []
producers = [asyncio.create_task(producer(i, q, stop_event, rng)) for i in range(n_producers)]
consumers = [asyncio.create_task(consumer(i, q, processed)) for i in range(n_consumers)]
await asyncio.sleep(run_time)
stop_event.set()
await asyncio.gather(*producers)
await q.join() # wait until every produced item is processed
for _ in consumers:
await q.put(SENTINEL)
await asyncio.gather(*consumers)
assert len(processed) == sum(1 for _ in processed) # every produced item made it through
asyncio.run(run_pipeline())
A representative run reported "total produced: 62, total consumed: 62"; the invariant produced == consumed holds on every run because q.join() will not return until all items are drained, the specific count itself is not reproducible (it depends on wall-clock scheduling) and is not the claim being made here.
Trade-offs & pitfalls
- Complexity: O(1) per item for queue operations; space is bounded by
maxsizefor the queue itself, plus O(k) for the k in-flight producer/consumer tasks. - Edge case: if a consumer raises an exception before calling
task_done(),q.join()hangs forever; wrap the processing body intry/finallysotask_done()always runs. - Edge case: sending fewer sentinels than there are consumers leaves some consumers blocked on
get()forever; one sentinel per consumer is required, not one total. - Choosing
notify()when the change actually affects multiple waiters (rare for a single bounded queue, more common in custom condition-based coordination) silently starves the other waiters; if you are not certain only one waiter's condition changed,notify_all()is the safer default even though it wakes more coroutines/threads than strictly necessary. - The asyncio version scales to many more concurrent producers/consumers than the threading version for the same memory budget, since coroutines are far cheaper than OS threads, but it only helps if the "work" inside producer/consumer is itself non-blocking; a CPU-bound consumer inside an asyncio pipeline will stall the whole event loop exactly like any other CPU-bound coroutine.
Given a Pandas DataFrame with columns 'timestamp' (timezone-aware) and 'value', write code to resample to hourly bins in UTC, forward-fill missing values, and ensure the result preserves timezone awareness and dtype. Explain pitfalls around DST transitions.
Sample Answer
Goal: resample timezone-aware timestamps to hourly UTC bins, forward-fill, preserve tz-aware dtype.
Code:
import pandas as pd
def resample_hourly_utc(df):
# ensure tz-aware
ts = pd.to_datetime(df['timestamp'])
if ts.dt.tz is None:
raise ValueError('timestamp must be timezone-aware')
# convert to UTC
df = df.copy()
df['timestamp'] = ts.dt.tz_convert('UTC')
df = df.set_index('timestamp').sort_index()
# resample hourly and forward-fill
out = df.resample('h').ffill() # lowercase 'h': pandas removed the uppercase hour alias
# ensure index tz-aware and dtype preserved
out.index = out.index.tz_localize(None).tz_localize('UTC') if out.index.tz is None else out.index
return out
Worked example, verified with pandas on CPython 3.12:
df = pd.DataFrame({
"timestamp": pd.to_datetime(
["2024-11-03 04:15", "2024-11-03 06:40", "2024-11-03 08:05"]
).tz_localize("America/New_York"),
"value": [10, 20, 30],
})
print(resample_hourly_utc(df))
Output:
value
timestamp
2024-11-03 09:00:00+00:00 NaN
2024-11-03 10:00:00+00:00 10.0
2024-11-03 11:00:00+00:00 10.0
2024-11-03 12:00:00+00:00 20.0
2024-11-03 13:00:00+00:00 20.0
The three original local readings, converted internally to UTC before resampling, become five hourly UTC bins: the first bin has no data yet before the first real reading, so it stays NaN (nothing earlier to forward-fill from), and each later hour repeats the most recent value until a newer one arrives (10.0 covers two consecutive bins because the next real reading does not land until the following hour).
DST pitfalls
- Convert to UTC before resampling to avoid ambiguous/missing local times.
- If you must resample in local tz, handle ambiguous times with tz_localize(ambiguous=...) and nonexistent with nonexistent=...; otherwise conversions across DST will produce duplicate or missing hours.
Why DST actually produces duplicate or missing local hours, made concrete: on 2024-11-03, US Eastern clocks "fall back" from 2:00 AM EDT to 1:00 AM EST, so local wall-clock time 01:00-01:59 happens twice, once while still on EDT (UTC-4) and once on EST (UTC-5):
naive = pd.Timestamp("2024-11-03 01:30:00")
first = naive.tz_localize("America/New_York", ambiguous=True) # earlier occurrence, still EDT
second = naive.tz_localize("America/New_York", ambiguous=False) # later occurrence, now EST
print(first, "->", first.tz_convert("UTC"))
print(second, "->", second.tz_convert("UTC"))
Output:
2024-11-03 01:30:00-04:00 -> 2024-11-03 05:30:00+00:00
2024-11-03 01:30:00-05:00 -> 2024-11-03 06:30:00+00:00
Both print the identical local wall-clock string, 01:30:00, but they are genuinely different instants, one hour apart in UTC; without ambiguous=, pandas has no way to know which of the two real moments a bare "2024-11-03 01:30:00" local string refers to, and that unresolved ambiguity is the "duplicate hour" pitfall.
The mirror case, a nonexistent time: on 2025-03-09, US Eastern clocks "spring forward" from 2:00 AM directly to 3:00 AM, so the entire local hour 02:00-02:59 never occurs at all:
try:
pd.Timestamp("2025-03-09 02:30:00").tz_localize("America/New_York") # expected: 02:30 never happened locally that day
except ValueError as e:
print(f'raises as expected: ValueError: {e}')
print(pd.Timestamp("2025-03-09 02:30:00").tz_localize("America/New_York", nonexistent="shift_forward"))
# 2025-03-09 03:00:00-04:00
Asking for 02:30 local time on that date raises by default, since no such wall-clock moment ever happened; nonexistent="shift_forward" is one explicit policy for handling it (push the timestamp forward past the gap), rather than silently guessing what was meant.
This approach keeps data consistent for ETL and downstream aggregation.
Write a function that removes duplicates from a list while preserving the first-seen order. It only needs to handle hashable elements. Then explain the time and space complexity of your solution and how you'd adapt it if the input were far too large to hold a full result list in memory.
Sample Answer
Approach
Since the elements are guaranteed hashable (hashable means it can be used as a dict key or set member: numbers, strings, and tuples made entirely of hashable values all qualify; lists, dicts, and sets themselves do not, since they can change after creation), a set gives O(1) average membership checks, so the whole pass stays linear: walk the input once, keep a set of everything already seen, and only append an item to the output the first time it appears.
Code (Python 3.12)
def unique_preserve_order(seq):
"""Return items from seq in first-seen order with duplicates removed.
Assumes every element of seq is hashable.
"""
seen = set()
out = []
for item in seq:
if item not in seen:
seen.add(item)
out.append(item)
return out
print(unique_preserve_order([3, 1, 3, 2, 1, 4]))
# [3, 1, 2, 4]
print(unique_preserve_order(["b", "a", "b", "c"]))
# ['b', 'a', 'c']
print(unique_preserve_order([]))
# []
The identical logic also dedupes an already-sorted input (a "unique_sorted" ask is the same function; sortedness only changes what the output looks like, not the algorithm) and works the same way on any hashable payload, such as deployment IDs:
deployment_ids = ["d-9", "d-3", "d-3", "d-11", "d-9", "d-2"]
print(unique_preserve_order(deployment_ids))
# ['d-9', 'd-3', 'd-11', 'd-2']
Key points
- The
setmembership check is what keeps this linear; checking membership against the growing output list instead (if item not in out) would silently work but degrade to O(n2), since list membership is a linear scan. - Order is preserved because
outis only ever appended to, never reordered; thesetis used purely for the O(1) "have I seen this" check and never influences output order.
Complexity
Time: O(n) average, one hash-table lookup and possible insert per element. Space: O(n) worst case (all elements unique), for both seen and out.
Edge cases
- Empty input returns
[]. - All-duplicate input returns a single-element list.
- Unhashable elements (a
listordictinsideseq) raiseTypeErrorfromitem not in seen/seen.add(item); per the problem statement this function is only contracted to handle hashable elements, so that is expected and correct behavior, not a bug to work around here.
Adapting for input too large to hold a full result list in memory
The seen set still has to track every distinct value you've encountered (there is no way to know something is a duplicate without remembering it), so the real constraint to relax is holding the full result in memory at once, not the full seen-set. Turn the function into a generator that yields as it goes, so a caller can stream results to a sink (a file, a socket, a downstream queue) instead of building out as a list:
def unique_preserve_order_stream(iterable):
seen = set()
for item in iterable:
if item not in seen:
seen.add(item)
yield item
print(list(unique_preserve_order_stream([3, 1, 3, 2, 1, 4])))
# [3, 1, 2, 4]
This removes the O(n) out list (memory now scales only with the number of distinct items seen so far, in seen, plus whatever the consumer of the generator chooses to retain), and it lets a downstream writer start consuming the first unique item immediately instead of waiting for the whole input to be processed. If even seen itself cannot fit in memory (billions of distinct values), the exact-uniqueness guarantee has to be traded for an approximate one (a probabilistic structure such as a Bloom filter, which can have false positives), which is a genuinely different problem from what is asked here and worth naming as the next escalation rather than silently downgrading correctness.
Your operations team gets weekly status items from different managers, but the same item can appear with different capitalization, extra spaces, or punctuation. In Python, write a function that normalizes the titles, removes duplicates while preserving the first occurrence, and returns the cleaned list. Assume a few thousand strings at most.
Sample Answer
Approach
I’d normalize by lowercasing, removing punctuation, and collapsing repeated spaces. Then I’d use a set of normalized keys to keep only the first occurrence. A set is a data structure that gives fast membership checks, so this stays simple and efficient for a few thousand strings.
import re
from typing import List
def normalize_title(title: str) -> str:
cleaned = re.sub(r'[\W_]+', ' ', title.lower())
return ' '.join(cleaned.split())
def dedupe_titles(titles: List[str]) -> List[str]:
seen = set()
result = []
for title in titles:
norm = normalize_title(title)
if norm in seen:
continue
seen.add(norm)
result.append(norm)
return result
titles = [' Weekly Update!', 'weekly update', 'Budget Review', 'Budget review ']
print(dedupe_titles(titles))
The regex pattern r'[\W_]+' is what actually strips punctuation: \W (capital W) matches any character that is NOT a letter, digit, or underscore, so it catches punctuation, symbols, and stray whitespace runs in one go; adding _ to the character class folds underscores into that same "replace with a space" rule too, since \W alone would NOT match an underscore (underscores count as word characters). The + means one-or-more, so any run of these unwanted characters collapses to a single space rather than leaving multiple spaces behind, and ' '.join(cleaned.split()) then trims leading/trailing spaces and collapses any remaining internal runs down to exactly one space each.
Example
Input: [' Weekly Update!', 'weekly update', 'Budget Review', 'Budget review ']
Output:
['weekly update', 'budget review']
That is the literal stdout of print(dedupe_titles(titles)) from the code block above.
Why this works: the first weekly update is kept, and the later duplicate is skipped because it normalizes to the same key.
Complexity: O(n * m) time, where m is string length, and O(n) extra space for the set.
If the team wants to keep original casing, I would store the first original string in result instead of the normalized version.
Given orig = [{'a': [1, 2]}, {'b': [3]}], what happens if you do sh = list(orig) and then mutate the nested list inside one of orig's dicts? What if you'd used copy.deepcopy(orig) instead? Trace through both.
Sample Answer
Approach
list(orig) is a shallow copy: it allocates a new outer list, but every element it holds is the exact same dict object as in orig, so mutating something inside one of those shared dicts (like the list stored under 'a') is visible through both orig and sh. copy.deepcopy(orig) recursively duplicates the dicts and the lists inside them, so dp shares nothing with orig at any level and is unaffected by mutating orig's nested state.
Code (Python 3.12)
import copy
orig = [{'a': [1, 2]}, {'b': [3]}]
sh = list(orig)
dp = copy.deepcopy(orig)
orig[0]['a'].append(99) # mutate the nested list two levels deep
print("orig:", orig)
print("sh: ", sh)
print("dp: ", dp)
print("sh[0] is orig[0]:", sh[0] is orig[0])
print("dp[0] is orig[0]:", dp[0] is orig[0])
Output:
orig: [{'a': [1, 2, 99]}, {'b': [3]}]
sh: [{'a': [1, 2, 99]}, {'b': [3]}]
dp: [{'a': [1, 2]}, {'b': [3]}]
sh[0] is orig[0]: True
dp[0] is orig[0]: False
Trace through both
sh = list(orig): a new list object is created for sh, but sh[0] is literally orig[0], the same dict, and sh[1] is literally orig[1]. orig[0]['a'] is a list object nested inside that shared dict, so sh[0]['a'] is that same list too, three references deep (orig[0]['a'], sh[0]['a'], and the object itself are all one thing). Calling .append(99) mutates that one shared list in place, so the change is visible through orig, sh, or any other name bound to the same dict, because nothing about list(orig) ever touched anything below the outer list.
dp = copy.deepcopy(orig): deepcopy walks into orig, sees it is a list, creates a new list, then for each element (a dict) creates a new dict, then for each value inside that dict (a list) creates a new list too, copying [1, 2]'s contents into a brand-new list object. By the time orig[0]['a'].append(99) runs, dp[0]['a'] is an entirely separate list from orig[0]['a'], so the mutation has nowhere to propagate to.
Key points
- The depth of sharing in a shallow copy is exactly one level: the container is new, its immediate contents are shared.
list(orig),orig.copy(),orig[:], andcopy.copy(orig)all produce the same one-level-shallow result for a plain list. deepcopyuses an internal memo dictionary keyed byid()to avoid copying the same shared object twice or infinitely recursing on cyclic structures; that memoization is also why two elements oforigthat happen to reference the same nested object stay correctly linked to each other in the deep copy, rather than becoming two independent copies that have drifted apart.
Complexity and edge cases
list(orig) is O(k) where k is the number of top-level elements (just copying references). copy.deepcopy(orig) is O(n) where n is the total number of objects in the entire nested structure, since every one of them gets visited and duplicated.
- If
orig[0]andorig[1]both referenced the same inner dict object (not the case in this example, but a common real bug), a shallow copy would obviously still share it; a deep copy would still correctly produce two shallow-copy targets that both point to one newly-copied dict, preserving the original aliasing relationship rather than accidentally splitting a shared object into two independent ones. - Tuples, sets, and custom objects nested inside the structure are handled by
deepcopytoo (it dispatches on type, and respects a class's own__deepcopy__if one is defined);list(orig)-style shallow copying has no equivalent generalization beyond "copy this one container".
Unlock Full Question Bank
Get access to all Python Programming interview questions and detailed answers.
Sign in to ContinueJoin thousands of developers preparing for their dream job.