Business Acumen and Commercial Context Questions

Direct answer

Pick one objective, translate it into a measurable change in how your team decides and spends its time, and measure it in the business's own units (revenue, margin, retention), with a credit rule agreed in advance so the result survives scrutiny. The story below uses an engineering team, but the same shape fits an analytics or product function.

The story shape

Situation. The company's objective for the year was to win more enterprise customers (larger customers on bigger contracts). Engineering's own goals were about delivery speed and reliability, and nobody had said what the objective meant for our daily work.

Translation (the specific actions).

1. I asked sales and customer teams which deals had stalled or been lost on a technical requirement (single sign-on, so staff log in with the company's own identity system; audit logs, a record of who did what; data residency, keeping data in a required country or region; uptime commitments, contractual availability levels), and had them tag these in the customer relationship management (CRM) system, with the buyer's written requirement attached.
2. I reserved 25% of the team's capacity for those blockers: 2 of 8 engineers.
3. We changed intake: a weekly triage (sorting incoming items by urgency and value) of tagged blockers ranked by deal value and by how many deals shared the same gap, and a definition of done (the checklist a piece of work must meet to count as finished) that required the requirement to be demonstrable to a buyer.
4. We changed one rule: reliability work was justified by contract requirements, not only by internal incident counts.

Measuring impact in business terms (illustrative, derived)

Over four quarters, 6 closed enterprise deals had a tagged technical blocker we then fixed, worth $180k a year each, or $1,080k of yearly contract value. Sales and I agreed the credit rule beforehand: count a deal only if the buyer's written requirement named the capability and the deal closed after we shipped it. That held for 4 of the 6, so credited revenue is 4 x $180k = $720k.

• Cost: 2 engineers x $200k loaded yearly cost (salary plus overhead) = $400k.
• Credited revenue to cost: 720 / 400 = 1.8.
• At an 80% gross margin, credited gross profit is $576k, 1.44 times the cost (576 / 400).

I also tracked leading indicators (early signals that move before revenue does): the number of tagged blockers open and the median age of a blocker, because deal results arrive months later.

What got in the way

• Roadmap tension. Product lost a quarter of our capacity. I showed the cost (what the 2 engineers would have shipped) next to the credited revenue so the trade was visible.
• Over-tagging. Within a month, everything was a "blocker". The written-requirement rule fixed that.
• Attribution disputes. Sales felt every win was theirs. A pre-agreed credit rule, not a debate afterwards, ended it.
• Slow signal. Enterprise cycles run long. Leading indicators kept the team and leadership confident.
• Morale. Engineers worried they had become a feature factory (a team that ships requested features without checking they matter). Linking each fix to a named customer need helped.

The same shape in an analytics team

For a team of 8 analysts, reserve 25% (2 analysts) for the pricing and renewal-risk analyses that sales tagged as blocking renewals. Agree the credit rule in advance in the same way: a renewal counts only if the customer's written reason named the question the analysis answers and the analysis shipped before the renewal date. Compute credited revenue against the two analysts' loaded cost exactly as above, and track leading indicators such as open tagged requests and their median age.

What I would do differently

Set the credit rule with finance as well as sales, and review at six months whether the 25% reservation still matched the backlog of blockers.

Pitfalls

Vague stories ("we aligned with strategy") fail. The interviewer wants the specific change in how you worked, a result in business units, and an honest obstacle. Do not claim the whole company outcome as yours; claim the share you can defend.

Direct answer

The value of a latency cut is the extra conversions it produces times what each conversion is worth, minus the cost of delivering the speed-up. You need five inputs: traffic and baseline conversion, contribution per conversion, (most important) a measured relationship between latency and conversion from your own site, which is called the elasticity: the percentage change in conversion for each 100 ms of speed gained, which latency and which users the speed-up applies to, and the cost of delivering it. Such estimates usually go wrong by borrowing someone else's elasticity, extrapolating a straight line, confusing correlation with cause, and forgetting the running cost.

What you need to know

1. Traffic and conversion. Sessions per month (10,000,000 in the example) and baseline conversion (2.5%, so 250,000 conversions a month). Conversion here means a session ending in a purchase.
2. Value per conversion. Contribution per conversion (revenue less variable cost), $40 in the example. Using revenue overstates the benefit.
3. The latency-to-conversion relationship. The relative change in conversion for each 100 ms saved. This is the number everything hangs on and it cannot be assumed.
4. Which latency and which users. Median or 95th percentile (p95, the speed that 95% of loads beat)? Which pages (product page versus checkout) and which devices? A 500 ms to 200 ms improvement on the median may be a 3000 ms to 2900 ms change for the slowest users.
5. Cost to deliver. A caching layer costs $18,000 a month in infrastructure and upkeep, in this example.

How to get the relationship

Best: a randomised experiment that adds an artificial delay (for example 100 or 300 ms) to a small slice of traffic and measures conversion loss. Next best: a natural experiment (a change nobody planned as a test but that you can compare before and after, such as a deploy that happened to change speed), keeping the device mix the same and adjusting for confounders, meaning other things that changed at the same time, such as a sale or a new ad campaign. Weakest: correlation across pages, because fast pages are usually simple pages and fast users have good devices.

Worked example

SESSIONS_PER_MONTH = 10_000_000
CONV_RATE = 0.025
CONTRIBUTION_PER_CONV = 40.0     # dollars per conversion after variable costs
CACHE_COST_PER_MONTH = 18_000    # infrastructure plus upkeep of the caching layer
MS_SAVED = 300

conversions = SESSIONS_PER_MONTH * CONV_RATE
print(f"baseline conversions per month: {conversions:,.0f}")
print("assumed relative conversion gain per 100 ms saved -> value and net per month")
for per_100ms in (0.0025, 0.005, 0.01, 0.015):
    relative_gain = per_100ms * MS_SAVED / 100
    value = conversions * relative_gain * CONTRIBUTION_PER_CONV
    print(f"{per_100ms:.2%} per 100 ms: +{relative_gain:.2%} = {conversions*relative_gain:>7,.0f} conversions, ${value:>8,.0f}/month, net ${value - CACHE_COST_PER_MONTH:>8,.0f}")

needed_conversions = CACHE_COST_PER_MONTH / CONTRIBUTION_PER_CONV
total_gain = needed_conversions / conversions
print(f"break-even: {needed_conversions:,.0f} extra conversions a month = {total_gain:.2%} total = {total_gain / (MS_SAVED / 100):.2%} per 100 ms")

Output:

baseline conversions per month: 250,000
assumed relative conversion gain per 100 ms saved -> value and net per month
0.25% per 100 ms: +0.75% =   1,875 conversions, $  75,000/month, net $  57,000
0.50% per 100 ms: +1.50% =   3,750 conversions, $ 150,000/month, net $ 132,000
1.00% per 100 ms: +3.00% =   7,500 conversions, $ 300,000/month, net $ 282,000
1.50% per 100 ms: +4.50% =  11,250 conversions, $ 450,000/month, net $ 432,000
break-even: 450 extra conversions a month = 0.18% total = 0.06% per 100 ms

• At an assumed 0.5% relative conversion gain per 100 ms, saving 300 ms gives +1.5%, which is 3,750 extra conversions and $150,000 a month; net of the $18,000 cache cost, $132,000 a month, a return on cost of about 7.3 times ($132,000 / $18,000).
• The result is very sensitive to the assumed elasticity: it swings from $57,000 to $432,000 net a month across the range tried, while break-even is only 0.06% per 100 ms. So the decision to build is robust, but the headline benefit number is not and should be quoted as a range.

Where such estimates usually go wrong

• Borrowed statistics. Another company's "100 ms = 1%" is a story about its users and pages.
• Linear extrapolation. Gains tend to shrink below a speed where users stop noticing, and may be larger for slow users; a straight line over 300 ms overstates.
• Correlation as cause. Fast pages convert better partly because of what they are.
• Averages versus tails. The benefit may sit in the p95 population, which the median hides.
• Double counting. Double counting means claiming the same benefit twice, for example counting both higher conversion and lower bounce rate (the share of visitors who leave after one page; it is a symptom of the same effect), or adding SEO gains (better search-engine ranking from faster pages) that the experiment cannot see and that you have not measured.
• Forgetting running cost and opportunity cost. The cache needs maintenance, invalidation bugs (cache invalidation is deciding when stored copies of a page are out of date and must be refreshed) can serve stale prices, and the engineers could be doing something else.
• Pulled-forward demand. Some conversions would have happened anyway, later or through another channel; a faster page may only make a shopper buy today instead of next week, so the net gain is smaller than the count of extra conversions in the window.

What I would present

A range with a stated method ("measured 0.5% per 100 ms from a delay experiment, interval 0.3% to 0.7%"), the cost, the break-even elasticity, and the one thing that would change the call. Spoken version: "We measured that every 100 ms we remove lifts conversion by about 0.5% relative, with a plausible range of 0.3% to 0.7%. Cutting 300 ms is therefore worth about $150,000 a month, between $90,000 and $210,000, against $18,000 a month to run. It stops paying for itself only if the true effect is below 0.06% per 100 ms, far under anything we measured. The one thing that would change my call is if the effect turns out to be concentrated in users the cache does not reach."

Direct answer

I would default to buying or renting when the capability is not what sets the company apart and volume is below the point where owning becomes cheaper; I would build when it is core to the product, when volume is high, or when data or regulation rules a vendor out; and I would consider a partnership in between. I decide with three tools together: a three-year total cost of ownership (TCO: every cost of the option over the period, not just the sticker price), a break-even volume (the monthly volume at which two options cost the same), and a weighted score on non-cost axes. For a language model for customer support at 30,000 conversations a month, my illustrative numbers favour buying, and the call flips if volume passes about 38,900 conversations a month or if strategic value is weighted heavily.

Visible and hidden costs

Worked example: 3-year TCO (all figures illustrative)

Assume 30,000 conversations a month.

• Buy: $0.80 per conversation (36 months x 30,000 x $0.80 = $864,000), $60,000 integration once, and 0.5 of an engineer at $180,000 a year loaded (salary plus benefits and overhead) to run it ($90,000 a year, $270,000 over 3 years). Total $1,194,000. The vendor-operations engineer is costed at $180,000 and the build engineers at $200,000 because the run role is a more junior, partly shared position; that is a judgement, and on a common $200,000 basis the buy total is $1,224,000, buying is still about $178,000 cheaper, and break-even moves only slightly, to about 37,600 conversations a month ($880,000 / $23.40).
• Build on a hosted foundation-model API (a large pre-trained language model rented by the call from a provider, which you prompt, connect and evaluate rather than train from scratch): 3 engineers at $200,000 a year in year 1 ($600,000), 1.5 engineers for upkeep in years 2 and 3 ($300,000 a year), $40,000 evaluation and labelling, and $0.15 per conversation inference ($54,000 a year). Total $600,000 + $40,000 + $600,000 + $162,000 = $1,402,000.

At today's volume buying is about $208,000 cheaper over three years. Break-even volume V (conversations a month) comes from writing each three-year cost as a fixed part plus a part that grows with volume. Buying: fixed $60,000 integration + $270,000 to run the vendor = $330,000; per monthly conversation, $0.80 x 36 months = $28.80, so buying costs $330,000 + $28.80 x V. Building: fixed $600,000 + $40,000 + $600,000 = $1,240,000; per monthly conversation, $0.15 x 36 = $5.40, so building costs $1,240,000 + $5.40 x V. Setting them equal gives V = ($1,240,000 - $330,000) / ($28.80 - $5.40) = $910,000 / $23.40 = about 38,900 a month. Above that, building is cheaper. If the vendor raised its price to $1.00 per conversation, break-even would fall to about 29,700, below today's volume, so a vendor price rise is the biggest cost risk of buying.

Scored decision matrix (hypothetical scores, 1 to 5, where 5 is best for us)

Partner here means a co-development or managed-service arrangement, for example a vendor that tunes its model on our conversations under a shared roadmap and shared commercial terms. I have not costed it in the TCO because its terms vary widely, so its scores are a judgement between the two pure options until a quote exists.

Each score is multiplied by its weight and divided by 100. The reasons behind the scores, which are judgements to challenge: Build scores 2 on cost ($1.40M against $1.19M), 2 on time to value (hiring and building take months) and 2 on integration (we write every connection to our help desk and data systems ourselves), but 4 on regulatory and data risk (data never leaves our systems) and 5 on strategic value (we own the capability). Buy scores 4 on cost (cheapest at today's volume), 5 on time (live in weeks), 4 on integration (vendor connectors exist), 3 on regulatory and data risk (it depends on the vendor's terms) and 2 on strategic value (competitors can buy the same thing). Partner sits at 3 on most axes and 4 on strategic value because of the shared roadmap. Buy wins. If strategic value gets weight 40 and the others become 20, 15, 10 and 15, the scores become Build 3.50, Buy 3.20, Partner 3.40, and build wins. That shows the decision rests on one question: is this capability a source of lasting advantage for us?

My recommendation and what changes it

Buy now, on a contract with data-export rights and a price cap, and keep the conversation data and evaluation set in-house so a later switch to build costs weeks rather than a year. Revisit if volume nears 38,900 a month, if the vendor cannot meet data-residency (customer data must stay in a stated country or region) or audit requirements, if the vendor's price rises, or if the company decides this capability is its edge.

Pitfalls

• Comparing a vendor's price with only build salaries, ignoring maintenance, or the reverse.
• A weighted matrix hides disagreement about the weights, so I show the sensitivity to weights.
• Treating build as irreversible, or buy as free to leave.

Direct answer

Make the invisible visible by pricing it: convert "40% fewer outages" into avoided cost per year, add the benefits customers never see but the company does (engineer toil, on-call load, delivery speed), and be honest about payback. Then prove it in a pilot, but not with outage counts alone: outages are too rare for a short pilot to show a 40% change, so the pilot must be judged on leading indicators that occur often enough to measure, with a control group and success criteria agreed beforehand.

Making the case to executives

Frame it as insurance: a known cost now against an expected loss that falls. Illustrative inputs: 10 major outages a year, 50 minutes each, $300 of lost revenue per minute, $20,000 per outage of response time and service credits, and a $220,000 one-off project cost.

OUTAGES_PER_YEAR = 10
MINUTES_PER_OUTAGE = 50
REVENUE_LOST_PER_MINUTE = 300
RESPONSE_AND_CREDITS_PER_OUTAGE = 20_000    # engineer time plus SLA credits, illustrative
PROJECT_COST = 220_000                       # one-off, illustrative
REDUCTION = 0.40

per_outage = MINUTES_PER_OUTAGE * REVENUE_LOST_PER_MINUTE + RESPONSE_AND_CREDITS_PER_OUTAGE
annual = OUTAGES_PER_YEAR * per_outage
avoided = annual * REDUCTION
print(f"cost per outage ${per_outage:,}; yearly outage cost ${annual:,}")
print(f"outages avoided per year {OUTAGES_PER_YEAR * REDUCTION:.0f}; value ${avoided:,.0f} a year")
print(f"payback on outage cost alone: {PROJECT_COST / avoided * 12:.1f} months")

Output:

cost per outage $35,000; yearly outage cost $350,000
outages avoided per year 4; value $140,000 a year
payback on outage cost alone: 18.9 months

On outage cost alone, payback is 18.9 months, which is a sober number to show rather than hide. Then add what that number leaves out:

• Tail risk. The average hides the rare bad day; one 4-hour outage costs 4 x 60 x $300 = $72,000 in lost revenue before response costs. Reducing the chance of the long tail is worth more than the average suggests.
• Engineering time. Hours spent in incident response, postmortems and manual recovery, and on-call burnout (attrition costs are real but must be estimated, not asserted).
• Delivery speed. Fewer failed releases means more deploys, which is a feature-velocity benefit.
• Customer-visible proxies. Support tickets, SLA credits and renewal risk for accounts that have had incidents.

Ask for a staged decision: fund the pilot on the two most incident-prone services, with a go or no-go review at a set date. The pitch an executive would hear: "Outages cost us about $350,000 a year. This project removes about 40% of them, roughly $140,000 a year, and also cuts the pages that wake our engineers. It costs $220,000 once, so on outages alone it pays back in about 19 months, sooner once engineering time is counted. I am asking you to fund a pilot on two services, with success criteria we agree today and a review date. If we miss them, we stop."

Proving value in a pilot

Design. Pick pilot services and comparable control services that do not get the project in the same period (same traffic, similar change rate). Agree the metrics, the period and the success threshold before starting.

Why outage counts are not enough. A pilot's expected outages are few. In plain words, the code below answers one question: if the project really does cut outages by 40%, how often would a pilot of a given size show it convincingly, rather than looking like luck? That chance is called power. Suppose the treated and control groups have equal exposure, outage counts are Poisson (random, independent events), and the treated group truly has 40% fewer. The chance that a one-sided exact test at the 5% level detects it (the power) depends on how many outages the control group is expected to have over the pilot:

import math

def pois_pmf(lam, kmax):
    p = [math.exp(-lam)]
    for k in range(1, kmax + 1):
        p.append(p[-1] * lam / k)
    return p

def power(control_events, reduction=0.4, alpha=0.05):
    """Expected events in the control group over the pilot; treated group has
    (1 - reduction) times as many. Equal exposure. One-sided exact test: given the
    total N, the treated count is Binomial(N, 0.5) under no effect."""
    lc, lt = control_events, control_events * (1 - reduction)
    kmax = int(lc + 12 * math.sqrt(lc) + 20)
    pc, pt = pois_pmf(lc, kmax), pois_pmf(lt, kmax)
    # largest treated count that is significant, for each total N
    cache = {}
    def crit(N):
        if N not in cache:
            cum, best = 0.0, -1
            for i in range(N + 1):
                cum += math.comb(N, i) / 2 ** N
                if cum <= alpha:
                    best = i
                else:
                    break
            cache[N] = best
        return cache[N]
    total = 0.0
    for nt in range(kmax + 1):
        for nc in range(kmax + 1):
            if nt <= crit(nt + nc):
                total += pt[nt] * pc[nc]
    return total

for ctrl in (5, 10, 20, 50, 100):
    print(f"control events {ctrl:>3}, treated expected {ctrl*0.6:>5.1f}, power {power(ctrl):.2f}")

Output:

control events   5, treated expected   3.0, power 0.09
control events  10, treated expected   6.0, power 0.19
control events  20, treated expected  12.0, power 0.35
control events  50, treated expected  30.0, power 0.69
control events 100, treated expected  60.0, power 0.93

With 5 or 10 expected outages in the control group, power is 9% to 19%; you would need about 100 to be 93% likely to detect the effect. So a quiet pilot proves nothing either way.

How to read the code. The two groups' outage counts are drawn at random from the Poisson probabilities (pois_pmf), which give the chance of 0, 1, 2, ... outages when the average is known. For every pair of counts, the test asks whether the treated group's share of the combined outages is low enough to be surprising. The exact test works like this: if the project did nothing, each outage would be a coin flip between the two groups, so the treated count out of N total outages follows Binomial(N, 0.5). With N = 20 total outages, the treated group having 5 or fewer happens by chance about 2% of the time (6 or fewer is about 6%, which is over the 5% cut-off), so 5 is the critical value for N = 20; the function crit finds that cut-off for each N and caches it. The final double loop adds up the probability of every pair of counts that lands at or below the cut-off. "One-sided" means we only ask whether the treated group is lower, and "5% level" means that when nothing has changed the test wrongly declares success no more than 5% of the time.

Use measures that happen often:

• Change failure rate (the share of deployments causing an incident or rollback).
• Pages (alerts that wake an engineer) per week, and the share that are actionable.
• Faults caught by the new safeguards before customers saw them (near-misses), such as a bad deploy that automatic rollback reverted in two minutes before anyone noticed, or injected failures (a server deliberately switched off in a test) that the system survived.
• Time to detect and time to recover in incidents that do occur.
• Customer-facing minutes of impact, SLA credits (refunds promised in a service-level agreement) and support tickets as lagging confirmation.

A common target is 80% power: an 80% chance of detecting a real 40% effect. With 50 expected events in the control group, power is about 69%, just short; with about 70 it is about 82%, and with 100 it is 93%. So choose indicators that should produce roughly 70 or more events in the control group over the pilot (about 5 a week over a 13-week pilot), and treat anything below that as too thin to read.

Trade-offs and pitfalls

• Choosing the pilot services because they are easy makes the result unrepresentative; choose by incident history and report that choice.
• Regression to the mean (extreme results drift back toward normal on their own): if a service had 8 outages last quarter when it usually has about 4, it will probably have 4 or 5 next quarter with no project at all, which would look like a 40% improvement. A control group chosen the same way shows the drift, which is why it matters.
• Declaring victory on a quiet quarter: absence of outages is the expected outcome in many quarters.
• Agree the kill criteria too: what result would make you stop?

Direct answer

Lead with the customer and money, not the bug. Say: "We found a fault that is making some customers fail to pay us. Fixing it first takes about two weeks and moves the new feature by two weeks. Every week we leave it unfixed puts up to $24,000 of orders at risk, against about $15,000 a week of expected feature revenue, and the feature is only delayed, not lost. If many failed customers retry, the true loss is smaller, so I will measure the retry rate and confirm." Then give the new date, the cost of each option, and the one decision you need from them.

How to build the message

1. Translate the regression into a business symptom. A "reliability regression" means something that used to work got worse: slower pages, more errors, more downtime. Name the symptom the CEO already cares about: failed checkouts, support tickets, churn risk, a contract with an availability promise.
2. Put a number on the cost of leaving it. Use rates and money, not technical measures.
3. Put a number on the cost of the delay. Be honest that a delay is revenue deferred, not revenue destroyed, unless a launch window or a customer commitment makes it truly lost.
4. Offer a choice, with a recommendation. Two or three options, each with date, cost and risk, and your pick.
5. Say what you will report and when, so trust does not depend on the CEO understanding the fix.

Worked example (illustrative numbers)

A shop has 20,000 checkout attempts a week at an average order of $80. After last release, 1.5% of attempts fail with an error.

Some failed customers retry, so $24,000 is an upper bound; say so, and say you will measure the real retry rate. The comparison with the $15,000 a week the feature earns holds only if fewer than about 37.5% of failed checkouts are retried and completed (1 - 15,000 / 24,000 = 37.5%); above that, the fault costs less per week than the feature earns, and the case for fixing first rests on the risk growing and on the feature running on the same checkout. The script in the room:

"A change we shipped last month is making about 1 in 67 checkouts fail. That is roughly $24,000 of orders at risk every week. If we fix it first, the new feature lands two weeks later and about $30,000 of its revenue moves later in the year. If we ship the feature first, the checkout problem stays live and the risk keeps growing, and the feature itself runs on the same checkout. I recommend fixing first. I will send you the failed-checkout rate every Friday until it is back to normal."

(The 1 in 67 is 1 divided by 0.015.)

Trade-offs and pitfalls

• Jargon. Do not say "latency regression", "error budget" or "tech debt" to a CEO. Say "customers waiting", "failures", "money at risk".
• Do not hide the delay inside a vague date. Give a date with a confidence level ("two weeks, and I will confirm by Wednesday once we have traced the cause").
• Do not oversell the fix. If the cause is not yet known, say the range and when you will know.
• What changes the recommendation: if the feature is tied to a dated launch, a contract or a partner event that would be lost by a two-week slip, or if the regression only hits a small, low-value slice, the comparison changes. Say that you checked.