Tactical Excellence, Strategic Truth: Who Checks Whom

A Line That Stays Open

Think of two people on a phone call. One placed the call and the other answered, and the line stays open between them whether or not anyone is talking.

The caller may put the phone down and walk away, put the call on hold to take another one, or drive through a tunnel and lose the signal. The caller knows which of those happened. The person who answered knows only that the line is still open or that it has gone dead, and cannot tell whether anyone is still listening. They might hang up after a stretch of silence, or the phone company might cut the line for inactivity. Either way the call ends, and at most one person knows why.

Now ask a simple question: are dropped calls getting better? Engineering shows a chart where drops are down. QA has a count of scenarios that failed again after release. Operations has a different chart built from a different query. Analytics has a monthly trend line that nobody on the call quite trusts yet. Forty minutes later the group has debated four numbers and decided nothing.

When I find myself in that meeting, the tooling is rarely the problem. Each dashboard is probably accurate for the person whose side of the call it records. The trouble is that every group is answering every question, and nobody has agreed which question belongs to whom.

Three Questions, Three Owners

The cleanest fix is a division of labor. Developers improve the system. QA checks that the improvement is real. Analytics measures whether it mattered. It helps me to write the questions down, because they sound similar and are not.

Development owns diagnosis and action. That means finding the problem, tracing the root cause, and designing the fix, along with building enough operational visibility to see what the system is doing. QA owns the claim that the fix holds. Analytics owns the longer view, which is whether the thing everyone agreed was worth fixing got better in a way a customer would notice.

Development and QA share a horizon, which is why they often get blurred together. They are still different jobs. One person believing a fix works and another person proving it works are separate acts.

Each End of the Line Gets Its Own Dashboard

Tactical thinking is about the present. What is broken right now, where are users getting stuck, what changed after the last deployment. Both people on the call have events that lead to a drop, so each team builds a dashboard of what its own person observed. One team owns the caller's side and sees what the caller experienced: the phone set down, the hold, the tunnel. The other team owns the answering side and sees when the line was hung up or timed out.

The tools that serve this are the ones engineers already live in: Application Insights, Kusto queries, operational dashboards, error monitoring, and the QA test suites. A good tactical dashboard is messy, specific, and changes as the problems change.

Strategic thinking asks a slower question. Is quality improving? Are drops declining over a quarter, not an afternoon? Neither team can answer that alone, because each holds one person's half of the conversation. The answer requires both accounts stitched together over time, and that work lives in enterprise analytics platforms, a warehouse like Snowflake, and the executive dashboards built on top of it. Those dashboards should be stable and boring, because a trend line only means something if the definition under it stays put.

The stitching forces a decision the tactical teams never had to make: what counts as a dropped call? A caller who walked away and a line that timed out on the answering side may both be drops on one dashboard and different events on another. Settling that once, in the open, is analytics' job.

The mistake runs in both directions. A monthly stitched trend cannot tell you why one particular call died, so using it to troubleshoot an incident goes nowhere. A count from one side can shift just because that team changed what it records as a drop, so using it for executive reporting produces a number that moves for the wrong reasons.

Why the Fixer Shouldn't Also Be the Judge

The group that makes a change shouldn't be the only group that decides whether the change worked.

I'm a developer, and I know how it feels to ship a fix. The error disappears from the logs, the deployment goes green, and the part of my brain that wants the work to be finished quietly agrees that it is. Every team wants its own work to succeed, and that wanting shapes what it looks for.

So the same fix gets three different questions. Developers say, "We fixed it." QA asks whether they can prove it. Analytics asks, weeks later, whether the trend improved.

Go back to the dropped calls. A team ships a change to how its person handles a lost line. The developers verify what they can see: the error no longer appears, the code behaves as intended, the deployment succeeded. From where they sit, the problem is solved.

QA's job is to doubt that conclusion politely and with evidence. They can recreate the conditions that actually end a call, such as a caller who walks away mid-conversation, a call placed on hold, or a signal that disappears for a minute and returns. They might find that one of those still ends the call badly, that the requirement was understood slightly differently than it was written, or that the fix changed behavior somewhere else. Any of those findings could come from a strong development team. QA sees how the fix behaves in the conditions of a real call, a view the developers' own checks may not include.

I wrote about the handoff side of this in QA Was Always My Last Battle Front, which is about how often I made QA reconstruct what I meant before they could even begin validating it. The separation of roles only works if each side gets what it needs from the other.

The Trend Is Someone Else's Question

Suppose the dropped-call fix passes QA. The tests are green, stakeholders are pleased, and the release is called a success. Analytics is the group that comes back weeks later with both people's accounts stitched together and asks whether the monthly trend moved.

A fix can work exactly as intended and still leave the trend roughly where it was, because the calls it addressed were a small share of the whole. It can also go the other way, where a modest fix has an outsized effect on the trend. Neither result is visible from inside the release, and neither contradicts the tactical dashboards, which were answering a different question. Without that longer view, an organization can't tell work completed from value delivered, and it will count the first as the second.

This is also where the temptation to let engineering self-report gets expensive. I explored a version of it in The Engineering Zoo, where the dashboard improved steadily while the thing the dashboard was supposed to represent stayed exactly where it was.

Analytics and development read the same data differently because they ask from different horizons. The analytic contract is one way I've found to make each side's assumptions visible before a report breaks, since the difference is structural.

Whom the Product Owner Asks

The practical payoff shows up when a product owner needs an answer. Dueling dashboards come from that same missing agreement. The caller's dashboard and the dashboard for the person who answered can show different counts and both be correct. The meeting then turns into an audit of the numbers instead of a decision about the work.

Assigning the questions to owners reduces most of that:

Those two questions sound like one, and they run on different clocks. The first can be answered this week. The second may take months of stitched data before the answer means anything. When someone brings a number to a meeting, the first question to ask becomes which kind of number it is and whether it is being used for the right job.

It also changes the character of a disagreement. If QA says the fix failed and analytics says the trend improved, both statements can be true, because they cover different horizons. The conversation moves from who is right to what each result says about the work.

The Loop, and Where It Can Break

When the three roles work, they form a feedback loop. Developers change how one person on the line handles a lost connection, QA validates the change under real conditions, analytics measures the trend, leadership uses that to decide whether dropped calls deserve the next round of investment, and the decision points developers at the next improvement. Then it starts again.

Each handoff in the loop can fail. If developers skip QA's validation, bad behavior reaches customers while the dashboard looks fine. If analytics lacks a stable definition of a dropped call, leadership gets a trend nobody trusts, and priorities get set by whoever argues loudest. If leadership never acts on what analytics reports, the reports were produced and then ignored.

Leadership acting on the results is the handoff most likely to fail. The other three groups can each do excellent work and the loop still stalls if nobody with authority is willing to be redirected by what the measurement says, a tension I looked at from another angle in Accountability and Authority.

The design does not depend on any group being wise or unbiased. Each group wants its own work to succeed, so the design puts a second set of ears at each stage. That seems a more realistic foundation than hoping everyone is objective. The real question for any team, mine included, is who can hear both sides of the call, and who is willing to say when a fix the team is proud of left the trend where it was.

Explore More

• QA Was Always My Last Battle Front — the handoff problem between developers and independent validation
• The Engineering Zoo: When the Dashboard Gets Better but the Zoo Does Not — what happens when measurement and the work it describes drift apart
• Stop Digging Through Logs. Start Designing for Learning. — making the analytics and development relationship explicit
• Accountability and Authority: Walking the Tightrope — responsibility without the power to act on it