The Complete Guide to Test Strategy: Decision Criteria Instead of the Pyramid Debate

• Introduction
• 1. Why Tests Exist: Regression Defense or Design Pressure
• 2. Pyramid and Trophy — the Real Axes of the Argument
  • 2-1. What the two sources actually say
  • 2-2. Both camps say the ratio debate is not the point
  • 2-3. The four axes it actually turns on
  • 2-4. What to check before you copy a model
• 3. Nobody Ever Agreed What "Unit" Means
  • 3-1. The spectrum of definitions
  • 3-2. Name the properties, not the category
• 4. Seeing It as a Cost Function: Writing, Running, Maintaining, False Alarms
• 5. What to Replace With a Mock
  • 5-1. The two positions
  • 5-2. The axis, and a working rule
  • 5-3. The doubles you can reach for instead
• 6. What to Do With Coverage Numbers
• 7. Where Defects Actually Come From — Setting the Ratio From Data
  • 7-1. Four weeks of records is enough
  • 7-2. Read it alongside delivery metrics
  • 7-3. Production is a verification layer too
• 8. Operational Rules for Slow and Flaky Tests
• 9. Writing Your Team's Test Strategy on One Page
• Quiz: Check Your Understanding
• Closing
• References
• Further reading

Introduction

This blog already carries several Korean complete guides on testing. The Complete Guide to Software Testing Strategies covers what kinds of tests exist and how to use them, Property-Based Testing in Practice covers one specific technique, and Designing Verification — Tests as Grounds for Trust treats verification as an individual skill. All three are about which tests exist and how to write them.

This post is different. It is about how a team decides its own ratio. It does not pick between the pyramid and the trophy; it decomposes the argument into the axes it actually turns on. Once you know the axes you can answer "which side is our stack on?" yourself, and that lasts longer than copying somebody else's conclusion.

The conclusion up front: the ratio is not an answer, it is a result. Measure where defects come from, how expensive it is to stand up an integration environment, and how often your E2E layer cries wolf, and the ratio derives itself.

1. Why Tests Exist: Regression Defense or Design Pressure

When a testing argument goes in circles, it is usually because the participants assume different purposes. Tests exist for roughly four reasons, and a different purpose implies a different shape of good test.

• Regression defense: tells you whether something that worked yesterday broke today. For this purpose a test must attach to externally observable behavior, not to the implementation. A test you have to edit alongside every refactoring is not a regression asset, it is a liability.
• Design pressure: code that is hard to test is usually tightly coupled code. Writing the test first shapes the interface from the caller's point of view. Here fine-grained unit tests help, and some coupling to the implementation is accepted as the price.
• Specification and documentation: records what the code guarantees in executable form. Being read is the point, so naming and placement matter.
• Release confidence: makes it possible to ship without a human sign-off. Integration, contract and E2E carry more weight here.

The first conflict is already visible. Prioritizing regression defense pulls tests away from the implementation; prioritizing design pressure pushes them toward it. When the two purposes collide in one codebase, a single question resolves it: "is it acceptable for this test to be edited during a refactoring?" For a regression test, no. For a design-pressure test, yes. Naming that distinction in a team halves the amount of arguing in review.

2. Pyramid and Trophy — the Real Axes of the Argument

2-1. What the two sources actually say

The test pyramid is most often cited via Martin Fowler. The core is one sentence: "Write lots of small and fast unit tests. Write some more coarse-grained tests and very few high-level tests that test your application from end to end." The higher you go, the slower and more expensive, so keep the count down.

What deserves attention is that the same article admits the limits of its own model. It says "the concept of the test pyramid falls a little short if you take a closer look," notes of the middle layer that "service test is a term that is hard to grasp," and adds that in modern front-end setups "UI tests don't have to be on the highest level." Of the top layer it says E2E tests are "notoriously flaky and often fail for unexpected and unforeseeable reasons" and "require a lot of maintenance and run pretty slowly."

The testing trophy is Kent C. Dodds' formulation. It splits into static analysis, unit, integration and E2E, and makes integration the largest section. The justification is confidence per unit of investment, in his words: "The more your tests resemble the way your software is used, the more confidence they can give you."

2-2. Both camps say the ratio debate is not the point

Crucially, the trophy side does not treat this as a search for the right answer either. In the same article Dodds notes that definitions of unit test diverge to the tune of "24 different definitions of unit test," and writes "Any attempt to come to a single definition for all these terms is a futile endeavor." He then quotes Justin Searls: "People love debating what percentage of which type of tests to write, but it's a distraction."

Fowler says something equivalent: "If you ask three different people what 'unit' means in the context of unit tests, you'll probably receive four different, slightly nuanced answers."

So the flagship text of each camp says the ratio debate is not the substance. The reason the argument nevertheless never ends in practice is that several different conditions hide behind that one number.

2-3. The four axes it actually turns on

Axis 1  Where you draw the boundary of a "unit"
        one function = a unit    → unit count explodes, looks like a pyramid
        one module  = a unit     → the same tests get filed as "integration",
                                   looks like a trophy

Axis 2  What an integration environment costs to stand up
        30 seconds in a container → raising the integration share is rational
        a dedicated staging env   → raising it destroys the feedback loop

Axis 3  The false-alarm rate of your E2E layer
        under 1%                  → E2E can be used as grounds for confidence
        over 10%                  → noise, not signal; shrink the layer itself

Axis 4  What the tests are primarily for
        regression defense        → behavior-centric, larger units, few mocks
        design feedback           → close to the implementation, small units,
                                    mocks allowed

Plug your team's real values into the four axes and the ratio falls out without an argument. If the container-based integration environment comes up in 30 seconds, the E2E false-alarm rate is 8%, and the team's main purpose is regression defense, the result looks like a trophy. If domain logic is thick and the integration environment is expensive, it looks like a pyramid. Neither model is wrong; the two articles simply assumed codebases with different axis values.

2-4. What to check before you copy a model

Dodds himself notes that the trophy targets an individual codebase and may not transfer directly to microservice or serverless setups. Whichever model you adopt, carry the context its author assumed along with it. Copying only the picture means setting your ratio from somebody else's axis values.

3. Nobody Ever Agreed What "Unit" Means

3-1. The spectrum of definitions

Half the ratio argument is a vocabulary problem. The same test gets filed as a unit test by one team and an integration test by another.

narrow ←──────────────────────────────────────────────→ wide

one function     one class       one module        one process
all collaborators some mocked    internals real    only external doubled
mocked                           DB in-memory      DB in a container

As both Fowler and Dodds point out, there is no standard boundary anywhere on this spectrum. So a goal of "70% unit tests" means nothing until the boundary is fixed. Moving the boundary alone swings the number for the same codebase from 20% to 80%.

3-2. Name the properties, not the category

The workaround that holds up in practice is to abandon the category names and describe the properties instead.

• Isolation level: how much real code does this test actually execute.
• Runtime: milliseconds, seconds, or minutes.
• Environment requirement: does it finish in-process, need another process, or need the network.
• Determinism: does the same input always produce the same result.

Once those four values are fixed, the label is unnecessary. Structure your CI stages by those values rather than by names. "Finishes in three seconds with no external dependency" runs on every commit, "needs a container" runs before merge, and "needs staging" runs in the deployment pipeline.

4. Seeing It as a Cost Function: Writing, Running, Maintaining, False Alarms

Treating a test not as an asset but as a contract with both a cost and a return makes decisions easier. The cost has four parts.

• Writing cost: paid once. The most visible part, and usually the smallest share of total cost.
• Running cost: paid on every commit. In a pipeline that runs 50 times a day, one test getting two seconds slower burns hours per year. The bigger loss is developers losing context while they wait.
• Maintenance cost: paid every time the code changes. A test bolted to implementation detail sends an invoice with every refactoring. This item frequently dominates the total.
• False-alarm cost: the time spent deciding whether a failure is real, plus the cost of lost trust. That last part is the most expensive. Once people habitually re-run a red pipeline, that suite has stopped being an asset.

There is a single return item: the expected value of defects prevented — the probability of a defect that would have reached production, times its cost.

Put numbers on it. Say an E2E test takes four hours to write, 40 seconds to run, and two hours of fixing twice a quarter. In a pipeline that runs 30 times a day, execution alone burns about 68 hours a year, and with maintenance the annual total passes 80 hours. Five or six unit tests covering the same logic often total less than half that. Compare only writing cost and E2E looks cheap; compare total cost and it usually is not.

That yields a practical rule. If two layers catch the same defect, delete the expensive one. If unit tests already cover every logic branch an E2E exercises, that E2E is pure cost. Leave in E2E only what lower layers cannot see in principle: wiring between layers such as routing, authentication, serialization and configuration.

The opposite mistake is just as common. Forcing a check down into a unit test when it cannot live there means that the moment a mock diverges from reality, everything is green and production is broken. The criterion is "is this defect visible at this layer in principle," not "is this layer cheap."

5. What to Replace With a Mock

How far to take mocking is one of the oldest arguments in this field, and it is still unsettled.

5-1. The two positions

• The broad-mocking side: replacing collaborators with doubles makes tests fast and narrows the cause of failure. The name of the failing test alone tells you which unit broke. You can also design against a collaborator that does not exist yet.
• The narrow-mocking side: a mock verifies your assumption, not the other side's actual behavior. When the other side changes, the mock stays quiet, which manufactures the all-green-but-production-broken situation. Mocks that assert call order and arguments also bolt themselves to the implementation and block refactoring.

5-2. The axis, and a working rule

The axis of the argument is whether the thing being replaced is under your control. The following rule generally works well.

Replace with a double            Use the real thing
─────────────────────────       ─────────────────────────
external systems (payments,     collaborators inside the same codebase
mail)                           pure computation and value objects
non-determinism (clock,         stores you can start in a container
random)                         stores replaceable by an in-memory version
failures hard to reproduce
slow I/O (seconds or more)

Two further rules matter. First, do not hand-write mocks for interfaces you do not own. Wrap the external SDK in a thin adapter and double the adapter instead. Freezing an imagined version of an external SDK into a mock means nothing fails when that imagination turns out wrong. Second, if you mocked something, confirm at another layer that the mock matches reality. Contract tests or a small number of real-communication tests play that role.

5-3. The doubles you can reach for instead

Lumping everything under "mock" makes the argument longer than it needs to be. Doubles come in kinds with different maintenance costs.

• Stub: returns a fixed value and nothing more. It does not verify how it was called, so it is robust to implementation change. In most cases this is enough.
• Fake: a lightweight implementation satisfying the same contract as the real thing — an in-memory store is the classic example. Expensive to build, but shared across many tests, so total cost drops, and the contract survives as code you can verify.
• Strict mock: asserts call counts and arguments. Use it only when "did the side effect actually happen" is the thing under test. Otherwise it attaches to the implementation and works against refactoring.
• A real container: for a store or a message broker that starts in tens of seconds, using the real thing is simply more accurate.

"Which double is the minimum this check needs?" is a better question than "should we mock?" Most of the argument starts from the habit of reaching for a strict mock by default.

6. What to Do With Coverage Numbers

Coverage targets are another area without consensus. Lay out the axes first.

• The pro-target side: without a number, untested regions grow quietly. A floor creates a minimum of discipline.
• The anti-target side: coverage counts lines executed, not behavior verified. The moment the target becomes a metric, workarounds appear — tests with no assertions, getter tests, generated code counted in.

Both are right. The realistic move is to keep the number but change how you use it.

• A delta gate instead of an absolute target: rather than "80% overall," look at the coverage of lines newly added by this change. It manages the quality of new code without touching legacy debt, and the incentive to game it is far weaker.
• Use low numbers only as an investigation signal: a low-coverage module is a subject for inquiry, not a penalty. It is often a signal that the structure is hard to test.
• Do not use a high number as grounds for confidence: 90% does not mean "10% is risky," it means "90% was executed." Execution and verification are different things.
• Look at branch coverage too: line coverage alone hides a conditional where only one side was exercised.

The question coverage cannot answer is "does this test actually catch defects?" There is a way to measure that directly: automatically plant small mutations in the code and see whether the tests catch them. A large number of surviving mutations means a lot of code is executed but not verified. It is expensive to run continuously across a whole suite, but running it once over a core domain module exposes the regions where assertions are missing.

7. Where Defects Actually Come From — Setting the Ratio From Data

7-1. Four weeks of records is enough

The most reliable way to set a ratio is a record, not an argument. Attach just two fields to production defects and rollbacks and the direction becomes visible within four weeks.

• Which layer should have caught it: unit / integration / contract / E2E / none of them (a design or requirements problem).
• Why it was missed: no test existed / a test existed but the mock differed from reality / environment difference / data difference / concurrency and timing.

The distribution across those two fields is your investment plan. If half say "integration should have caught it," raising integration is correct. If half say "the mock differed from reality," reducing mocks or adding contract tests is correct. If half say none of them, adding tests will not reduce defects. That is when investment should move toward requirements definition and design review.

7-2. Read it alongside delivery metrics

DORA defines four metrics: deployment frequency, change lead time (the time for a change to go from "committed to version control" to "deployed in production"), change fail rate ("The ratio of deployments that require immediate intervention following a deployment"), and failed deployment recovery time.

Judge a test strategy by the combination of change fail rate and lead time. If you added tests and the change fail rate did not move, you invested in the wrong layer; if the fail rate dropped but lead time ballooned, cost exceeded return. DORA says these two are not a trade-off: "DORA's research has repeatedly demonstrated that speed and stability are not tradeoffs." If one of them got worse, treat it as a design problem rather than an unavoidable compromise.

7-3. Production is a verification layer too

You cannot catch every defect before release. The Google SRE Workbook defines canarying as "a partial and time-limited deployment of a change in a service and its evaluation," with the remaining fleet as the control. One calculation from it is especially useful: a 5% canary population with a 20% error rate yields only a 1% overall error rate. A canary is therefore not a device for hiding defects from users but a device for keeping the exposure inside the error budget.

The same document warns that change over time is one of the largest confounders in observed metrics, so compare against a concurrently running control rather than a before-and-after snapshot. Test layers and release layers are complements, not substitutes, and "add more E2E" versus "make the canary sharper" are competing claims on the same budget. The error-budget framing is covered in SLI/SLO/Error Budget.

8. Operational Rules for Slow and Flaky Tests

These operational rules affect real quality more than any strategy document.

• Give every flaky test an owner and a deadline: the moment you wave one through with a re-run, trust in the whole suite drops. Collect failure rates automatically, open an issue when a threshold is crossed, and forbid quarantining without an assignee and a date.
• Quarantine is scheduled deletion: attach an expiry date to every quarantined test. If it is not fixed by then, delete it. A test that is permanently disabled inflates coverage numbers and protects nothing.
• Set a time budget per stage: put a ceiling such as "5 minutes at commit, 15 minutes before merge," and when it is exceeded, decide what to remove before adding anything. Without a budget a pipeline only grows.
• Review the slowest 20 regularly: the top of the runtime list is usually dominated by a handful of tests. Fixing only those often cuts total time substantially.
• Treat failure messages as an asset: a test that leaves only "expected true but got false" does not help when it fails. Recording the input and state at failure time cuts debugging time.
• Test code is review material too: it is the part most often skipped in review, yet most of the maintenance cost originates there.
• Leave no shared state between tests: order-dependent tests break randomly the moment parallel execution is enabled. Turning on randomized test ordering surfaces that dependency early.
• Inject time and randomness: if code reads the current clock or a random source directly, that test will eventually break. Failures that erupt at month end, on a leap day, or at a time-zone transition mostly start here.

Fowler calling E2E "notoriously flaky" describes a structural property of that layer, not a tooling problem. Treat E2E as a layer to keep narrow and manage tightly, not one to grow.

9. Writing Your Team's Test Strategy on One Page

A strategy is an agreement, not a document. Past one page, nobody reads it.

1. What we are buying with tests
   (priority among regression defense / design feedback / release confidence)

2. Layer definitions and where they run
   fast layer:        under 3s, no external dependency  → every commit
   integration layer: needs a container, under 30s      → before merge
   E2E layer:         needs staging, under 10 minutes   → deploy pipeline

3. Where we mock and where we do not
   doubled: external payments and mail, clock, randomness
   real:    our own stores (container), internal modules

4. Coverage rule
   only a delta-coverage floor is a gate; no overall target

5. Time budget
   5 min at commit / 15 min before merge; over budget means remove before adding

6. Flaky handling
   automatic issue above a failure-rate threshold;
   quarantine requires an assignee and an expiry date

7. Review cadence
   once a quarter; update items 2 and 3 from defect distribution data

The value of this document is less in its content than in its revision history. When the defect distribution shifts, the ratio must shift too, and the reasoning has to be on record for the next person to understand it. Read it alongside The Complete Guide to Software Testing Strategies.

Quiz: Check Your Understanding

Closing

There is no answer to "is the pyramid or the trophy correct." The authors of both models say so themselves, and the line Dodds quotes is blunt: the percentage debate is a distraction.

The questions that do have answers are these. Where is the unit boundary in our team? What does standing up an integration environment cost? What is our E2E false-alarm rate? Which layer should have caught last quarter's defects? Answer those four with numbers and the ratio comes out as a calculation, not a conclusion.

And that calculation is never done once. When the stack changes, the tooling gets faster, or the defect distribution moves, the ratio must move with it. The quality of a test strategy depends on its revision cadence, not on the precision of its ratio.

References

• The Practical Test Pyramid — Martin Fowler — quoted for the core pyramid sentence, the article's own acknowledgement of the model's limits, the observation that people disagree on what a unit is, and its assessment of E2E flakiness and maintenance cost. Checked 2026-08-15.
• The Testing Trophy and Testing Classifications — Kent C. Dodds — quoted for the four-layer structure and integration-heavy argument, the confidence-through-resemblance line, the futility of a single unit definition, the Justin Searls quote, and the individual-codebase premise. Checked 2026-08-15.
• DORA — Four keys metrics — quoted for the definitions of change lead time and change fail rate, and the finding that speed and stability are not trade-offs. Checked 2026-08-15.
• Google SRE Workbook — Canarying Releases — quoted for the definition of canarying, the calculation that a 5% population with a 20% error rate yields 1% overall, and the recommendation to use a concurrent control rather than a before-and-after comparison. Checked 2026-08-15.
• The four axes, the four-part cost function, the two defect-record fields, and the one-page strategy template do not appear in the sources above — they are procedures assembled in this post.

Further reading

• Related post on this blog: The Complete Guide to Software Testing Strategies
• Related post on this blog: Property-Based Testing in Practice
• Related post on this blog: Designing Verification — Tests as Grounds for Trust
• Related post on this blog: Reliability Engineering with SLI/SLO/Error Budget
• Related tool: SLO Error Budget Calculator

Complete Guide Series

• Previous: The Complete Guide to Concurrency: The Order in Which You Shrink Shared State
• Next: The Complete Guide to Code Review: Designing Review as a Process