Root cause · Customer quality · October 11, 2026

How the best tech companies find the real cause — and what it means for customer quality

Amazon, Google, Anthropic, Meta and Toyota describe the same discipline in different words. Here is what their public sources actually say, how it maps onto 8D and CAPA, and what I would do with it on a customer-quality team.

By Sina Nezhadian — customer quality and root-cause problem-solving leader, Lean Six Sigma Black Belt. Every company claim below links to that company’s own published material. Where a company has not published its method, I say so.

1. Why symptom-fixing fails customers

A customer complaint arrives as a symptom. The fastest way to make it go away is to fix the symptom: replace the part, reset the setting, close the ticket. The ticket count goes down. The customer, though, does not experience your closure rate. They experience whether the failure comes back.

On BART’s ~$2.2B Fleet of the Future — 775 railcars — I led ~15 8D investigations and tracked ~20 customer non-conformances and corrective actions to verified closure. A fleet makes the cost of symptom-fixing obvious. When hundreds of units share one design, a fix applied to the unit that happened to fail, without removing the cause, is a decision to see the same failure again on the next unit. That is why every fix I closed on that program was verified before closure, and why I measure a quality problem as solved only when the failure has stopped for the customer.

The companies that run the largest software and hardware fleets in the world have reached the same conclusion and written it down. Their vocabulary differs. The discipline is remarkably consistent.

2. How leading companies find the real cause

I used only public primary sources: each company’s own documentation, engineering blog or book. I did not use former-employee accounts or press coverage to describe internal practice.

Amazon: the Correction of Error (COE)

AWS describes the COE as its standard post-incident analysis: establish the sequence of events, find the root causes, measure the impact on customers and the business, and track action items that prevent recurrence. The analysis begins only once the impact has been mitigated, and AWS states plainly that it is not a process for assigning blame (AWS, “Why you should develop a correction of error”). The document has a fixed shape — summary, impact, timeline, metrics, questions about detection, diagnosis, mitigation and prevention, a Five Whys analysis, and action items that each carry an owner and a due date.

A companion walkthrough builds a COE for a fictional incident. Its “Five Whys” needs six whys to reach the cause, and each incident question asks how the team could cut the time to detect, diagnose or mitigate in half (AWS, “Creating a correction of errors document”). AWS also calls the COE a closed-loop mechanism used for any significant event, even one customers never saw, and feeds COE learnings into Operational Readiness Review checklists so other teams can avoid the same risk (AWS Well-Architected, Operational Readiness Reviews). Its operational-excellence guidance asks teams to review every customer-impacting incident and track every resulting action to delivery (AWS Well-Architected, OPS11-BP02).

Google: blameless postmortems

Google’s SRE book sets postmortem triggers before any incident happens: user-visible downtime or degradation past a threshold, any data loss, on-call intervention such as a rollback, resolution time beyond a limit, or a monitoring failure that left a human to discover the problem. Any stakeholder can also ask for one. The postmortem is blameless — it looks for contributing causes and assumes people acted in good faith on the information they had, because blame suppresses reporting. Senior engineers review each draft for complete data, a real root cause and a sound action plan before it is shared, and the book is blunt that an unreviewed postmortem has little value. The chapter quotes one engineer’s summary: “The cost of failure is education.” (Google SRE book, ch. 15, “Postmortem Culture”).

Anthropic: postmortems that admit what detection missed

In September 2025 Anthropic published a postmortem of three infrastructure bugs that degraded some Claude responses: a routing error that sent some requests to servers configured for a different context length, made worse by a load-balancing change; a TPU misconfiguration that produced stray characters; and a compiler bug in approximate top-k sampling. The most useful part for a quality engineer is the section on why detection was slow — evaluations missed the degradation, the model often recovered from isolated mistakes, privacy controls limited engineers’ view of user conversations, and the bugs showed different symptoms on different platforms, so reports looked random. The fixes included more sensitive evaluations, continuous quality evaluation on production systems and faster, privacy-preserving debugging of user feedback (Anthropic, “A postmortem of three recent issues”).

An April 2026 update on Claude Code quality follows the same pattern: three separate changes each hit a different slice of traffic on a different schedule, early reports looked like normal variation, and one defect passed code review, automated tests and internal use. The corrective actions were systemic — per-model evaluations for prompt changes, soak periods and gradual rollouts (Anthropic, “An update on recent Claude Code quality reports”). In both cases, customer reports were the signal before the internal metrics were.

Meta: a public outage analysis, a private review process

Meta’s write-up of its October 4, 2021 outage traces the failure to a maintenance command, meant to assess backbone capacity, that disconnected the backbone because a bug in an audit tool failed to stop it; DNS servers then withdrew their route advertisements. Recovery was slowed by the loss of out-of-band access and by security hardening, and the post credits earlier “storm” drills with helping (Meta Engineering, “More details about the October 4 outage”). Meta’s incident (“SEV”) review process is often discussed, but the only Meta-authored public artifact I found that names it is a 2016 conference talk listing with no published content (SREcon16 Europe, “Facebook’s SEV Process”). I won’t describe a process Meta has not published.

Toyota, A3 and the origin of 8D

Toyota describes its production system as resting on jidoka — stopping the line the moment something is wrong — and just-in-time, with kaizen as daily improvement by everyone (Toyota, Toyota Production System). The Lean Enterprise Institute’s lexicon describes the 5 Whys as asking why until the root cause appears, using Taiichi Ohno’s example of a stopped machine traced back to a missing strainer (LEI, 5 Whys), and the A3 as one sheet holding the problem, its analysis, the countermeasures and the evidence the issue was resolved, with a named owner (LEI, A3 Report). 8D itself comes from Ford’s Team Oriented Problem Solving manual, first published in 1987, with the D0 emergency-response step added in the 1990s (AllAboutLean, history of 8D — a secondary source; I found no public Ford original).

Apple and NVIDIA: not published

I looked for public primary sources describing how Apple and NVIDIA run root-cause analysis on customer or product-quality failures. Apple’s supply-chain pages describe corrective action for supplier compliance — labor, safety and environment — not product failure analysis (Apple, Supply Chain). I found no NVIDIA primary source on the subject. Neither company has published its internal method, so I make no claims about either.

3. The shared pattern

Read side by side, the published methods agree on seven things:

Triggers are decided in advance. Google defines them before any incident; AWS runs a COE even when customers saw nothing.
Contain first, analyze second. AWS starts the COE after mitigation; 8D separates containment from permanent correction.
Blameless, systems-level causes. Both AWS and Google say it outright: the goal is the cause, not the culprit.
Ask why past the first answer. The 5 Whys, and an AWS example that needs six.
Owned, dated, tracked actions. Every AWS action item has an owner and a due date; Google reviews the action plan before publishing.
Detection is part of the root cause. AWS asks how to halve detection time; Google treats a monitoring miss as a trigger; Anthropic’s write-ups spend as much space on why the problem escaped as on the bug itself.
Learning has to travel. AWS turns COEs into readiness checklists for other teams; Google circulates postmortems widely; Toyota uses the A3 to coach.

4. What this means for 8D and CAPA in hardware fleets

None of this is foreign to a hardware quality engineer — it is 8D with better habits around detection and sharing. Mapped discipline by discipline:

D0/D3 — emergency response and containment match mitigation before analysis. Protect the customer first, then investigate.
D2 — problem description matches the impact and timeline sections. Write it from the customer’s side: what they saw, when, how often, and on which configuration.
D4 — root cause should carry two answers, as the tech postmortems do: why the failure occurred, and why it escaped detection. In 8D language, the occurrence root cause and the escape point.
D5/D6 — permanent corrective action and verification match owned, dated action items — with the verification method decided before the fix is chosen.
D7 — prevent recurrence matches AWS’s readiness checklists and Google’s shared repository: change the design rule, the test or the process, not just the unit.
D8 — recognize the team is where blamelessness lives. People report what they can safely report.

AI-hardware fleets sharpen two of these. First, symptoms vary by configuration — Anthropic’s bugs looked random until they were grouped by platform — so the problem description has to slice failures by firmware, build, site and workload before anyone argues about cause. Second, verification has to happen at fleet scale. I have not worked in a data center; what transfers from a 775-car rail fleet is the habit of treating one repaired unit as a sample, not as proof, and closing only when the failure has stopped across the population.

5. What I would do in my first 90 days on a customer-quality team

Days 1–30: see the problem from the customer’s side

Read the last two quarters of customer complaints, non-conformances and returns, and rank recurring failure modes by customer impact rather than by count. Sit in the customer quality reviews. Agree, in writing, the triggers that make an issue a formal 8D — borrowed directly from Google’s idea of deciding before the incident, not during it.

Days 31–60: find the cause the system is hiding

Take the two or three recurring failures that hurt customers most and run them as full 8Ds: containment first, whys past the first answer, and an explicit escape-point analysis. Put one template in place, with an owner and a date on every action, and add a review before anything is shared — the step Google says gives a postmortem its value.

Days 61–90: close only when the evidence holds

Define the verification for each corrective action before it is approved, then measure recurrence in the field. Turn what we learn into checklist questions other product lines can use. Report monthly to leadership with one view of open issues, risks and decisions — the kind of program report senior leadership relied on as its main view of the $4.9B LAX Automated People Mover when I wrote it.

The best companies differ in vocabulary, not in discipline. Find the problem the customer is living with, find the cause the system is hiding, and close it only when the evidence holds.