Reading Time: 8 min
Why We Design Failure Before Features
What production failures teach us about building systems that can survive reality
There is a moment in almost every software project when the conversation becomes exciting.
The team has understood the problem. The product vision is clear. The feature list is taking shape. Someone starts talking about the user experience, another person is already thinking about the technology stack, and suddenly everyone wants to discuss what the system should do.
That’s usually when we ask a different question at Prologic.
What happens when it doesn’t?
It sounds like a pessimistic way to begin a project.
It isn’t.
When you’re building software that sits inside a healthcare workflow, processes a financial transaction, coordinates a retail operation, or supports an AI-driven decision, failure isn’t an edge case waiting somewhere at the bottom of the requirements document.
Failure is part of the operating environment.
Networks fail. APIs timeout. Services become unavailable. Data arrives late. Users do unexpected things. Third-party providers change their behaviour. Models return low-confidence results. Traffic suddenly exceeds what we planned for.
The interesting engineering question isn’t whether these things will happen.
It’s what the system does when they do.
The Feature Works. Now What?
A feature demo is usually performed under ideal conditions.
The user follows the expected journey. The API responds. The database is available. The payment succeeds. The AI model returns an answer. Everything moves from left to right exactly as the architecture diagram suggested.
And that’s useful.
A successful happy path tells us that the system can work.
It doesn’t tell us whether the system can survive.
Production has a habit of testing the second question much more aggressively.
Imagine a healthcare application where a consultation has been completed, but the network drops before the prescription is transmitted. Has the consultation been recorded? Can the clinician safely retry the action? Could the patient receive the prescription twice?
Or consider an intelligent commerce platform where an order is placed while inventory information is temporarily stale. The payment succeeds, but the product is no longer available. Which system owns the truth? What happens to the customer? When does the refund begin?
These aren’t theoretical engineering puzzles.
They are the situations that turn a seemingly simple feature into a production problem.
Failure Changes How We Design the Feature
This is why we prefer to think about failure before we finalize the feature.
Not after development.
Not after QA finds an unexpected scenario.
And certainly not after a customer discovers it in production. Take something as ordinary as a login flow. The obvious requirement is straightforward: a user enters credentials and gets access.
But an engineer thinking about failure immediately starts asking different questions.
What if the authentication provider is unavailable?
What if the request succeeds but the response is lost?
What if the session expires halfway through an important workflow?
What if the user’s device reconnects after being offline?
What if two requests arrive simultaneously?
What if the user’s permissions change while the session is active?
Suddenly, authentication isn’t just a screen and an API.
It is a system of states and transitions. And that way of thinking changes the architecture before a line of code is written.
The Happy Path Is Only One Path
One of the simplest techniques we use during architecture and engineering discussions is to deliberately separate the happy path from the failure paths.
The happy path answers:
“What should happen when everything works?”
Failure analysis asks:
“What should happen when something doesn’t?”
That second question forces us to think about states that aren’t visible in the product demo.
Pending, Retrying, Partially completed,
Timed out, Rejected, Unavailable, Unknown.
These states are where many production bugs originate.
A payment that fails is easy to understand.
A payment where the gateway says “timeout” but the bank has actually charged the customer is much harder.
The system isn’t dealing with a simple failure anymore.
It’s dealing with uncertainty
– And uncertainty requires architecture.
Some Failures Should Be Invisible
Good failure design doesn’t necessarily mean showing the user an error message every time something goes wrong.
Sometimes the best failure handling is the failure the user never notices.
A service temporarily becomes unavailable, so another mechanism retries the request.
A downstream system responds slowly, so the workflow moves into a controlled pending state rather than failing completely.
A message cannot be delivered immediately, so it is queued and processed when the dependency recovers.
The user shouldn’t have to understand the architecture to benefit from it.
That’s an important distinction;
Resilience isn’t about making systems look sophisticated
– It’s about making them behave predictably when the environment isn’t.
But Some Failures Must Be Visible
The opposite is equally important.
There are situations where hiding failure creates more risk than exposing it.
Consider healthcare.
If a clinical record hasn’t been successfully synchronized, telling a clinician that everything is fine could be dangerous. The system needs to make the uncertainty visible so that the clinician can make an informed decision.
The same principle applies to financial transactions, identity verification, inventory, and AI-assisted decisions.
The engineering challenge isn’t simply to “handle errors.”
It is to determine which failures can be absorbed, which can be retried, and which must be surfaced to a human being.
That is a much more meaningful design conversation.
AI Makes This Even More Important
AI-native systems introduce another category of failure that traditional software teams didn’t have to deal with in quite the same way.
Software traditionally expects deterministic behaviour.
Given the same input, the system generally produces the same output.
AI systems can produce outputs that are probabilistic, uncertain, incomplete, or simply wrong.
That means an AI feature cannot be designed only around the question:
“What should the model produce?”
We also have to ask:
What happens when confidence is low?
What happens when the model cannot answer?
What happens when the answer conflicts with business rules?
What happens when the model produces an unexpected output?
When should the system ask for human intervention?
These aren’t AI model questions alone.
They’re system architecture questions.
And they need to be answered before the AI capability becomes part of a production workflow.
Failure Is Also a Business Decision
There’s another reason we bring failure into the design conversation early.
Not every failure deserves the same engineering response.
A recommendation engine being unavailable for thirty seconds is very different from a payment being processed twice.
A delayed marketing notification is very different from a clinical alert not reaching a care provider.
The appropriate response depends on business impact.
That means engineers need to understand the business consequence of failure- not just the technical error.
We need to know what is recoverable. What is reversible. What is tolerable.
What requires immediate intervention. What must never happen twice.
Once those questions are understood, architectural decisions become much clearer.
Designing Failure Creates Better Features
There is an unexpected benefit to this approach.
Thinking about failure often improves the feature itself.
When engineers understand what can go wrong, they are forced to define states more precisely. Product teams become clearer about what the user should see. Business stakeholders become more explicit about priorities. Architects discover hidden dependencies earlier.
In other words, failure analysis doesn’t slow feature development.
It removes ambiguity from it.
And ambiguity is usually much more expensive when discovered after the system has gone live.
The Prologic Perspective
At Prologic, we don’t consider failure analysis a separate exercise that happens after architecture.
It is part of architecture.
When we’re designing a system, we want to understand not only how information moves when everything works, but also what happens when a dependency disappears, a response arrives late, data becomes inconsistent, or a decision cannot be made with sufficient confidence.
That thinking becomes even more important when we’re building connected healthcare and intelligent commerce platforms, where multiple systems and external dependencies are involved.
The more connected the system becomes, the more ways it can fail.
But connectivity also gives us more opportunities to design graceful recovery.
That is where engineering discipline matters.
Build for the Day Things Go Wrong
I’ve come to believe that one of the best indicators of engineering maturity isn’t how impressive a system looks when everything is working.
It’s how calmly it behaves when something isn’t.
Can it recover?
Can it isolate the failure?
Can it protect data?
Can it prevent duplicate actions?
Can it preserve the user’s state?
Can it tell a human when intervention is genuinely required?
Can the team understand what happened afterward?
Those questions don’t belong in the post-mortem alone. They belong in the design meeting.
Because by the time a failure reaches production, the cheapest opportunity to design for it has already passed.
Closing Thoughts
Every product team wants to talk about what the system will do.
That’s natural.
Features are visible. They are easy to demonstrate, easy to put on roadmaps, and easy to celebrate when they ship.
Failure is different.
Nobody puts “gracefully handle a downstream timeout” on a launch announcement. But those decisions often determine whether a product earns trust after launch.
A system isn’t production-ready simply because its features work. It is production-ready when the team has thought carefully about what happens when those features don’t.
That’s why we design failure before features. Not because we expect the system to fail.
Because we expect reality to.
FAQs
Failure-First Engineering is an architectural approach that designs systems around predictable failures instead of assuming everything will always work. It focuses on resilience, graceful degradation, recovery, and maintaining essential functionality when components become unavailable.
Every production system eventually experiences failures, whether due to infrastructure, networks, third-party services, or user actions. Designing for these scenarios early results in more reliable software and reduces costly production incidents.
Graceful degradation allows an application to continue delivering its core functionality even when certain features or services are unavailable. Instead of failing completely, the system provides a reduced but usable experience.
AI systems operate with uncertainty. Models may return low-confidence responses or lack sufficient context. Failure-First Engineering ensures there are fallback mechanisms, human review processes, or alternative workflows instead of blindly accepting unreliable outputs.
It may require additional architectural planning, but it significantly reduces operational risk, production downtime, and maintenance costs over the life of the product. Building resilience early is usually less expensive than retrofitting it later.
Common practices include retries, circuit breakers, health checks, idempotent operations, observability, fallback workflows, redundancy, and clear human escalation paths. Together, these techniques help systems recover predictably and maintain user trust.
Further Reading
Previous Reading
Every AI Project Has One Constraint Nobody Mentions
Why Context, Not Models, Determines Success Executive Summary Every AI project begins with excitement. The conversation quickly turns to mod...
Related Reading
Retail AI Begins After Checkout
Why the Greatest Opportunities in Commerce Start When the Payment Is Complete Executive Summary For decades, retailers have invested heavily...
