The New Reality of Incident Response

On October 20, 2025, AWS’s us-east-1 region experienced a cascading failure lasting more than 15 hours. A DNS issue in DynamoDB triggered cascades affecting over 1,000 services globally. Banking applications went dark. Ring doorbells stopped recording. The internet collectively held its breath.

The outage revealed two entirely different types of engineering teams.

The first type barely noticed. Multi-region architectures tested quarterly meant automated failover executed within minutes. Users experienced slight latency increases for seconds. Engineers returned to planned work.

The second type entered crisis mode. All-hands emergency calls. Revenue stopped. Engineers discovered their disaster recovery plans, written months ago and never tested, didn’t account for the failure modes they faced. Routine failover became extended outage.

The difference wasn’t luck. It was preparation.

This outage tested two capabilities every organisation needs: disaster recovery planning (architectural decisions made months ago determining what’s possible when infrastructure fails) and incident response execution (diagnosing problems under pressure, making good decisions quickly, restoring service systematically).

What makes this relevant now is how AI intersects with both. AI services behaved unpredictably during infrastructure stress, exhibiting failure modes traditional monitoring missed. Yet ironically, teams that recovered fastest used AI tools to accelerate response: correlating patterns across millions of log lines, identifying root causes faster, validating recovery procedures.

This article explores both sides. How AI creates new failure modes your plans need to address. How AI tools enhance every response phase when applied thoughtfully. How to build capabilities your team needs for inevitable failures.

Perfect reliability is impossible. Effective disaster recovery and incident response are absolutely achievable.

The Foundations That Haven’t Changed

Before discussing AI or new tools, we need to establish what good disaster recovery and incident response have always required. These fundamentals haven’t changed. They remain the foundation that everything else builds upon.

Planning for Failure

Good disaster recovery starts with accepting that failures will occur. Always. Without exception. The question isn’t whether systems fail, but how gracefully and how quickly you recover.

Core concepts define what recovery means:

• Recovery Time Objective (RTO): How long you can tolerate being down
• Recovery Point Objective (RPO): How much data you can afford to lose

These are business decisions with revenue implications, not just technical metrics. The numbers sound good until you calculate actual downtime:

A system “available” 95% of the time is down for more than a full day every month.

Eliminating single points of failure requires systematic thinking. Database without replica? Single point of failure. Region without failover target? Single point of failure. Monitoring hosted in infrastructure it monitors? Single point of failure.

Multi-region deployment provides geographic distribution through patterns like active-passive (standby ready), active-active (continuous load distribution), backup and restore (slower but simpler), or pilot light (minimal infrastructure that scales rapidly). Each trades off cost, complexity, and recovery speed differently.

Key decisions happen before disasters. Geographic distribution? Graceful degradation? Independent monitoring? Accessible recovery procedures? Documentation and automation determine whether plans work under pressure. Automated procedures execute consistently. Manual procedures executed by stressed humans fail.

Testing reveals whether plans work. Untested plans are fantasies. Teams testing quarterly discover gaps whilst stakes are low. Understanding dependencies matters. Your DR is only as strong as your weakest dependency.

The hard truth: DR is expensive and inconvenient. But the alternative is discovering plans don’t work whilst users are affected and revenue stops.

Executing Under Pressure

Disaster recovery architecture determines what’s possible during an incident. Incident response capability determines how effectively you execute those possibilities. Both matter equally.

The incident response lifecycle has six phases:

1. Detection: Knowing problems quickly
2. Assessment: Understanding scope and impact
3. Investigation: Finding root cause under pressure
4. Remediation: Restoring service
5. Recovery: Full restoration and validation
6. Learning: Post-mortems and improvements

Organisational elements enable effective response. Clear roles prevent confusion:

• Incident Commander: Coordinates response
• Technical Lead: Drives investigation
• Communications Lead: Handles updates
• Subject Matter Experts: Contribute domain knowledge

Communication structures keep signal separate from noise. Documented procedures provide structure when thinking becomes difficult. Runbooks, escalation processes, and rollback procedures guide decisions under stress.

Practiced response builds muscle memory. Game days discover gaps whilst stakes are low. Chaos engineering reveals actual system behaviour during failures. Practice transforms abstract procedures into concrete skills.

Cultural elements determine whether teams learn or repeat failures. Blameless post-mortems focus on systems, not individuals. Psychological safety enables honest discussion. Your DR architecture determines available options. Your incident response capability determines how effectively you execute them. You need both.

The Challenge of Modern Complexity

Even with solid foundations, incident response faces new challenges. Modern architectural practices deliver enormous benefits: scalability, resilience, team autonomy, deployment velocity. These same practices introduce complexities that stress traditional incident response approaches.

Microservices architecture enables team autonomy and independent deployment, but introduces failure mode complexity. Hundreds of services mean hundreds of potential failure combinations. Partial degradation becomes harder to detect: 3% of requests through one path failing whilst everything else works often doesn’t trigger alerts. Users experience breakage whilst dashboards show green.

Cascading failures compound rapidly. DNS issues in one service cascade through dependents, then their dependents, spreading faster than teams can track. Each service needs its own DR strategy, but strategies must work coherently. Service A failing over doesn’t help if dependency Service B can’t also fail over.

Multi-cloud strategies reduce vendor lock-in and improve resilience, whilst introducing coordination challenges. Different providers offer different capabilities with no unified control plane. Your monitoring might be affected by failures you’re diagnosing. Comprehensive observability becomes a data volume challenge: millions of log lines per minute make human information processing the bottleneck.

Continuous deployment accelerates feature delivery and reduces deployment risk through smaller changes, whilst making incident investigation more complex. “What changed?” becomes harder when multiple teams deploy multiple services multiple times daily. The triggering change might have happened hours or days ago in a dependency three layers removed. Architecture evolves continuously, requiring DR plans and documentation to keep pace.

Distributed ownership allows teams to move independently, but creates coordination overhead during incidents. Three affected teams require coordination among three groups. Twelve teams create complexity that slows everything. System complexity naturally exceeds individual understanding. No one person can comprehend entire systems. Team turnover means knowledge distribution becomes critical. “Have we seen this before?” shouldn’t require tracking down specific individuals.

Third-party services enable teams to focus on core business value rather than building everything, whilst creating dependencies outside your control. Your DR doesn’t control third-party availability. Services depend on infrastructure services which depend on other infrastructure services, creating transitive dependencies that require careful mapping.

The AI Factor: New Failure Modes

AI introduces failure modes traditional approaches weren’t designed to handle. Third-party AI dependencies mean your DR depends on providers’ DR with limited visibility. Stateful AI components (fine-tuned models, vector databases, embeddings) create recovery challenges stateless services don’t face. Cost implications become unpredictable: different providers and regions have different rates and rate limits. Non-deterministic behaviour across regions creates subtle consistency issues.

Quality degradation doesn’t trigger traditional uptime monitoring. During stress, an AI recommendation service might remain “available” (200 OK responses) whilst quality degrades from 85% to 45% relevance. Traditional monitoring shows green. Users experience breakage. Teams discover issues through support tickets, not alerts.

Investigation reveals AI using degraded fallback models under load: technically working but producing poor results. This failure mode (graceful degradation degrading too far) doesn’t exist in traditional services. New incident vectors appear: rate limiting during traffic shifts, unpredictable behaviour under unusual load, subtle quality degradation accumulating gradually.

Non-deterministic failures are harder to reproduce. Traditional bugs follow predictable patterns. AI issues might be probabilistic, appearing only in certain contexts. Limited remediation options create frustration; you can’t fix third-party services. Communication complexity increases: explaining quality degradation is harder than explaining “down.”

Traditional challenges (distributed systems, rapid change, knowledge gaps) combine with AI complications (quality degradation, non-deterministic behaviour, third-party dependencies) creating maximum difficulty. But the same AI creating challenges offers capabilities to address broader difficulties when applied thoughtfully.

Working Smarter During Crisis

AI doesn’t fix poor architecture or weak processes. No amount of sophisticated tooling can compensate for missing fundamentals. But when your foundations are solid, AI provides significant leverage at every phase of incident response.

Detection and Assessment

• Intelligent Anomaly Detection: Traditional monitoring relies on static thresholds, generating many false positives and false negatives. AI-powered anomaly detection recognises patterns across hundreds of metrics simultaneously. It learns normal behaviour including daily and weekly cycles. It detects subtle anomalies humans miss. Modern platforms identify unusual patterns 15-20 minutes before outages manifest, providing time for proactive response rather than reactive firefighting.

  Start with AI surfacing potential issues whilst humans decide severity. Build trust gradually.

• Natural Language Log Analysis: Addresses a fundamental challenge: needing information quickly from millions of log lines. You can ask “show me payment failures with database timeout errors” in plain language. AI translates this intent to proper queries across different formats and services. This takes seconds versus 15 minutes crafting perfect queries manually.

• Rapid Impact Assessment: Synthesises information that takes humans 15-30 minutes to gather manually. AI can immediately provide: “3,200 enterprise customers affected, EU region primarily, estimated £75,000 per hour.” This enables appropriate urgency, correct escalation, and better resource allocation from the start.

Investigation

• Cross-System Correlation: Solves one of distributed systems’ hardest investigation problems. Requests flow through 15 services, any of which could be the root cause. AI analyses patterns across failed requests automatically, correlating traces, logs, and metrics across entire request paths.

  Example: After a multi-region failover with 1,000 failing requests, AI identifies in minutes that all failures require database writes. This narrows investigation to the write path immediately. Root cause: replication lag. Time to identify: 5 minutes with AI versus 45 minutes manually. Every minute of investigation delay is another minute of user impact.

• Historical Pattern Matching: Makes “have we seen this before?” answerable regardless of who’s on call. AI searches previous post-mortems, incident reports, and runbooks. It suggests solutions that worked in similar situations. Team B facing an incident can benefit from Team A’s solution six months ago, even if the teams never talked.

• Change Correlation: Addresses deployment velocity challenges. “What changed?” gets answered by AI correlating changes across all services with incident timing. Real example: a configuration change 45 minutes before the incident increased health check frequency from 30 to 5 seconds in a dependency service. Harmless alone, but causing connection exhaustion when multiplied across hundreds of callers.

Remediation and Learning

• Intelligent Runbook Assistance: Suggests relevant runbooks based on symptoms and context. Provides context-specific guidance: “here’s the failover procedure; based on current state, steps 3 and 7 can automate whilst step 5 requires manual intervention due to replication lag.”

• Decision Support: Helps evaluate options during incidents. AI provides relevant data for each option, comparative analysis, risk assessment from similar past incidents, and time estimates. Critical principle: AI informs whilst humans decide, particularly for high-stakes decisions requiring business context AI lacks.

• Recovery Validation: Monitors recovery across all metrics simultaneously, alerting if recovery isn’t proceeding as expected. Real example: after one remediation, AI detected that error rates had recovered whilst latency remained 30% higher. The backup region was under-provisioned. The team scaled capacity before declaring full recovery.

• Automated Timeline Generation: Assembles chronological timelines correlating system events with human actions. This reduces timeline creation from hours to minutes, enabling faster post-mortem completion.

• Deeper Root Cause Analysis: Identifies contributing factors beyond immediate triggers. Example output: “This is the fourth database connection exhaustion this quarter; pattern suggests systematic capacity planning issue.” This shifts discussion from tactical fixes to systemic solutions.

• Action Item Intelligence: Extracts items from post-mortem discussions automatically. Tracks completion across incidents. Identifies recurring themes. Surfaces incomplete items from previous incidents.

• Knowledge Synthesis: Makes “how did we handle last multi-region failover?” answerable by querying past incidents instead of tracking down individuals. Organisational knowledge becomes accessible rather than trapped in individual memories.

When AI Itself Fails

AI components in your systems require specific approaches to disaster recovery planning, monitoring, runbooks, and testing.

Architecture for Resilience

• Multi-Provider Strategy: Reduces single-provider dependency risk. Architect with a primary provider and fallback to an alternative. Example: e-commerce might use one provider for personalised recommendations with automatic fallback if the primary becomes unavailable.

• Graceful Degradation Paths: Answer “what happens when AI is unavailable?” Every AI feature needs tiered fallback strategies:

  Personalised AI recommendations
    ↳ Different provider
      ↳ Bestsellers
        ↳ Curated lists
          ↳ Category browse
  

  Each tier provides less sophistication whilst maintaining core workflows.

• AI-Specific RTO/RPO Considerations: Model fine-tuning restoration takes time. Vector database synchronisation might mean significant data transfer. Embedding regeneration for large datasets could take hours. Model version consistency across regions matters for user experience.

• Cost-Aware DR: Acknowledges unpredictable cost implications. Different providers and regions have different rates and rate limits. Circuit breakers prevent surprise bills during failover scenarios.

Monitoring AI Health

Quality metrics must be first-class monitoring alongside availability:

• Track response quality continuously
• Alert on degradation: “Accuracy dropped from 95% to 87%” should trigger investigation even if technically available
• Monitor cost patterns, alerting when approaching rate limits
• Track confidence score distributions, alerting when patterns change

AI-Specific Runbooks

AI-specific runbooks follow consistent patterns: symptoms, investigation, remediation, prevention, and success criteria.

Model Quality Degradation

• Symptoms: Quality metrics decreasing below threshold, user complaints about poor AI results
• Investigation: Compare current metrics to baseline, check for data drift, review recent model changes, check AI provider status
• Remediation: Revert model version, adjust prompts, activate fallback provider, or scale back usage
• Prevention: Continuous quality monitoring with alerting, automated testing before deployment, gradual rollouts
• Success: Quality drop detected within 5 minutes, fallback activated automatically, users experience less than 30 seconds of degraded service

Rate Limiting/Quota Exhaustion

• Symptoms: Timeout errors, HTTP 429 responses from AI provider
• Investigation: Check current usage patterns versus normal, identify traffic source causing spike, review recent changes
• Remediation: Activate circuit breakers, switch to fallback provider if available, request quota increase, fix bugs causing usage spikes
• Prevention: Usage monitoring with headroom alerts, burst capacity planning, multi-provider architecture
• Success: Approaching rate limit triggers alert, automatic fallback before user impact, quota increased proactively

AI Vendor Outage

• Symptoms: Complete unavailability or widespread degradation from AI provider
• Investigation: Check vendor status page immediately, test endpoints directly, assess scope of impact
• Remediation: Activate graceful degradation path, switch to backup provider, enable aggressive response caching, communicate degraded functionality to users clearly
• Prevention: Multi-vendor architecture, response caching, circuit breakers
• Success: Vendor outage detected within 60 seconds, degradation path activated automatically, service continuity maintained

Prompt Injection Response

• Symptoms: Unexpected AI behaviour, potential information exposure, unusual outputs that don’t match expected patterns
• Investigation: Review recent queries for malicious patterns, assess what information might have been exposed, check logs for similar patterns
• Remediation: Update input guards, implement additional filtering, notify security team, strengthen system prompts
• Prevention: Input validation on all user-provided content, adversarial testing, security review
• Success: Suspicious patterns detected and blocked automatically, security team notified, exposure contained

Testing AI Resilience

Chaos Engineering for AI

Validates that your disaster recovery and degradation paths actually work:

• Simulate AI provider unavailability and verify fallback mechanisms activate correctly
• Test rate limiting responses by deliberately exceeding quotas in test environments
• Validate that monitoring detects AI-specific issues before users do
• Test cost controls to prevent runaway spending during incidents

Regular DR Testing

Follows the same principles as traditional DR testing but includes AI-specific scenarios:

• Quarterly AI component failover testing builds muscle memory
• Annual full disaster recovery exercises including AI dependencies reveal gaps in planning
• Document learnings and update runbooks after each exercise

How to Actually Do This

Moving from principles to practice requires a structured approach. Sequential phases build capability progressively.

Phase 1: Assess and Strengthen Foundations

Before adding AI tools, audit your fundamentals honestly:

• Do you have comprehensive observability across all systems and services?
• Are incident roles and escalation paths clear and documented?
• Do runbooks exist and are they tested regularly?
• Have you practiced incident response recently?
• Is your post-mortem process generating actual improvements?

The brutal truth here is that if your fundamentals are weak, AI won’t save you. It will amplify dysfunction rather than creating capability. Fix fundamentals first. The good news is that if your fundamentals are solid, AI provides massive leverage.

For engineering leaders, this assessment reveals where to invest before scaling AI adoption. For practitioners, strengthening foundations makes your job easier regardless of whether AI gets involved.

Phase 2: Start with Augmentation

Begin with low-risk AI additions to existing processes that accelerate without fundamentally changing workflows:

• Use AI to query logs during investigations, reducing time to find relevant information without changing investigation methodology.
• Let AI draft post-mortem timelines, reducing documentation burden whilst humans review and refine.
• Add anomaly detection alongside existing alerts, providing additional signal whilst traditional alerting continues.
• Employ AI-powered log analysis to find patterns faster across distributed services.

The key principle in this phase is that humans review all AI outputs before taking action. Don’t trust AI blindly, especially early in adoption. Build trust gradually by learning tool strengths and limitations through experience. Some patterns AI excels at detecting. Others generate noise. Tune based on false positive rates in your specific environment. Understand when AI helps versus when traditional approaches work better. This learning period is essential: rushing to automation before understanding capabilities leads to problems.

For engineering leaders, Phase 2 provides opportunity to build organisational experience without high risk. For practitioners, it’s chance to develop intuition about AI tool effectiveness before stakes increase.

Phase 3: Guided Automation

The next level of integration involves AI suggesting actions whilst humans approve before execution. AI recommends remediation approaches based on similar past incidents, but humans decide whether to execute. This combines AI’s pattern matching across historical incidents with human judgment about current context. Automated runbook execution includes human checkpoints at critical decision points: AI can execute routine steps whilst humans validate before irreversible actions.

What to avoid in this phase: fully automated remediation without human oversight in critical paths. The temptation is strong when AI suggestions look correct. Resist it for high-stakes decisions. AI can inform and accelerate, but critical decisions still need human judgment, particularly those requiring business context that AI lacks. Should we failover knowing it causes 30 seconds of downtime, or fix in place taking 15 minutes? That decision requires understanding current business context (is this peak shopping hours? Are we in critical sales period?) that AI doesn’t have.

Phase 4: Build AI-Specific Capabilities

For teams with AI in production systems, this phase addresses the unique challenges AI components create. Implement quality monitoring for AI features that goes beyond uptime to track actual output quality. Traditional availability monitoring misses AI quality degradation entirely. Create AI-specific runbooks for common failure modes like model quality degradation, rate limiting, and vendor outages. These failures patterns differ from traditional service failures and require different investigation and remediation approaches.

Design and test graceful degradation paths so AI features can fail partially rather than completely. Every AI-powered feature needs answer to “what happens when AI is unavailable?” Regular chaos testing of AI components and dependencies validates that degradation paths work in practice and monitoring actually catches AI-specific issues before users do.

The Skills Your Team Needs

New tools require evolved skills, but core capabilities remain essential.

Foundational Human Skills

• Systems Thinking and Troubleshooting: Form the foundation of effective incident response. Understanding how components interact, isolating failures in complex systems, and reasoning about cascading effects are skills that no amount of tooling eliminates. AI can accelerate investigation, but humans must understand systems well enough to ask good questions and validate AI findings.

• Calm Under Pressure: Enables clear thinking during crisis. The ability to prioritise when everything seems urgent, avoid panic-driven decisions, and maintain focus despite stress remains fundamentally human.

• Clear Communication: Matters enormously during incidents. Status updates for executives require different detail than updates for engineers. Knowing when to escalate and how to document during chaos require human judgment about context and audience.

• Cross-Team Coordination: Becomes more critical as systems become more distributed. Working across organisational boundaries and collaborative problem-solving when incident response requires expertise from multiple teams aren’t skills that AI handles.

• Learning from Failure: Through blameless post-mortems, systematic improvement, and sharing knowledge across teams builds organisational capability over time.

New AI-Era Skills

Working with AI Tools

Effective prompting and question formulation determine whether you get useful answers or confusing noise. “Show me errors” is too vague. “Show me authentication failures in the payment service in the last 30 minutes with HTTP 401 responses” gets useful results.

Validating AI outputs and suggestions critically prevents blindly trusting AI in high-stakes situations. AI can be confidently wrong. Humans must verify recommendations before executing them, especially during high-pressure incidents.

Understanding AI tool limitations comes from experience using them in various scenarios: knowing when to trust AI analysis versus when to fall back to traditional investigation methods.

AI-Specific Domain Knowledge

For systems with AI components, responders need knowledge beyond traditional troubleshooting. Understanding model behaviour and limitations helps diagnose when AI components are causing versus experiencing incidents. Is the AI service failing, or is upstream infrastructure failing and AI is just a symptom?

Interpreting confidence scores and quality metrics requires knowing what those metrics actually mean for your specific use cases. A confidence score of 0.7 might be excellent for one application but concerning for another.

Recognising prompt injection and adversarial patterns requires understanding how AI systems can be manipulated: knowing what suspicious patterns look like in logs and user inputs.

Building Team Capability

For Engineering Leaders

Building team capability requires structured approaches beyond just providing tools:

• Incident simulations including AI failure modes provide practice in low-stakes environments where mistakes don’t affect real users
• Run game days that include AI quality degradation, rate limiting, and vendor outages
• Chaos engineering for AI components validates that teams know how to respond when AI fails
• Post-mortem reviews highlighting AI learnings ensure the organisation captures and shares knowledge systematically

Knowledge Sharing

Makes individual learning organisational capability that compounds over time:

• Cross-team incident reviews spread expertise beyond teams that experienced specific incidents
• Team A’s hard-won lessons from an AI vendor outage become accessible to Team B before they face similar issues
• Internal talks on AI incident patterns make tribal knowledge accessible to new team members
• Runbook libraries with AI-specific procedures provide reference during actual incidents when stress makes recall difficult

Better Prepared, Better Outcomes

Incident response is harder than ever. System complexity, deployment velocity, and AI integration create challenges that traditional approaches struggle to address. Perfect reliability remains impossible regardless of investment. But effective disaster recovery and incident response are absolutely achievable for teams willing to invest in both foundations and modern capabilities.

The AI Opportunity

AI tools help manage complexity at every stage of incident response:

• Detection becomes faster and more comprehensive through intelligent anomaly detection
• Investigation accelerates through automated correlation and pattern matching
• Remediation benefits from context-aware guidance
• Learning improves through automated timeline generation and deeper root cause analysis

The result is augmented human capability, not replacement of human judgment.

The Disciplined Path Forward

• Ensure Solid Fundamentals: Teams that rush to adopt AI tools without fixing basic observability and incident response processes find AI amplifies existing dysfunction rather than creating new capability. Fix fundamentals first.

• Build Incrementally: Add AI capabilities incrementally and deliberately, building experience at each phase before progressing to the next. Keep humans in control of critical decisions, particularly those requiring business context that AI cannot access.

• Address AI-Specific Challenges: Build AI-specific monitoring and procedures that address unique failure modes AI introduces. Quality degradation monitoring, fallback testing, and AI-specific runbooks aren’t optional extras: they’re essential components of operating AI in production.

• Invest in Skills: Effective incident response requires both foundational capabilities and new AI-specific knowledge. Engineers need to understand traditional systems thinking and troubleshooting whilst also developing skills in working with AI tools and investigating AI-specific failures. This combination takes time to build. Learn from each incident systematically, using both human reflection and AI-powered analysis.

Constant Principles

Key principles remain regardless of tools or technologies:

• AI augments humans: Rather than replacing judgment, particularly during high-stakes decisions where business context and nuanced understanding matter enormously
• Culture matters most: Good incident response culture (blameless post-mortems, psychological safety, learning orientation) matters more than sophisticated tooling
• No silver bullets: The best tools in the world can’t compensate for a culture where people hide problems, fear blame, or don’t learn from failures
• Continuous improvement: Never stops because systems and requirements constantly evolve, creating new failure modes and new opportunities to build resilience

What Success Looks Like

The goal isn’t zero incidents. That’s unrealistic for complex systems operating at scale. The goal is:

• Resilient systems that fail gracefully when components inevitably fail
• Teams that respond effectively under pressure, making good decisions despite stress
• Organisations that learn systematically from failures, implementing improvements that prevent recurrence
• Users who experience minimal disruption when problems occur because systems degrade gracefully and teams recover quickly

Learning from Recent Outages

Recent major infrastructure outages remind us that incidents happen to everyone, regardless of size, sophistication, or investment. The teams that recovered fastest shared common characteristics:

• They’d invested in solid disaster recovery architecture with tested failover procedures
• They’d built strong incident response capabilities through practice and continuous improvement
• They’d augmented their capabilities with appropriate tooling, including AI where it provided clear value
• They maintained human judgment and oversight whilst using automation to reduce toil

Teams that master this combination will build more reliable systems that serve users effectively. They’ll deliver better outcomes for customers through faster recovery and less disruption. They’ll create better working conditions for engineers through reduced firefighting and more systematic approaches. They’ll compound advantages over time as incident response capability and system resilience improve together.

The Choice

The fundamentals haven’t changed. Clear roles, documented procedures, practiced response, and blameless learning remain essential. AI provides new tools to execute fundamentals more effectively. Better detection through pattern recognition. Faster investigation through automated correlation. More informed decisions through context-aware guidance. Deeper learning through systematic analysis.

The question isn’t whether to invest in disaster recovery and incident response capabilities. Every organisation faces that necessity. The question is whether you’ll build both the foundations and the augmentation that modern complexity requires. Whether you’ll invest in team skills alongside tooling. Whether you’ll maintain the discipline to keep humans in control whilst benefiting from AI acceleration.

The path is clear for teams willing to walk it. Assess foundations honestly. Strengthen weak areas before adding complexity. Introduce AI capabilities progressively. Build team skills systematically. Learn from every incident. Maintain discipline about human oversight. The teams making these investments will handle the inevitable incidents that all complex systems experience whilst maintaining reliability, serving users effectively, and building sustainable incident response capability that improves continuously.

---

About The Author

Tim Huegdon is the founder of Wyrd Technology, a consultancy focused on helping engineering teams achieve operational excellence through strategic AI adoption. With over 25 years of experience in software engineering and technical leadership at companies including Yahoo! Eurosport and Amazon Prime Video, Tim specialises in building resilient systems and the organisational capabilities needed to maintain them under pressure. His work focuses on the practical intersection of traditional reliability engineering and modern AI capabilities, helping teams build incident response practices that work when it matters most.