data center, cable, information, technology, technician, server, system, room, engineer, work, rack, man, computer, network, industry, data, connection, equipment, fixing, optical, wireless, pc, data center, data center, data center, data center, data center, server. Platform outages risks: a one-page continuity plan
Photo by cookieone on Pixabay

Industry

Part of Platform outages: down for you or down for everyone, and how to tell

Platform outages risks: a one-page continuity plan

Platform outages risks: how to find the choke points where one failure stops several activities, and a small operation's continuity plan that fits on a page.

Your exposure to platform outages risks is not proportional to how much you use a platform. It is proportional to how much of what you do stops when it stops. That is a different quantity, usually larger.

This page shows how to find those points before they find you, judge the exposure without pretending to a precise number, and build a response a small operation can maintain.

What to take away

  • Find the places where one failure stops several unrelated activities. Those are worth money to fix; everything else usually is not.
  • Most continuity planning fails because it was written for an operation larger than yours. A one-page plan you will actually follow beats a good one you will not.
  • The cost of an outage is mostly not lost revenue. It is the confidence you spend explaining it and the attention you lose to it.

Finding the choke points

A single point of failure is a component whose failure stops the whole. In a small operation they are rarely dramatic, and they are almost never where people look.

In a small operation, the same few dependencies tend to appear: Amazon Web Services (AWS), Cloudflare, Meta (Facebook, Instagram, WhatsApp), Google Workspace, Slack, or a payment processor. The 2021 Facebook outage is a real example: the outage lasted about six hours, and businesses that used Facebook Login, WhatsApp Business, or Instagram scheduling lost those channels at the same time.

From activities to choke points

  1. List your activities
  2. Ask which dependency each needs
  3. Repeat for a dozen activities
  4. Same two or three names appear
  5. Dependency used by nine = plan around it

Find them by listing your activities, not your dependencies. Then ask which dependency each activity needs. That list is the dependency inventory in its most useful form.

Do that for a dozen activities and the same two or three names repeat. That is your answer. A dependency used by one activity is an inconvenience; one used by nine is what to plan around.

Two properties turn a dependency into a real exposure.

It has no substitute reachable in the time you would have. Not "no substitute exists" but "no substitute you could stand up this week".

Its failure is silent to you. Something that breaks loudly gets handled; something that degrades quietly costs you days before anyone notices, which is why detection matters more than redundancy for most small operations.

For detection, small operations can use UptimeRobot, Pingdom, Datadog, or Better Stack to watch their own endpoints. For a public status page, Atlassian Statuspage, Instatus, or a static page hosted outside your main infrastructure keeps the message away from the failure.

Judging the cost

CostHow to estimate itUsual size
Work stoppedHours lost times what an hour is worthSmaller than expected
Recovery workBacklog, reconciliation, cleanupOften larger than the outage itself
CommunicationTime spent explaining, plus the messages you sendUnderestimated every time
ConfidenceWhether people trust the thing next timeReal, and not estimable
CompoundingThe other work that did not happen that dayThe largest and least visible

The bottom two rows resist arithmetic. Pretending otherwise yields a number that looks rigorous but is not; the honest exercise gives a range for the top three and a written note that the bottom two exist. Use your own hourly cost in the first row; this page does not supply a generic one.

That is enough to judge whether a mitigation is worth its cost, the only decision the estimate must support.

Redundancy, and when it is not the answer

The engineering response is to have two of everything. Redundancy works and it is expensive, and for most small operations the expense is not in the second copy but in the ongoing cost of keeping it current. A standby that has not been exercised in a year is not a standby.

Redundancy versus fault tolerance

Redundancy

Question asked
Can we have two?
Main cost
Keeping the copy current
Failure mode
Unexercised standby is not one
Example
Second system

Fault tolerance

Question asked
Can it keep working reduced?
Main cost
Often no duplication
Failure mode
Degrades but continues
Example
Local copy, held queue

Fault tolerance is the more useful frame. It asks a softer question: can the thing keep working, in reduced form, when a part fails? That is often achievable without duplication.

Publishing from a local copy, holding a queue until the far side returns, and having a way to reach your audience that does not depend on the surface are all fault tolerance without a second system.

The general trade is simple: concentration is not automatically wrong, but it should be priced and it should not be accidental. What a platform shift costs is useful further reading.

A continuity plan that fits on a page

Ready.gov's material on business continuity planning is written for larger organizations and is worth reading for its structure, which reduces to four questions you can answer in an afternoon. In the US, CISA also publishes free continuity and cyber resources; if you handle personal data, your state breach-notification law and the GDPR in the EU set notification duties.

What must keep working, in order. Most operations have two things that genuinely cannot wait and a dozen that can wait a week, and writing the order down is most of the work.

What each of those depends on. Taken from the activity list above.

What you would do if each dependency were gone for a day. Not a procedure: a sentence. The value is in discovering which ones have no sentence.

Who decides and who tells people. Naming this in advance removes the worst hour of any incident, which is the one spent deciding who is in charge. What that hour should leave behind is a plain written record: what broke, when it started, what was decided, and who was told.

Review it once a year, in an hour, and change the parts that have gone stale. A plan that gets reviewed is worth more than a thorough one nobody opens.

The one-page template

Fill this in on a single page and keep it where you can find it during an incident.

Field / What to write

Activities that cannot wait
Ranked list, two or three items
Dependency each needs
Platform, provider, or tool
Fallback for one day
One sentence per dependency
Recovery target (RTO/RPO)
What you can tolerate, in your own words
Decision owner
Name and backup
Communicator
Name and channel
Review date
Once a year

Incident log fields:

| What broke | When it started | What was decided | Who was told |

The exposure you cannot remove

Some of it is structural. If your audience is reachable only through one surface, that surface's failure is your failure, and no amount of local redundancy changes it. The honest response is to know the size of the exposure and to reduce it slowly by building a route you own, rather than to pretend a plan covers it.

Similarly, a provider's failure to communicate is not something you can mitigate. What you can do is decline to depend on their page for your own decisions. That is the argument for running your own detection rather than waiting to be told.

Common questions

How much should I spend on this?

Use your own hourly figures in the top three rows of the cost table, summed over the outages you actually expect in a year. For most small operations that number is small, which is the honest and unpopular answer.

Is a second provider worth it?

Only if switching is genuinely fast, which means you have written down an RTO and tested the cutover against it. A second provider you cannot cut over to within that RTO is an expense, not a mitigation. Test the cutover once before you believe in it.

What about a status page of my own?

Worth having if people depend on you, and it should be hosted somewhere that does not share your failure modes. A status page on the infrastructure that just went down is the classic version of this mistake.

Does any of this reduce the frequency of outages?

No. It reduces how long they last for you and how much they cost, which is the whole of what is available. Frequency is set by other people's engineering, and the tests that tell a real change from noise are the only thing that will tell you whether it is getting worse.

More in Industry

Latest from Records Desk