Fragile Redundancy

Backup systems can come to share dependencies because many of the same pressures shape both the primary system and its alternatives. They may be bought from the same suppliers, certified under the same standards, run by the same staff, or routed through the same upstream services. The choices that made the primary attractive can also shape the backup.

Shared dependencies may remain difficult to see until they are tested. Two independent data centres may draw power from the same underlying substation. Two independent providers may rely on the same submarine cable. Two services can appear independent at the contract level while still sharing physical or institutional dependencies underneath.

Contingency plans can drift from reality as the world around them changes. Staff move on, vendors are replaced, configurations change, and assumptions written into a plan may no longer match the operating system. Without regular use or review, the gap between the plan and the system can grow without attracting much attention.

Exercises can expose some of this drift by testing assumptions against current arrangements. They may still take place under controlled conditions that do not reproduce the timing, uncertainty, or combination of problems present in a real disruption.

One reason failovers may behave unexpectedly is that they are exercised under conditions that differ from those in which they are eventually needed. They may be tested in isolation, on a schedule, with the cooperation of the system being failed away from. Real incidents can involve several problems at once, inconvenient timing, and incomplete information.

Knowing that a backup exists is different from understanding how it behaves when the primary system is unavailable. This distinction is examined further in Critical Dependencies Bulletin — Issue 02: When Redundancy Isn't Really Redundancy.