Fragile Redundancy

Backup systems share dependencies because the same forces that shape a primary system shape its alternatives. They are bought from the same suppliers, certified under the same standards, run by the same staff, and routed through the same upstream services. The conditions that made the primary attractive make the same choices attractive for the backup.

Common-mode dependencies are often invisible until they are tested. Two independent data centres may run on power drawn ultimately from the same substation. Two independent providers may rely on the same submarine cable. Independence at the contract level rarely survives close examination of the physical or institutional layer beneath.

Contingency plans decay because the world around them does not stand still. Staff move on, vendors are replaced, configurations drift, and the assumptions baked into the plan slowly diverge from operating reality. Without continuous use, the gap between what the plan describes and what the system actually does grows quietly each year.

Exercises slow this decay but do not stop it. The plans that survive longest are those embedded in routine operations — used often enough that their assumptions are continually checked — rather than those preserved in documents intended for emergencies.

Failovers fail because they are exercised under conditions that differ in important ways from the conditions in which they are eventually needed. They are tested in isolation, on schedules, with the cooperation of the system being failed away from. Real incidents arrive in combinations, at inconvenient times, in the presence of partial information.

The systems that recover well from disruption are usually those that exercise their alternative paths as part of normal operation, rather than treating them as reserves to be activated only in extremis. Redundancy that is used is redundancy that works.