[LMB] Sorcerous Architects (NO spoilers for most current Pen&Des)
skyefire at skyefire.org
Tue Apr 4 22:07:13 BST 2017
On 4/4/2017 4:32 PM, Brad Jones wrote:
> For a modern, humorous take on this idea, there's a current Disney animated
> series (US airings on Disney XD) called "Milo Murphy's Law". The idea is
> that the titular Milo Murphy is a direct descendant of the coiner of
> Murphy's Law. Milo suffers from "Extreme Hereditary Murphy's Law", meaning
> that...well, chaos appears around him as everything that can go wrong, does.
> His father (also afflicted with EHML) works as a safety inspector, because
> having him go near complicated machinery acts exactly as you suggest for
> the sorcerer checking the ropes, rollers, and hoists.
In RL, Netflix has the Chaos Monkey tool, and the bigger Chaos
Kong. Long version short, they run an automated tool that
(semi-?)randomly *crashes* parts of their server pool, to force their
failover measures to be constantly tested and exercised. It seems like
one of those terribly counterintuitive ideas, but it actually makes
sense -- the number of times I've seen "backup systems" that failed b/c
they hadn't been tested in too long, and they either aged badly or the
primary system evolved beyond the backup's ability to cover... in the
Netflix model (and, I believe, certain other large-scale data-center
type systems), the concept is that rather than "primary" and "backup"
systems, *all* the various servers are load-sharing, and deliberately
forcing failover events on a regular basis ensures that each system, and
the various load-sharing/balancing procedures, are constantly tested
during normal operation.
More information about the Lois-Bujold