Designing distributed systems means considering failure scenarios—both likely and less so. Will the network let you down? (Almost assuredly.) Will some portion of your IaaS misbehave? (Have you met computers?) We build in graceful degradation for much of our automation but often neglect the (just-as-essential) human interactions.
The classic “hard problems” of cache invalidation and naming things revolve around our understanding of what’s correct and true and our agreements with one another on scope and relevance. Communication is essential for making context-dependent decisions.
Whether we’re attempting to determine the current state of reality or distinguish logical boundaries, democratized observability is key to answering our questions. As the fractal complexity of our distributed systems grows, we need to mindfully choose practices that work with our tooling. You can’t buy a silver bullet, but you can forge one from the collaborative efforts of your team.
Bridget Kromhout is a Principal Technologist for Cloud Foundry at Pivotal. Her CS degree emphasis was in theory, but she now deals with the concrete (if ‘cloud’ can be considered tangible). After years as an operations engineer (most recently at DramaFever), she traded in oncall for more travel. A frequent speaker at tech conferences, she helps organize tech meetups at home in Minneapolis, serves on the program committee for Velocity, and acts as a global core organizer for devopsdays. She podcasts at Arrested DevOps, occasionally blogs at bridgetkromhout.com, and is active in a Twitterverse near you.