Something Audacious + Nobody Noticing = Awesome!

I remember watching a documentary (or maybe it was Pushing Tin) about air traffic controllers who passed around small pieces of paper with flight information. As an aircraft would transition from approach to ground control to departure the triangles would be passed from person to person.

Last night my team moved 27 critical pieces of equipment that run our entire business It took us countless hours of prep and five hours of execution. We needed a way to keep the entire company informed of our progress, coordinate three different teams and make sure our bosses had enough information to keep off our backs. So we created a Trello board with lists for each stage a piece of equipment would transition through and cards for each piece of equipment. It worked like a charm!

I was able to walk through the datacenter and know which machine was ready to be moved. By clicking on the card I could see where the machine was and if I would need help removing it from the rack. I could see if the cabling team was falling behind and needed to be augmented. I could see if servers we not coming up cleanly and our remote team needed assistance. Perhaps most importantly our bosses could see progress, watch us figh fires and have confidence in our ability to not destroy the company.

There is much more I want to say about this. Why we did take an outage. How many wrenches got thrown in our planning. Why nobody noticing is awesome. But this was just the first part of the first phase of a master plan to harden and upgrade our infrastructure. We’re off to a flying start!

Title Credit: https://twitter.com/michaelpryor/status/320962983192825857