Every time I manage a change to a customer network I have a chance to taste the many shades of possible IT Operations maturity levels. I collected some best practices over the years about how to reduce risk and speed-up the change and testing process. I’ll share some in this post. Improvements and suggestions are welcome in the comments of the post or on my Twitter account.
As most IT professionals I usually configure network devices in a lab environment before the actual installation at customer site. I try to limit the installation as much as possible to a simple box moving process, spending most of the change window in a previously defined validation process. In this particular case I deal with a data center core network that includes 8 Nexus 9k switches configured in 4 VPC pairs and a bunch of links between them.
This story starts with a phone call at night. If you worked in IT long enough you know what it means. Customer’s HQ network is down and since the day before I’ve replaced a pair of data center switch in a remote site I’m somehow involved based on the well-known principle “last one who made changes is responsible”. I state that all the facts took place with my telephone support, without any remote access to the machines.
At Cisco Live Europe in Barcelona I had a chance to see Cisco Candid (Network Assurance Engine) in action. I shared my views on GestaltIT Tech Talks. Full video of TFDx session:
A couple of days ago Cisco released a Security Advisory. No big deal so far, level was informational so I didn’t read it right away. Title is impressive: Cisco Best Practices to Harden Devices Against Cyber Attacks Targeting Network Infrastructure so i read it during a lunch break just to be aware of the contents. Management sessions to network devices provide the ability to view and collect information about a device and its operations.
Automation and programmability is not a new topic for me. Having studied Information Technology in High School I’ve always coded somehow, never making it my primary focus but always using it as a tool to make my life easier. I remember a script I did in Pascal to create a menu to load custom maps for Doom II instead of using the CLI. It would be great to find it again but it’s very unlikely because I trashed so many PCs and hard drives since, well, at least I hadn’t bitcoins stored there!
For a Network Engineer living and working on the field has some challenges that are not common in office environments. I have a set of tools, hardware and software, that I bought or built over the years that allow me to accomplish my job in more effective way. I used to carry a small Access Point to provide connectivity inside a datacenter or campus when the rack is located in odd places (you know what I mean).
We live in a time of intent, automation, orchestration and a lot of wonderful tools that promise to make the life of network engineers easier. Sometimes reality is simpler and maybe less fascinating, real problems need to be solved quickly with small budget. The specific case I discuss here is a medium network, around a hundred devices. The problem is to create an inventory of all the devices, backup configurations and verify all the boxes have the correct syslog, ntp and timezone configuration.
I had the honor and pleasure of being invited again to attend Tech Field Day, this time for an Extra event at Cisco Live Europe in Barcelona. Cisco Live is a week full of product announcements, technical session, (social) networking with fellow network engineers, meetings with colleagues and customers, discussions with Cisco engineers about products and roadmap. This is exhausting and exciting at the same time but it definitely worth the effort.
In the previous post of this AirPiConsole series (part1, part2) I used Autossh to create a reverse tunnel from the device to a cloud VPS to permit remote access. The VPS I use is cheap but unreliable so the tunnel was down most of the time so I started looking for a better alternative. The solution came from the Packet Pushers podcast episode PQ134 about ZeroTier. What is ZeroTier?