Troubleshooting intermittent internet outages
I recently ran into an issue at a client that underscored the importance of keeping networking firmware current. We had replaced a single, aging Cisco ASA Firewall with redundant Internet connections and a High Availability pair of new Cisco ASA Firewalls. In order to split the incoming Internet connections to the two firewalls, we put a pair of Cisco 2960 switches in front of the firewalls. This implementation was working well for a few weeks until the firm started to experience sporadic Internet outages.
After some investigation, we found that the 2960 edge switches were rebooting intermittently. Because of the redundancy built into the system, the firm only noticed an Internet outage when both switches happened to reboot at the same time. According to the switch logs, the switches were rebooted because of a power event. That led us down the path of investigating the power. While we did find issues with the UPS units, resolving them didn’t fix our issue. After applying additional monitoring, we were able to determine that the switches were rebooting at a consistent repeating interval.
This led me to look for a software issue. Reviewing the switch configurations, I noticed that Smart Install and PnP Connect were enabled. As these features were not needed, I disabled them, and the reboots stopped. With some additional research, I found that there was a Denial of Service vulnerability discovered in March, and patched with a recent update. We installed the new firmware and have been fine ever since. We also left the configuration features disabled, just to be safe.
This was a tricky situation, as the equipment was new and the firmware was pretty up to date. Most IT pros know that it’s important to install Windows Updates, but many people don’t keep networking equipment firmware as current as they should. Networking equipment runs operating systems like desktop PCs do, has its own security vulnerabilities, and needs to be kept up to date. It is especially critical to keep Internet-facing equipment like firewalls and edge switches updated.