A few months ago, I stepped into an IT role at my workplace, and it has been a fascinating experience filled with both highs and lows. One of the most intriguing aspects has been the timing of my entry, as we were facing significant internet issues, experiencing disruptions internally several times a week. We had previously investigated these problems with the former IT head and determined that the root cause was a network loop, but identifying the exact location of the loop was challenging.
So, what is a network loop? You can think of it as a tank with water pouring in from the top. Instead of allowing the water to flow out, two pipes are connected to each other on the side, causing the water to circulate indefinitely. Eventually, the tank overflows.
Similarly, in a network, one cable might bring in the same data repeatedly. Since the data can’t flow out like the water in our tank, it creates saturation, leading to network congestion and, ultimately, downtime or disruptions.
With support from one of our internet providers, we carefully inspected the cables connected to the main switches. We unplugged each cable and monitored the traffic for dropouts. Eventually, we identified a cable that was causing the loop by reconnecting back to the same switch.
That should have resolved our issue, right?
Not exactly. A few days later, the network went down again. While the connection from our Internet Service Providers (ISPs) was functioning correctly, internal operations were still facing problems. I had to reevaluate the situation since something else was clearly causing the outages.
Currently, we are using FortiGate to manage our firewall along with Unifi switches and access points throughout the network. I say “currently” because I believe we may switch from FortiGate in the future. Unifi has made significant improvements in their firewall policies, and honestly, it’s easier to manage devices from a single manufacturer. More details will follow on this in a later post.
I logged into the UniFi dashboard to check the main switches for any signs of a loop. To my surprise, there were none. I reviewed the network logs from FortiGate and found nothing unusual.
Feeling a bit apprehensive, I identified the switch connected to the “faulty devices” and removed it from the network. As a result, the network stabilized. I then moved the active devices from that switch to a nearby one, as there were a few available. Days turned into weeks and then a month, and everything seemed fine. I attempted to reintroduce the switch to the network and even tried resetting it, but to no avail. It became clear that we had a faulty switch at the core of our network (no wonder STP wasn’t working as expected).
However, if you’re familiar with IT, you’ll know that this wasn’t the end of our troubles. Another problem loomed on the horizon, the Wi-Fi 😂. While it’s currently working well for the staff, the situation for the rest is a story for another day.
Sign up