If you are reading this hoping for a solution, I don't have one at this time. Just documenting the things we've tried in case someone has some suggestions or is running into the same issues we are.
We are currently in the process of trying to resolve this issue with Cisco. They appear to have a Cisco TAC case (CSCtg86211) on the high CPU usage, but it doesn't mention the activity lights on the switches fast flashing on all connected ports. We also tried upgrading the IOS with no success.
Problem (listed together because I think they are related):
High CPU usage on the switch by "Hulc LED Process"
Fast blinking lights on the ports, in unison, as though there are constant broadcasts
We have two stacks of four Cisco 2960S switches. One stack is running 12.2(58)SE2, the other 12.2(55)SE3. Both stacks sit at around 20% HULC LED CPU usage and exhibit the odd uniform LED flashing as though there is a constant stream of broadcasts. Packet captures don't show nearly the amount of broadcast activity that the lights make it appear to have.
Here is the short version of the tests we did further down in the post:
We booted up the stack with all
switches disconnected from each other (each switch is independent), all servers
are plugged in and all workstations are plugged in. The traffic looks normal
until you either introduce the firewall connection (which starts the issue on
the switch you connect it to) or if you uplink the switches then the issue
presents itself.
Unplugging uplinks (stacking cables, uplink ports, firewalls) does not affect the blink speed of the ports still connected. The issue remains until you reboot the switch. After reboot, it starts right after you re-introduce some sort of uplink to the switch.
I'm by no means a Cisco expert, but the fact that the lights behave normally until you introduce some form of uplink makes me think the issue is related to a spanning tree type function that goes into some sort of loop or is just buggy. It would makes sense that the reason the flashing light activity doesn't go back to normal when you remove the uplink is because once the issue gets triggered by the uplink, it just goes into a loop.
Here is some testing we did on the odd flashing lights/LED issue (not looking at CPU usage):
Change: Updated one of the stacks from 12.2(55) to 12.2(58). (rebooted)
Result: We can't tell for sure, but we think the lights are blinking faster than the stack running the older IOS. Flashing is still uniform.
Change: Removed all stacking cables from switch stack while it was exhibiting behavior, no reboot.
Result: All connected port lights still flashing at the exact same speed/frequency/uniformity, even though now there was nothing linking the switches together.
Change: Unplugged almost all devices from a single switch while it was exhibiting behavior, no reboot.
Result: All connected port lights still flashing at the same exact speed/frequency/uniformity.
Change 1a: Rebooted single switch, nothing plugged in but one computer.
Result: Normal LED/light behavior
Change 1b: Rebooted single switch, two computers plugged in.
Result: Normal LED/light behavior
Change 1c: Plugged in a firewall to the above switch with only two other computers connected
Result: Fast flashing LED/light behavior
Change 1d: Unplugged firewall from switch (so it's only the original two computers plugged in.
Result: Still fast flashing LED/light behavior
Change: Connected another 2960S switch through regular RJ45 uplink to a freshly booted 2960S switch with two computers plugged (and LEDs behaving normally).
Result: Fast flashing LED/light behavior. Like the above scenario, the problem doesn't go away when you unplug the connection that triggered the fast flashing.
