Stuck in Cisco Hell

April 1, 2008 10:54 AM

I have a recently installed (upgraded) wireless network composed of 11 Cisco Aironet 1200 Access Points controlled by a Cisco 4400 WLC. The access points are running on the slim firmware, and are being controlled by the WLC. We are using the WLC to broadcast several different SSIDs from the access points, spanned across several different V-Lans We are having massive intermittent connectivity problems.

All antennas broadcast the same SSIDs, but on different channels to minimize crosstalk.
Right now we have around 77 wireless clients connected throughout the plant. This is a mix of laptops, and desktops. Most of the desktops have Linksys WUSB54GC, though we have several desktops with Cisco wireless PCI cards in them. The laptops have a mix of intel and broadcom based chipsets.
All switches in the plant are Cisco, with a Cisco ASA firewall, etc, etc

The general problems are as follow:
1 - Once connected to one of the SSIDs, the pc will after some time switch to another antenna in the plant with a much weaker signal. Same SSID, just another completely crap signal antenna.
2 - Regardless of what AP it is connected to, we will see the transfer speeds of the machine drop down to 1-2, and the web becomes unusable. This happens regardless of machine and card setup. The machine can be feet away from the AP and experience the same performance as one a hundred or more feet away. Usually right click and repair fixes this for a random amount of time.

Things we have tried:
- Switching all the cards from the vendor supplied software to Windows Zero Config.
- Changing the cards from b/g to b-only
- Disabling USB sleep and low power settings
- Many other things over the last week I can't recall at this point.

General question points:
- Any suggestions on things we can try further to fix these issues?
- Do any of you work with a similar setup, and have any experience with similar problems?
- How many devices should we expect to be able to have on each access point? The users will be web browsing only.
- What other tests or diagnostics should we be doing to narrow down where the problem is?

posted by JonnyRotten to computers & internet (6 comments total)

Historically there was a bug where WZC would poll every x number of seconds and that caused problems, but it seems like you've crossed that off.

Do your users authenticate using radius/certificate/etc or is it a static PSK defined in the configs?

I would enable debugging on a couple of access points and the WLC and point it at a loghost and see if you can do some event correlation. I'd run a show tech or any messages that show up through the output interpreter on CCO as well.

You should be able ot get some detail on the disconnection and the clients leaving/joining from the WLAN nodes and the WLC that will give you more of a starting point You appear to e in try random stuff mode and have changed thing materially from where you started out chasing one problem and possibly introducing a new set.

Lock things down to your expected configuration, and get some solid data via logs/messages from there.

You don't mention your campus size, but you should be fine with the number of AP's you have.

The WLC is the probable culprit here, it controls the RF and essentially everything else associated with the AP's. It's a long shot but have you insured that you don't have intermitant connectivity out to the AP's from the WLC? If you are experiencing intermitant failures within the wired LAN it can impact the ability of the WLC to manage the AP's.
posted by iamabot at 11:38 AM on April 1

I have tested it with both WEP, WPA, and completely open SSIDs, and the results are about the same. I am looking into enabling debugging right now.
Our factory is pretty large (I am not sure of square footage), but when we did a site survey using Cisco's site survey software, we do not have many weak signal areas. Cisco today mentioned that we should mount the APs on the wall and point them sideways.
posted by JonnyRotten at 12:08 PM on April 1

Missed that final point on your post iamabot. We are not seeing any other failures within the wired lan, but I have a couple new GBICs on the way to swap out on the connections for the WLC, just to rule that out.
posted by JonnyRotten at 12:14 PM on April 1

Yeah that's goin to be dificult, I'd really start with taking an AP out of the rotation if possible and testing from there with as much debugging turned on as possible on the WLC and the AP to see if you get any interesting messages.

Certainly run the configs through the output interpreter. As it is a factory have you considered that perhaps you have some heavy interference in the 2.4 ghz band ? Might try switching to 11.a which in the 5ghz band and may provide some relief.
posted by iamabot at 1:15 PM on April 1

I've tried to replicate any interference that may be happening on the shop floor. Ran towmotors back and forth next to them, passed the crane back and forth over the press and below the AP.
I am looking into adding another AP, so that I can turn on debugging. I've contacted the company that we contract some of this stuff out to about turning it on the current APs. I need to set up a cisco account to get the output intrepreter, but sounds like it will help alot.

Thank you so far!!
posted by JonnyRotten at 2:06 PM on April 1

If you have your cisco support contract number you can ring them up and get it set up in about 5 minutes. Entitlement will work fairly quickly if you're experiencing problems.
posted by iamabot at 2:28 PM on April 1