We left Brisbane bright and early on Thursday the 16th of July to spend a day with the guys from GCCEC to get to the bottom of this latest issue.

Step 1: Knowing what the first step is!

Random reboots, firmware updates, and twiddling of settings does not constitute logical fault finding. If you think there is a problem with a system there is only one place to start and that is at producing a clinical and unambiguous statement of the problem at hand. We almost always work in tricky multi-supplier scenarios at these events and so it is really important to reduce the problem to a clinical document we can all look at so you’re not fingering individuals or companies – but rather working together with all parties to nail the main pain.

With this in mind we set out allocating the first half of our day to an entirely tedious but necessary exercise of conducting a complete access point by access point, radio interface by radio interface survey of the venue (by hand ).  The goal of the survey was not to measure the RF characteristics of the wireless network (been there, done that) but rather the real-world throughput characteristics.

Step 2: Sanity check the fibre/copper network and backhaul before looking the wifi performance

GCCEC’s copper and fibre network is a relatively modern star topology of a single gigE fibre core servicing a number of 10/100 copper edge switches with fibre backhaul. This is good as exhibition venues that have grown over time tend to have a lot of zany and miscellaneous long fibre runs with random Vendor X fibre transceivers all over the place making you unsure of what exactly it is that you’re measuring. The GCCEC network on the other hand is simple to test definitively due to a sane overall architecture.

Additionally, the MDF that houses the core features in-room Foxtel. I could have done with that in the basement of the Sheraton Mirage at Port Douglas for last year’s Australian Partner Conference.

GCCEC MDF - Rather impressive as it is the first machine room I've been into with Foxtel installed, though I couldn't find the bar fridge !

Testing Rig @ MDF/network core

We deployed a high performance x64 machine with 4GB of RAM and IIS7 directly connected to the core switch via 1000BaseTX copper. We deployed several files to this IIS7 server being 1 megabyte, 10 megabytes, 50 megabytes, and 100 megabytes in size.

Consistency in client testing

We used the same clunky Lenovo (waves at the other David ) for all client tests to ensure that we were always using a consistent platform on both ends of the equation. This slowed us down somewhat because it meant we could not split up to make things faster, but I felt strongly that being able to say that all results have as high a degree of consistency as possible was very important. We configured the Lenovo for maximum power wastage performance on all interfaces to ensure that results would not be inconsistent if we were to conduct some tests off battery versus off mains. Wherever possible we used mains power for tests.

Testing procedure

We used a simple script with wget downloading the test files from the testing rig in the MDF three times. We recorded these results together with some other data at each test point (more on that later).

Validating the test rig and core

A Lenovo notebook was directly connected to a port on the core switch via 1000BaseTX copper. We did several transfer tests between the test rig and the Lenovo receiving approximately 800mbps of real world throughput which is pretty good with a notebook as a client any day of the week – and certainly sufficient for our tests.

Validating the edge switch at the first test point

The last step before heading off on the Tedium Express was to test the first edge port that services the first access points we were to test. Again, this was the case of running the test scripts from the Lenovo directly connected to the edge switch on the same VLAN as our private test network. These reported 99.99% of the maximum throughput one would expect from a 100 meg link – so that was certainly very good.

The performance of the copper and fibre network validated as satisfactory.

The next step was to confirm whether or not there was a systemic issue with wifi performance (as per my hunch) or whether there was a simple problem with a loose TNC connector on the one of two APs we saw as problematic during the last exercise.

Anyone who has worked with me knows that I’m a bit like an old dog with a bone when it comes to pushing for a quality outcome and ensuring that things do what they say on the tin. Given that I tend to push damned hard for what I think is the right thing to do – it also means that I was experiencing rather some trepidation at the survey as I’d possibly have some explaining to do regarding billable time if my hunch proved to be false.

Step 3: Conduct the wireless survey

The wireless component of the GCCEC network is comprised of the following elements:

• Cisco 4402 wireless lan controller
• 50 x Cisco 1252 abgn wifi access points
• Each access point has a 2.4GHz and 5GHz radio interface – you can think of these two radios as being akin to having two separate Ethernet ports in your PC … if you’re troubleshooting they BOTH need testing.

Nathan at GCCEC did a wonderful job of building us a mobile testing rig out of a road case, trestle table and ratchet straps + onboard GPOs a 50m power lead and anti-slip mats! Nath – you’re a legend! Christened the Tedium Express, this was our working environment as we tested each AP.

Toot Toot! All aboard the Tedium Express!

We wanted to ensure that we could conduct a single sweep of the entire venue and not have to go back out to reassess it as we only had funding clearance for a day on site. With this in mind we prepared a comprehensive spreadsheet listing:

1. each copper MAC address
2. each base radio MAC address
3. the associated IDF and edge switch port

The process we used was to park the mobile rig under (or as close as possible to) the antennas under test and collect:

1. the advertised radio link speed
2. the current channel allocation
3. three real-world download performance data points
4. Repeat steps 1-3 for 2.5GHz and 5GHz.

This meant 50 access points * 2 radios * 5 data points = 500 collected values.

The first dozen or so radios took nearly an hour and a half to collect the data from, primarily because the Lenovo was taking so long to switch between 2.4GHz and 5GHz radios. We therefore chose to survey the venue twice, once at 2.4GHz and a second time at 5GHz. Despite having to cover twice the distance we ended up doing the collection much more quickly as we weren’t waiting on client WLAN configuration.

Out of interest we chose the Lenovo as the client test rig as the ThinkPad connections software allows you to view a series of access points listed by MAC address and then choose a MAC address to associate to as your preferred access point. Given that we had a pre-prepared list of all of the access points in the venue together with their MAC addresses, this made validation of what we were testing at any point a snap.

Also, David Cormack from CBO was helpfully driving the wlan controller software and printed installation maps to further validate what we were doing. The whole process worked pretty well but the ‘law of big numbers’ meant that it was going to take a while no matter what – there were 100 interfaces to survey and that will never happen in an hour.

The entire survey took from approximately 0830 to 1430 to complete with four people.

I see dead radios

During the course of the survey we found the 5GHz radio in AP-24 to be faulty. It was extremely difficult to obtain an association with the 5GHz interface and when we did the radio link speed was 9mbps and real world download speeds were 3-4 kilobytes (no, that is not a typo) per second. We had to bench it to work out what was going on.

I don’t have a problem with heights per se as I’m fine on the top of a mountain or on a high rise balcony or observation deck. What I do have a problem with is these damned scissor lifts that are rickety and dodgy and wobbly. If that is not bad enough they actually can drive around when you’re 10m in the air.

I edited out the bit where we started percussive maintenance (I’m talking about the AP, not your forehead David Cormack ). As an aside, it is a pretty good view of the hall from up there. During the video you’ll hear a faint rasping sound … that is the GCCEC staff scraping every last skerrick of masking/gaff tape off the raw concrete floor to ensure the venue is spotless for the next show. Keep this view in mind when you’re walking around the nice carpeted expo halls with fancy games and Xboxes and other stuff … there are innumerable venue and staging guys really go to an amazing amount of work to transform that concrete shell into the event you know and love. While we did our survey they blacked out the entire hall to do metered comparisons of a new super-bright ‘green’ compact fluorescent lamp for the hall to ensure they’re a satisfactory brightness and colour temperature compared to the existing metal-halide units. In short – there is just so much stuff that goes on to prepare the minutiae of detail at these events that hours and hours of labour go into light globe selection alone – let alone the what technology team do.

Here’s another view of the area tech•ed will consume from ground level (opposite end of the hall from the previous video):

Step 4: Analyse the results

I digress … where was I? That’s right – the survey.

So after a couple of laps of the venue we came up with this:

GCCEC Wireless Throughput Survey Results

Some of you might see quoted figures of 2 megabytes or 5 megabytes per second and think that is pretty good for wireless, and truth be told if you were getting sustained rates like that out of some dinky home router I’d probably agree. Our expectations are somewhat higher and you have to remember that we’re really the only people at the venue there are only two acceptable performance outcomes:

• saturation of the RF
• saturation of the edge port

The results paint an entirely different picture though. On average, the RF segments of the network are only providing 17.80% of associated RF link speed on 2.4GHz and 16.14% of associated RF link speed on 5GHz.

The other problem with these results is that they are highly inconsistent. Across each of the three runs the individual tests could vary by double or half.

So it turned out that my initial hunch from the last day on site was correct … and we have a serious wifi performance problem on our hands. Ruh Roh!

Step 5: Find the culprit

By this stage we had chewed up most of the day. We had ascertained:

• there was a serious performance problem with wifi and we had quantified it in a clear and clinical way as was our original intention.
• that the problem lay between the access points and the wireless lan controller, or possibly the way the wireless lan controller terminated in the venue core switch.

While this was progress, we really wanted to provide more definitive findings that would further narrow down the source of the problem. I must admit I was very disappointed at the prospect of leaving the venue with only problems identified and no positive prescriptive advice.

Further isolation testing

We had AP-24 in hand so we retreated to the centre of the venue and proceeded to shut down each of the wireless access points. After doing this we patched AP-24 into the wireless lan controller VLAN and reconfigured it in 2.4GHz 802.11n mode (turning off it’s flaky 5GHz radio altogether). We re-ran the same test suite and found performance to be the same as any of the other access points in the venue during our initial survey. This was helpful as it showed us that the problem was not in some way related to load on the wireless controller or the number of access points talking to the controller.

Reflashing AP-24 as autonomous

You will recall from the previous article that we discussed how the access points in the network are running in lightweight mode and so can only run in conjunction with the wireless LAN controller (see this if you’re interested: http://www.cisco.com/en/US/products/hw/wireless/ps430/products_qanda_item09186a00806a4da3.shtml). We decided it would be nothing if not informative to ‘downgrade’ the access point from lightweight mode to autonomous mode.

Autonomous mode firmware gives the access point a more complete IOS feature set and allows it to bridge the 802.11 (wifi) and 802.3 (Ethernet) networks without a controller. After some mucking around we re-flashed AP-24 into autonomous mode and directly connected it to our test VLAN bypassing the controller VLAN all together.

Surprise surprise!

AP-24 Throughput with Autonomous Firmware - Now we're cooking!

With a link speed of 144 mbps you can expect approximately half of that in terms of real-world download speed, which would be 72mbps in this case. Let’s do some sums. Remember that hard drive sizes and network throughput are universally measured using base 10 arithmetic (1 megabyte is 8,000,000 bits), while file sizes on disk and memory consumption and so on are measured using base 2 arithmetic  (1 megabyte is 8,388,608 bits) … I know, I know I could use SI units for kibibytes and kilobytes but most people reading this would have no idea what I was talking about.

Anyway, ignoring framing overhead and so on to make this easier:

1. 72 x 1000 x 1000 = 72,000,000 bps
2. 72,000,000 / 8 / 1024 /1024 = 8.5ish megabytes of files downloaded per second.

Above we see 9.8, 8.9, and 9.1 as the figures from the test of AP-24 in autonomous mode. This is what I would call EXCELLENT!

Conclusion

We made a lot of progress in the day on-site.

1. We proved and documented that there is a systemic problem with wifi performance at the venue.
2. We found and removed a dead AP.
3. We proved that the edge and core networks are functioning as expected.
4. We proved that the poor wifi performance was not related to channel/RF intereference by our first isolation test.
5. We proved that the access points provide excellent performance in autonomous mode.

In short, we turned a hunch into a known and well documented problem.

We provided a written report of our findings back to the venue and they have taken it back to the installers to get the core issue looked at and the installers have basically said that the issue will be resolved ‘no matter what’ – Thank you guys – I love commitment.

We’ll provide a further update when we know definitively what the root cause was and when the matter is corrected to our satisfaction.

Don’t worry, we have a plan B if the above doesn’t get resolved. Fortunately for you this means you will get as good performance as possible at the event. Unfortunately for me it means that damned scissor lift and reflashing 50 access points to autonomous mode with individual configurations before the event and back again afterwards. Hmmm David Eagles; What are you doing this time next month?

Per AP TX/RX figures with channel utilisation across the ground floor of GCCEC

Number of Client Associations across the ground floor of GCCEC

I was given logon access to the WCS console at GCCEC at the start of May this year. Shortly (10 minutes) later I started e-mailing “DANGER, WILL ROBINSON!” messages to the venue and the tech•ed technology team.

What’s wrong?

The first image shows the current receive and transmit utilisation (Rx. Util. and Tx. Util. respectively) for a given access point (we’re still trying to get to the bottom of understanding how they’re calculated given there are multiple radios in each access point). The Controller also has the access points doing a passive scan in the background to determine the actual RF spectrum utilisation on the channel to which the access point is assigned (that’s the third “Channel Util.”  figure off each access point).

The second image shows the number of client associations across the same area.

See the problem? The venue is between events. No one is there. But there is a massive RF utilisation across the building. Some of the access points are yakking their heads off to no one and there is not much bandwidth left over for any users.

When we raised the issue, Cisco TAC and the installers of the network were of the opinion that the issue was caused by external interference. I rejected this explanation immediately because:

1. GCCEC is on the coast of Queensland and so constructed to withstand cyclones and severe tropical storms. This means robust reinforced construction materials throughout.
2. The venue is bounded by water on three sides and the Gold Coast highway (building foyer + drive way + park out front + 4 lane road + block of shops before you even get to the nearest residential building)

Gold Coast Convention and Exhibition Centre (from Bing Maps)

Anyone who has diagnosed complex problems with multiple suppliers in the mix knows that getting traction from people in resolving problems is sometimes hard, especially when people invariably have a foregone conclusion as to a root cause in their minds. Our logical fault finding steps needed to be clear and bulletproof to gain traction and ownership from all involved.

Step 1 – Isolate the cause of interference as internal or external to the building

This part was pretty easy. There is a company called Metageek that sells a great little device called a Wi-Spy (it presents itself to Windows as a HID device so there is no mucking around with special drivers and other nonsense) and a companion piece of software called Chanalyzer. Chanalyzer and the Wi-Spy together allow you to see peak and average utilisation of the entire 2.4 ghz spectrum (there’s a version you can buy now that does 5 ghz). You can simultaneously use your lappy’s onboard WLAN NIC to grab a list of SSIDs with the corresponding channels and signal strength information and then overlay that over the actual RF activity on the network.

Central Room A Spectrum Utilisation - not you can clearly see the signature of the 802.11n network on channel 1 (the squarish pattern with a dip in the middle)

There are two useful views here. The Spectral View (aka waterfall) shows a time series utilisation graph of the RF spectrum. You can adjust the sampling period and play back different periods. The other useful view is the topographic view that shows the signature of the RF utilisation pattern overlaid with the SSIDs found using the WLAN NIC in your laptop.

As I mentioned above, it was amply clear that the interference was inside the building because the WCS console was saying as much, but we really needed a smoking gun. This was too easy to produce (by sheer brute force):

1. Run up wi-spy, and grab a sample of what the spectrum is doing;
2. Shut down the entire wireless system in the venue;
3. Repeat step one and compare.

Step 1 showed a lot of RF utilisation. Step 3 showed nearly none. Case closed: The interference was the wireless system in the building. Now we just had to work out why!

Step 2 – What is the RF interference exactly?

After two months and a number of Cisco TAC cases the utilisation figures at the venue were still unacceptably high. We had not received a decent explanation from the parties involved as to the true root cause (well, not one that would satisfy me anyway) so I chose to employ brute force again.

Brute force this time came in the form of an embedded wireless platform that allowed us raw and unfettered access to the underlying WLAN NIC to do some packet captures of the RF-side of the wireless interface. We needed to use this specialised platform to capture packets due to limitations within the Windows kernel in which 802.11 traffic is presented to applications as 802.3 (Ethernet) traffic as it moves up the driver stack. Therefore, under Windows, it is not possible to capture raw management frames unless you use devices that use a proprietary raw miniport driver that bypasses most of Window’s normal networking. These drivers are never certified.

The embedded device we used is normally stuck on mining vehicles with neodymium magnets (David Eagles from iVolve brought it in a nice green Coles friends-of-the-Earth recycled shopping bag and told everyone not to put your laptop near it unless you wanted a blank hard drive).

Running packet capture of the RF in North-West hall.

Running RF capture in North-West hall - the four circles on the brackets on the device will erase your lappy.

We were pretty much the only users of the WLAN in the North-West of the centre. We ran a packet capture on the RF to see what on Earth was going on and fed the raw file to Wireshark. The results were very revealing:

• We ran the packet capture for 185 seconds
• 39193 frames were captured (remember no one is using the network at this point!)
• 38,088 frames were 802.11 beacons … !
• Only 1105 frames were not 802.11 beacons … !!

Further from this you can work out:

• There were approximately 220 beacons per second with a size of 258 bytes each.

At this point we knew we were onto something … but why so much traffic?

Step 3 – Analyse the logs

Beacon frames are sent as a normal part of 802.11 management traffic. Normally an access point will send (about) 10 frames per second to advertise their SSID and various information about the capabilities offered. That would account for but a small fraction of the traffic above. We were only pulling traffic from channel 6 in this case so there could not possibly be sufficient access points to generate that much traffic.

Remember we’re looking at 220 beacons per second. A single access point should only generate 10.

Wireshark Trace from Central Room A showing the beacon frame spam

The Wireshark traces showed that GCCEC has 5 SSIDs being advertised for use (their public one, internal, one for Telstra and some other stuff). Each of these were being advertised in its own beacon packet. This is helpful as it shows us now to expect 5x the number of beacons per access point and importantly we’re now in the realm of feasibly accounting for the quantity of beacon packets being seen in our packet captures (i.e. our packet capture device would easily see 4-5 access points on channel 6).

This answers part of the problem as to why there was so many frames from an ‘unused’ wireless network. Now we just needed to answer the original question we came on site for – why so much RF utilisation?

Step 4 – Punch some numbers into a calculator

To understand the nature of the problem we need to understand a bit about data rates and 802.11 networking. There are a number of bit rates defined for 802.11 networking and clients will choose a bit rate based on signal strength, configuration of the base station, and other things.

The important thing to note here is that all management traffic is sent at the lowest bit rate supported by the base station. In this case that would be … 1 mbps.

A 1 mbps bit rate gives you typical data throughput speeds of 500 kbps.

Let’s go back to those figures again:

• 220 beacon frames per second;
• 258 bytes each;
• multiplies out to 454,080 bits per second;
• Typical throughput for 802.11b at 1 mbps is about half-a-megabit … which would be about 500,000 bits per second.

BINGO!

We now can account for 80-90% RF utilisation figures based on beacon frames alone. All of these marry up more or less and so now we understand the problem.

Recommendation for tech•ed

There are a few very logical outcomes from this exercise that provide ‘easy wins’.

1. We will turn off all advertised BSSIDs except for MicrosoftEvent;
2. We will get GCCEC to make their corporate network’s WLAN access require a probe request so it is not causing another SSID to be advertised;
3. We will disable 802.11b at the event (sorry to all of you with an iMate Jamin, but it might be time for an upgrade! );
4. We will up the basic rate to 18 mbps. This alone will ensure that management traffic will take up 1/18th of the RF spectrum that it was before.

Hooray! Beers all around at Q1 … not quite

In a case of “solve one problem, find another” we unfortunately did uncover a fair few more issues while conducting the work above over a full day on-site. The main outstanding issue that we have now is that we noticed that some of the radio interfaces in particular access points perform very poorly (3 mbps typical throughput even on 802.11n). This is now our critical concern with the wireless at GCCEC and we will be scheduling another visit onsite as soon as possible to get to the bottom of these problematic radio interfaces to work out what the issue is. At this stage it looks like a real puzzler because we did some quick tests in the waning hours of our time on site and we received excellent performance from other radios of the same access point, and excellent performance from the edge switchports that service the access points.

We like a good mystery and hopefully solving this next one won’t bring up more problems so we can have those beers at Q1.

Its no secret we’ve been spending a lot of time checking out the venue to ensure we’re getting adequate access to wireless and network facilities. On our last site visit we had the venue network engineers working with codify to figure out ‘black spots’ in the network and work towards a root cause. We were lucky enough to have a loner machine from Nick to test out the profile, the results of which you will see later, but the size factor of the machines make them really easy to cart around the venue, and I generally averaged about 330 Mbps on the network – of course that’s really easy to do when you’re the only person connected….

While the codify guys started getting down to business, we were tasked with another very important job;

Tasting the tech•ed and APC delegate menus….

Some days you have to work very hard, and I guess this was one of those days; leaving the codify guys to continue working on the network, we dragged the jomablue guys down for opinions on the menu. Now I’m not going to go into detail on all the items served but there was an amazing range that was presented to us, such as:

• Beef with Ginger and green peppercorns
• Ocean Trout
• Beef and mushroom Risotto
• Fruit Shots
• Seafood Paella
• Lamb Korma
• Lamb Cutlets

Now bear in mind that this was for both events so we can’t guarantee you’ll see any of them (unless you are going to both..) oh.. there was bunch of wines as well, but best leave you to enjoy that for the event itself. Some shots below…

The team led by “Fletch” at GCCEC did an awesome job of exciting us about what the delegates are going to experience food wise.

Those photos above don’t really do it justice and after we ate till we couldn’t eat no more, we rolled/staggered our way back to find the codify guys still working hard, apparently testing what appeared to be some strange speed results from a couple of the wireless access points.

So as you can see we’re working VERY HARD to give you the best level of service possible for tech•ed and APC…. well at least some of us are

- jorke

This has worked really well for Microsoft in the past as we have only ever really officially offered wireless in select common foyer areas and the exhibition hall areas. This year, however, things have been ratcheted up a whole number of notches due to the giveaway of the HP Mini 2140 devices.

Wireless hardware for tech•ed 2009

GCCEC has recently installed a new Cisco 4404 Wireless LAN Controller with 50 x Aironet 1252 lightweight access points with abgn radios installed throughout the entire complex.

This differs greatly from most wireless network scenarios you’re probably used to in that:

• The Wireless LAN Controller is a centralised management device that manages all of the access points.
• All of the access points are ‘dumb’ in that they receive firmware and configuration information directly from the controller.
• All traffic from the individual access points is delivered to the Controller over copper Ethernet and then distributed to the local area network by the Controller using 2 x gigabit fibre uplinks.

There are a few key benefits in this architecture, the primary being that there is a single device (the Controller) with a complete understanding of all access points on the network. This means that:

• The Controller can manage the channel selection and power selection settings across the network. The Controller will therefore (allegedly) ensure that near-by access points are not assigned to conflicting channels.
• The Controller works proactively to understand and close coverage gaps in the network.
• The Controller understands where all the walls are, roughly what they’re made out of (so it understands the db loss caused by the wall), ceiling heights and so on.
• In some scenarios the Controller can detect if clients at the edge of the network are having trouble connecting and so crank up the power levels accordingly.

“Wow! That sounds amazing … 50 APs will handle anything!” I wish that were true however as you’ll learn in coming posts, there is no such thing as a free lunch (well there is if you work as crew at the event but you don’t want to see what we get fed out back).

There are a number of issues with the WLAN deployment at GCCEC that we have to resolve before the event. This series of articles will keep you updated on how we’re doing with root cause analysis and implementing the necessary fixes.

Coming up next:

• Diagnosing and resolving extremely high RF utilisation
• Diagnosing and resolving intermittent throughput problems on individual radio interfaces (Scheduled to be onsite to crack this one on Thursday … write up will follow that)