The slide deck from our presentation is available here:

Infrastructure Saturday tech•ed Backstage 2010

Ben Parker presented quite a few slides on the enterprise software stack on DemoNet – content that we really missed out on delivering at the event.

We hope you enjoyed the talk – No evals through yet but in the wrap up a guy said that he gave us all “1″s because he thought that was the top score (not 5!) Doh!

The video is available here: http://www.msteched.com/2010/Australia/voc208

The whole talk was very much ‘by the seat of our pants’ as we were insanely busy. Ben Parker, unfortunately, could not make it on stage due to other work commitments at the event. This was a real shame as he is ‘the man’ when it comes to the IT Pro side of the MS Enterprise software stack.

Unbelievably (given the rushed presentation and that most of the slide deck was a re-heated user group presentation I delivered a few months ago because we ran out of time ), we managed to score the #2 spot out of 165 sessions at the event, with an “Overall Satisfaction” of 4.74 out of 5. This is a great result but I cannot help but think there was a lot of other more deserving content at the event that probably should have rated higher.

We had a lot of good laughs – half of them at my expense. The notorious review comment from ARC301 came back to haunt me (2m 20s in … Thanks Jorke )

Evaluation scores follow:

Overall Results

Rollup Results

Comments

I will be presenting tech•ed backstage at the Brisbane Infrastructure Group. Guess appearances may include Nathan Wright from GCCEC and Ben Parker from Parker Tech Pty Ltd

Venue
Microsoft, Level 9, 1 Eagle Street, Waterfront Place, Brisbane City.
Time
1700 for Pizza.
Date
Tuesday, 13th of July 2010

Further details on #BigAU.

I’ll be covering off stuff that happened last year, stuff that’s happening this year + our wins and losses. Shock/Horror! I actually have some good news re Cisco WLCs this time around too!

See you there!

Cisco 4404 Marketing Poster

Long time readers of this blog (and the tech•ed 2009 team who have to put up with my whinging) will know that we had a significant number of challenges getting the wifi at GCCEC up to a scratch. Today I am happy to announce, however, that we have had a win. The 5508 delivers in the areas the 4404 didn’t.

Catch up on the complete history:

• http://www.techedbackstage.net/2009/07/15/diagnosing-and-resolving-extremely-high-rf-utilisation/
• http://www.techedbackstage.net/2009/07/22/resolving-a-hunch-wifi-performance-gccec/
• http://www.techedbackstage.net/2009/08/26/when-i-said-16-percent-i-meant-16-percent/

David Connors and David Eagles posing for a rare post survey photo op. "Weeze frum Queensland and weeze makin wirelesses!", commented Connors during a recent interview.

The bottom line issue was that all of our investigative work lead to the inescapable conclusion that the Cisco 4404 is a crock. I’d like to say that it was just a matter of the 4404 not supporting 802.11n data rates, but as the previous posts have shown, it would only barely perform at 802.11b rates in some cases.

We engaged with the original installers and Cisco TAC as well as local Cisco reps and pretty much got nowhere. Despite some early positive noises, they certainly did not want to own the issue and every time we would go back to them with detailed reports the responses we received were totally unacceptable (95% RF idle utilisation caused by co-channel interference on a network with 1 access point active and 1 user). Yes, I am still annoyed at the comment about Cisco being “amazed if there was a fault of this nature … especially if someone from Queensland found it.”

The most frustrating thing for me about the whole endeavour is that we spent a LOT of time and both Microsoft and GCCEC spent a LOT of money in having to go back and rectify someone else’s work. We also went to great lengths to ensure that we provided detailed reports that covered off exactly the clinical steps we went through to provide a final conclusion as to the problematic performance characteristics at the deployment.

In the end, we decided to remove the 4404 from the equation and deploy Cisco autonomous IOS images to all of the access points. This resolved the majority of the issues and, to us at least, confirmed that the 4404 was the root cause of all of our pain.

In the time since tech•ed 2009, GCCEC has replaced their 4404 WLC with a new 5508 unit (http://www.cisco.com/en/US/products/ps10325/). The product literature for the unit talks a lot about supporting 802.11n data rates.

O’RLY? What was the 4404 with AGN-capable Aironets on it supposed to do then?

4404 vs Autonomous

The move away from the WLC to autonomous firmware last year was a bit of a painful thing for us to do. We were torn because:

• The autonomous firmware would demonstrably provide 500%+ performance improvement of ther 4404 WLC.
• In moving to autonomous, we had to give up:
  • centralised management
  • AP-assisted roaming via CCX (see http://www.techedbackstage.net/2009/07/17/what-is-ccx-and-why-should-you-care/)
  • centralised wireless -> wired egress

We wanted to use the WLC but the performance characteristics were so dreadful it was not an option.

4404 in action

5508 Performance Characteristics

On the 26th of March 2010 we were engaged by Microsoft to validate the new 5508. David Eagles, Jorke and I spent the whole day on site with Nathan from the convention centre rerunning the entire survey from scratch. I am delighted to have to report that the 5508 has resolved the performance issues.

Simply swapping out the 4404 for a 5508 (and adding nothing to the RF delivery side of the wireless network) yieled:

• 484% average throughput improvement over 2.4GHz
• 340% average throughput improvement over 5GHz

26 March 2010 GCCEC Wireless Survey 2.4GHz Performance Comparison

26 March 2010 GCCEC Wireless Survey 5GHz Performance Comparison

2.4GHz Performance Comparison Across Aironet Deployment Scenarios

5GHz Performance Comparison Across Aironet Deployment Scenarios

So there you have it. There is still a bit of tuning for us to do regarding channel allocation (the same stuff we did for tech•ed 2009 – but was lost in the 5508 deployment) but it looks very promising.

A heartfelt THANK YOU to the GCCEC guys for the capital investment for the new unit for tech•ed 2010 – we really appreciate it.

Delegate Network 

• 99.998 uptime (was a 45 second outage from a GBIC flapping..)
• ~1300 wireless clients was the highest amount connected.
• ~2TB of data downloaded (in 4 days!)
• 300GB send out.

Demonstration Network (the glass box of doom with the servers inside) 

• 100% uptime on services with over 80 virtual machines hosted for speaker and other demos. This was highly under-utilized.

Major issues 

Oops

• Intel Wireless drivers – escalated to Intel who were sponsoring -
  they were kind enough to help us out
• Bit Torrent – Played network cop – Rickrolled access to key sites –  click here (I dare you)
• RRAS port exhaustion – escalated to our support team in India
• Live ID creation – another escalated to our support team…
• Netbook deployment – finished this with brute force
• Session Recording – Expression Encoder crashes
• Wrong URL on netbook images – network hack
• Well i think this is minor, but there was some slight damage of a netbook box…

As you can image there was so much more that went on, we’ve added a lot of these things to our planning process which has already started; 

“Life is like riding a bicycle. To keep your balance you must keep moving” – Einstein 

So here go, we’re hitting the reset button – we’ll do our best to keep you update-to-date from our end – no promises of course as we’re all focused on delivering to a higher quality outcome than before. 

- jorke 

We don’t shape the traffic on the network at all and nor do we see it as our role to police or restrict delegate use of the network (a policy that will, no doubt, change for tech•ed 2010 based on last year). Given this, we had to do a bit of seat-of-our-pants work to identify who the main culprits were and then implement counter-measures to at least lessen their impact on the delegates who were at the event to learn and share (rather than download). The back story is interesting as it involves a collaboration between us and the team at Microsoft who own the ipnat.sys driver that we were using to do the network address translation.

Where is all that data going?

The first thing we noticed was a couple of brief periods during the day where a very small number of clients (maybe 5 or 10) would experience no Internet connectivity. This was odd:

• the network was happily switching a couple of hundred megabits per second of data (that’s the Internet link, not the core).
• there was plenty of CPU and other resource headroom on the servers in question.
• nothing was showing up as unusual on the core switches.
• performance tests carried out in network operations showed plenty of excess capacity.
• we did all of our sums on port and CPU requirements and the RRAS team did a simulation for us (see: http://www.techedbackstage.net/2009/08/05/windows-server-2008-r2-nat-performance-guest-post-by-the-windows-team/) so we did do all of our homework up front.

I was puzzled. I don’t like fault reports of any kind and while the number of affected users was small, we really did need to get to the bottom of it. There was also the matter of the amount of data we were pulling from the Internet. We normally run tech•ed with a 100-150 mbps link with headroom to spare – but this year we were seeing sustained peaks twice the historical norm. Odd and uncharacteristic.

tech•ed 2009 Internet link port utilisation expressed as megabytes per second (base10, multiple by 10 to get a rough base10 mbps figure)

The quickest way to get to the bottom of anything like this is to point Wireshark at the network and see what is happening. As the network is switched, we need to set one of the ports on the core into SPAN mode (Switch Port ANalyser mode) which instructs the switch to send everything on a given VLAN or port out a nominated destination port, regardless of whether it is actually destined for that port. Once you have all of that data, Wireshark does a fantastic (if not always slow) job of breaking down the data into the corresponding protocols visually so you can see what is going on. The “Top Talkers” feature in Wireshark is awesome as it quickly lets you identify which clients are consuming a bulk of the traffic.

This quickly showed us two things:

1. A small number of clients were causing a lot of data utilisation.
2. The protocol was BitTorrent.

At this point you have to remember that we have a heap of bandwidth available. Some clients chomping through a lot of bandwidth isn’t a problem and running BitTorrent isn’t a problem per se. The aforementioned work on port utilisation planning was already done – but I had not looked at that empirically and so that was the next step.

Port Exhaustion

We pulled up the NAT table for the relevant interfaces in RRAS and the problem was immediately obvious. The table was HUGE (approaching 65K ports). The RRAS machine was experiencing port exhaustion.

Let’s step back and look at the issue a bit from first principles to understand what is going on. Like any corporate network, tech•ed 2009 runs private address space. Private address space is special IP address space allocated by IANA and intended solely for IP networks that do not route directly to and from the global Internet.

In order to provide Internet access at the event, we use Network Address Translation on the default gateway. The gateway has two network interfaces:

• one facing the private address space (and the IP address of this is your default route if you do a netstat -r on your PC).
• the other network interface is directly connected to our IP transit provider (in this case it was Telstra for delegates and Over The Wire for staff and speakers) and this interface has a normal public IP address on it.

The following happens when your client tries to send a packet to an Internet destination:

1. Your local IP stack sends the packet to the default gateway specified in your current IP config.
2. The NAT server receives this and pretends that it is going to route it to the Internet.
3. The NAT server alters the contents of the packet so that the source of the packet is the NAT server’s public IP address and a possibly a different IP port on that public interface.
4. When doing step 3, the NAT server makes a note of what it has done in what is called a translation table.
5. The NAT server sends the packet on its way to wherever it is meant to go.
6. A reply from the Internet host will come back to the NAT server on the public IP and port that the server provided.
7. The NAT server alters the contents of that packet to be destined to your private IP address and port and then sends the packet to you.

So the upshot of the above is that you can ‘hide’ a thousand people behind a few public IP addresses. You don’t need to obtain a public IP address for each user, which is a good thing as IPv4 address space is running out. There are a few limitations on this scheme:

1. There is a limit to how much you can store in the translation table. The translations need to be available for kernel mode device drivers and so cannot be swapped out to virtual memory – this means they reside in Nonpaged Pool in the NT kernel (an area of memory that is always resident).  Each mapping consumes around 256bytes of Nonpaged pool. Assuming 15 mappings by 2000 clients = 30000 translation table entries. That’s 7.5 megabytes of nonpaged pool memory out of a nonpaged pool of say 256 meg. Not a problem.
2. There is a limit to how much CPU grunt the machine has. The CPU time spent making translation table look ups and re-writing the packets on the way in and out of the NAT quickly add up when you’re talking about thousands and thousands of packets per second.
3. IP ports are represented as two bytes. This means that the maximum number of 65536 TCP and 65536 UDP ports are available on a given IP address. The actual number available is a bit lower as the server will have a number of ports allocated for its own use depending on what is installed.
4. RRAS, we found, only uses the machine base IP address for the outside of the NAT. It will not use additional IP addresses in the public address pool, no matter how many IP addresses are in that pool.

Analysing the ipnat.sys address translation table

The wireless network only had a maximum capacity of 1500 users and we only hit that peak very briefly. Even if we were at that peak, surely it cannot be the case that all users on the network are using an average of 40+ TCP ports in the NAT translation table?! I am currently running Skype, MSN Messenger, Google Talk, WinAMP (streaming 256kbps Internet radio), Chrome (16 tabs including this, two Google Apps instances, Salesforce.com, YouTube and a bunch of other reference material for this post), VPN to our intranet + whatever Windows is doing itself. I have only 20 ports open to Internet destinations that traverse the NAT in our office. There is no way the average user is going to be using double my ‘power user’ scenario.

We really needed to get our hands on the NAT table to analyse what was going on with the address translation table. “No problem!”, I think, “I’ve done that heaps of times!” Unfortunately, what I did not realise was the following:

• There is no way to export the translation table from the management snap-in.
• ipnat.sys does not have a WMI provider.
• ipnat.sys does not have any native and easily accessible Win32 APIs.
• The only way to pull the table was via MSRPC Remote Procedure Call interface.

The Windows RPC interface is architecturally … shall we say … “dated” to be polite. It is overly complex and not remotely easy to write software against (especially when you consider how simple our requirement is). We started to write an extraction utility but it was getting pretty late and you can’t really keep working until the wee hours of the morning when you have to be on site at 6:30am. We handed the code off to the owners of ipnat.sys as they said they had plenty of steam left (being in India) so they would finish it off for us.

In the meantime, we implemented certain, ahem, ‘interim countermeasures’. We quickly built a list of all of the top torrent trackers around and got the nod from Jorke to add them all to the local DNS resolver and point them at a local web server containing some RickRoll scripts.

How professional network administrators deal with Torrent users.

It killed me that I didn’t see anyone getting done by this first hand, but there were hundreds of impressions in the server logs containing the the Rick Roll scripts so I did get a fair amount of satisfaction at least. It was the most evil of evil Rick Roll scripts too – worse than any that anyone has used to get me in the past.

The next morning we found that ipnat.sys developer (being in India) suffered all sorts of Internet and power problems overnight and was unable to finish the utility. It was a new day when we received this news so we completed the utility ourselves (thanks Paul – using software engineers as network admins has some benefits ) and so were able to pull the translation table via MSRPC and dump that out to text.

The results were revealing and really made me want to go on a rampage. We were indeed hitting port exhaustion as an issue, and the distribution curve of who was using what looked like this:

The distribution of the number of translation table entries used on a per-client basis. I wonder who's using BitTorrent?

As soon as you see that – it is a no-brainer. The right-hand side of that graph represents a small number of users who were using 700-800 translation table entries each at the time. We had a one minute sample file that showed a particular individual using 2500 translation table entries! Argh!

So we scheduled this script to run each minute to generate a list of offending MAC addresses. It took a few goes to get the analysis right but we ended up generating a ‘naughty factor’ based on the number of port mappings, number of distinct hosts on the other end of the mapping, idle time, and so on to give us a number between 0.0 and 1.0. 1.0 meant you were very naughty. 0.0 meant you were very good. We reasoned that if you had a lot of mappings, and that a large proportion of those mappings were to a lot of distinct remote hosts, and largely not idle, that you are probably a Torrenter. OTOH, if you had, say, 20 connections open to a single host or a low number of hosts then this is probably quite fine.

These scripts output a list of bad MACs, that we then just dropped into a block list in the core switches. The logic proved to be quite sound as this is what happened when we blocked a couple of dozen particular users with a high naughtiness factor:

The result of applying a block to a couple of dozen users.

And there you have it. The culprits fingered and booted off the network. Of course, they then just changed their MAC addresses, in which case they were then re-identified as soon as their utilisation crept up, and the new MAC was banned.

2010

So this year, the users who did the above have driven me to recommend the following this Friday when we meet at Microsoft Brisbane for the first technology team meeting:

1. Opt-in “allows list” basis for access to the network. You will need to register your MAC in CommNet if you want access, and we will apply a quota across all of your devices; and/or
2. Mandatory rate limiting on a per MAC basis across all users; and/or
3. Packeteer/Allot/etc based deep packet inspection and traffic shaping.

It’d be better if we could provide peak throughput to any given person at any time should they need it – but the above shows that a small number of people ruin the experience for a large number of prople. My annual argument that users will respect the network resources and behave sensibly will no longer wash with the rest of the team.

</rant>

Anyway, I wanted to write a bit about a particular position I advocate each year and how disappointing to was to see that position abused by a very small number of network users to the detriment of other users at peak usage times.

At the outset of network planning, there is almost always someone who advocates locking down the network environment so that user access is restricted by protocol or all user traffic is forced through an HTTP proxy. I have always dug my heals in arguing that unfettered network access is the best possible delegate experience, and that there will always be delegates with oddball use cases for network access where HTTP/HTTPS only access is not sufficient. In recent tech•ed years the network has been structured like this as private IP address space with NAT – and we’ve never had complaints from users. Good old RRAS does the job!

This year, however, the scenario was a lot different. We ended up with some peak times of network congestion – admittedly only a few times – and the cause was users with their new NetBooks running BitTorrent in a way that showed a complete disregard for other users of the network.

We saw a fair few people huddled around furtively running BitTorrent. It was really disappointing to have to approach users to ask them to be more considerate to others, only to get “But I’m not running BitTorrent” as a reply. Come on people: The network ops people at tech•ed didn’t come down in the last shower.

There was even a case where one guy took his Netbook to the helpdesk to get his network access restored (we had some automated scripts blocking torrent users – but that is a post in itself). He too, allegedly, wasn’t running BitTorrent apparently (there was a folder called “TV Shows” on the desktop of his brand new Netbook).

Running BitTorrent is one thing, but there were even people who removed all of their natural limits from their client. There was one guy who would have 2000-3000 peers running at once.

All in all it was a bit depressing as I really felt like we were having our good nature taken advantage of.

BitTorrent Protocol

The first thing that people suggested (and kept suggesting to the point that Jorke and I were going to start throwing chairs at people )  is that we simply just ‘block them’ or ‘block BitTorrent’ or even funnier “block the BitTorrent port” (You know who you are..) . Unfortunately, this is not so easy.

Broadly, there are two parts to BitTorrent. The first is the tracker protocol, and the second is the peer protocol. For the purposes of this article, I’ll use BitTorrent to refer to a number of peer to peer protocols that behave similarly. I’m also going to gloss over a lot of detail to keep this post under control otherwise it will turn out like one of my wireless ones and take all day.

Tracker Protocol

The BitTorrent tracker protocol is a relatively simple HTTP/HTTPS protocol that is used on a centralised server to:

• Advertise torrents via an announce URL
• Centralise information about the peers participating in the torrent
• Collect statistics about the health of the torrent

Typically you will access a tracker over HTTP to search for torrents via a web page (this is not the Tracker Protocol obviously). Once you find a torrent you download a .torrent file which stores the torrent meta data, including the announce URL. You then launch a BitTorrent client that uses this metadata to talk to the tracker.

This is very light weight and does not really have much of a discernible network impact.

Peer Protocol

Once the client has located the tracker, it asks for a list of peers for the relevant announcement. The tracker comes back with a bunch of data, but importantly this includes the IP addresses and remote ports of peers participating.

With a list of peers in hand, the client will then attempt to communicate with those peers in order to start exchanging chunks of normally around 16 kilobytes in size. The protocol is fairly nuanced and allows clients to sniff each other out a bit like two dogs (whether they are interested in the chunks each other has, or whether one of the clients thinks the other is unfairly leeching and wants intends to throttle that client back). I wonder if that dog metaphore will make it past StalinJorke.

Filtering BitTorrent

The problem with the conceptual overview above is that you’ll see the protocol is very flexible in terms of end-point negotiation.

• There is no fixed BitTorrent port (true, use TCP ports 6881-6889 by default but anyone who is going to specify infinity for a number of concurrent peers is probably  not going to use the default port range).
• There is no fixed relationship between the tracker and the members of the Torrent. Once the client has a list of peers from the tracker it can continue to exchange chunks of data for a long period of time and blocking the tracker will not necessarily be effective.
• Deep packet inspection does not help when you have Azureus, Bitcomet, and uTorrent all cooperating to implement transport level encryption.

User Behaviour

As I said above, I’ve always advocated the point of least interference in the network in the interests of a better customer experience. This has always worked really well and so it is interesting to note that tech•ed 2009 represented such a massive departure from previous tech•ed events. I am not sure whether or not to attribute this to:

1. the fact we had really high speed Internet connectivity; or
2. the fact we gave out Netbooks and so there were a number of users at the event with PCs who normally would not have PCs in that environment.

Regardless, the really disappointing thing to see what a lack of consideration for other users and then lying about it when cornered.

Original article on ZDNet: http://www.zdnet.com.au/insight/hardware/soa/Backstage-tour-at-Tech-Ed/0,139023759,339298840,00.htm

[ I knew we'd get him on camera at one point, btw the guy you whos head you don't see is Nathan from GCCEC - jorke]

Well we are now under way with the session recording and the pressure is starting to subside; especially now we have a stable platform to work with.

Let me take you through some of the issues we ran into during the setup of tech•ed.

Over the last week we have had countless problems with Expression Encoder 3, this software was to be used in conjunction with the ADVC55 Canopus to capture session recording both audio & video.

The software proved to be unstable and difficult to use.

The encoding would start well, but around the 15-17, 32 minute mark of recording it would crash, freeze and lock up the PC.

At this stage the only we could overcome the problem was to reset the ADVC55 Canopus and reboot the PC; this went on for three days.

We changed and tested every possible video & audio setting available but problems persisted. Expression Encoder 3 would allow you to record past the previous crash times and on preview the quality was great. After pressing stop and reviewing the recorded footage it showed really poor quality images, the only thing working well was the audio. Unfortunately the session wasn’t usable with only audio.

The only thing we could do at this stage of the event was look to a solid program that had as many features as expression 3. The main features we were looking for was stability, video compression and compatibility with the ADVC55 Canopus.

We came up with Windows Media Encoder 9. This saved the day! Upon connecting the Canopus device, the software immediately launched. The software automatically detected the signal being supplied to the Canopus and set all the video compression settings automatically.

Press record and off we went. One hour later the software was still solid, the images were quality, the audio was in sync and the playback was DVD Quality.

We had found a winner. This software is highly recommended in all facets, easy to operate, great playback quality and stable.

- Brian “The Video Guy”

They’ve been turned into public space for Brisbane Park(ing) Day initiative by the Architects of Brisbane office of BVN – Check out the concept pics below – This will be built on 133 Mary Street Brisbane 7-10pm – so swing by and check it out. Tell the guys that are there that your netbook used to sit on those

-jorke

We’ll wrap things up over the next couple of weeks. There is some pretty interesting content coming so please stay with us.

The helpful chaps on the Intel stand managed to come up with signed drivers that correct the issue (significantly higher version number than the in-box ones with Win7 RTM) however they’re still trying to get us the okay to release these. Interestingly they suggested that the work around was to disable one of the radios and the escalation technician said to turn off 5GHz – but please don’t do that.

For now, if you have an Intel wifi NIC, I recommend turning off the 2.4GHz radio as there is far greater channel capacity and bandwidth on the 5GHz radios.

The Netbooks have a different chipset that are unaffected by this issue however you will likely experience better performance if you turn off 2.4GHz.

1. Wifi becomes unresponsive including being stuck ‘on’ so that hardware switch does not turn off the interface.
2. Blue-screen-of-death (only one person on the technology team experienced this).

We did a lot of investigative work and determined that the issue is related to the driver stack in the affected clients. The issue only affects uses who are using Intel wireless NICs. The issue does not affect Broadcom (in the netbooks), Atheros, etc.

We managed to latch on to a very helpful man on the Intel stand in the exhibition who confirmed that this is a known issue where:

1. There are a large number of access points.
2. There are a large number of clients.
3. There is roaming.

Sound familiar?

The underlying issue is a limitation in a table that manages the number of access points in range – when this is exceeded the driver stack will crash. This is exacerbated by roaming.

We turned off CCX4 extensions today and that seemed to helped the issue a small amount by being less agressive in how it manages client roaming.

We’ll turn CCX4 on tomorrow for the keynote and then turn it off for the rest of the event. Hopefully Intel can give us a beta driver that fixes the issue. If we get it, we’ll put it up on the blog.

AWESOME!

It is actually Gianpaolo Carraro, Microsoft’s DPE Director. We did a cook’s tour of the venue for him and his family during set and all enjoyed a nice BBQ that evening as a rare bit of downtime. He’s an extremely good sport for doing this and letting me post it.

The F12 madness:

Jorke taking some servers out for a spin:

If you were trying to create a LiveID from onsite at Tech.Ed yesterday would would have seen a message along the lines of “limit exceeded..” – all our Tech.Ed networks are behind NAT, and although we could have cycled the NAT gateway IP’s around, that would only get us a few hundred go’s.

We escalated to the Live product team, in particular our favourite aussie member of the live team, Angus Logan who helped us sort it.

if you do notice any funkiness like this happening, please don’t hesitate to escalate to the Ops team.

-jorke

I woke up at 4:00am this morning in a cold sweat. And I haven’t slept since. In the wee hours of the morning my brain, as is its want, was mulling a worst-case scenario: “What if we got the image or imaging wrong. How are we ever going to re-image these 2575 machines?”

In our planning, the imaging process was always date constrained. Whilst David Haysom had planned an extra 2 days, just in case, as there are always strange things that could bite you on the bum.

First day of Netbook handout. Today. 9am. Within an hour, we had handed out about 10% of the total.

And a problem appeared. Thankfully not hardware; the opening and recharging of each machine tickled out any weird hardware issues. In fact, our out-of-box hardware failure rate was half of the planned amount. HP repaired these quickly, and all was well.

Boot to nothing. Ouch!

The problem: missing images on a low, single percentage number of machines.

After reviewing our WDS/MDT2010 imaging server and technique by local and Redmond experts: this was deemed not the root cause for the missing images. Another possibility was Netbooks booting in the box, running out of power, and getting into a weird state. Again, this was discounted as there was just no data on the drives.

The Current Theory

The imaging process ends with the Netbook shutting down. From F12 to network boot to shutdown is one keypress: F12 to “boot from network”. At the completion of the multi-cast and unpacking in the WDS/MDT process, the Netbook shuts itself down to a black screen.

On the first two days of imaging last week, the imaging area had power issues: essentially, fluctuating power at the setup trestle tables. This caused machines to be left powered off. To a black screen.

Our thinking is that the black screen at the end of this process led to false-positives. Netbooks that had power issues, left on a black screen, were accidentally deemed “imaged” during the first few days and packed up.

Learning 1:  Leave setup machines with a static, on-state saying “OK” would be our recommendation if you are planning such a large, time constrained install.

The QA process we had during the imaging week was to test every 80 to 100 machines: that is, go through the out of box experience to test build quality.

Learning 2: factor in a slightly higher rate of QA checks than you think necessary.

How are we Fixing this?

Handing over a known good, working Windows 7 Netbook to each attendee is a must. No compromise. Therefore, today, we instituted a quickly developed solution to freshly image these machines.

After re-evaluation this evening, we are dedicating a team to process any machines with missing images. There is also another, faster process that is being investigated that may be implemented (via USB imaging)

So, worst case: we have upset attendees whilst we fix their machines. All I can say is “Sorry, and we are making it right” and I am confident this process will work tomorrow. With a little patience, all will be well.

Best case: our current glass-half-full thinking is that the machines that were handed out today were from the early in the week batch: where we had power issues, and black screen false-positives.

Tomorrow will tell.

Thanks for your patience and understanding.

And if you want to complain or vent at someone: find me.

My name is Nathan, I’m the Technical Services Manager here at GCCEC.  A lot of people ask me what Microsoft need off the venue and assume it’s plugging some computers into the venue’s network and all is done however this isn’t as easy as it sounds.  As you can see from the other posts on backstage we have run into a few hiccups with Tech.Ed 09.

My office at the top of the arena.. 'the loft'

First up,  the guys at codify specified to have their own independent network from edge to core.  In the past they have had some issues using the venues current production network  so the decision was made to drop a separate network in.  Codify and Jomablue sourced the appropriate hardware and we installed all the switches in the MDF and the 13 IDF’s around the place, and used our fibre infrastructure for the back bone. All the main backbones are 1GB fibre and from there it breaks down to trunked 100MB ports to the managed switches out on the floor and breakout rooms. The codify team has network monitoring of the entire network as well as remote alarms .

I’ve also had the fun task of dealing with all the contracts relating to the installation of new services within the building specifically for this event and yes we have a lot of bandwidth with redundant links and if all else fails,  then have  an adsl2 backup (might be a bit slow for the 3000+ devices on the network).  Internally we have been planning for this event for well over 12 months, and technically I have been working for the last 5 months on this event. This event by far is the most stressful technology event we have at GCCEC however the team running the event makes it all go very smooth for us.  There are many things that go on behind the scenes to make this event possible for you guys from all the food (you all eat so much) to the 1000’s of bottles of drinks you will drink.

Another large element of Tech.Ed is the AV production to make it all look cool. Apart from the pallets of computers there is 1 shipping container & 3 semi-trailer loads of sound/vision & lighting equipment setup to make Tech.Ed happen.  There is also a lot of work done in relation to power & load sharing.  Do the math.. 2500 netbooks.. 400 odd PC’s = a lot of power + all the production equipment means you need to do a lot planning around this.  There has been a team of workers working in the keynote room for 3 days installing power boards into the seating so you can recharge your netbooks whilst you are in the keynote room and this is just one example of the things that go on outside the technology realm to make Tech.Ed what it is today.

Foxtel on the BIG Screen

- Nathan

There will be a time, after TechEd, where we triage our original planning: time, budget, hand-offs, tools and techniques. As a Quality Manager at one stage in my working life: project review, documenting them and absorbing learnings are critical.

On a lighter note this Friday: here are two youtubes for your viewing pleasure:

So you can get a sense of scale of our operation our professional photographer came in and shot some great photos..

Moving on today we’re starting building the showcase with our great HP demo servers – more on that later

- jorke

First day on site – and first post! My name is Kyle Rosenthal – and I am the content manager for HynesITe, the group that puts together the Hands on Labs(HOL) and Instructor led labs(ILL) for TechEd – here in AU, North America and now in Europe. This is our 5th TechEd Australia – starting all the way back at TechEd Canberra.

So – what goes into the deployment of the Hands-On area? A lot of time and effort! Basically we will be deploying to the HOL/ILL area on a total of 5Tb of data – to 180 machines – in 3 days.

This starts at my place generally for TechEd AU – Once a year my wife’s patience is tested as I take over the kitchen table and suck a year’s worth of power in one week.

This year is sort of different – we have moved to a new house and I have invested in a new table.

However – when Kim and Ted arrived – the kitchen table was back in use!

The setup this year for us is Windows Server 2008 R2, Hyper-V, and our custom support front end that will assist you the attendee in getting to your lab as quickly as possible. This does mean that in the process of getting ready  for the event we will start, save, snapshot and export about 300 virtual machines. In addition we then will touch all the lab manuals that help you to get through your lab.

To do this we have 8 machines in the back end that form our core infrastructure, with gigabit network everywhere.

So we loaded up the car and headed to the coast, the drive from Brisbane to the Gold Coast meant another hour’s worth of work time available for Kim and Corey – one in the front seat and one in the back seat – with laptops in use. This meant that in true geek style when a file needed to be transferred between their laptops – no usb device was needed. So – if you happened to be beside the Pacific Highway and saw an Ad-Hoc wireless network named “Driving” appear and then disappear – that was us.

Now we are complete and on site ready to start the next stage of the deployment.

Office in setup

Ted at work

More as we start our roll out.

Cheers
Kyle

After meeting Jack Morton’s Patrick, and donning a hi-vis vest – its into the Technical Learning Centre to witness the setup of the 2575 HP Mini-notes/netbooks.

There are about 13-14 long trestle tables, each with and average of 20 machines laid out. 8 staff are out and about unwrapping each box, putting it on the table, and initiating the Windows 7 TechEd image. This is a sight to behold. It feels somewhat like an Industrial Revolution-era factory. A study in time and motion.

Time is critical now. We cannot delay TechEd, nor stop the setup of the Technical Learning Centre – so its all hands to the wheel to complete the imaging tonight. Being a production process that is now in production, and costing per hour – it is way too late to change anything. Either the image, the process or technology involved in the imaging.

To give you a vision of what is going on – here is a quick youtube I created within 10 minutes of arrival.

Telstra has a major ongoing outage this morning: http://forums.whirlpool.net.au/forum-replies.cfm?t=1274075

Early reports are that it has taken out Next G, BigPond & Telstra Internet Direct – and certainly the 100mbps fibre service at the venue is dead.

Luckily, we are running APC off our redundant link which is a private circuit back to Pipe DC2 we blogged about earlier. This service is using different intercapital backhaul to Telstra so fortunately APC is completely unaffected (funnily enough we might patch the venue’s network over to ours so that they have Internet access ).

Telstra will be in tomorrow to install the 500mbps link but it is good to know we have the diversity up our sleeve in case something goes catastrophically wrong like this. Everyone thought we were bananas for ordering two 7200VXR routers for APC at Port Douglas last year … only to find the first one blew up within an hour of being turned on.

One of these things is not like the other one…

We recently mentioned that we completed deleting the WLC from the wireless equation at the venue. In rolling our the custom configs to the 50 autonomous devices we found that one of them returned an error.

One of these things is not like the other ones

It turned out that the radios were mounted in the chassis back-to-front. This changes the way that IOS enumerates the radios so that dot11Radio 0 and dot11Radio 1 are the opposite of all of the other APs. Consequently our templates scripts would fail on this unit.

Luckily the radios are completely modular with respect to the chassis:

1252AG with modular radios removed

And so popping them around the right way did not involve an RMA.

1252AG modular radio slot

I was rather impressed by the whole thing (we normally use 1232AGs which are not like this) … there had to be at least one sunny/bright point from these units given all of the brain damage they’ve inflicted on everyone over the past three months.

We previously posted performance metrics that were from a controlled set of tests with the APs patched into the venue network. We just did final tests yesterday (31 Aug) with the wireless network using our replacement Catalyst edge and core and the performance is consistent with what we saw earlier (after our rectification work, not before ). A good lappy will get 8MB/s (megabytes not megabits) from a 144 mbps associated link speed.

We (well David Eagles from iVolve) made some final recommendations regarding the RF utilisation on 5GHz. In the end we made the call to limit the maximum channel bandwidth on the 5GHz network to 20 MHz. This will ensure that there are 12 unique channels available at the expense of limiting the radio link speed for any client to 144mbps (instead of 300mbps). In practical terms this means a maximum download speed to your netbook of 8 megabytes per second instead of 16. Besides, it was a bit pointless having 300mbps radio link speeds supported when the edge of the network is 100mbps.

So in the final analysis we’re able to drive the wifi hard enough to saturate the RF – as it should be.

Oh yeah, and on those 400% throughput improvements we mentioned, there was one AP where we showed 700% improvement by making them all autonomous. Yay!

So we’re basically in production now for the folks attending the Australian Partner Conference. There is still a lot of work to do before tech•ed 2009:

• heaps of imaging work and issues that are keeping Jorke up until 2am
• bringing up and cutting over to a new 500mbps Internet service that is going in on Friday (yes, that is cutting it fine)
• Setting up the load balanced RRAS servers to cope with the expected load at tech•ed (we’re just using one for APC as that’s all we need to translate 100mbps)

We had to work through a few interesting EDNS0 issues with RRAS over the past few days – hopefully I’ll get a chance to write that up before Haysom overwrites those parts of my brain with helpdesk tickets.

Some random photos of work  starting in the main halls follow.

NetBook imaging

Rigging begins for tech•ed

More tech•ed rigging

Empty trestle tables, waiting for PCs in the Technical Learning Centre

Show machines setup

HP 2140’s being imaged and recharged

Jorke in his High-Vis Vest. How cute is he?

These photos were attached to an email from Simone Hulse:

Well it’s been an exciting few days for the build crew at Tech.Ed.  The technology team are very well established as they have been here for over a week.

• The mini-notes are imaging
• The show machines are complete
• CommNet and recharge is set in the foyers ready for APC (editor: Australian partner conference)
• The TLC is well underway (editor: TLC = Technical learning centre)
• Windows 7 Ideas Exchange lighting truss is in – and will be incredibly impressive once the lighting is complete
• The showcase banners and lighting rig is in
• The APC dinner staging is almost complete
• The rig is completely set in the roof, so now we can start building the expo from the ground up.
• Today is all about data cabling and power, tomorrow is about flooring and more cabling and then we build the expo on Friday

For those of you who are not hear yet.  The weather is warm in the day 22-30 degrees, but gets cool at night…. bring layers (and closed shoes!)

Very exciting.  Another edition later in the week.

In an event such as tech•ed, managing data cables is in itself, a job which could challenge the best of logistical minds.  There are over 1,200 Cat5e UTP cables to install or over 8,200m to lay down, plug into switches, test, secure to desks & stage plinths and of course plug into computers at the other end. Don’t forget the task of recollecting them all at the other end in a manner that doesn’t result in the biggest bunch of copper and plastic spaghetti in the world.

To help with the taming of this cable mammoth, a colour coding system is used to make what would normally be a challenging task just that little easier.  With experience, we’ve established the best combination of lengths. The colours were chosen based on what are standard available colours without the need for excessive custom makes :

• Grey Cables : 2m
• Yellow Cables : 3m
• Green Cables : 5m
• Orange Cables : 8m
• White Cables : 10m
• Purple Cables : 15m
• Blue Cables : 20m and over

The result is cable installers and the technology team can quickly identify the appropriate cable. It hasn’t always been like this. Before we had a colour coding system for the lengths we had a huge wastage issue with excess length. So much a number of years ago we reduced the cabling costs by 30%.

What happens to all of the cables after the event? You would appreciate the cost involved when replacing cables, not to mention the environmental impact. To maximise integrity, cables are used for no more than 2 events and so costs are amortised. If cables have been in trafficable areas, such as where trolleys and feet run over them, they are replaced after each event.  At the end of the event, the task of recollecting, rolling and sorting patch leads is job that may leave you chrysophobic, but the method of colour for length has made our “Where does that cable go” jigsaw easier to piece together.

• Making Wireless Work
• Diagnosing and resolving extremely high RF utilisation
• Resolving a Hunch– Wi-Fi Performance @ GCCEC

Insufficient Traction Error

We had been working extensively with a number of suppliers to get the WLC issues resolved. To be honest, I reached the end of my tether some time ago as every call seemed to revolve around either co-channel interference (so much so that is now the standard butt of jokes on the tech team … car won’t start? co-channel inteference :\), or some discussion of why there were methodological defects in the reports we’d written rather than just focusing on the obvious @#(*&@ issue.

We’d provided really rock solid documented reports showing throughput tests done in isolation with RF scans at the same time to show no interference.

At no time could we get more than 16-20% of the RF throughput we should. We could never saturate a 100mbps edge port even with 300mbps association speed. Despite our best efforts we have really been worn down by the whole experience. At one point I was told on a teleconference (paraphrased) “We would be amazed if there was a defect of this nature … especially so if someone from Queensland found it.” I took offence at that since we’ve had electricity and flushing toilets here since 1997.

Plan B

As mentiond in the linked articles above, we had a plan B all along to remove the APs from the WLC and install autonomous firmware on them all.

David Eagles just finished implementing that plan today (there is a bit of clean up to do tomorrow) … but well whaddaya know Jethro? In between tending to the cows and riding kangaroos, it turns out the country folk from Queensland might have been right all along:

Percentage of Theoretical Throughput, pre and post rectification work

Left is before, middle is after our work, right is theoretical throughput.

Look ma! Weeze yokles from North did good!

500% improvement FTW.

Life would be a lot simpler (well mine would anyway) if people took the time to read and understand the reports you put in front of them and work collaboratively on quality commercial outcomes.

davidc

TechOps (or will be) - This is the room from which we will run all technology related technical activities during APC and tech•ed 2009!

It fills me with a combination of dread and anticipation – not sure which is overriding. Anyway – about the Internets – we have an order in play to get a 500mbps service installed for tech•ed. We have also been installing a private 300mbps circuit between the Gold Coast Convention and Exhibition Centre and Pipe Networks DC2 as a diversity option and Plan B. For all our problems with City Pac and microwave options, it turned out that, upon actual examination of their records, Pipe were surprised to find they had fibre running RIGHT PAST THE FRONT DOOR.

We’re commissioning the Internet services on this link tomorrow (Tuesday the 25th of August) with the helpful guys from Over The Wire (the data centre we’re procuring the Internet connectivity from).

I’ll post some of the facts and figures from the commissioning exercise when we have collected the performance data.

P.S. Macca *possibly* could have made the effort to suggest I relocate. “Yeaaah mate … I heard the water in the pipe gurgling…” @#&^%@#@!

This post briefly explores the sorts of issues we have with DHCP on a large scale temporary network, and the sorts of things that go wrong.

Quick recap

DHCP is an application layer protocol that allows clients to negotiate for an IP address lease. The protocol also allows clients to obtain additional configuration parameters as a part of the lease process (e.g. an embedded device client might obtain the IP address of a server from which it can subsequently download firmware).

You might think that DHCP uses some sort of low-level trickery (hey, it must! You don’t even have an IP address yet!) but in fact it is a very simple UDP protocol that works via broadcasts. So, you are in fact using IP to obtain an address so that you can use IP.

The protocol works roughly as follows:

1. The client sends a DHCPDISCOVER broadcast in order to locate all DHCP servers on the local subnet. The client sends a list of options in this request for items it wants back from relevant DHCP servers.
2. Any and all servers respond to the client with a DHCPOFFER response. At the time of sending the offer, the server(s) will set aside an IP address for the client. The offer also includes additional dhcp options that have been requested by the client.
3. The client receives all of the responses and then chooses one to request as a lease (DHCPREQUEST). Once that is sent, any server that is NOT the server that offered the address that the client chose will return the offered address back to their pool for use by other clients.
4. The server that recognises the response (via a unique transaction ID used throughout the conversation) will respond with a DHCP acknowledgement and ‘lease’ the IP address to the client for the time specified in the policy on the server.

When DHCP Servers go bad

Every tech•ed there will be someone who brings:

1. A dinky little home router with a view to starting their own rogue wireless network
2. An entire enterprise infrastructure in VMs, including DHCP servers and so on, with their network adapter set to bridge the virtual switch with the local network

In our experience, we’ll get at least two to three rogue DHCP servers popping up at each tech•ed.

As you can see from the brief protocol overview above, there isn’t really any authentication or trust between the clients and the DHCP server. Therefore any rogue DHCP server will be treated with the same authority as our own and start handing out IP addresses for a different subnet and with a different router and so on. Once this happens, a large number of clients can aquire an incorrect IP address quite quickly and this creates a massive PITA for the helpdesk guys to tell everyone to do a release and renew to get an IP address that works.

Trusted and authorised servers

You might have noticed that when you use Windows DHCP server there are a few behaviours that might strike a networking traditionalist such as myself as odd:

1. If your DHCP server is a member of a domain, it will not start until someone with administrative credentials ‘authorises’ the server to serve addresses. You do that using the DHCP server MMC snap-in.
2. If your DHCP server is on a standalone server, and it is running, and it sees another DHCP server that is also running and that server is a member of a domain … then your standalone DHCP server will shut down.

I believe that both of these approaches are a little bit silly for a number of reasons, the primary of which being that rogue DHCP servers are rarely Windows Servers. A little busybox machine or home wifi router isn’t going to play in the above scheme and so you’re still exposed to IP addressing integrity issues.

We’re really dealing with a layer 3 network security/integrity issue here and the correct place to solve it is in the network itself, not in the operating system of some select servers that exist on that network.

DHCP Snooping

DHCP Snooping is a catch-all term that covers a number of techniques for ensuring the security and integrity of certain aspects of the edge of your network. You can think of DHCP Snooping as a type of light-weight firewall that sits in every switch on your network.

The main feature we’re interested from the various features in the DHCP Snooping arsenal is where by individual ports can be marked as trusted or untrusted for the purposes of DHCP traffic.

We are deploying a large number of Cisco Catalysts throughout the venue for the event. The standard units in the IDFs will have 24 x 10/100 copper ports, and 1 x fibre GBIC to connect the switch back to the core of the network. The standard configuration we will deploy to these switches will set all 24 10/100 copper ports as untrusted. We will nominate the fibre GBIC as trusted.

The result of this configuration is that packets such as a DHCPOFFER received on one of the untrusted ports are simply dropped. They never enter the network. It does not matter if the device on the edge port is a massive Windows Data Centre Edition server or a $89 home wifi unit … the treatment is at the packet level and the result is the same.

Problem solved!

Since we started insisting on Catalysts in all untrusted/edge scenarios, we’ve never had a repeat of the issues of the tech•eds of many years ago. The solution described above is implemented at the right place in the OSI networking model, and at the right place in terms of the overall topology of your network.

If you have rogue DHCP server problems then it is well worth looking into the benefits of DHCP Snooping. There are a lot of features under that name and I’ve only briefly touched on one that is relevant to the event here.

Further reading:

• http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/snoodhcp.html
• http://tools.ietf.org/html/rfc1531
• http://en.wikipedia.org/wiki/DHCP_Snooping

I recommend the first link as a very comprehensive overview of all of the aspects of DHCP Snooping that you can add to your network security arsenal.

The week started with an integration meeting. A group of us from across  the project (logistics, construction, technology, catering) lock ourselves away for half a day. We review each task from the moment the first person sets foot at the venue to the moment the last truck departs at the conclusion of the event. It’s a mind bending job, but on a project like this with literally hundreds of people making things happen, it’s an important milestone to have this integration meeting. You can imagine we always uncover a few issues where the schedules haven’t matched up entirely during the previous months of decisions. The document, with over 1,000 line items, then undergoes updates by everyone and then we come together to do it all again later this week.

Last week ended with another important milestone for the technology team. We locked down the technology requirements / change requests on Friday night. While the project owners signed off the statement of requirements months ago, we obviously have a period where we need to cater for changes to the needs of the various event owners and Exhibitors. This week it’s all about our documentation, training crew, preparing final briefs and this Thursday we send the first shipment of technology gear from Sydney to for the venue.

Pop over and have your say.

At this stage we should note we leaving the Netbooks open for you to install whatever you like; Microsoft or non-Microsoft. These are your machines – and we’d love you to give Windows 7 RTM a really good test.

Everyone loves fast Internets. Providing really fast Internets at tech•ed this year has provide to be a complete pain in the proverbial.

Give, and ye shall receive (unfortunately)

The Gold Coast Convention and Exhibition Centre has a permanent 100 meg service to Telstra Internet Direct. We know from last year that we were saturating the 100 meg service at the Sydney Convention Centre (we actually put on more bandwidth during the event) and so with that experience, and 2500 people wanting to download the lastest build of Chrome Vizact we know we’re in for a rough time if that is the only connectivity option available to us.

Our first port of call was obviously upgrading the service in the building. We learned however that that would require an infrastructure upgrade from Southport back to Brisbane. The original estimate for this was … lets just say well and truly out of our price range (we could have bought another 2500 netbooks, but we did manage to get it reduced significantly) however with the GFC and what not there is no way we could budget in the hundreds of thousands of dollars to ramp a 100 meg service up to a 500 meg service.

As luck would have it (or not as you will learn), On The Net has a dedicated DWDM fibre service in the City Pac building at 2 Miama Keys (next door, more or less). In the past few weeks we managed to do the following:

• Gain an agreement from On The Net that we could use the service in the City Pac building and that they would provide a fully managed layer 2 service from there back to Pipe Network’s DC3 or DC2 in Brisbekistan. Once you’re in a Pipe facility your connectivity options are opened up considerably as they’re telco neutral.
• Gain approval from Gold Coast City Council together with an affordable cabling estimate for us to run temporary fibre from the convention centre, across the bridge that connects T.E. Peter’s Drive to the little island to the left of the convention centre, and then hop off that onto a council light pole and into City Pac.
• Over The Wire then organised temporary upgrades with connectivity for the Australian Partner Conference and then a burst up to 300 mbps for the duration of tech•ed. Brent Paddon from Over The Wire had even orgnised some temporary upgrades on the Brisbane to Sydney intercapital link from one of their upstream providers for us.
• We received a verbal agreement from City Pac for access to their MDF and data centre.

“Sounds Great!” I hear you say … “What could possibly go wrong?”

City Pac went into receivership on the 31st of July.

I’ve spent the last week grovelling to the morticians at PPB Receivers who are doing the wind up (nice folk, Receivers … not). We’ve tried everything including offering to rent the space, providing an indemnification from Microsoft. David Haysom even tried to get a call set up between Gianpaolo Carraro (Microsoft’s DPE Director) and the CEO/MD of PPB – no dice. They just want to put the place on the market.

It is an understatement to say that I am annoyed that the team had the entire shooting match organised and we’ve come unstuck at the door to an MDF across the road from the convention centre.

Where to now?

I’ve spent pretty much the last ten days with a phone stuck to my head and talked with everyone with an Internet we can use. A number of options have surfaced, two of which are reasonably firm committments from a supplier that we can do 400mbps microwave from the comms deck at the convention centre to here:

Corporate Centre 1 as seen from the Gold Coast Convention Centre communications deck.

We’d jump onto the roof of that place, then do another microwave link across to Royal Pines. The other plan is we do more or less the same thing to get to Austar’s building. Both of those facilities can provide the fibre back to Brisbane.

There is a final terrestrial option that we are exploring in the next few days and I’d really like to see that come to fruition over using a radio solution. I’ll post an update if we manage to pull that off.

If we do end up with a radio solution to get the extra bandwidth, knowing our luck of late we’ll end up with a freak dust storm/storm/aliens attacking during the show.

The fallback scenario is of course the Telstra Internet Direct service that is already there – and that is not a bad outcome however it might mean congestion at peak break times.

Anyway, we have plans hatching and I’m sure we’ll be able to pull something off.

We’ve not been defeated at a tech•ed yet … though this one is trying my patience.

we shall fight on the comms deck, we shall fight in the IDFs and in the MDFs, we shall fight in the data centres; we shall never surrender.

“It won’t cost that much…”

I have to say, i’m constantly impressed by the commitment these guys have to quality outcomes and the sheer amount of effort to every aspect of the delegate experience. There are quite a few unique challenges that mandate we need super speedy internets; such as the thought of 2500 HP Mini-notes turned on at once to Windows Update or Youtube.. and i’m sure over the next series of posts you’ll see the huge amount of effort the guys have put into giving you the fastest internet connection for tech•ed Australia ever.

- jorke

All of this said, the following LOLCAT appeared to me today. Don’t panic. There can be no delays.

see more Lolcats and funny pictures

[ The following performance analysis was submitted to tech•ed backstage by Arpan Gupta and the RRAS team. They own the RRAS/ipnat.sys components of Windows 7/Windows Server 2008 R2. Given that we're using it for address translation at tech•ed 2009, it is rather handy having them on board. We challenged the RRAS team to validate using RRAS as a NAT solution for 3000 concurrent clients; here is their excellent and helpful response. They did all the hard work with a physical simulation too! -- David ]

Background

NAT protocol is a well known solution to provide simplified internet connection sharing across multiple clients. Theoretically NAT supports 65536 TCP and same number of UDP address-port mappings over a single public address. Practically there’re some hardware bottlenecks which limits its performance. NAT, which is a part of ‘Routing and Remote Access’ services that come as a part of NPAS role with WS08R2, allows multiple hosts to connect to the Internet via a single or multiple external IP addresses. In home networking scenario, number of machines hardly crosses a limit of hundreds. But the number may reach to thousands if we deploy RRAS-NAT in enterprise networking. In some middle scale technical meets or conferences we require to manage 3000-4000 clients behind NAT. With RRAS-NAT we can easily achieve this target. A DHCP server can be used to provide private addresses dynamically along with, DNS server & default gateway (NAT private interface) and NAT will forward the packets to internet. We can also distribute the load via deploying multiple NAT boxes with unique DHCP address scopes. As said earlier, RRAS-NAT also supports a scenario where ISP assigns multiple external IP addresses (instead of one) but in this post we’ll target single external IP address and list out its performance.

Hardware Specification

For a set of 1500+ clients we’ve tested on a WS08R2 machine with below specification:

• 64 bit Quad-Core with 2GHz Clock processor
• RAM – 8 GB
• 2 NICs (Gigabit Ethernet)
• Assuming that external interface supports sufficient web access bandwidth (= Number of clients X Required average bandwidth per client).

Measured Performance

Use case scenario: 1500⁺ clients each having 5 open internet sessions and each of these sessions consumes 20 Kbps (Total 100 Kbps) bandwidth.

Result:

• We’ve observed up to 100 Mbps average data transfer speed across public interface of NAT flawlessly.
• With 1100 mappings (each of 100 Kbps sessions) CPU consumption was measured in order of 15%~20% and for 1500⁺ it was in order of 25%~30%.

Steps to configure a NAT box:

• Connect external NIC (Say X) to internet and assign a static IP A.B.C.D to another interface (Say Y).
• Install RRAS role and configure it via NAT path.

• Choose ‘X’ for internet NIC and ‘Y’ for intranet NIC.
• Install DHCP role and configure it. While configuring bind it to ‘Y’ NIC.
• Add a scope with 255.0.0.0 mask. (Please note that A.B.C.D should fall under it.)
• During various steps fill A.B.C.D as preferred DNS server & default gateway.

• Now connect the clients to ‘Y’ NIC using hub or switch.
• Clients will get IP address, default gateway & DNS server through DHCP server and NAT will route them to web.
• In case of more number of clients, deploy & configure more WS08R2 boxes. Please use unique address scope for each DHCP server.

Acknowledgements

David Connors, from TechEd, who came up with this requirement. R.Sai Ganesh, who arranged & configured required test setup. Mahmoud Elhaddad for providing technical details about IPNAT driver.

Further information

You can stay up-to-date with the RRAS team by pointing your browser or RSS reader here: http://blogs.technet.com/rrasblog/

We left Brisbane bright and early on Thursday the 16th of July to spend a day with the guys from GCCEC to get to the bottom of this latest issue.

Step 1: Knowing what the first step is!

Random reboots, firmware updates, and twiddling of settings does not constitute logical fault finding. If you think there is a problem with a system there is only one place to start and that is at producing a clinical and unambiguous statement of the problem at hand. We almost always work in tricky multi-supplier scenarios at these events and so it is really important to reduce the problem to a clinical document we can all look at so you’re not fingering individuals or companies – but rather working together with all parties to nail the main pain.

With this in mind we set out allocating the first half of our day to an entirely tedious but necessary exercise of conducting a complete access point by access point, radio interface by radio interface survey of the venue (by hand ).  The goal of the survey was not to measure the RF characteristics of the wireless network (been there, done that) but rather the real-world throughput characteristics.

Step 2: Sanity check the fibre/copper network and backhaul before looking the wifi performance

GCCEC’s copper and fibre network is a relatively modern star topology of a single gigE fibre core servicing a number of 10/100 copper edge switches with fibre backhaul. This is good as exhibition venues that have grown over time tend to have a lot of zany and miscellaneous long fibre runs with random Vendor X fibre transceivers all over the place making you unsure of what exactly it is that you’re measuring. The GCCEC network on the other hand is simple to test definitively due to a sane overall architecture.

Additionally, the MDF that houses the core features in-room Foxtel. I could have done with that in the basement of the Sheraton Mirage at Port Douglas for last year’s Australian Partner Conference.

GCCEC MDF - Rather impressive as it is the first machine room I've been into with Foxtel installed, though I couldn't find the bar fridge !

Testing Rig @ MDF/network core

We deployed a high performance x64 machine with 4GB of RAM and IIS7 directly connected to the core switch via 1000BaseTX copper. We deployed several files to this IIS7 server being 1 megabyte, 10 megabytes, 50 megabytes, and 100 megabytes in size.

Consistency in client testing

We used the same clunky Lenovo (waves at the other David ) for all client tests to ensure that we were always using a consistent platform on both ends of the equation. This slowed us down somewhat because it meant we could not split up to make things faster, but I felt strongly that being able to say that all results have as high a degree of consistency as possible was very important. We configured the Lenovo for maximum power wastage performance on all interfaces to ensure that results would not be inconsistent if we were to conduct some tests off battery versus off mains. Wherever possible we used mains power for tests.

Testing procedure

We used a simple script with wget downloading the test files from the testing rig in the MDF three times. We recorded these results together with some other data at each test point (more on that later).

Validating the test rig and core

A Lenovo notebook was directly connected to a port on the core switch via 1000BaseTX copper. We did several transfer tests between the test rig and the Lenovo receiving approximately 800mbps of real world throughput which is pretty good with a notebook as a client any day of the week – and certainly sufficient for our tests.

Validating the edge switch at the first test point

The last step before heading off on the Tedium Express was to test the first edge port that services the first access points we were to test. Again, this was the case of running the test scripts from the Lenovo directly connected to the edge switch on the same VLAN as our private test network. These reported 99.99% of the maximum throughput one would expect from a 100 meg link – so that was certainly very good.

The performance of the copper and fibre network validated as satisfactory.

The next step was to confirm whether or not there was a systemic issue with wifi performance (as per my hunch) or whether there was a simple problem with a loose TNC connector on the one of two APs we saw as problematic during the last exercise.

Anyone who has worked with me knows that I’m a bit like an old dog with a bone when it comes to pushing for a quality outcome and ensuring that things do what they say on the tin. Given that I tend to push damned hard for what I think is the right thing to do – it also means that I was experiencing rather some trepidation at the survey as I’d possibly have some explaining to do regarding billable time if my hunch proved to be false.

Step 3: Conduct the wireless survey

The wireless component of the GCCEC network is comprised of the following elements:

• Cisco 4402 wireless lan controller
• 50 x Cisco 1252 abgn wifi access points
• Each access point has a 2.4GHz and 5GHz radio interface – you can think of these two radios as being akin to having two separate Ethernet ports in your PC … if you’re troubleshooting they BOTH need testing.

Nathan at GCCEC did a wonderful job of building us a mobile testing rig out of a road case, trestle table and ratchet straps + onboard GPOs a 50m power lead and anti-slip mats! Nath – you’re a legend! Christened the Tedium Express, this was our working environment as we tested each AP.

Toot Toot! All aboard the Tedium Express!

We wanted to ensure that we could conduct a single sweep of the entire venue and not have to go back out to reassess it as we only had funding clearance for a day on site. With this in mind we prepared a comprehensive spreadsheet listing:

1. each copper MAC address
2. each base radio MAC address
3. the associated IDF and edge switch port

The process we used was to park the mobile rig under (or as close as possible to) the antennas under test and collect:

1. the advertised radio link speed
2. the current channel allocation
3. three real-world download performance data points
4. Repeat steps 1-3 for 2.5GHz and 5GHz.

This meant 50 access points * 2 radios * 5 data points = 500 collected values.

The first dozen or so radios took nearly an hour and a half to collect the data from, primarily because the Lenovo was taking so long to switch between 2.4GHz and 5GHz radios. We therefore chose to survey the venue twice, once at 2.4GHz and a second time at 5GHz. Despite having to cover twice the distance we ended up doing the collection much more quickly as we weren’t waiting on client WLAN configuration.

Out of interest we chose the Lenovo as the client test rig as the ThinkPad connections software allows you to view a series of access points listed by MAC address and then choose a MAC address to associate to as your preferred access point. Given that we had a pre-prepared list of all of the access points in the venue together with their MAC addresses, this made validation of what we were testing at any point a snap.

Also, David Cormack from CBO was helpfully driving the wlan controller software and printed installation maps to further validate what we were doing. The whole process worked pretty well but the ‘law of big numbers’ meant that it was going to take a while no matter what – there were 100 interfaces to survey and that will never happen in an hour.

The entire survey took from approximately 0830 to 1430 to complete with four people.

I see dead radios

During the course of the survey we found the 5GHz radio in AP-24 to be faulty. It was extremely difficult to obtain an association with the 5GHz interface and when we did the radio link speed was 9mbps and real world download speeds were 3-4 kilobytes (no, that is not a typo) per second. We had to bench it to work out what was going on.

I don’t have a problem with heights per se as I’m fine on the top of a mountain or on a high rise balcony or observation deck. What I do have a problem with is these damned scissor lifts that are rickety and dodgy and wobbly. If that is not bad enough they actually can drive around when you’re 10m in the air.

I edited out the bit where we started percussive maintenance (I’m talking about the AP, not your forehead David Cormack ). As an aside, it is a pretty good view of the hall from up there. During the video you’ll hear a faint rasping sound … that is the GCCEC staff scraping every last skerrick of masking/gaff tape off the raw concrete floor to ensure the venue is spotless for the next show. Keep this view in mind when you’re walking around the nice carpeted expo halls with fancy games and Xboxes and other stuff … there are innumerable venue and staging guys really go to an amazing amount of work to transform that concrete shell into the event you know and love. While we did our survey they blacked out the entire hall to do metered comparisons of a new super-bright ‘green’ compact fluorescent lamp for the hall to ensure they’re a satisfactory brightness and colour temperature compared to the existing metal-halide units. In short – there is just so much stuff that goes on to prepare the minutiae of detail at these events that hours and hours of labour go into light globe selection alone – let alone the what technology team do.

Here’s another view of the area tech•ed will consume from ground level (opposite end of the hall from the previous video):

Step 4: Analyse the results

I digress … where was I? That’s right – the survey.

So after a couple of laps of the venue we came up with this:

GCCEC Wireless Throughput Survey Results

Some of you might see quoted figures of 2 megabytes or 5 megabytes per second and think that is pretty good for wireless, and truth be told if you were getting sustained rates like that out of some dinky home router I’d probably agree. Our expectations are somewhat higher and you have to remember that we’re really the only people at the venue there are only two acceptable performance outcomes:

• saturation of the RF
• saturation of the edge port

The results paint an entirely different picture though. On average, the RF segments of the network are only providing 17.80% of associated RF link speed on 2.4GHz and 16.14% of associated RF link speed on 5GHz.

The other problem with these results is that they are highly inconsistent. Across each of the three runs the individual tests could vary by double or half.

So it turned out that my initial hunch from the last day on site was correct … and we have a serious wifi performance problem on our hands. Ruh Roh!

Step 5: Find the culprit

By this stage we had chewed up most of the day. We had ascertained:

• there was a serious performance problem with wifi and we had quantified it in a clear and clinical way as was our original intention.
• that the problem lay between the access points and the wireless lan controller, or possibly the way the wireless lan controller terminated in the venue core switch.

While this was progress, we really wanted to provide more definitive findings that would further narrow down the source of the problem. I must admit I was very disappointed at the prospect of leaving the venue with only problems identified and no positive prescriptive advice.

Further isolation testing

We had AP-24 in hand so we retreated to the centre of the venue and proceeded to shut down each of the wireless access points. After doing this we patched AP-24 into the wireless lan controller VLAN and reconfigured it in 2.4GHz 802.11n mode (turning off it’s flaky 5GHz radio altogether). We re-ran the same test suite and found performance to be the same as any of the other access points in the venue during our initial survey. This was helpful as it showed us that the problem was not in some way related to load on the wireless controller or the number of access points talking to the controller.

Reflashing AP-24 as autonomous

You will recall from the previous article that we discussed how the access points in the network are running in lightweight mode and so can only run in conjunction with the wireless LAN controller (see this if you’re interested: http://www.cisco.com/en/US/products/hw/wireless/ps430/products_qanda_item09186a00806a4da3.shtml). We decided it would be nothing if not informative to ‘downgrade’ the access point from lightweight mode to autonomous mode.

Autonomous mode firmware gives the access point a more complete IOS feature set and allows it to bridge the 802.11 (wifi) and 802.3 (Ethernet) networks without a controller. After some mucking around we re-flashed AP-24 into autonomous mode and directly connected it to our test VLAN bypassing the controller VLAN all together.

Surprise surprise!

AP-24 Throughput with Autonomous Firmware - Now we're cooking!

With a link speed of 144 mbps you can expect approximately half of that in terms of real-world download speed, which would be 72mbps in this case. Let’s do some sums. Remember that hard drive sizes and network throughput are universally measured using base 10 arithmetic (1 megabyte is 8,000,000 bits), while file sizes on disk and memory consumption and so on are measured using base 2 arithmetic  (1 megabyte is 8,388,608 bits) … I know, I know I could use SI units for kibibytes and kilobytes but most people reading this would have no idea what I was talking about.

Anyway, ignoring framing overhead and so on to make this easier:

1. 72 x 1000 x 1000 = 72,000,000 bps
2. 72,000,000 / 8 / 1024 /1024 = 8.5ish megabytes of files downloaded per second.

Above we see 9.8, 8.9, and 9.1 as the figures from the test of AP-24 in autonomous mode. This is what I would call EXCELLENT!

Conclusion

We made a lot of progress in the day on-site.

1. We proved and documented that there is a systemic problem with wifi performance at the venue.
2. We found and removed a dead AP.
3. We proved that the edge and core networks are functioning as expected.
4. We proved that the poor wifi performance was not related to channel/RF intereference by our first isolation test.
5. We proved that the access points provide excellent performance in autonomous mode.

In short, we turned a hunch into a known and well documented problem.

We provided a written report of our findings back to the venue and they have taken it back to the installers to get the core issue looked at and the installers have basically said that the issue will be resolved ‘no matter what’ – Thank you guys – I love commitment.

We’ll provide a further update when we know definitively what the root cause was and when the matter is corrected to our satisfaction.

Don’t worry, we have a plan B if the above doesn’t get resolved. Fortunately for you this means you will get as good performance as possible at the event. Unfortunately for me it means that damned scissor lift and reflashing 50 access points to autonomous mode with individual configurations before the event and back again afterwards. Hmmm David Eagles; What are you doing this time next month?

Maybe I should step back a bit, back to November last year. Or further back.

So, this is my 10th Tech Ed as Technology Project Manager. Honestly I have the coolest job on the project (Kleefie always said he did). My role is to work across the business owners, sponsors, venue and technical guys where anything technology is involved. Pulling a truck load of bits together into some type of plan. That is under the watchful eye of Jorke as the technology lead for the project and David Connors as the Engineering lead.

We have a structured 5 phases approach to the delivery of large scale event technology. Long before you get to know about the good stuff like mini notebooks (possibly the coolest take away ever. So much so it has other Tech Ed events around the world wishing they had the idea first), we conduct a feasibility phase. Various Microsofties, the event agency team and the Engineering team comes together to look at how the project could work be at the proposed venue. We don’t work with a load of detail at that stage, but we look for the obvious pitfalls around timing, location, connectivity etc. This phase brings with it long brainstorming sessions and usually lots of research tasks. That in turn means I and a swag of others spend ages writing reports on the if, but, and may etc. All necessary stages for the business guys to make the big decisions. Once the decision is done setting dates, locations etc in stone we start the longer road of understanding what the business folks need the technology to deliver.

So, no Jorke I’m not going to be the International Man of Mystery and neither are the rest of the team. Stay tuned and over the next few weeks we will share more of what’s going on.

- DavidH

Cisco Compatible Extensions are a set of additions to the 802.11 wireless standards that cover a myriad of topics as discussed here:

http://www.cisco.com/web/partners/pr46/pr147/program_additional_information_new_release_features.html

an incomplete list of compatible devices is here:

http://www.cisco.com/web/partners/pr46/pr147/partners_pgm_partners_0900aecd800a7907.html

Some of these are esoteric and a lot revolve around security, VOIP, and so on. The one feature we’re all interested in here is AP-assisted roaming.

AP-assisted roaming is a mechanism whereby the access point can provide intelligent advice to clients when they roam between access points. In the context of the overall wireless network in the venue, this means that the centralised wireless controller will be able to bump you off a busy access point/radio and on to one that is less congested, ensuring smoother operation for users of the wireless network.

Without CCX drivers, you may roam from a busy access point to a more congested one!

In testing the first view Mininote 2140s, Jorke found that the native driver in Windows 7 and also the driver downloadable from hp.com were not CCX compliant – however the Windows XP driver was! Jeffa is working with the guys at HP (or “I’ll place a call to Huston” as he likes to say ) to get access to native CCX compliant drivers for the 2140s to be loaded into the base image … if not we might have to put on the XP version!

Bottom line: CCX will give you a better wireless experience and will help us even out the distribution of users across the access points. Make sure you bring compliant drivers on your lappy!

Per AP TX/RX figures with channel utilisation across the ground floor of GCCEC

Number of Client Associations across the ground floor of GCCEC

I was given logon access to the WCS console at GCCEC at the start of May this year. Shortly (10 minutes) later I started e-mailing “DANGER, WILL ROBINSON!” messages to the venue and the tech•ed technology team.

What’s wrong?

The first image shows the current receive and transmit utilisation (Rx. Util. and Tx. Util. respectively) for a given access point (we’re still trying to get to the bottom of understanding how they’re calculated given there are multiple radios in each access point). The Controller also has the access points doing a passive scan in the background to determine the actual RF spectrum utilisation on the channel to which the access point is assigned (that’s the third “Channel Util.”  figure off each access point).

The second image shows the number of client associations across the same area.

See the problem? The venue is between events. No one is there. But there is a massive RF utilisation across the building. Some of the access points are yakking their heads off to no one and there is not much bandwidth left over for any users.

When we raised the issue, Cisco TAC and the installers of the network were of the opinion that the issue was caused by external interference. I rejected this explanation immediately because:

1. GCCEC is on the coast of Queensland and so constructed to withstand cyclones and severe tropical storms. This means robust reinforced construction materials throughout.
2. The venue is bounded by water on three sides and the Gold Coast highway (building foyer + drive way + park out front + 4 lane road + block of shops before you even get to the nearest residential building)

Gold Coast Convention and Exhibition Centre (from Bing Maps)

Anyone who has diagnosed complex problems with multiple suppliers in the mix knows that getting traction from people in resolving problems is sometimes hard, especially when people invariably have a foregone conclusion as to a root cause in their minds. Our logical fault finding steps needed to be clear and bulletproof to gain traction and ownership from all involved.

Step 1 – Isolate the cause of interference as internal or external to the building

This part was pretty easy. There is a company called Metageek that sells a great little device called a Wi-Spy (it presents itself to Windows as a HID device so there is no mucking around with special drivers and other nonsense) and a companion piece of software called Chanalyzer. Chanalyzer and the Wi-Spy together allow you to see peak and average utilisation of the entire 2.4 ghz spectrum (there’s a version you can buy now that does 5 ghz). You can simultaneously use your lappy’s onboard WLAN NIC to grab a list of SSIDs with the corresponding channels and signal strength information and then overlay that over the actual RF activity on the network.

Central Room A Spectrum Utilisation - not you can clearly see the signature of the 802.11n network on channel 1 (the squarish pattern with a dip in the middle)

There are two useful views here. The Spectral View (aka waterfall) shows a time series utilisation graph of the RF spectrum. You can adjust the sampling period and play back different periods. The other useful view is the topographic view that shows the signature of the RF utilisation pattern overlaid with the SSIDs found using the WLAN NIC in your laptop.

As I mentioned above, it was amply clear that the interference was inside the building because the WCS console was saying as much, but we really needed a smoking gun. This was too easy to produce (by sheer brute force):

1. Run up wi-spy, and grab a sample of what the spectrum is doing;
2. Shut down the entire wireless system in the venue;
3. Repeat step one and compare.

Step 1 showed a lot of RF utilisation. Step 3 showed nearly none. Case closed: The interference was the wireless system in the building. Now we just had to work out why!

Step 2 – What is the RF interference exactly?

After two months and a number of Cisco TAC cases the utilisation figures at the venue were still unacceptably high. We had not received a decent explanation from the parties involved as to the true root cause (well, not one that would satisfy me anyway) so I chose to employ brute force again.

Brute force this time came in the form of an embedded wireless platform that allowed us raw and unfettered access to the underlying WLAN NIC to do some packet captures of the RF-side of the wireless interface. We needed to use this specialised platform to capture packets due to limitations within the Windows kernel in which 802.11 traffic is presented to applications as 802.3 (Ethernet) traffic as it moves up the driver stack. Therefore, under Windows, it is not possible to capture raw management frames unless you use devices that use a proprietary raw miniport driver that bypasses most of Window’s normal networking. These drivers are never certified.

The embedded device we used is normally stuck on mining vehicles with neodymium magnets (David Eagles from iVolve brought it in a nice green Coles friends-of-the-Earth recycled shopping bag and told everyone not to put your laptop near it unless you wanted a blank hard drive).

Running packet capture of the RF in North-West hall.

Running RF capture in North-West hall - the four circles on the brackets on the device will erase your lappy.

We were pretty much the only users of the WLAN in the North-West of the centre. We ran a packet capture on the RF to see what on Earth was going on and fed the raw file to Wireshark. The results were very revealing:

• We ran the packet capture for 185 seconds
• 39193 frames were captured (remember no one is using the network at this point!)
• 38,088 frames were 802.11 beacons … !
• Only 1105 frames were not 802.11 beacons … !!

Further from this you can work out:

• There were approximately 220 beacons per second with a size of 258 bytes each.

At this point we knew we were onto something … but why so much traffic?

Step 3 – Analyse the logs

Beacon frames are sent as a normal part of 802.11 management traffic. Normally an access point will send (about) 10 frames per second to advertise their SSID and various information about the capabilities offered. That would account for but a small fraction of the traffic above. We were only pulling traffic from channel 6 in this case so there could not possibly be sufficient access points to generate that much traffic.

Remember we’re looking at 220 beacons per second. A single access point should only generate 10.

Wireshark Trace from Central Room A showing the beacon frame spam

The Wireshark traces showed that GCCEC has 5 SSIDs being advertised for use (their public one, internal, one for Telstra and some other stuff). Each of these were being advertised in its own beacon packet. This is helpful as it shows us now to expect 5x the number of beacons per access point and importantly we’re now in the realm of feasibly accounting for the quantity of beacon packets being seen in our packet captures (i.e. our packet capture device would easily see 4-5 access points on channel 6).

This answers part of the problem as to why there was so many frames from an ‘unused’ wireless network. Now we just needed to answer the original question we came on site for – why so much RF utilisation?

Step 4 – Punch some numbers into a calculator

To understand the nature of the problem we need to understand a bit about data rates and 802.11 networking. There are a number of bit rates defined for 802.11 networking and clients will choose a bit rate based on signal strength, configuration of the base station, and other things.

The important thing to note here is that all management traffic is sent at the lowest bit rate supported by the base station. In this case that would be … 1 mbps.

A 1 mbps bit rate gives you typical data throughput speeds of 500 kbps.

Let’s go back to those figures again:

• 220 beacon frames per second;
• 258 bytes each;
• multiplies out to 454,080 bits per second;
• Typical throughput for 802.11b at 1 mbps is about half-a-megabit … which would be about 500,000 bits per second.

BINGO!

We now can account for 80-90% RF utilisation figures based on beacon frames alone. All of these marry up more or less and so now we understand the problem.

Recommendation for tech•ed

There are a few very logical outcomes from this exercise that provide ‘easy wins’.

1. We will turn off all advertised BSSIDs except for MicrosoftEvent;
2. We will get GCCEC to make their corporate network’s WLAN access require a probe request so it is not causing another SSID to be advertised;
3. We will disable 802.11b at the event (sorry to all of you with an iMate Jamin, but it might be time for an upgrade! );
4. We will up the basic rate to 18 mbps. This alone will ensure that management traffic will take up 1/18th of the RF spectrum that it was before.

Hooray! Beers all around at Q1 … not quite

In a case of “solve one problem, find another” we unfortunately did uncover a fair few more issues while conducting the work above over a full day on-site. The main outstanding issue that we have now is that we noticed that some of the radio interfaces in particular access points perform very poorly (3 mbps typical throughput even on 802.11n). This is now our critical concern with the wireless at GCCEC and we will be scheduling another visit onsite as soon as possible to get to the bottom of these problematic radio interfaces to work out what the issue is. At this stage it looks like a real puzzler because we did some quick tests in the waning hours of our time on site and we received excellent performance from other radios of the same access point, and excellent performance from the edge switchports that service the access points.

We like a good mystery and hopefully solving this next one won’t bring up more problems so we can have those beers at Q1.

Its no secret we’ve been spending a lot of time checking out the venue to ensure we’re getting adequate access to wireless and network facilities. On our last site visit we had the venue network engineers working with codify to figure out ‘black spots’ in the network and work towards a root cause. We were lucky enough to have a loner machine from Nick to test out the profile, the results of which you will see later, but the size factor of the machines make them really easy to cart around the venue, and I generally averaged about 330 Mbps on the network – of course that’s really easy to do when you’re the only person connected….

While the codify guys started getting down to business, we were tasked with another very important job;

Tasting the tech•ed and APC delegate menus….

Some days you have to work very hard, and I guess this was one of those days; leaving the codify guys to continue working on the network, we dragged the jomablue guys down for opinions on the menu. Now I’m not going to go into detail on all the items served but there was an amazing range that was presented to us, such as:

• Beef with Ginger and green peppercorns
• Ocean Trout
• Beef and mushroom Risotto
• Fruit Shots
• Seafood Paella
• Lamb Korma
• Lamb Cutlets

Now bear in mind that this was for both events so we can’t guarantee you’ll see any of them (unless you are going to both..) oh.. there was bunch of wines as well, but best leave you to enjoy that for the event itself. Some shots below…

The team led by “Fletch” at GCCEC did an awesome job of exciting us about what the delegates are going to experience food wise.

Those photos above don’t really do it justice and after we ate till we couldn’t eat no more, we rolled/staggered our way back to find the codify guys still working hard, apparently testing what appeared to be some strange speed results from a couple of the wireless access points.

So as you can see we’re working VERY HARD to give you the best level of service possible for tech•ed and APC…. well at least some of us are

- jorke

This has worked really well for Microsoft in the past as we have only ever really officially offered wireless in select common foyer areas and the exhibition hall areas. This year, however, things have been ratcheted up a whole number of notches due to the giveaway of the HP Mini 2140 devices.

Wireless hardware for tech•ed 2009

GCCEC has recently installed a new Cisco 4404 Wireless LAN Controller with 50 x Aironet 1252 lightweight access points with abgn radios installed throughout the entire complex.

This differs greatly from most wireless network scenarios you’re probably used to in that:

• The Wireless LAN Controller is a centralised management device that manages all of the access points.
• All of the access points are ‘dumb’ in that they receive firmware and configuration information directly from the controller.
• All traffic from the individual access points is delivered to the Controller over copper Ethernet and then distributed to the local area network by the Controller using 2 x gigabit fibre uplinks.

There are a few key benefits in this architecture, the primary being that there is a single device (the Controller) with a complete understanding of all access points on the network. This means that:

• The Controller can manage the channel selection and power selection settings across the network. The Controller will therefore (allegedly) ensure that near-by access points are not assigned to conflicting channels.
• The Controller works proactively to understand and close coverage gaps in the network.
• The Controller understands where all the walls are, roughly what they’re made out of (so it understands the db loss caused by the wall), ceiling heights and so on.
• In some scenarios the Controller can detect if clients at the edge of the network are having trouble connecting and so crank up the power levels accordingly.

“Wow! That sounds amazing … 50 APs will handle anything!” I wish that were true however as you’ll learn in coming posts, there is no such thing as a free lunch (well there is if you work as crew at the event but you don’t want to see what we get fed out back).

There are a number of issues with the WLAN deployment at GCCEC that we have to resolve before the event. This series of articles will keep you updated on how we’re doing with root cause analysis and implementing the necessary fixes.

Coming up next:

• Diagnosing and resolving extremely high RF utilisation
• Diagnosing and resolving intermittent throughput problems on individual radio interfaces (Scheduled to be onsite to crack this one on Thursday … write up will follow that)

As you can well imagine there are many factors we need to consider when we’re expecting ~2500 machines to be delivered to end users in 5 days… And this will be a main theme of quite a few posts on this blog as it presents quite a few unique challenges.

• Wireless Network planning – a LOT of work is been done to mitigate risks around this
• Help Desk – Providing support around these machines including DOA’s
• Power and Charging – Ensuring these machines will be charged up enough at hand out and there are enough power points around the venue.
• Handing Out to Delegates – People flow!
• Imaging – how to get the image onto the machines, luckily we have Jeffa helping us here as the “Build Czar”

This is a start to the planning we’ve made, let us know what you want to hear about. To start with, the question everyone wants to hear about.. WIRELESS!

- jorke

We’re going to give the warts and all view of what’s involved in doing this focusing around the pragmatic approach of logistic and planning for this event and those related such as:

• Products and technology used for the event
• How the logistical planning is done
• How to you build a network for such and event and all the prep work.
• Tons of other information in mind numbing detail!

Plus of course lots of statistics and prediction that all go into delivering this kind of technology for the event.

So first of all some introductions for the core delivery team;

Jorke Odolphi – Microsoft – Infrastructure Architecture Evangelist /  tech•ed Technology Guy – responsible for the delivery of the infrastructure technology and ensuring and enabling that Microsoft has the best technology on show.

Other duties include attending long meetings and staying awake, breaking down barriers in delivery, helping the business understand the logistics and food/wine tasting.

(source: flickr.com/photos/heroeshappen)

Codify – International Men of Mystery – These guys actually DO the delivery of all the core infrastructure to tech•ed such as network connectivity, Wireless Networks, building machines for speaker rooms and connecting everyone to the internet. Simply put if you’re going to be putting anything on the network at  tech•ed  these guys are involved in some form. As a result they’re slaves to the network for the event!

To protect their identity (and they’re camera shy) the picture on the right is an artists rendition.. (source wikipedia.org)

Jomablue – International Men of Mystery – again another group of camera shy individuals. These guys are the core of the logistics for the event. When any piece of technology is placed in a location in the event these are the guys bumping it in, plugging it in and making sure the the power is right and in fact everything is right.

To protect their identity (and they’re camera shy) the picture on the left is an artists rendition.. (source wikipedia.org)

Of course we’re only scraping the surface of what we all do, and we can’t forget all the other people we interact with such as the tech•ed core team, Gold Coast Convention and Exhibition Centre staff and if course all the awesome staging staff. We’ll be hearing from all those people leading up to the event, so sit back take it in and tell us what you’d love to hear from the bluelit parts of teched.

- jorke