This is how it is done.

We had just shy of 1000 concurrent users in the keynote. All the while everyone was tweeting, uploading photos and we were still getting 1-2ms pings to Charlotte Street exchange in Brisbane.

As Borat would say, “Niiiiicee.”

I finished off the physical design for the MDF and IDFs a while ago and thought I would take the time to post these in case you wanted to build your own 3000 person corporate conference at home. I am only covering off the network side of things below – there is a massive HP + Microsoft private cloud solution going in at the event that will need post(s) in its own right.

In terms of fast Internets

• 1000 mbps of Internet connectivity provided by Telstra Internet Direct. This is a high performance e-line service back to Telstra Internet Direct in the Charlotte St exchange (i.e. it is a wholesale/service-provider quality service, uncontended back to their core).
• An additional 500 mbps of Internet connectivity from Over The Wire as a redundancy option.
• BGP routing under our own AS9984 to make the best use of the combined link.
• IPv4 and IPv6.

In terms of WiFi

• Cisco 5508 WLC
• 50 x Cisco Aironet 1252 access points
• Each AP has 2 x radio interfaces (2.4 and 5GHz) for a total of 100 radios supporting 802.11a, b, g, and n

In terms of physical infrastructure

• 2 x Brocade CER units for core routing
• 2 gbps LACP core trunking between the routers, core switches and wireless LAN controller
• 1 gbps distribution network
• 77 switches deploying 2000+ ports of connectivity

IDF Physical Design

MDF Physical Design

Also available as a PDF export of the original VSD:

Project 3031 – Tech·Ed 2011 Network Design and Implementation (Microsoft Pty Ltd) – Network Design

If you have any questions then post them below.

Each year I’ve challenged our technical team to try something risky to keep us honest about leading with first rate technology implementation and practice. You can see our success on this blog and numerous press articles.

The first challenge I’ve identified this year is to move the whole event from a NAT’d network to a full blown public routed network, similar to what you see with an ISP. We’ve based this looking at last years usage pattern on our networks , and after some research we’ve come to this conclusion due to the lack of sensibly priced devices that can support 8000+ clients with many port mappings.

To some extent we achieved part of this last year with giving everyone a public routable ipv6 address with all ipv4 still being NAT’d. Our key requirement to support approximately 2500 delegates a few hundred staff and many, many ip enabled devices it to acquire a large range of pubic ip’s in the order of a /19,  /18 or /17 giving us 8192 or 16384, 32768 respectively, possible clients to support (if you don’t understand see CIDR).

Now the impending exhaustion of ipv4 address space commonly referred to as the IPocalypse you can imagine this kind of addressable space is not easily obtainable  – and can sometimes even go for a high price .  Last year we requested a temporary allocation from APNIC out of their portable/temporary range, which we used for a couple of weeks and shortly after returned. As a side note that same temporary range was used by LinuxConf in Brisbane a few months later, however sites like speedtest.net still cached the Microsoft tech•ed entry at the event… 

In our manner of planning ahead, last week we initiated our request to APNIC for a /17 or /18 or /19, as well as a temporary ipv6 allocation.. This was the response from APNIC.(note that the names and contact details have been scrubbed to protect the not so innocent – of course read from the ) ;

From: xxxxx@xxxxx
Date: Wed, Mar 23, 2011 at 11:47 AM
Subject: [APNIC #XXXXXX] Temporary IPv4 and IPv6 allocation for Microsoft TechEd
To: xxxxx@xxxx.xxx

Dear XXXXXXX,

Thank you for your email reply.

There is no IPv4 address space available for temporary allocation.

This reserved address space is no longer available as it has been placed
back in the APNIC free pool for distribution.

Please let us know if you wish to continue to obtain a temporary /48
IPv6 assignment from APNIC.

We look forward to hear back from you.
If you have and further questions, please let us know.

Kind Regards,
--
____________________________________________________________________
APNIC       sip:

http://www.apnic.net

_____________________________________________________________________

* Sent by email to save paper. Print only if necessary.
On Wed Mar 23 10:06:01 2011, xxxx@xxxx.xx wrote:
> Hi XXXX,
>
> What is the largest v4 block you can offer?
>
> I was under the understanding that APNIC had a block of space parked for
> temporary uses like this event (the netblock we used at TE last year was
> used at LinuxConf this year).
>
> XXXXXX.
>
> On Wed, Mar 23, 2011 at 10:01 AM, XXXX XX via RT  wrote:
>
> > Dear XXXXX,
> >
> > Thank you for your email.
> >
> > In regards to your request, due to the APNIC reaching Stage 2 of the
> > IPv4 Exhaustion plan, APNIC are unable to provide a temporary /17 IPv4
> > allocation.
> >
> > However, we are able to provide you a temporary /48 IPv6 assignment.
> >
> > Please let us know if you wish to continue to obtain a temporary /48
> > IPv6 assignment from APNIC.
> >
> > We look forward to hear back from you.
> >
> > If you have any questions, please let us know.
> >
> > Kind Regards,
> >
> > --
> > _____________________________________________________________________
> > APNIC       sip:
> > http://www.apnic.net                 phone:
> >                                       fax:
> > _____________________________________________________________________
> >
> > * Sent by email to save paper. Print only if necessary.
> > On Tue Mar 22 11:39:03 2011, xxxx@xxxx.xxx wrote:
> > > Hi there,
> > >
> > > We are in planning stages for Microsoft TechEd 2011 to be held in the
> > > Gold Coast Convention Centre this year. As per last year we would like
> > > to obtain a temporary allocation for use at the event.
> > >
> > > We are seeking the following:
> > >
> > > * IPv6 /48
> > > * IPv4 /17
> > >
> > > The earliest date we would be using these addresses is the 1st of July
> > > 2011.
> > >
> > > We would return the addresses on the 16th of September 2011.
> > >
> > > Could you please advise what you require for this allocation and when
> > > it might be approved so that we can do our physical network design.

So that’s it then – no more ipv4 space for us. We’ve committed to the path of deploying a fully routed network so it looks like it will be an:  IPv6 ONLY network!

We are working really hard to make sure that the key resources for the event (the schedule builder, for example) are available over IPv6.

As for other purposes not directly related to the event, we will not be offering IPv4 connectivity.

There will be an option to purchase time on Telstra NextG USB dongles at the event and we are working hard on coming up with a good sponsorship arrangement to help out with the cost of IPv4 network access during tech•ed.

- Jorke and the network team aka David Connors

The slide deck from our presentation is available here:

Infrastructure Saturday tech•ed Backstage 2010

Ben Parker presented quite a few slides on the enterprise software stack on DemoNet – content that we really missed out on delivering at the event.

We hope you enjoyed the talk – No evals through yet but in the wrap up a guy said that he gave us all “1″s because he thought that was the top score (not 5!) Doh!

Where: Microsoft Brisbane Office (map)
When: November 6th, 2010
Register: http://infrasat.eventbrite.com
Cost: FREE!

There are a heap of other great sessions (see more here: http://infrastructuresaturday.org/sessions/) on at the event so be sure to mark the day in your calendar!

TechEd 2010 Community Wall

In case you missed it, there was a giant 6 metre wide plasma wall at TechEd that streamed tweets (from Twitter obviously) and various images from Flickr. We took the following feeds:

1. Anything with the #auteched hashtag in it appeared on the tweet slide;
2. Anything with #news and #auteched hashtags in it from the user @auteched (the ‘official’ news feed for the event) was used for the blue news slide;
3. Anything from the TechEd Australia Flickr set was used for the photo sets.

The app itself was written using .NET 4 + WPF and Visual Studio 2010.

The PC that ran the display (“riceputer”, my blingy gaming rig, named after four-banger rice cars):

• Intel Core i7 920 @ 2.67GHz
• Radeon HD5850 (1GB RAM)
• 3GB of triple channel RAM

The display formed a centre-piece for the community in the expo hall. It was received really well by the delegates and hopefully this will become a permanent fixture at tech•eds of the future. The expo wall was used as backing on the main stage and there were a heap of beanbags and comfy couches around in front of it – the whole area had a really nice community feel. We saw heaps of people sitting around and tweeting to see if their tweets would pop up on the display (come on guys, as if we’d smoke-and-mirrors you – you’re all too smart for that )

Technical Direction Company (the audio visual supplier to the event) provided the display and the necessary matrix hardware. We provided a 1080p DVI feed (1920 x 1080) and they used a device called a Sypder (from Vista Systems) to split the top 729 pixels of that feed across all of the plasmas. You can see the initial calibration of the display in this video (sorry for the end bit, Patrick ).

In terms of application development (and it will be no surprise to any of the developers reading this): WPF is a thirsty beast. The application had to run 12 hours a day without a glitch or leak and this proved to be very difficult to achieve with WPF.

The individual character animations on the tweet screen were achieved by:

1. Downloading the tweets in a background thread and scrubbing that against a profanity filter;
2. Using GetGlyphTypeface() on the TextBlock object to read the metrics for the individual characters in the tweet (see this http://i.msdn.microsoft.com/dynimg/IC26720.png).
3. Create a separate TextBlock control for each of the characters in the tweet. We did this while honouring all of the kerning etc in the original font glyphs;
4. Animate each of the characters individually to achieve arbitrary layout effects and animations.

In retrospect, this was a LOT of work that probably was lost on people watching the display. If we had our time over we might have just stuck with animating an individual TextBlock or Canvas and transitioning the tweets in in one hit (as was the case on the blue News @ TechEd slide).

We found that using storyboard based animations was very problematic. Each time the tweets appear we bring in four at a time, in four sets for a total of 16 tweets meaning a worst case scenario of ~2500 individual items in the storyboard. We tried a lot of different approaches to interacting with the WPF storyboards but in all cases we found that there were subtle leaks, even for items that were well out of scope. Over time (and remember we’re running this all day and night) memory usage would creep up.

In the end, we resolved to ditch storyboards altogether and manually created the WPF animations and queued them up ourselves. Once we did this, the memory leakage issues disappeared and the app would sit on a steady 111 meg of RAM usage indefinitely.

We found that the static storyboarding in WPF was excellent for prototyping (i.e. just writing up XAML by hand) but as soon as you start modifying those in code you’re in for a world of pain. That isn’t really an isolated experience as far as we can tell – check out the memory usage on Blu, MetroTwit, etc.

For next year, it would be great to do this again using the new Twitter streaming APIs and XNA.

The video is available here: http://www.msteched.com/2010/Australia/voc208

The whole talk was very much ‘by the seat of our pants’ as we were insanely busy. Ben Parker, unfortunately, could not make it on stage due to other work commitments at the event. This was a real shame as he is ‘the man’ when it comes to the IT Pro side of the MS Enterprise software stack.

Unbelievably (given the rushed presentation and that most of the slide deck was a re-heated user group presentation I delivered a few months ago because we ran out of time ), we managed to score the #2 spot out of 165 sessions at the event, with an “Overall Satisfaction” of 4.74 out of 5. This is a great result but I cannot help but think there was a lot of other more deserving content at the event that probably should have rated higher.

We had a lot of good laughs – half of them at my expense. The notorious review comment from ARC301 came back to haunt me (2m 20s in … Thanks Jorke )

Evaluation scores follow:

Overall Results

Rollup Results

Comments

• http://www.noisetosignal.com.au/franklyspeaking/?p=253

• Microsoft Tech.Ed 2010 network shouldered the load, IT Wire:
  http://www.itwire.com/business-it-news/networking/41617-microsoft-teched-2010-network-shouldered-the-load
• Tech Ed tames torrent leechers, IT News:
  http://www.itnews.com.au/News/229928,tech-ed-tames-torrent-leechers.aspx
• Microsoft Tech.Ed 2010 goes IPv6, ZDNet:
  http://www.zdnet.com.au/microsoft-tech-ed-2010-goes-ipv6-339305258.htm
• How do they do IT: Microsoft’s Tech.Ed goes IPv6, Computer World:
  http://www.computerworld.com.au/article/357238/how_do_they_do_it_microsoft_tech_ed_goes_ipv6/

This is a pretty good summary of the event in general:

• http://www.itwire.com/business-it-news/technology/41427-tripping-through-tasty-tech-at-microsoft-australias-teched-2010

More write-ups coming … no, really!

Why are we doing this?

It’s well know that ipv4 address space is running out, while this is problematic for service providers and telcos – in our opinion it also as just as big a problem for application developers.

So this is our challenge this year – giving every delegate the chance to experience this first hand. We’ll be giving full native IPv6 addresses for each and every device that can support it.

We’ve learnt a LOT about the ins and outs of IPv6, those learning will be transparently displayed on this blog as per our normal policy.

If you’re at Tech.Ed you can check this out by going to http://ip6.me or http://www.v6.facebook.com and see the IP address you get!

I also hope you’ll enjoy our planned IPv6 celebration hours we’ll have during the event, where only IPv6 will be available on the network!

- The Backstage team.

Your friendly technology team

Thank you to everyone for the kind feedback on our recent BIGAU presentation both at and after the event. I couldn’t resist including this picture again as a) I love it, and b) it was the only politically correct on in the slide deck (sorry Princess Parker!).

We’re on site now and work has begun in earnest on provisioning the infrastructure for the event. Jorke and Ben are still in Brisbane busily setting up all of the HP blades for DemoNet and these will ship down to the venue on Wednesday.

Pile of work

We’re going to try and keep the blog up to date as we build out the network infrastructure – but as always, delivery of the event will take priority.

If you have any cool ideas for topics you would like to see covered on the blog, post a comment at the end of this article and we’ll see what we can whip up in response.

I will be presenting tech•ed backstage at the Brisbane Infrastructure Group. Guess appearances may include Nathan Wright from GCCEC and Ben Parker from Parker Tech Pty Ltd

Venue
Microsoft, Level 9, 1 Eagle Street, Waterfront Place, Brisbane City.
Time
1700 for Pizza.
Date
Tuesday, 13th of July 2010

Further details on #BigAU.

I’ll be covering off stuff that happened last year, stuff that’s happening this year + our wins and losses. Shock/Horror! I actually have some good news re Cisco WLCs this time around too!

See you there!

Cisco 4404 Marketing Poster

Long time readers of this blog (and the tech•ed 2009 team who have to put up with my whinging) will know that we had a significant number of challenges getting the wifi at GCCEC up to a scratch. Today I am happy to announce, however, that we have had a win. The 5508 delivers in the areas the 4404 didn’t.

Catch up on the complete history:

• http://www.techedbackstage.net/2009/07/15/diagnosing-and-resolving-extremely-high-rf-utilisation/
• http://www.techedbackstage.net/2009/07/22/resolving-a-hunch-wifi-performance-gccec/
• http://www.techedbackstage.net/2009/08/26/when-i-said-16-percent-i-meant-16-percent/

David Connors and David Eagles posing for a rare post survey photo op. "Weeze frum Queensland and weeze makin wirelesses!", commented Connors during a recent interview.

The bottom line issue was that all of our investigative work lead to the inescapable conclusion that the Cisco 4404 is a crock. I’d like to say that it was just a matter of the 4404 not supporting 802.11n data rates, but as the previous posts have shown, it would only barely perform at 802.11b rates in some cases.

We engaged with the original installers and Cisco TAC as well as local Cisco reps and pretty much got nowhere. Despite some early positive noises, they certainly did not want to own the issue and every time we would go back to them with detailed reports the responses we received were totally unacceptable (95% RF idle utilisation caused by co-channel interference on a network with 1 access point active and 1 user). Yes, I am still annoyed at the comment about Cisco being “amazed if there was a fault of this nature … especially if someone from Queensland found it.”

The most frustrating thing for me about the whole endeavour is that we spent a LOT of time and both Microsoft and GCCEC spent a LOT of money in having to go back and rectify someone else’s work. We also went to great lengths to ensure that we provided detailed reports that covered off exactly the clinical steps we went through to provide a final conclusion as to the problematic performance characteristics at the deployment.

In the end, we decided to remove the 4404 from the equation and deploy Cisco autonomous IOS images to all of the access points. This resolved the majority of the issues and, to us at least, confirmed that the 4404 was the root cause of all of our pain.

In the time since tech•ed 2009, GCCEC has replaced their 4404 WLC with a new 5508 unit (http://www.cisco.com/en/US/products/ps10325/). The product literature for the unit talks a lot about supporting 802.11n data rates.

O’RLY? What was the 4404 with AGN-capable Aironets on it supposed to do then?

4404 vs Autonomous

The move away from the WLC to autonomous firmware last year was a bit of a painful thing for us to do. We were torn because:

• The autonomous firmware would demonstrably provide 500%+ performance improvement of ther 4404 WLC.
• In moving to autonomous, we had to give up:
  • centralised management
  • AP-assisted roaming via CCX (see http://www.techedbackstage.net/2009/07/17/what-is-ccx-and-why-should-you-care/)
  • centralised wireless -> wired egress

We wanted to use the WLC but the performance characteristics were so dreadful it was not an option.

4404 in action

5508 Performance Characteristics

On the 26th of March 2010 we were engaged by Microsoft to validate the new 5508. David Eagles, Jorke and I spent the whole day on site with Nathan from the convention centre rerunning the entire survey from scratch. I am delighted to have to report that the 5508 has resolved the performance issues.

Simply swapping out the 4404 for a 5508 (and adding nothing to the RF delivery side of the wireless network) yieled:

• 484% average throughput improvement over 2.4GHz
• 340% average throughput improvement over 5GHz

26 March 2010 GCCEC Wireless Survey 2.4GHz Performance Comparison

26 March 2010 GCCEC Wireless Survey 5GHz Performance Comparison

2.4GHz Performance Comparison Across Aironet Deployment Scenarios

5GHz Performance Comparison Across Aironet Deployment Scenarios

So there you have it. There is still a bit of tuning for us to do regarding channel allocation (the same stuff we did for tech•ed 2009 – but was lost in the 5508 deployment) but it looks very promising.

A heartfelt THANK YOU to the GCCEC guys for the capital investment for the new unit for tech•ed 2010 – we really appreciate it.

Delegate Network 

• 99.998 uptime (was a 45 second outage from a GBIC flapping..)
• ~1300 wireless clients was the highest amount connected.
• ~2TB of data downloaded (in 4 days!)
• 300GB send out.

Demonstration Network (the glass box of doom with the servers inside) 

• 100% uptime on services with over 80 virtual machines hosted for speaker and other demos. This was highly under-utilized.

Major issues 

Oops

• Intel Wireless drivers – escalated to Intel who were sponsoring -
  they were kind enough to help us out
• Bit Torrent – Played network cop – Rickrolled access to key sites –  click here (I dare you)
• RRAS port exhaustion – escalated to our support team in India
• Live ID creation – another escalated to our support team…
• Netbook deployment – finished this with brute force
• Session Recording – Expression Encoder crashes
• Wrong URL on netbook images – network hack
• Well i think this is minor, but there was some slight damage of a netbook box…

As you can image there was so much more that went on, we’ve added a lot of these things to our planning process which has already started; 

“Life is like riding a bicycle. To keep your balance you must keep moving” – Einstein 

So here go, we’re hitting the reset button – we’ll do our best to keep you update-to-date from our end – no promises of course as we’re all focused on delivering to a higher quality outcome than before. 

- jorke 

We don’t shape the traffic on the network at all and nor do we see it as our role to police or restrict delegate use of the network (a policy that will, no doubt, change for tech•ed 2010 based on last year). Given this, we had to do a bit of seat-of-our-pants work to identify who the main culprits were and then implement counter-measures to at least lessen their impact on the delegates who were at the event to learn and share (rather than download). The back story is interesting as it involves a collaboration between us and the team at Microsoft who own the ipnat.sys driver that we were using to do the network address translation.

Where is all that data going?

The first thing we noticed was a couple of brief periods during the day where a very small number of clients (maybe 5 or 10) would experience no Internet connectivity. This was odd:

• the network was happily switching a couple of hundred megabits per second of data (that’s the Internet link, not the core).
• there was plenty of CPU and other resource headroom on the servers in question.
• nothing was showing up as unusual on the core switches.
• performance tests carried out in network operations showed plenty of excess capacity.
• we did all of our sums on port and CPU requirements and the RRAS team did a simulation for us (see: http://www.techedbackstage.net/2009/08/05/windows-server-2008-r2-nat-performance-guest-post-by-the-windows-team/) so we did do all of our homework up front.

I was puzzled. I don’t like fault reports of any kind and while the number of affected users was small, we really did need to get to the bottom of it. There was also the matter of the amount of data we were pulling from the Internet. We normally run tech•ed with a 100-150 mbps link with headroom to spare – but this year we were seeing sustained peaks twice the historical norm. Odd and uncharacteristic.

tech•ed 2009 Internet link port utilisation expressed as megabytes per second (base10, multiple by 10 to get a rough base10 mbps figure)

The quickest way to get to the bottom of anything like this is to point Wireshark at the network and see what is happening. As the network is switched, we need to set one of the ports on the core into SPAN mode (Switch Port ANalyser mode) which instructs the switch to send everything on a given VLAN or port out a nominated destination port, regardless of whether it is actually destined for that port. Once you have all of that data, Wireshark does a fantastic (if not always slow) job of breaking down the data into the corresponding protocols visually so you can see what is going on. The “Top Talkers” feature in Wireshark is awesome as it quickly lets you identify which clients are consuming a bulk of the traffic.

This quickly showed us two things:

1. A small number of clients were causing a lot of data utilisation.
2. The protocol was BitTorrent.

At this point you have to remember that we have a heap of bandwidth available. Some clients chomping through a lot of bandwidth isn’t a problem and running BitTorrent isn’t a problem per se. The aforementioned work on port utilisation planning was already done – but I had not looked at that empirically and so that was the next step.

Port Exhaustion

We pulled up the NAT table for the relevant interfaces in RRAS and the problem was immediately obvious. The table was HUGE (approaching 65K ports). The RRAS machine was experiencing port exhaustion.

Let’s step back and look at the issue a bit from first principles to understand what is going on. Like any corporate network, tech•ed 2009 runs private address space. Private address space is special IP address space allocated by IANA and intended solely for IP networks that do not route directly to and from the global Internet.

In order to provide Internet access at the event, we use Network Address Translation on the default gateway. The gateway has two network interfaces:

• one facing the private address space (and the IP address of this is your default route if you do a netstat -r on your PC).
• the other network interface is directly connected to our IP transit provider (in this case it was Telstra for delegates and Over The Wire for staff and speakers) and this interface has a normal public IP address on it.

The following happens when your client tries to send a packet to an Internet destination:

1. Your local IP stack sends the packet to the default gateway specified in your current IP config.
2. The NAT server receives this and pretends that it is going to route it to the Internet.
3. The NAT server alters the contents of the packet so that the source of the packet is the NAT server’s public IP address and a possibly a different IP port on that public interface.
4. When doing step 3, the NAT server makes a note of what it has done in what is called a translation table.
5. The NAT server sends the packet on its way to wherever it is meant to go.
6. A reply from the Internet host will come back to the NAT server on the public IP and port that the server provided.
7. The NAT server alters the contents of that packet to be destined to your private IP address and port and then sends the packet to you.

So the upshot of the above is that you can ‘hide’ a thousand people behind a few public IP addresses. You don’t need to obtain a public IP address for each user, which is a good thing as IPv4 address space is running out. There are a few limitations on this scheme:

1. There is a limit to how much you can store in the translation table. The translations need to be available for kernel mode device drivers and so cannot be swapped out to virtual memory – this means they reside in Nonpaged Pool in the NT kernel (an area of memory that is always resident).  Each mapping consumes around 256bytes of Nonpaged pool. Assuming 15 mappings by 2000 clients = 30000 translation table entries. That’s 7.5 megabytes of nonpaged pool memory out of a nonpaged pool of say 256 meg. Not a problem.
2. There is a limit to how much CPU grunt the machine has. The CPU time spent making translation table look ups and re-writing the packets on the way in and out of the NAT quickly add up when you’re talking about thousands and thousands of packets per second.
3. IP ports are represented as two bytes. This means that the maximum number of 65536 TCP and 65536 UDP ports are available on a given IP address. The actual number available is a bit lower as the server will have a number of ports allocated for its own use depending on what is installed.
4. RRAS, we found, only uses the machine base IP address for the outside of the NAT. It will not use additional IP addresses in the public address pool, no matter how many IP addresses are in that pool.

Analysing the ipnat.sys address translation table

The wireless network only had a maximum capacity of 1500 users and we only hit that peak very briefly. Even if we were at that peak, surely it cannot be the case that all users on the network are using an average of 40+ TCP ports in the NAT translation table?! I am currently running Skype, MSN Messenger, Google Talk, WinAMP (streaming 256kbps Internet radio), Chrome (16 tabs including this, two Google Apps instances, Salesforce.com, YouTube and a bunch of other reference material for this post), VPN to our intranet + whatever Windows is doing itself. I have only 20 ports open to Internet destinations that traverse the NAT in our office. There is no way the average user is going to be using double my ‘power user’ scenario.

We really needed to get our hands on the NAT table to analyse what was going on with the address translation table. “No problem!”, I think, “I’ve done that heaps of times!” Unfortunately, what I did not realise was the following:

• There is no way to export the translation table from the management snap-in.
• ipnat.sys does not have a WMI provider.
• ipnat.sys does not have any native and easily accessible Win32 APIs.
• The only way to pull the table was via MSRPC Remote Procedure Call interface.

The Windows RPC interface is architecturally … shall we say … “dated” to be polite. It is overly complex and not remotely easy to write software against (especially when you consider how simple our requirement is). We started to write an extraction utility but it was getting pretty late and you can’t really keep working until the wee hours of the morning when you have to be on site at 6:30am. We handed the code off to the owners of ipnat.sys as they said they had plenty of steam left (being in India) so they would finish it off for us.

In the meantime, we implemented certain, ahem, ‘interim countermeasures’. We quickly built a list of all of the top torrent trackers around and got the nod from Jorke to add them all to the local DNS resolver and point them at a local web server containing some RickRoll scripts.

How professional network administrators deal with Torrent users.

It killed me that I didn’t see anyone getting done by this first hand, but there were hundreds of impressions in the server logs containing the the Rick Roll scripts so I did get a fair amount of satisfaction at least. It was the most evil of evil Rick Roll scripts too – worse than any that anyone has used to get me in the past.

The next morning we found that ipnat.sys developer (being in India) suffered all sorts of Internet and power problems overnight and was unable to finish the utility. It was a new day when we received this news so we completed the utility ourselves (thanks Paul – using software engineers as network admins has some benefits ) and so were able to pull the translation table via MSRPC and dump that out to text.

The results were revealing and really made me want to go on a rampage. We were indeed hitting port exhaustion as an issue, and the distribution curve of who was using what looked like this:

The distribution of the number of translation table entries used on a per-client basis. I wonder who's using BitTorrent?

As soon as you see that – it is a no-brainer. The right-hand side of that graph represents a small number of users who were using 700-800 translation table entries each at the time. We had a one minute sample file that showed a particular individual using 2500 translation table entries! Argh!

So we scheduled this script to run each minute to generate a list of offending MAC addresses. It took a few goes to get the analysis right but we ended up generating a ‘naughty factor’ based on the number of port mappings, number of distinct hosts on the other end of the mapping, idle time, and so on to give us a number between 0.0 and 1.0. 1.0 meant you were very naughty. 0.0 meant you were very good. We reasoned that if you had a lot of mappings, and that a large proportion of those mappings were to a lot of distinct remote hosts, and largely not idle, that you are probably a Torrenter. OTOH, if you had, say, 20 connections open to a single host or a low number of hosts then this is probably quite fine.

These scripts output a list of bad MACs, that we then just dropped into a block list in the core switches. The logic proved to be quite sound as this is what happened when we blocked a couple of dozen particular users with a high naughtiness factor:

The result of applying a block to a couple of dozen users.

And there you have it. The culprits fingered and booted off the network. Of course, they then just changed their MAC addresses, in which case they were then re-identified as soon as their utilisation crept up, and the new MAC was banned.

2010

So this year, the users who did the above have driven me to recommend the following this Friday when we meet at Microsoft Brisbane for the first technology team meeting:

1. Opt-in “allows list” basis for access to the network. You will need to register your MAC in CommNet if you want access, and we will apply a quota across all of your devices; and/or
2. Mandatory rate limiting on a per MAC basis across all users; and/or
3. Packeteer/Allot/etc based deep packet inspection and traffic shaping.

It’d be better if we could provide peak throughput to any given person at any time should they need it – but the above shows that a small number of people ruin the experience for a large number of prople. My annual argument that users will respect the network resources and behave sensibly will no longer wash with the rest of the team.

</rant>

Anyway, I wanted to write a bit about a particular position I advocate each year and how disappointing to was to see that position abused by a very small number of network users to the detriment of other users at peak usage times.

At the outset of network planning, there is almost always someone who advocates locking down the network environment so that user access is restricted by protocol or all user traffic is forced through an HTTP proxy. I have always dug my heals in arguing that unfettered network access is the best possible delegate experience, and that there will always be delegates with oddball use cases for network access where HTTP/HTTPS only access is not sufficient. In recent tech•ed years the network has been structured like this as private IP address space with NAT – and we’ve never had complaints from users. Good old RRAS does the job!

This year, however, the scenario was a lot different. We ended up with some peak times of network congestion – admittedly only a few times – and the cause was users with their new NetBooks running BitTorrent in a way that showed a complete disregard for other users of the network.

We saw a fair few people huddled around furtively running BitTorrent. It was really disappointing to have to approach users to ask them to be more considerate to others, only to get “But I’m not running BitTorrent” as a reply. Come on people: The network ops people at tech•ed didn’t come down in the last shower.

There was even a case where one guy took his Netbook to the helpdesk to get his network access restored (we had some automated scripts blocking torrent users – but that is a post in itself). He too, allegedly, wasn’t running BitTorrent apparently (there was a folder called “TV Shows” on the desktop of his brand new Netbook).

Running BitTorrent is one thing, but there were even people who removed all of their natural limits from their client. There was one guy who would have 2000-3000 peers running at once.

All in all it was a bit depressing as I really felt like we were having our good nature taken advantage of.

BitTorrent Protocol

The first thing that people suggested (and kept suggesting to the point that Jorke and I were going to start throwing chairs at people )  is that we simply just ‘block them’ or ‘block BitTorrent’ or even funnier “block the BitTorrent port” (You know who you are..) . Unfortunately, this is not so easy.

Broadly, there are two parts to BitTorrent. The first is the tracker protocol, and the second is the peer protocol. For the purposes of this article, I’ll use BitTorrent to refer to a number of peer to peer protocols that behave similarly. I’m also going to gloss over a lot of detail to keep this post under control otherwise it will turn out like one of my wireless ones and take all day.

Tracker Protocol

The BitTorrent tracker protocol is a relatively simple HTTP/HTTPS protocol that is used on a centralised server to:

• Advertise torrents via an announce URL
• Centralise information about the peers participating in the torrent
• Collect statistics about the health of the torrent

Typically you will access a tracker over HTTP to search for torrents via a web page (this is not the Tracker Protocol obviously). Once you find a torrent you download a .torrent file which stores the torrent meta data, including the announce URL. You then launch a BitTorrent client that uses this metadata to talk to the tracker.

This is very light weight and does not really have much of a discernible network impact.

Peer Protocol

Once the client has located the tracker, it asks for a list of peers for the relevant announcement. The tracker comes back with a bunch of data, but importantly this includes the IP addresses and remote ports of peers participating.

With a list of peers in hand, the client will then attempt to communicate with those peers in order to start exchanging chunks of normally around 16 kilobytes in size. The protocol is fairly nuanced and allows clients to sniff each other out a bit like two dogs (whether they are interested in the chunks each other has, or whether one of the clients thinks the other is unfairly leeching and wants intends to throttle that client back). I wonder if that dog metaphore will make it past StalinJorke.

Filtering BitTorrent

The problem with the conceptual overview above is that you’ll see the protocol is very flexible in terms of end-point negotiation.

• There is no fixed BitTorrent port (true, use TCP ports 6881-6889 by default but anyone who is going to specify infinity for a number of concurrent peers is probably  not going to use the default port range).
• There is no fixed relationship between the tracker and the members of the Torrent. Once the client has a list of peers from the tracker it can continue to exchange chunks of data for a long period of time and blocking the tracker will not necessarily be effective.
• Deep packet inspection does not help when you have Azureus, Bitcomet, and uTorrent all cooperating to implement transport level encryption.

User Behaviour

As I said above, I’ve always advocated the point of least interference in the network in the interests of a better customer experience. This has always worked really well and so it is interesting to note that tech•ed 2009 represented such a massive departure from previous tech•ed events. I am not sure whether or not to attribute this to:

1. the fact we had really high speed Internet connectivity; or
2. the fact we gave out Netbooks and so there were a number of users at the event with PCs who normally would not have PCs in that environment.

Regardless, the really disappointing thing to see what a lack of consideration for other users and then lying about it when cornered.

Original article on ZDNet: http://www.zdnet.com.au/insight/hardware/soa/Backstage-tour-at-Tech-Ed/0,139023759,339298840,00.htm

[ I knew we'd get him on camera at one point, btw the guy you whos head you don't see is Nathan from GCCEC - jorke]

Well we are now under way with the session recording and the pressure is starting to subside; especially now we have a stable platform to work with.

Let me take you through some of the issues we ran into during the setup of tech•ed.

Over the last week we have had countless problems with Expression Encoder 3, this software was to be used in conjunction with the ADVC55 Canopus to capture session recording both audio & video.

The software proved to be unstable and difficult to use.

The encoding would start well, but around the 15-17, 32 minute mark of recording it would crash, freeze and lock up the PC.

At this stage the only we could overcome the problem was to reset the ADVC55 Canopus and reboot the PC; this went on for three days.

We changed and tested every possible video & audio setting available but problems persisted. Expression Encoder 3 would allow you to record past the previous crash times and on preview the quality was great. After pressing stop and reviewing the recorded footage it showed really poor quality images, the only thing working well was the audio. Unfortunately the session wasn’t usable with only audio.

The only thing we could do at this stage of the event was look to a solid program that had as many features as expression 3. The main features we were looking for was stability, video compression and compatibility with the ADVC55 Canopus.

We came up with Windows Media Encoder 9. This saved the day! Upon connecting the Canopus device, the software immediately launched. The software automatically detected the signal being supplied to the Canopus and set all the video compression settings automatically.

Press record and off we went. One hour later the software was still solid, the images were quality, the audio was in sync and the playback was DVD Quality.

We had found a winner. This software is highly recommended in all facets, easy to operate, great playback quality and stable.

- Brian “The Video Guy”

They’ve been turned into public space for Brisbane Park(ing) Day initiative by the Architects of Brisbane office of BVN – Check out the concept pics below – This will be built on 133 Mary Street Brisbane 7-10pm – so swing by and check it out. Tell the guys that are there that your netbook used to sit on those

-jorke

We’ll wrap things up over the next couple of weeks. There is some pretty interesting content coming so please stay with us.

The helpful chaps on the Intel stand managed to come up with signed drivers that correct the issue (significantly higher version number than the in-box ones with Win7 RTM) however they’re still trying to get us the okay to release these. Interestingly they suggested that the work around was to disable one of the radios and the escalation technician said to turn off 5GHz – but please don’t do that.

For now, if you have an Intel wifi NIC, I recommend turning off the 2.4GHz radio as there is far greater channel capacity and bandwidth on the 5GHz radios.

The Netbooks have a different chipset that are unaffected by this issue however you will likely experience better performance if you turn off 2.4GHz.

1. Wifi becomes unresponsive including being stuck ‘on’ so that hardware switch does not turn off the interface.
2. Blue-screen-of-death (only one person on the technology team experienced this).

We did a lot of investigative work and determined that the issue is related to the driver stack in the affected clients. The issue only affects uses who are using Intel wireless NICs. The issue does not affect Broadcom (in the netbooks), Atheros, etc.

We managed to latch on to a very helpful man on the Intel stand in the exhibition who confirmed that this is a known issue where:

1. There are a large number of access points.
2. There are a large number of clients.
3. There is roaming.

Sound familiar?

The underlying issue is a limitation in a table that manages the number of access points in range – when this is exceeded the driver stack will crash. This is exacerbated by roaming.

We turned off CCX4 extensions today and that seemed to helped the issue a small amount by being less agressive in how it manages client roaming.

We’ll turn CCX4 on tomorrow for the keynote and then turn it off for the rest of the event. Hopefully Intel can give us a beta driver that fixes the issue. If we get it, we’ll put it up on the blog.

AWESOME!

It is actually Gianpaolo Carraro, Microsoft’s DPE Director. We did a cook’s tour of the venue for him and his family during set and all enjoyed a nice BBQ that evening as a rare bit of downtime. He’s an extremely good sport for doing this and letting me post it.

The F12 madness:

Jorke taking some servers out for a spin:

If you were trying to create a LiveID from onsite at Tech.Ed yesterday would would have seen a message along the lines of “limit exceeded..” – all our Tech.Ed networks are behind NAT, and although we could have cycled the NAT gateway IP’s around, that would only get us a few hundred go’s.

We escalated to the Live product team, in particular our favourite aussie member of the live team, Angus Logan who helped us sort it.

if you do notice any funkiness like this happening, please don’t hesitate to escalate to the Ops team.

-jorke

I woke up at 4:00am this morning in a cold sweat. And I haven’t slept since. In the wee hours of the morning my brain, as is its want, was mulling a worst-case scenario: “What if we got the image or imaging wrong. How are we ever going to re-image these 2575 machines?”

In our planning, the imaging process was always date constrained. Whilst David Haysom had planned an extra 2 days, just in case, as there are always strange things that could bite you on the bum.

First day of Netbook handout. Today. 9am. Within an hour, we had handed out about 10% of the total.

And a problem appeared. Thankfully not hardware; the opening and recharging of each machine tickled out any weird hardware issues. In fact, our out-of-box hardware failure rate was half of the planned amount. HP repaired these quickly, and all was well.

Boot to nothing. Ouch!

The problem: missing images on a low, single percentage number of machines.

After reviewing our WDS/MDT2010 imaging server and technique by local and Redmond experts: this was deemed not the root cause for the missing images. Another possibility was Netbooks booting in the box, running out of power, and getting into a weird state. Again, this was discounted as there was just no data on the drives.

The Current Theory

The imaging process ends with the Netbook shutting down. From F12 to network boot to shutdown is one keypress: F12 to “boot from network”. At the completion of the multi-cast and unpacking in the WDS/MDT process, the Netbook shuts itself down to a black screen.

On the first two days of imaging last week, the imaging area had power issues: essentially, fluctuating power at the setup trestle tables. This caused machines to be left powered off. To a black screen.

Our thinking is that the black screen at the end of this process led to false-positives. Netbooks that had power issues, left on a black screen, were accidentally deemed “imaged” during the first few days and packed up.

Learning 1:  Leave setup machines with a static, on-state saying “OK” would be our recommendation if you are planning such a large, time constrained install.

The QA process we had during the imaging week was to test every 80 to 100 machines: that is, go through the out of box experience to test build quality.

Learning 2: factor in a slightly higher rate of QA checks than you think necessary.

How are we Fixing this?

Handing over a known good, working Windows 7 Netbook to each attendee is a must. No compromise. Therefore, today, we instituted a quickly developed solution to freshly image these machines.

After re-evaluation this evening, we are dedicating a team to process any machines with missing images. There is also another, faster process that is being investigated that may be implemented (via USB imaging)

So, worst case: we have upset attendees whilst we fix their machines. All I can say is “Sorry, and we are making it right” and I am confident this process will work tomorrow. With a little patience, all will be well.

Best case: our current glass-half-full thinking is that the machines that were handed out today were from the early in the week batch: where we had power issues, and black screen false-positives.

Tomorrow will tell.

Thanks for your patience and understanding.

And if you want to complain or vent at someone: find me.

My name is Nathan, I’m the Technical Services Manager here at GCCEC.  A lot of people ask me what Microsoft need off the venue and assume it’s plugging some computers into the venue’s network and all is done however this isn’t as easy as it sounds.  As you can see from the other posts on backstage we have run into a few hiccups with Tech.Ed 09.

My office at the top of the arena.. 'the loft'

First up,  the guys at codify specified to have their own independent network from edge to core.  In the past they have had some issues using the venues current production network  so the decision was made to drop a separate network in.  Codify and Jomablue sourced the appropriate hardware and we installed all the switches in the MDF and the 13 IDF’s around the place, and used our fibre infrastructure for the back bone. All the main backbones are 1GB fibre and from there it breaks down to trunked 100MB ports to the managed switches out on the floor and breakout rooms. The codify team has network monitoring of the entire network as well as remote alarms .

I’ve also had the fun task of dealing with all the contracts relating to the installation of new services within the building specifically for this event and yes we have a lot of bandwidth with redundant links and if all else fails,  then have  an adsl2 backup (might be a bit slow for the 3000+ devices on the network).  Internally we have been planning for this event for well over 12 months, and technically I have been working for the last 5 months on this event. This event by far is the most stressful technology event we have at GCCEC however the team running the event makes it all go very smooth for us.  There are many things that go on behind the scenes to make this event possible for you guys from all the food (you all eat so much) to the 1000’s of bottles of drinks you will drink.

Another large element of Tech.Ed is the AV production to make it all look cool. Apart from the pallets of computers there is 1 shipping container & 3 semi-trailer loads of sound/vision & lighting equipment setup to make Tech.Ed happen.  There is also a lot of work done in relation to power & load sharing.  Do the math.. 2500 netbooks.. 400 odd PC’s = a lot of power + all the production equipment means you need to do a lot planning around this.  There has been a team of workers working in the keynote room for 3 days installing power boards into the seating so you can recharge your netbooks whilst you are in the keynote room and this is just one example of the things that go on outside the technology realm to make Tech.Ed what it is today.

Foxtel on the BIG Screen

- Nathan

There will be a time, after TechEd, where we triage our original planning: time, budget, hand-offs, tools and techniques. As a Quality Manager at one stage in my working life: project review, documenting them and absorbing learnings are critical.

On a lighter note this Friday: here are two youtubes for your viewing pleasure:

So you can get a sense of scale of our operation our professional photographer came in and shot some great photos..

Moving on today we’re starting building the showcase with our great HP demo servers – more on that later

- jorke

First day on site – and first post! My name is Kyle Rosenthal – and I am the content manager for HynesITe, the group that puts together the Hands on Labs(HOL) and Instructor led labs(ILL) for TechEd – here in AU, North America and now in Europe. This is our 5th TechEd Australia – starting all the way back at TechEd Canberra.

So – what goes into the deployment of the Hands-On area? A lot of time and effort! Basically we will be deploying to the HOL/ILL area on a total of 5Tb of data – to 180 machines – in 3 days.

This starts at my place generally for TechEd AU – Once a year my wife’s patience is tested as I take over the kitchen table and suck a year’s worth of power in one week.

This year is sort of different – we have moved to a new house and I have invested in a new table.

However – when Kim and Ted arrived – the kitchen table was back in use!

The setup this year for us is Windows Server 2008 R2, Hyper-V, and our custom support front end that will assist you the attendee in getting to your lab as quickly as possible. This does mean that in the process of getting ready  for the event we will start, save, snapshot and export about 300 virtual machines. In addition we then will touch all the lab manuals that help you to get through your lab.

To do this we have 8 machines in the back end that form our core infrastructure, with gigabit network everywhere.

So we loaded up the car and headed to the coast, the drive from Brisbane to the Gold Coast meant another hour’s worth of work time available for Kim and Corey – one in the front seat and one in the back seat – with laptops in use. This meant that in true geek style when a file needed to be transferred between their laptops – no usb device was needed. So – if you happened to be beside the Pacific Highway and saw an Ad-Hoc wireless network named “Driving” appear and then disappear – that was us.

Now we are complete and on site ready to start the next stage of the deployment.

Office in setup

Ted at work

More as we start our roll out.

Cheers
Kyle

After meeting Jack Morton’s Patrick, and donning a hi-vis vest – its into the Technical Learning Centre to witness the setup of the 2575 HP Mini-notes/netbooks.

There are about 13-14 long trestle tables, each with and average of 20 machines laid out. 8 staff are out and about unwrapping each box, putting it on the table, and initiating the Windows 7 TechEd image. This is a sight to behold. It feels somewhat like an Industrial Revolution-era factory. A study in time and motion.

Time is critical now. We cannot delay TechEd, nor stop the setup of the Technical Learning Centre – so its all hands to the wheel to complete the imaging tonight. Being a production process that is now in production, and costing per hour – it is way too late to change anything. Either the image, the process or technology involved in the imaging.

To give you a vision of what is going on – here is a quick youtube I created within 10 minutes of arrival.

Telstra has a major ongoing outage this morning: http://forums.whirlpool.net.au/forum-replies.cfm?t=1274075

Early reports are that it has taken out Next G, BigPond & Telstra Internet Direct – and certainly the 100mbps fibre service at the venue is dead.

Luckily, we are running APC off our redundant link which is a private circuit back to Pipe DC2 we blogged about earlier. This service is using different intercapital backhaul to Telstra so fortunately APC is completely unaffected (funnily enough we might patch the venue’s network over to ours so that they have Internet access ).

Telstra will be in tomorrow to install the 500mbps link but it is good to know we have the diversity up our sleeve in case something goes catastrophically wrong like this. Everyone thought we were bananas for ordering two 7200VXR routers for APC at Port Douglas last year … only to find the first one blew up within an hour of being turned on.

One of these things is not like the other one…

We recently mentioned that we completed deleting the WLC from the wireless equation at the venue. In rolling our the custom configs to the 50 autonomous devices we found that one of them returned an error.

One of these things is not like the other ones

It turned out that the radios were mounted in the chassis back-to-front. This changes the way that IOS enumerates the radios so that dot11Radio 0 and dot11Radio 1 are the opposite of all of the other APs. Consequently our templates scripts would fail on this unit.

Luckily the radios are completely modular with respect to the chassis:

1252AG with modular radios removed

And so popping them around the right way did not involve an RMA.

1252AG modular radio slot

I was rather impressed by the whole thing (we normally use 1232AGs which are not like this) … there had to be at least one sunny/bright point from these units given all of the brain damage they’ve inflicted on everyone over the past three months.

We previously posted performance metrics that were from a controlled set of tests with the APs patched into the venue network. We just did final tests yesterday (31 Aug) with the wireless network using our replacement Catalyst edge and core and the performance is consistent with what we saw earlier (after our rectification work, not before ). A good lappy will get 8MB/s (megabytes not megabits) from a 144 mbps associated link speed.

We (well David Eagles from iVolve) made some final recommendations regarding the RF utilisation on 5GHz. In the end we made the call to limit the maximum channel bandwidth on the 5GHz network to 20 MHz. This will ensure that there are 12 unique channels available at the expense of limiting the radio link speed for any client to 144mbps (instead of 300mbps). In practical terms this means a maximum download speed to your netbook of 8 megabytes per second instead of 16. Besides, it was a bit pointless having 300mbps radio link speeds supported when the edge of the network is 100mbps.

So in the final analysis we’re able to drive the wifi hard enough to saturate the RF – as it should be.

Oh yeah, and on those 400% throughput improvements we mentioned, there was one AP where we showed 700% improvement by making them all autonomous. Yay!

So we’re basically in production now for the folks attending the Australian Partner Conference. There is still a lot of work to do before tech•ed 2009:

• heaps of imaging work and issues that are keeping Jorke up until 2am
• bringing up and cutting over to a new 500mbps Internet service that is going in on Friday (yes, that is cutting it fine)
• Setting up the load balanced RRAS servers to cope with the expected load at tech•ed (we’re just using one for APC as that’s all we need to translate 100mbps)

We had to work through a few interesting EDNS0 issues with RRAS over the past few days – hopefully I’ll get a chance to write that up before Haysom overwrites those parts of my brain with helpdesk tickets.

Some random photos of work  starting in the main halls follow.

NetBook imaging

Rigging begins for tech•ed

More tech•ed rigging

Empty trestle tables, waiting for PCs in the Technical Learning Centre

Show machines setup

HP 2140’s being imaged and recharged

Jorke in his High-Vis Vest. How cute is he?

These photos were attached to an email from Simone Hulse:

Well it’s been an exciting few days for the build crew at Tech.Ed.  The technology team are very well established as they have been here for over a week.

• The mini-notes are imaging
• The show machines are complete
• CommNet and recharge is set in the foyers ready for APC (editor: Australian partner conference)
• The TLC is well underway (editor: TLC = Technical learning centre)
• Windows 7 Ideas Exchange lighting truss is in – and will be incredibly impressive once the lighting is complete
• The showcase banners and lighting rig is in
• The APC dinner staging is almost complete
• The rig is completely set in the roof, so now we can start building the expo from the ground up.
• Today is all about data cabling and power, tomorrow is about flooring and more cabling and then we build the expo on Friday

For those of you who are not hear yet.  The weather is warm in the day 22-30 degrees, but gets cool at night…. bring layers (and closed shoes!)

Very exciting.  Another edition later in the week.

In an event such as tech•ed, managing data cables is in itself, a job which could challenge the best of logistical minds.  There are over 1,200 Cat5e UTP cables to install or over 8,200m to lay down, plug into switches, test, secure to desks & stage plinths and of course plug into computers at the other end. Don’t forget the task of recollecting them all at the other end in a manner that doesn’t result in the biggest bunch of copper and plastic spaghetti in the world.

To help with the taming of this cable mammoth, a colour coding system is used to make what would normally be a challenging task just that little easier.  With experience, we’ve established the best combination of lengths. The colours were chosen based on what are standard available colours without the need for excessive custom makes :

• Grey Cables : 2m
• Yellow Cables : 3m
• Green Cables : 5m
• Orange Cables : 8m
• White Cables : 10m
• Purple Cables : 15m
• Blue Cables : 20m and over

The result is cable installers and the technology team can quickly identify the appropriate cable. It hasn’t always been like this. Before we had a colour coding system for the lengths we had a huge wastage issue with excess length. So much a number of years ago we reduced the cabling costs by 30%.

What happens to all of the cables after the event? You would appreciate the cost involved when replacing cables, not to mention the environmental impact. To maximise integrity, cables are used for no more than 2 events and so costs are amortised. If cables have been in trafficable areas, such as where trolleys and feet run over them, they are replaced after each event.  At the end of the event, the task of recollecting, rolling and sorting patch leads is job that may leave you chrysophobic, but the method of colour for length has made our “Where does that cable go” jigsaw easier to piece together.

• Making Wireless Work
• Diagnosing and resolving extremely high RF utilisation
• Resolving a Hunch– Wi-Fi Performance @ GCCEC

Insufficient Traction Error

We had been working extensively with a number of suppliers to get the WLC issues resolved. To be honest, I reached the end of my tether some time ago as every call seemed to revolve around either co-channel interference (so much so that is now the standard butt of jokes on the tech team … car won’t start? co-channel inteference :\), or some discussion of why there were methodological defects in the reports we’d written rather than just focusing on the obvious @#(*&@ issue.

We’d provided really rock solid documented reports showing throughput tests done in isolation with RF scans at the same time to show no interference.

At no time could we get more than 16-20% of the RF throughput we should. We could never saturate a 100mbps edge port even with 300mbps association speed. Despite our best efforts we have really been worn down by the whole experience. At one point I was told on a teleconference (paraphrased) “We would be amazed if there was a defect of this nature … especially so if someone from Queensland found it.” I took offence at that since we’ve had electricity and flushing toilets here since 1997.

Plan B

As mentiond in the linked articles above, we had a plan B all along to remove the APs from the WLC and install autonomous firmware on them all.

David Eagles just finished implementing that plan today (there is a bit of clean up to do tomorrow) … but well whaddaya know Jethro? In between tending to the cows and riding kangaroos, it turns out the country folk from Queensland might have been right all along:

Percentage of Theoretical Throughput, pre and post rectification work

Left is before, middle is after our work, right is theoretical throughput.

Look ma! Weeze yokles from North did good!

500% improvement FTW.

Life would be a lot simpler (well mine would anyway) if people took the time to read and understand the reports you put in front of them and work collaboratively on quality commercial outcomes.

davidc

TechOps (or will be) - This is the room from which we will run all technology related technical activities during APC and tech•ed 2009!

It fills me with a combination of dread and anticipation – not sure which is overriding. Anyway – about the Internets – we have an order in play to get a 500mbps service installed for tech•ed. We have also been installing a private 300mbps circuit between the Gold Coast Convention and Exhibition Centre and Pipe Networks DC2 as a diversity option and Plan B. For all our problems with City Pac and microwave options, it turned out that, upon actual examination of their records, Pipe were surprised to find they had fibre running RIGHT PAST THE FRONT DOOR.

We’re commissioning the Internet services on this link tomorrow (Tuesday the 25th of August) with the helpful guys from Over The Wire (the data centre we’re procuring the Internet connectivity from).

I’ll post some of the facts and figures from the commissioning exercise when we have collected the performance data.

P.S. Macca *possibly* could have made the effort to suggest I relocate. “Yeaaah mate … I heard the water in the pipe gurgling…” @#&^%@#@!

This post briefly explores the sorts of issues we have with DHCP on a large scale temporary network, and the sorts of things that go wrong.

Quick recap

DHCP is an application layer protocol that allows clients to negotiate for an IP address lease. The protocol also allows clients to obtain additional configuration parameters as a part of the lease process (e.g. an embedded device client might obtain the IP address of a server from which it can subsequently download firmware).

You might think that DHCP uses some sort of low-level trickery (hey, it must! You don’t even have an IP address yet!) but in fact it is a very simple UDP protocol that works via broadcasts. So, you are in fact using IP to obtain an address so that you can use IP.

The protocol works roughly as follows:

1. The client sends a DHCPDISCOVER broadcast in order to locate all DHCP servers on the local subnet. The client sends a list of options in this request for items it wants back from relevant DHCP servers.
2. Any and all servers respond to the client with a DHCPOFFER response. At the time of sending the offer, the server(s) will set aside an IP address for the client. The offer also includes additional dhcp options that have been requested by the client.
3. The client receives all of the responses and then chooses one to request as a lease (DHCPREQUEST). Once that is sent, any server that is NOT the server that offered the address that the client chose will return the offered address back to their pool for use by other clients.
4. The server that recognises the response (via a unique transaction ID used throughout the conversation) will respond with a DHCP acknowledgement and ‘lease’ the IP address to the client for the time specified in the policy on the server.

When DHCP Servers go bad

Every tech•ed there will be someone who brings:

1. A dinky little home router with a view to starting their own rogue wireless network
2. An entire enterprise infrastructure in VMs, including DHCP servers and so on, with their network adapter set to bridge the virtual switch with the local network

In our experience, we’ll get at least two to three rogue DHCP servers popping up at each tech•ed.

As you can see from the brief protocol overview above, there isn’t really any authentication or trust between the clients and the DHCP server. Therefore any rogue DHCP server will be treated with the same authority as our own and start handing out IP addresses for a different subnet and with a different router and so on. Once this happens, a large number of clients can aquire an incorrect IP address quite quickly and this creates a massive PITA for the helpdesk guys to tell everyone to do a release and renew to get an IP address that works.

Trusted and authorised servers

You might have noticed that when you use Windows DHCP server there are a few behaviours that might strike a networking traditionalist such as myself as odd:

1. If your DHCP server is a member of a domain, it will not start until someone with administrative credentials ‘authorises’ the server to serve addresses. You do that using the DHCP server MMC snap-in.
2. If your DHCP server is on a standalone server, and it is running, and it sees another DHCP server that is also running and that server is a member of a domain … then your standalone DHCP server will shut down.

I believe that both of these approaches are a little bit silly for a number of reasons, the primary of which being that rogue DHCP servers are rarely Windows Servers. A little busybox machine or home wifi router isn’t going to play in the above scheme and so you’re still exposed to IP addressing integrity issues.

We’re really dealing with a layer 3 network security/integrity issue here and the correct place to solve it is in the network itself, not in the operating system of some select servers that exist on that network.

DHCP Snooping

DHCP Snooping is a catch-all term that covers a number of techniques for ensuring the security and integrity of certain aspects of the edge of your network. You can think of DHCP Snooping as a type of light-weight firewall that sits in every switch on your network.

The main feature we’re interested from the various features in the DHCP Snooping arsenal is where by individual ports can be marked as trusted or untrusted for the purposes of DHCP traffic.

We are deploying a large number of Cisco Catalysts throughout the venue for the event. The standard units in the IDFs will have 24 x 10/100 copper ports, and 1 x fibre GBIC to connect the switch back to the core of the network. The standard configuration we will deploy to these switches will set all 24 10/100 copper ports as untrusted. We will nominate the fibre GBIC as trusted.

The result of this configuration is that packets such as a DHCPOFFER received on one of the untrusted ports are simply dropped. They never enter the network. It does not matter if the device on the edge port is a massive Windows Data Centre Edition server or a $89 home wifi unit … the treatment is at the packet level and the result is the same.

Problem solved!

Since we started insisting on Catalysts in all untrusted/edge scenarios, we’ve never had a repeat of the issues of the tech•eds of many years ago. The solution described above is implemented at the right place in the OSI networking model, and at the right place in terms of the overall topology of your network.

If you have rogue DHCP server problems then it is well worth looking into the benefits of DHCP Snooping. There are a lot of features under that name and I’ve only briefly touched on one that is relevant to the event here.

Further reading:

• http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/snoodhcp.html
• http://tools.ietf.org/html/rfc1531
• http://en.wikipedia.org/wiki/DHCP_Snooping

I recommend the first link as a very comprehensive overview of all of the aspects of DHCP Snooping that you can add to your network security arsenal.

The week started with an integration meeting. A group of us from across  the project (logistics, construction, technology, catering) lock ourselves away for half a day. We review each task from the moment the first person sets foot at the venue to the moment the last truck departs at the conclusion of the event. It’s a mind bending job, but on a project like this with literally hundreds of people making things happen, it’s an important milestone to have this integration meeting. You can imagine we always uncover a few issues where the schedules haven’t matched up entirely during the previous months of decisions. The document, with over 1,000 line items, then undergoes updates by everyone and then we come together to do it all again later this week.

Last week ended with another important milestone for the technology team. We locked down the technology requirements / change requests on Friday night. While the project owners signed off the statement of requirements months ago, we obviously have a period where we need to cater for changes to the needs of the various event owners and Exhibitors. This week it’s all about our documentation, training crew, preparing final briefs and this Thursday we send the first shipment of technology gear from Sydney to for the venue.

Pop over and have your say.

At this stage we should note we leaving the Netbooks open for you to install whatever you like; Microsoft or non-Microsoft. These are your machines – and we’d love you to give Windows 7 RTM a really good test.

Everyone loves fast Internets. Providing really fast Internets at tech•ed this year has provide to be a complete pain in the proverbial.

Give, and ye shall receive (unfortunately)

The Gold Coast Convention and Exhibition Centre has a permanent 100 meg service to Telstra Internet Direct. We know from last year that we were saturating the 100 meg service at the Sydney Convention Centre (we actually put on more bandwidth during the event) and so with that experience, and 2500 people wanting to download the lastest build of Chrome Vizact we know we’re in for a rough time if that is the only connectivity option available to us.

Our first port of call was obviously upgrading the service in the building. We learned however that that would require an infrastructure upgrade from Southport back to Brisbane. The original estimate for this was … lets just say well and truly out of our price range (we could have bought another 2500 netbooks, but we did manage to get it reduced significantly) however with the GFC and what not there is no way we could budget in the hundreds of thousands of dollars to ramp a 100 meg service up to a 500 meg service.

As luck would have it (or not as you will learn), On The Net has a dedicated DWDM fibre service in the City Pac building at 2 Miama Keys (next door, more or less). In the past few weeks we managed to do the following:

• Gain an agreement from On The Net that we could use the service in the City Pac building and that they would provide a fully managed layer 2 service from there back to Pipe Network’s DC3 or DC2 in Brisbekistan. Once you’re in a Pipe facility your connectivity options are opened up considerably as they’re telco neutral.
• Gain approval from Gold Coast City Council together with an affordable cabling estimate for us to run temporary fibre from the convention centre, across the bridge that connects T.E. Peter’s Drive to the little island to the left of the convention centre, and then hop off that onto a council light pole and into City Pac.
• Over The Wire then organised temporary upgrades with connectivity for the Australian Partner Conference and then a burst up to 300 mbps for the duration of tech•ed. Brent Paddon from Over The Wire had even orgnised some temporary upgrades on the Brisbane to Sydney intercapital link from one of their upstream providers for us.
• We received a verbal agreement from City Pac for access to their MDF and data centre.

“Sounds Great!” I hear you say … “What could possibly go wrong?”

City Pac went into receivership on the 31st of July.

I’ve spent the last week grovelling to the morticians at PPB Receivers who are doing the wind up (nice folk, Receivers … not). We’ve tried everything including offering to rent the space, providing an indemnification from Microsoft. David Haysom even tried to get a call set up between Gianpaolo Carraro (Microsoft’s DPE Director) and the CEO/MD of PPB – no dice. They just want to put the place on the market.

It is an understatement to say that I am annoyed that the team had the entire shooting match organised and we’ve come unstuck at the door to an MDF across the road from the convention centre.

Where to now?

I’ve spent pretty much the last ten days with a phone stuck to my head and talked with everyone with an Internet we can use. A number of options have surfaced, two of which are reasonably firm committments from a supplier that we can do 400mbps microwave from the comms deck at the convention centre to here:

Corporate Centre 1 as seen from the Gold Coast Convention Centre communications deck.

We’d jump onto the roof of that place, then do another microwave link across to Royal Pines. The other plan is we do more or less the same thing to get to Austar’s building. Both of those facilities can provide the fibre back to Brisbane.

There is a final terrestrial option that we are exploring in the next few days and I’d really like to see that come to fruition over using a radio solution. I’ll post an update if we manage to pull that off.

If we do end up with a radio solution to get the extra bandwidth, knowing our luck of late we’ll end up with a freak dust storm/storm/aliens attacking during the show.

The fallback scenario is of course the Telstra Internet Direct service that is already there – and that is not a bad outcome however it might mean congestion at peak break times.

Anyway, we have plans hatching and I’m sure we’ll be able to pull something off.

We’ve not been defeated at a tech•ed yet … though this one is trying my patience.

we shall fight on the comms deck, we shall fight in the IDFs and in the MDFs, we shall fight in the data centres; we shall never surrender.