Each year I’ve challenged our technical team to try something risky to keep us honest about leading with first rate technology implementation and practice. You can see our success on this blog and numerous press articles.

The first challenge I’ve identified this year is to move the whole event from a NAT’d network to a full blown public routed network, similar to what you see with an ISP. We’ve based this looking at last years usage pattern on our networks , and after some research we’ve come to this conclusion due to the lack of sensibly priced devices that can support 8000+ clients with many port mappings.

To some extent we achieved part of this last year with giving everyone a public routable ipv6 address with all ipv4 still being NAT’d. Our key requirement to support approximately 2500 delegates a few hundred staff and many, many ip enabled devices it to acquire a large range of pubic ip’s in the order of a /19,  /18 or /17 giving us 8192 or 16384, 32768 respectively, possible clients to support (if you don’t understand see CIDR).

Now the impending exhaustion of ipv4 address space commonly referred to as the IPocalypse you can imagine this kind of addressable space is not easily obtainable  – and can sometimes even go for a high price .  Last year we requested a temporary allocation from APNIC out of their portable/temporary range, which we used for a couple of weeks and shortly after returned. As a side note that same temporary range was used by LinuxConf in Brisbane a few months later, however sites like speedtest.net still cached the Microsoft tech•ed entry at the event… 

In our manner of planning ahead, last week we initiated our request to APNIC for a /17 or /18 or /19, as well as a temporary ipv6 allocation.. This was the response from APNIC.(note that the names and contact details have been scrubbed to protect the not so innocent – of course read from the ) ;

From: xxxxx@xxxxx
Date: Wed, Mar 23, 2011 at 11:47 AM
Subject: [APNIC #XXXXXX] Temporary IPv4 and IPv6 allocation for Microsoft TechEd
To: xxxxx@xxxx.xxx

Dear XXXXXXX,

Thank you for your email reply.

There is no IPv4 address space available for temporary allocation.

This reserved address space is no longer available as it has been placed
back in the APNIC free pool for distribution.

Please let us know if you wish to continue to obtain a temporary /48
IPv6 assignment from APNIC.

We look forward to hear back from you.
If you have and further questions, please let us know.

Kind Regards,
--
____________________________________________________________________
APNIC       sip:

http://www.apnic.net

_____________________________________________________________________

* Sent by email to save paper. Print only if necessary.
On Wed Mar 23 10:06:01 2011, xxxx@xxxx.xx wrote:
> Hi XXXX,
>
> What is the largest v4 block you can offer?
>
> I was under the understanding that APNIC had a block of space parked for
> temporary uses like this event (the netblock we used at TE last year was
> used at LinuxConf this year).
>
> XXXXXX.
>
> On Wed, Mar 23, 2011 at 10:01 AM, XXXX XX via RT  wrote:
>
> > Dear XXXXX,
> >
> > Thank you for your email.
> >
> > In regards to your request, due to the APNIC reaching Stage 2 of the
> > IPv4 Exhaustion plan, APNIC are unable to provide a temporary /17 IPv4
> > allocation.
> >
> > However, we are able to provide you a temporary /48 IPv6 assignment.
> >
> > Please let us know if you wish to continue to obtain a temporary /48
> > IPv6 assignment from APNIC.
> >
> > We look forward to hear back from you.
> >
> > If you have any questions, please let us know.
> >
> > Kind Regards,
> >
> > --
> > _____________________________________________________________________
> > APNIC       sip:
> > http://www.apnic.net                 phone:
> >                                       fax:
> > _____________________________________________________________________
> >
> > * Sent by email to save paper. Print only if necessary.
> > On Tue Mar 22 11:39:03 2011, xxxx@xxxx.xxx wrote:
> > > Hi there,
> > >
> > > We are in planning stages for Microsoft TechEd 2011 to be held in the
> > > Gold Coast Convention Centre this year. As per last year we would like
> > > to obtain a temporary allocation for use at the event.
> > >
> > > We are seeking the following:
> > >
> > > * IPv6 /48
> > > * IPv4 /17
> > >
> > > The earliest date we would be using these addresses is the 1st of July
> > > 2011.
> > >
> > > We would return the addresses on the 16th of September 2011.
> > >
> > > Could you please advise what you require for this allocation and when
> > > it might be approved so that we can do our physical network design.

So that’s it then – no more ipv4 space for us. We’ve committed to the path of deploying a fully routed network so it looks like it will be an:  IPv6 ONLY network!

We are working really hard to make sure that the key resources for the event (the schedule builder, for example) are available over IPv6.

As for other purposes not directly related to the event, we will not be offering IPv4 connectivity.

There will be an option to purchase time on Telstra NextG USB dongles at the event and we are working hard on coming up with a good sponsorship arrangement to help out with the cost of IPv4 network access during tech•ed.

- Jorke and the network team aka David Connors

TechEd 2010 Community Wall

In case you missed it, there was a giant 6 metre wide plasma wall at TechEd that streamed tweets (from Twitter obviously) and various images from Flickr. We took the following feeds:

1. Anything with the #auteched hashtag in it appeared on the tweet slide;
2. Anything with #news and #auteched hashtags in it from the user @auteched (the ‘official’ news feed for the event) was used for the blue news slide;
3. Anything from the TechEd Australia Flickr set was used for the photo sets.

The app itself was written using .NET 4 + WPF and Visual Studio 2010.

The PC that ran the display (“riceputer”, my blingy gaming rig, named after four-banger rice cars):

• Intel Core i7 920 @ 2.67GHz
• Radeon HD5850 (1GB RAM)
• 3GB of triple channel RAM

The display formed a centre-piece for the community in the expo hall. It was received really well by the delegates and hopefully this will become a permanent fixture at tech•eds of the future. The expo wall was used as backing on the main stage and there were a heap of beanbags and comfy couches around in front of it – the whole area had a really nice community feel. We saw heaps of people sitting around and tweeting to see if their tweets would pop up on the display (come on guys, as if we’d smoke-and-mirrors you – you’re all too smart for that )

Technical Direction Company (the audio visual supplier to the event) provided the display and the necessary matrix hardware. We provided a 1080p DVI feed (1920 x 1080) and they used a device called a Sypder (from Vista Systems) to split the top 729 pixels of that feed across all of the plasmas. You can see the initial calibration of the display in this video (sorry for the end bit, Patrick ).

In terms of application development (and it will be no surprise to any of the developers reading this): WPF is a thirsty beast. The application had to run 12 hours a day without a glitch or leak and this proved to be very difficult to achieve with WPF.

The individual character animations on the tweet screen were achieved by:

1. Downloading the tweets in a background thread and scrubbing that against a profanity filter;
2. Using GetGlyphTypeface() on the TextBlock object to read the metrics for the individual characters in the tweet (see this http://i.msdn.microsoft.com/dynimg/IC26720.png).
3. Create a separate TextBlock control for each of the characters in the tweet. We did this while honouring all of the kerning etc in the original font glyphs;
4. Animate each of the characters individually to achieve arbitrary layout effects and animations.

In retrospect, this was a LOT of work that probably was lost on people watching the display. If we had our time over we might have just stuck with animating an individual TextBlock or Canvas and transitioning the tweets in in one hit (as was the case on the blue News @ TechEd slide).

We found that using storyboard based animations was very problematic. Each time the tweets appear we bring in four at a time, in four sets for a total of 16 tweets meaning a worst case scenario of ~2500 individual items in the storyboard. We tried a lot of different approaches to interacting with the WPF storyboards but in all cases we found that there were subtle leaks, even for items that were well out of scope. Over time (and remember we’re running this all day and night) memory usage would creep up.

In the end, we resolved to ditch storyboards altogether and manually created the WPF animations and queued them up ourselves. Once we did this, the memory leakage issues disappeared and the app would sit on a steady 111 meg of RAM usage indefinitely.

We found that the static storyboarding in WPF was excellent for prototyping (i.e. just writing up XAML by hand) but as soon as you start modifying those in code you’re in for a world of pain. That isn’t really an isolated experience as far as we can tell – check out the memory usage on Blu, MetroTwit, etc.

For next year, it would be great to do this again using the new Twitter streaming APIs and XNA.

The video is available here: http://www.msteched.com/2010/Australia/voc208

The whole talk was very much ‘by the seat of our pants’ as we were insanely busy. Ben Parker, unfortunately, could not make it on stage due to other work commitments at the event. This was a real shame as he is ‘the man’ when it comes to the IT Pro side of the MS Enterprise software stack.

Unbelievably (given the rushed presentation and that most of the slide deck was a re-heated user group presentation I delivered a few months ago because we ran out of time ), we managed to score the #2 spot out of 165 sessions at the event, with an “Overall Satisfaction” of 4.74 out of 5. This is a great result but I cannot help but think there was a lot of other more deserving content at the event that probably should have rated higher.

We had a lot of good laughs – half of them at my expense. The notorious review comment from ARC301 came back to haunt me (2m 20s in … Thanks Jorke )

Evaluation scores follow:

Overall Results

Rollup Results

Comments

• http://www.noisetosignal.com.au/franklyspeaking/?p=253

Your friendly technology team

Thank you to everyone for the kind feedback on our recent BIGAU presentation both at and after the event. I couldn’t resist including this picture again as a) I love it, and b) it was the only politically correct on in the slide deck (sorry Princess Parker!).

We’re on site now and work has begun in earnest on provisioning the infrastructure for the event. Jorke and Ben are still in Brisbane busily setting up all of the HP blades for DemoNet and these will ship down to the venue on Wednesday.

Pile of work

We’re going to try and keep the blog up to date as we build out the network infrastructure – but as always, delivery of the event will take priority.

If you have any cool ideas for topics you would like to see covered on the blog, post a comment at the end of this article and we’ll see what we can whip up in response.

I will be presenting tech•ed backstage at the Brisbane Infrastructure Group. Guess appearances may include Nathan Wright from GCCEC and Ben Parker from Parker Tech Pty Ltd

Venue
Microsoft, Level 9, 1 Eagle Street, Waterfront Place, Brisbane City.
Time
1700 for Pizza.
Date
Tuesday, 13th of July 2010

Further details on #BigAU.

I’ll be covering off stuff that happened last year, stuff that’s happening this year + our wins and losses. Shock/Horror! I actually have some good news re Cisco WLCs this time around too!

See you there!

Cisco 4404 Marketing Poster

Long time readers of this blog (and the tech•ed 2009 team who have to put up with my whinging) will know that we had a significant number of challenges getting the wifi at GCCEC up to a scratch. Today I am happy to announce, however, that we have had a win. The 5508 delivers in the areas the 4404 didn’t.

Catch up on the complete history:

• http://www.techedbackstage.net/2009/07/15/diagnosing-and-resolving-extremely-high-rf-utilisation/
• http://www.techedbackstage.net/2009/07/22/resolving-a-hunch-wifi-performance-gccec/
• http://www.techedbackstage.net/2009/08/26/when-i-said-16-percent-i-meant-16-percent/

David Connors and David Eagles posing for a rare post survey photo op. "Weeze frum Queensland and weeze makin wirelesses!", commented Connors during a recent interview.

The bottom line issue was that all of our investigative work lead to the inescapable conclusion that the Cisco 4404 is a crock. I’d like to say that it was just a matter of the 4404 not supporting 802.11n data rates, but as the previous posts have shown, it would only barely perform at 802.11b rates in some cases.

We engaged with the original installers and Cisco TAC as well as local Cisco reps and pretty much got nowhere. Despite some early positive noises, they certainly did not want to own the issue and every time we would go back to them with detailed reports the responses we received were totally unacceptable (95% RF idle utilisation caused by co-channel interference on a network with 1 access point active and 1 user). Yes, I am still annoyed at the comment about Cisco being “amazed if there was a fault of this nature … especially if someone from Queensland found it.”

The most frustrating thing for me about the whole endeavour is that we spent a LOT of time and both Microsoft and GCCEC spent a LOT of money in having to go back and rectify someone else’s work. We also went to great lengths to ensure that we provided detailed reports that covered off exactly the clinical steps we went through to provide a final conclusion as to the problematic performance characteristics at the deployment.

In the end, we decided to remove the 4404 from the equation and deploy Cisco autonomous IOS images to all of the access points. This resolved the majority of the issues and, to us at least, confirmed that the 4404 was the root cause of all of our pain.

In the time since tech•ed 2009, GCCEC has replaced their 4404 WLC with a new 5508 unit (http://www.cisco.com/en/US/products/ps10325/). The product literature for the unit talks a lot about supporting 802.11n data rates.

O’RLY? What was the 4404 with AGN-capable Aironets on it supposed to do then?

4404 vs Autonomous

The move away from the WLC to autonomous firmware last year was a bit of a painful thing for us to do. We were torn because:

• The autonomous firmware would demonstrably provide 500%+ performance improvement of ther 4404 WLC.
• In moving to autonomous, we had to give up:
  • centralised management
  • AP-assisted roaming via CCX (see http://www.techedbackstage.net/2009/07/17/what-is-ccx-and-why-should-you-care/)
  • centralised wireless -> wired egress

We wanted to use the WLC but the performance characteristics were so dreadful it was not an option.

4404 in action

5508 Performance Characteristics

On the 26th of March 2010 we were engaged by Microsoft to validate the new 5508. David Eagles, Jorke and I spent the whole day on site with Nathan from the convention centre rerunning the entire survey from scratch. I am delighted to have to report that the 5508 has resolved the performance issues.

Simply swapping out the 4404 for a 5508 (and adding nothing to the RF delivery side of the wireless network) yieled:

• 484% average throughput improvement over 2.4GHz
• 340% average throughput improvement over 5GHz

26 March 2010 GCCEC Wireless Survey 2.4GHz Performance Comparison

26 March 2010 GCCEC Wireless Survey 5GHz Performance Comparison

2.4GHz Performance Comparison Across Aironet Deployment Scenarios

5GHz Performance Comparison Across Aironet Deployment Scenarios

So there you have it. There is still a bit of tuning for us to do regarding channel allocation (the same stuff we did for tech•ed 2009 – but was lost in the 5508 deployment) but it looks very promising.

A heartfelt THANK YOU to the GCCEC guys for the capital investment for the new unit for tech•ed 2010 – we really appreciate it.

Delegate Network 

• 99.998 uptime (was a 45 second outage from a GBIC flapping..)
• ~1300 wireless clients was the highest amount connected.
• ~2TB of data downloaded (in 4 days!)
• 300GB send out.

Demonstration Network (the glass box of doom with the servers inside) 

• 100% uptime on services with over 80 virtual machines hosted for speaker and other demos. This was highly under-utilized.

Major issues 

Oops

• Intel Wireless drivers – escalated to Intel who were sponsoring -
  they were kind enough to help us out
• Bit Torrent – Played network cop – Rickrolled access to key sites –  click here (I dare you)
• RRAS port exhaustion – escalated to our support team in India
• Live ID creation – another escalated to our support team…
• Netbook deployment – finished this with brute force
• Session Recording – Expression Encoder crashes
• Wrong URL on netbook images – network hack
• Well i think this is minor, but there was some slight damage of a netbook box…

As you can image there was so much more that went on, we’ve added a lot of these things to our planning process which has already started; 

“Life is like riding a bicycle. To keep your balance you must keep moving” – Einstein 

So here go, we’re hitting the reset button – we’ll do our best to keep you update-to-date from our end – no promises of course as we’re all focused on delivering to a higher quality outcome than before. 

- jorke 

In an event such as tech•ed, managing data cables is in itself, a job which could challenge the best of logistical minds.  There are over 1,200 Cat5e UTP cables to install or over 8,200m to lay down, plug into switches, test, secure to desks & stage plinths and of course plug into computers at the other end. Don’t forget the task of recollecting them all at the other end in a manner that doesn’t result in the biggest bunch of copper and plastic spaghetti in the world.

To help with the taming of this cable mammoth, a colour coding system is used to make what would normally be a challenging task just that little easier.  With experience, we’ve established the best combination of lengths. The colours were chosen based on what are standard available colours without the need for excessive custom makes :

• Grey Cables : 2m
• Yellow Cables : 3m
• Green Cables : 5m
• Orange Cables : 8m
• White Cables : 10m
• Purple Cables : 15m
• Blue Cables : 20m and over

The result is cable installers and the technology team can quickly identify the appropriate cable. It hasn’t always been like this. Before we had a colour coding system for the lengths we had a huge wastage issue with excess length. So much a number of years ago we reduced the cabling costs by 30%.

What happens to all of the cables after the event? You would appreciate the cost involved when replacing cables, not to mention the environmental impact. To maximise integrity, cables are used for no more than 2 events and so costs are amortised. If cables have been in trafficable areas, such as where trolleys and feet run over them, they are replaced after each event.  At the end of the event, the task of recollecting, rolling and sorting patch leads is job that may leave you chrysophobic, but the method of colour for length has made our “Where does that cable go” jigsaw easier to piece together.

TechOps (or will be) - This is the room from which we will run all technology related technical activities during APC and tech•ed 2009!

It fills me with a combination of dread and anticipation – not sure which is overriding. Anyway – about the Internets – we have an order in play to get a 500mbps service installed for tech•ed. We have also been installing a private 300mbps circuit between the Gold Coast Convention and Exhibition Centre and Pipe Networks DC2 as a diversity option and Plan B. For all our problems with City Pac and microwave options, it turned out that, upon actual examination of their records, Pipe were surprised to find they had fibre running RIGHT PAST THE FRONT DOOR.

We’re commissioning the Internet services on this link tomorrow (Tuesday the 25th of August) with the helpful guys from Over The Wire (the data centre we’re procuring the Internet connectivity from).

I’ll post some of the facts and figures from the commissioning exercise when we have collected the performance data.

P.S. Macca *possibly* could have made the effort to suggest I relocate. “Yeaaah mate … I heard the water in the pipe gurgling…” @#&^%@#@!

This post briefly explores the sorts of issues we have with DHCP on a large scale temporary network, and the sorts of things that go wrong.

Quick recap

DHCP is an application layer protocol that allows clients to negotiate for an IP address lease. The protocol also allows clients to obtain additional configuration parameters as a part of the lease process (e.g. an embedded device client might obtain the IP address of a server from which it can subsequently download firmware).

You might think that DHCP uses some sort of low-level trickery (hey, it must! You don’t even have an IP address yet!) but in fact it is a very simple UDP protocol that works via broadcasts. So, you are in fact using IP to obtain an address so that you can use IP.

The protocol works roughly as follows:

1. The client sends a DHCPDISCOVER broadcast in order to locate all DHCP servers on the local subnet. The client sends a list of options in this request for items it wants back from relevant DHCP servers.
2. Any and all servers respond to the client with a DHCPOFFER response. At the time of sending the offer, the server(s) will set aside an IP address for the client. The offer also includes additional dhcp options that have been requested by the client.
3. The client receives all of the responses and then chooses one to request as a lease (DHCPREQUEST). Once that is sent, any server that is NOT the server that offered the address that the client chose will return the offered address back to their pool for use by other clients.
4. The server that recognises the response (via a unique transaction ID used throughout the conversation) will respond with a DHCP acknowledgement and ‘lease’ the IP address to the client for the time specified in the policy on the server.

When DHCP Servers go bad

Every tech•ed there will be someone who brings:

1. A dinky little home router with a view to starting their own rogue wireless network
2. An entire enterprise infrastructure in VMs, including DHCP servers and so on, with their network adapter set to bridge the virtual switch with the local network

In our experience, we’ll get at least two to three rogue DHCP servers popping up at each tech•ed.

As you can see from the brief protocol overview above, there isn’t really any authentication or trust between the clients and the DHCP server. Therefore any rogue DHCP server will be treated with the same authority as our own and start handing out IP addresses for a different subnet and with a different router and so on. Once this happens, a large number of clients can aquire an incorrect IP address quite quickly and this creates a massive PITA for the helpdesk guys to tell everyone to do a release and renew to get an IP address that works.

Trusted and authorised servers

You might have noticed that when you use Windows DHCP server there are a few behaviours that might strike a networking traditionalist such as myself as odd:

1. If your DHCP server is a member of a domain, it will not start until someone with administrative credentials ‘authorises’ the server to serve addresses. You do that using the DHCP server MMC snap-in.
2. If your DHCP server is on a standalone server, and it is running, and it sees another DHCP server that is also running and that server is a member of a domain … then your standalone DHCP server will shut down.

I believe that both of these approaches are a little bit silly for a number of reasons, the primary of which being that rogue DHCP servers are rarely Windows Servers. A little busybox machine or home wifi router isn’t going to play in the above scheme and so you’re still exposed to IP addressing integrity issues.

We’re really dealing with a layer 3 network security/integrity issue here and the correct place to solve it is in the network itself, not in the operating system of some select servers that exist on that network.

DHCP Snooping

DHCP Snooping is a catch-all term that covers a number of techniques for ensuring the security and integrity of certain aspects of the edge of your network. You can think of DHCP Snooping as a type of light-weight firewall that sits in every switch on your network.

The main feature we’re interested from the various features in the DHCP Snooping arsenal is where by individual ports can be marked as trusted or untrusted for the purposes of DHCP traffic.

We are deploying a large number of Cisco Catalysts throughout the venue for the event. The standard units in the IDFs will have 24 x 10/100 copper ports, and 1 x fibre GBIC to connect the switch back to the core of the network. The standard configuration we will deploy to these switches will set all 24 10/100 copper ports as untrusted. We will nominate the fibre GBIC as trusted.

The result of this configuration is that packets such as a DHCPOFFER received on one of the untrusted ports are simply dropped. They never enter the network. It does not matter if the device on the edge port is a massive Windows Data Centre Edition server or a $89 home wifi unit … the treatment is at the packet level and the result is the same.

Problem solved!

Since we started insisting on Catalysts in all untrusted/edge scenarios, we’ve never had a repeat of the issues of the tech•eds of many years ago. The solution described above is implemented at the right place in the OSI networking model, and at the right place in terms of the overall topology of your network.

If you have rogue DHCP server problems then it is well worth looking into the benefits of DHCP Snooping. There are a lot of features under that name and I’ve only briefly touched on one that is relevant to the event here.

Further reading:

• http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/snoodhcp.html
• http://tools.ietf.org/html/rfc1531
• http://en.wikipedia.org/wiki/DHCP_Snooping

I recommend the first link as a very comprehensive overview of all of the aspects of DHCP Snooping that you can add to your network security arsenal.

Everyone loves fast Internets. Providing really fast Internets at tech•ed this year has provide to be a complete pain in the proverbial.

Give, and ye shall receive (unfortunately)

The Gold Coast Convention and Exhibition Centre has a permanent 100 meg service to Telstra Internet Direct. We know from last year that we were saturating the 100 meg service at the Sydney Convention Centre (we actually put on more bandwidth during the event) and so with that experience, and 2500 people wanting to download the lastest build of Chrome Vizact we know we’re in for a rough time if that is the only connectivity option available to us.

Our first port of call was obviously upgrading the service in the building. We learned however that that would require an infrastructure upgrade from Southport back to Brisbane. The original estimate for this was … lets just say well and truly out of our price range (we could have bought another 2500 netbooks, but we did manage to get it reduced significantly) however with the GFC and what not there is no way we could budget in the hundreds of thousands of dollars to ramp a 100 meg service up to a 500 meg service.

As luck would have it (or not as you will learn), On The Net has a dedicated DWDM fibre service in the City Pac building at 2 Miama Keys (next door, more or less). In the past few weeks we managed to do the following:

• Gain an agreement from On The Net that we could use the service in the City Pac building and that they would provide a fully managed layer 2 service from there back to Pipe Network’s DC3 or DC2 in Brisbekistan. Once you’re in a Pipe facility your connectivity options are opened up considerably as they’re telco neutral.
• Gain approval from Gold Coast City Council together with an affordable cabling estimate for us to run temporary fibre from the convention centre, across the bridge that connects T.E. Peter’s Drive to the little island to the left of the convention centre, and then hop off that onto a council light pole and into City Pac.
• Over The Wire then organised temporary upgrades with connectivity for the Australian Partner Conference and then a burst up to 300 mbps for the duration of tech•ed. Brent Paddon from Over The Wire had even orgnised some temporary upgrades on the Brisbane to Sydney intercapital link from one of their upstream providers for us.
• We received a verbal agreement from City Pac for access to their MDF and data centre.

“Sounds Great!” I hear you say … “What could possibly go wrong?”

City Pac went into receivership on the 31st of July.

I’ve spent the last week grovelling to the morticians at PPB Receivers who are doing the wind up (nice folk, Receivers … not). We’ve tried everything including offering to rent the space, providing an indemnification from Microsoft. David Haysom even tried to get a call set up between Gianpaolo Carraro (Microsoft’s DPE Director) and the CEO/MD of PPB – no dice. They just want to put the place on the market.

It is an understatement to say that I am annoyed that the team had the entire shooting match organised and we’ve come unstuck at the door to an MDF across the road from the convention centre.

Where to now?

I’ve spent pretty much the last ten days with a phone stuck to my head and talked with everyone with an Internet we can use. A number of options have surfaced, two of which are reasonably firm committments from a supplier that we can do 400mbps microwave from the comms deck at the convention centre to here:

Corporate Centre 1 as seen from the Gold Coast Convention Centre communications deck.

We’d jump onto the roof of that place, then do another microwave link across to Royal Pines. The other plan is we do more or less the same thing to get to Austar’s building. Both of those facilities can provide the fibre back to Brisbane.

There is a final terrestrial option that we are exploring in the next few days and I’d really like to see that come to fruition over using a radio solution. I’ll post an update if we manage to pull that off.

If we do end up with a radio solution to get the extra bandwidth, knowing our luck of late we’ll end up with a freak dust storm/storm/aliens attacking during the show.

The fallback scenario is of course the Telstra Internet Direct service that is already there – and that is not a bad outcome however it might mean congestion at peak break times.

Anyway, we have plans hatching and I’m sure we’ll be able to pull something off.

We’ve not been defeated at a tech•ed yet … though this one is trying my patience.

we shall fight on the comms deck, we shall fight in the IDFs and in the MDFs, we shall fight in the data centres; we shall never surrender.

[ The following performance analysis was submitted to tech•ed backstage by Arpan Gupta and the RRAS team. They own the RRAS/ipnat.sys components of Windows 7/Windows Server 2008 R2. Given that we're using it for address translation at tech•ed 2009, it is rather handy having them on board. We challenged the RRAS team to validate using RRAS as a NAT solution for 3000 concurrent clients; here is their excellent and helpful response. They did all the hard work with a physical simulation too! -- David ]

Background

NAT protocol is a well known solution to provide simplified internet connection sharing across multiple clients. Theoretically NAT supports 65536 TCP and same number of UDP address-port mappings over a single public address. Practically there’re some hardware bottlenecks which limits its performance. NAT, which is a part of ‘Routing and Remote Access’ services that come as a part of NPAS role with WS08R2, allows multiple hosts to connect to the Internet via a single or multiple external IP addresses. In home networking scenario, number of machines hardly crosses a limit of hundreds. But the number may reach to thousands if we deploy RRAS-NAT in enterprise networking. In some middle scale technical meets or conferences we require to manage 3000-4000 clients behind NAT. With RRAS-NAT we can easily achieve this target. A DHCP server can be used to provide private addresses dynamically along with, DNS server & default gateway (NAT private interface) and NAT will forward the packets to internet. We can also distribute the load via deploying multiple NAT boxes with unique DHCP address scopes. As said earlier, RRAS-NAT also supports a scenario where ISP assigns multiple external IP addresses (instead of one) but in this post we’ll target single external IP address and list out its performance.

Hardware Specification

For a set of 1500+ clients we’ve tested on a WS08R2 machine with below specification:

• 64 bit Quad-Core with 2GHz Clock processor
• RAM – 8 GB
• 2 NICs (Gigabit Ethernet)
• Assuming that external interface supports sufficient web access bandwidth (= Number of clients X Required average bandwidth per client).

Measured Performance

Use case scenario: 1500⁺ clients each having 5 open internet sessions and each of these sessions consumes 20 Kbps (Total 100 Kbps) bandwidth.

Result:

• We’ve observed up to 100 Mbps average data transfer speed across public interface of NAT flawlessly.
• With 1100 mappings (each of 100 Kbps sessions) CPU consumption was measured in order of 15%~20% and for 1500⁺ it was in order of 25%~30%.

Steps to configure a NAT box:

• Connect external NIC (Say X) to internet and assign a static IP A.B.C.D to another interface (Say Y).
• Install RRAS role and configure it via NAT path.

• Choose ‘X’ for internet NIC and ‘Y’ for intranet NIC.
• Install DHCP role and configure it. While configuring bind it to ‘Y’ NIC.
• Add a scope with 255.0.0.0 mask. (Please note that A.B.C.D should fall under it.)
• During various steps fill A.B.C.D as preferred DNS server & default gateway.

• Now connect the clients to ‘Y’ NIC using hub or switch.
• Clients will get IP address, default gateway & DNS server through DHCP server and NAT will route them to web.
• In case of more number of clients, deploy & configure more WS08R2 boxes. Please use unique address scope for each DHCP server.

Acknowledgements

David Connors, from TechEd, who came up with this requirement. R.Sai Ganesh, who arranged & configured required test setup. Mahmoud Elhaddad for providing technical details about IPNAT driver.

Further information

You can stay up-to-date with the RRAS team by pointing your browser or RSS reader here: http://blogs.technet.com/rrasblog/

Maybe I should step back a bit, back to November last year. Or further back.

So, this is my 10th Tech Ed as Technology Project Manager. Honestly I have the coolest job on the project (Kleefie always said he did). My role is to work across the business owners, sponsors, venue and technical guys where anything technology is involved. Pulling a truck load of bits together into some type of plan. That is under the watchful eye of Jorke as the technology lead for the project and David Connors as the Engineering lead.

We have a structured 5 phases approach to the delivery of large scale event technology. Long before you get to know about the good stuff like mini notebooks (possibly the coolest take away ever. So much so it has other Tech Ed events around the world wishing they had the idea first), we conduct a feasibility phase. Various Microsofties, the event agency team and the Engineering team comes together to look at how the project could work be at the proposed venue. We don’t work with a load of detail at that stage, but we look for the obvious pitfalls around timing, location, connectivity etc. This phase brings with it long brainstorming sessions and usually lots of research tasks. That in turn means I and a swag of others spend ages writing reports on the if, but, and may etc. All necessary stages for the business guys to make the big decisions. Once the decision is done setting dates, locations etc in stone we start the longer road of understanding what the business folks need the technology to deliver.

So, no Jorke I’m not going to be the International Man of Mystery and neither are the rest of the team. Stay tuned and over the next few weeks we will share more of what’s going on.

- DavidH