Each year I’ve challenged our technical team to try something risky to keep us honest about leading with first rate technology implementation and practice. You can see our success on this blog and numerous press articles.

The first challenge I’ve identified this year is to move the whole event from a NAT’d network to a full blown public routed network, similar to what you see with an ISP. We’ve based this looking at last years usage pattern on our networks , and after some research we’ve come to this conclusion due to the lack of sensibly priced devices that can support 8000+ clients with many port mappings.

To some extent we achieved part of this last year with giving everyone a public routable ipv6 address with all ipv4 still being NAT’d. Our key requirement to support approximately 2500 delegates a few hundred staff and many, many ip enabled devices it to acquire a large range of pubic ip’s in the order of a /19,  /18 or /17 giving us 8192 or 16384, 32768 respectively, possible clients to support (if you don’t understand see CIDR).

Now the impending exhaustion of ipv4 address space commonly referred to as the IPocalypse you can imagine this kind of addressable space is not easily obtainable  – and can sometimes even go for a high price .  Last year we requested a temporary allocation from APNIC out of their portable/temporary range, which we used for a couple of weeks and shortly after returned. As a side note that same temporary range was used by LinuxConf in Brisbane a few months later, however sites like speedtest.net still cached the Microsoft tech•ed entry at the event… 

In our manner of planning ahead, last week we initiated our request to APNIC for a /17 or /18 or /19, as well as a temporary ipv6 allocation.. This was the response from APNIC.(note that the names and contact details have been scrubbed to protect the not so innocent – of course read from the ) ;

From: xxxxx@xxxxx
Date: Wed, Mar 23, 2011 at 11:47 AM
Subject: [APNIC #XXXXXX] Temporary IPv4 and IPv6 allocation for Microsoft TechEd
To: xxxxx@xxxx.xxx

Dear XXXXXXX,

Thank you for your email reply.

There is no IPv4 address space available for temporary allocation.

This reserved address space is no longer available as it has been placed
back in the APNIC free pool for distribution.

Please let us know if you wish to continue to obtain a temporary /48
IPv6 assignment from APNIC.

We look forward to hear back from you.
If you have and further questions, please let us know.

Kind Regards,
--
____________________________________________________________________
APNIC       sip:

http://www.apnic.net

_____________________________________________________________________

* Sent by email to save paper. Print only if necessary.
On Wed Mar 23 10:06:01 2011, xxxx@xxxx.xx wrote:
> Hi XXXX,
>
> What is the largest v4 block you can offer?
>
> I was under the understanding that APNIC had a block of space parked for
> temporary uses like this event (the netblock we used at TE last year was
> used at LinuxConf this year).
>
> XXXXXX.
>
> On Wed, Mar 23, 2011 at 10:01 AM, XXXX XX via RT  wrote:
>
> > Dear XXXXX,
> >
> > Thank you for your email.
> >
> > In regards to your request, due to the APNIC reaching Stage 2 of the
> > IPv4 Exhaustion plan, APNIC are unable to provide a temporary /17 IPv4
> > allocation.
> >
> > However, we are able to provide you a temporary /48 IPv6 assignment.
> >
> > Please let us know if you wish to continue to obtain a temporary /48
> > IPv6 assignment from APNIC.
> >
> > We look forward to hear back from you.
> >
> > If you have any questions, please let us know.
> >
> > Kind Regards,
> >
> > --
> > _____________________________________________________________________
> > APNIC       sip:
> > http://www.apnic.net                 phone:
> >                                       fax:
> > _____________________________________________________________________
> >
> > * Sent by email to save paper. Print only if necessary.
> > On Tue Mar 22 11:39:03 2011, xxxx@xxxx.xxx wrote:
> > > Hi there,
> > >
> > > We are in planning stages for Microsoft TechEd 2011 to be held in the
> > > Gold Coast Convention Centre this year. As per last year we would like
> > > to obtain a temporary allocation for use at the event.
> > >
> > > We are seeking the following:
> > >
> > > * IPv6 /48
> > > * IPv4 /17
> > >
> > > The earliest date we would be using these addresses is the 1st of July
> > > 2011.
> > >
> > > We would return the addresses on the 16th of September 2011.
> > >
> > > Could you please advise what you require for this allocation and when
> > > it might be approved so that we can do our physical network design.

So that’s it then – no more ipv4 space for us. We’ve committed to the path of deploying a fully routed network so it looks like it will be an:  IPv6 ONLY network!

We are working really hard to make sure that the key resources for the event (the schedule builder, for example) are available over IPv6.

As for other purposes not directly related to the event, we will not be offering IPv4 connectivity.

There will be an option to purchase time on Telstra NextG USB dongles at the event and we are working hard on coming up with a good sponsorship arrangement to help out with the cost of IPv4 network access during tech•ed.

- Jorke and the network team aka David Connors

The video is available here: http://www.msteched.com/2010/Australia/voc208

The whole talk was very much ‘by the seat of our pants’ as we were insanely busy. Ben Parker, unfortunately, could not make it on stage due to other work commitments at the event. This was a real shame as he is ‘the man’ when it comes to the IT Pro side of the MS Enterprise software stack.

Unbelievably (given the rushed presentation and that most of the slide deck was a re-heated user group presentation I delivered a few months ago because we ran out of time ), we managed to score the #2 spot out of 165 sessions at the event, with an “Overall Satisfaction” of 4.74 out of 5. This is a great result but I cannot help but think there was a lot of other more deserving content at the event that probably should have rated higher.

We had a lot of good laughs – half of them at my expense. The notorious review comment from ARC301 came back to haunt me (2m 20s in … Thanks Jorke )

Evaluation scores follow:

Overall Results

Rollup Results

Comments

Your friendly technology team

Thank you to everyone for the kind feedback on our recent BIGAU presentation both at and after the event. I couldn’t resist including this picture again as a) I love it, and b) it was the only politically correct on in the slide deck (sorry Princess Parker!).

We’re on site now and work has begun in earnest on provisioning the infrastructure for the event. Jorke and Ben are still in Brisbane busily setting up all of the HP blades for DemoNet and these will ship down to the venue on Wednesday.

Pile of work

We’re going to try and keep the blog up to date as we build out the network infrastructure – but as always, delivery of the event will take priority.

If you have any cool ideas for topics you would like to see covered on the blog, post a comment at the end of this article and we’ll see what we can whip up in response.

I will be presenting tech•ed backstage at the Brisbane Infrastructure Group. Guess appearances may include Nathan Wright from GCCEC and Ben Parker from Parker Tech Pty Ltd

Venue
Microsoft, Level 9, 1 Eagle Street, Waterfront Place, Brisbane City.
Time
1700 for Pizza.
Date
Tuesday, 13th of July 2010

Further details on #BigAU.

I’ll be covering off stuff that happened last year, stuff that’s happening this year + our wins and losses. Shock/Horror! I actually have some good news re Cisco WLCs this time around too!

See you there!

I woke up at 4:00am this morning in a cold sweat. And I haven’t slept since. In the wee hours of the morning my brain, as is its want, was mulling a worst-case scenario: “What if we got the image or imaging wrong. How are we ever going to re-image these 2575 machines?”

In our planning, the imaging process was always date constrained. Whilst David Haysom had planned an extra 2 days, just in case, as there are always strange things that could bite you on the bum.

First day of Netbook handout. Today. 9am. Within an hour, we had handed out about 10% of the total.

And a problem appeared. Thankfully not hardware; the opening and recharging of each machine tickled out any weird hardware issues. In fact, our out-of-box hardware failure rate was half of the planned amount. HP repaired these quickly, and all was well.

Boot to nothing. Ouch!

The problem: missing images on a low, single percentage number of machines.

After reviewing our WDS/MDT2010 imaging server and technique by local and Redmond experts: this was deemed not the root cause for the missing images. Another possibility was Netbooks booting in the box, running out of power, and getting into a weird state. Again, this was discounted as there was just no data on the drives.

The Current Theory

The imaging process ends with the Netbook shutting down. From F12 to network boot to shutdown is one keypress: F12 to “boot from network”. At the completion of the multi-cast and unpacking in the WDS/MDT process, the Netbook shuts itself down to a black screen.

On the first two days of imaging last week, the imaging area had power issues: essentially, fluctuating power at the setup trestle tables. This caused machines to be left powered off. To a black screen.

Our thinking is that the black screen at the end of this process led to false-positives. Netbooks that had power issues, left on a black screen, were accidentally deemed “imaged” during the first few days and packed up.

Learning 1:  Leave setup machines with a static, on-state saying “OK” would be our recommendation if you are planning such a large, time constrained install.

The QA process we had during the imaging week was to test every 80 to 100 machines: that is, go through the out of box experience to test build quality.

Learning 2: factor in a slightly higher rate of QA checks than you think necessary.

How are we Fixing this?

Handing over a known good, working Windows 7 Netbook to each attendee is a must. No compromise. Therefore, today, we instituted a quickly developed solution to freshly image these machines.

After re-evaluation this evening, we are dedicating a team to process any machines with missing images. There is also another, faster process that is being investigated that may be implemented (via USB imaging)

So, worst case: we have upset attendees whilst we fix their machines. All I can say is “Sorry, and we are making it right” and I am confident this process will work tomorrow. With a little patience, all will be well.

Best case: our current glass-half-full thinking is that the machines that were handed out today were from the early in the week batch: where we had power issues, and black screen false-positives.

Tomorrow will tell.

Thanks for your patience and understanding.

And if you want to complain or vent at someone: find me.

My name is Nathan, I’m the Technical Services Manager here at GCCEC.  A lot of people ask me what Microsoft need off the venue and assume it’s plugging some computers into the venue’s network and all is done however this isn’t as easy as it sounds.  As you can see from the other posts on backstage we have run into a few hiccups with Tech.Ed 09.

My office at the top of the arena.. 'the loft'

First up,  the guys at codify specified to have their own independent network from edge to core.  In the past they have had some issues using the venues current production network  so the decision was made to drop a separate network in.  Codify and Jomablue sourced the appropriate hardware and we installed all the switches in the MDF and the 13 IDF’s around the place, and used our fibre infrastructure for the back bone. All the main backbones are 1GB fibre and from there it breaks down to trunked 100MB ports to the managed switches out on the floor and breakout rooms. The codify team has network monitoring of the entire network as well as remote alarms .

I’ve also had the fun task of dealing with all the contracts relating to the installation of new services within the building specifically for this event and yes we have a lot of bandwidth with redundant links and if all else fails,  then have  an adsl2 backup (might be a bit slow for the 3000+ devices on the network).  Internally we have been planning for this event for well over 12 months, and technically I have been working for the last 5 months on this event. This event by far is the most stressful technology event we have at GCCEC however the team running the event makes it all go very smooth for us.  There are many things that go on behind the scenes to make this event possible for you guys from all the food (you all eat so much) to the 1000’s of bottles of drinks you will drink.

Another large element of Tech.Ed is the AV production to make it all look cool. Apart from the pallets of computers there is 1 shipping container & 3 semi-trailer loads of sound/vision & lighting equipment setup to make Tech.Ed happen.  There is also a lot of work done in relation to power & load sharing.  Do the math.. 2500 netbooks.. 400 odd PC’s = a lot of power + all the production equipment means you need to do a lot planning around this.  There has been a team of workers working in the keynote room for 3 days installing power boards into the seating so you can recharge your netbooks whilst you are in the keynote room and this is just one example of the things that go on outside the technology realm to make Tech.Ed what it is today.

Foxtel on the BIG Screen

- Nathan

There will be a time, after TechEd, where we triage our original planning: time, budget, hand-offs, tools and techniques. As a Quality Manager at one stage in my working life: project review, documenting them and absorbing learnings are critical.

On a lighter note this Friday: here are two youtubes for your viewing pleasure:

After meeting Jack Morton’s Patrick, and donning a hi-vis vest – its into the Technical Learning Centre to witness the setup of the 2575 HP Mini-notes/netbooks.

There are about 13-14 long trestle tables, each with and average of 20 machines laid out. 8 staff are out and about unwrapping each box, putting it on the table, and initiating the Windows 7 TechEd image. This is a sight to behold. It feels somewhat like an Industrial Revolution-era factory. A study in time and motion.

Time is critical now. We cannot delay TechEd, nor stop the setup of the Technical Learning Centre – so its all hands to the wheel to complete the imaging tonight. Being a production process that is now in production, and costing per hour – it is way too late to change anything. Either the image, the process or technology involved in the imaging.

To give you a vision of what is going on – here is a quick youtube I created within 10 minutes of arrival.

Empty trestle tables, waiting for PCs in the Technical Learning Centre

Show machines setup

HP 2140’s being imaged and recharged

Jorke in his High-Vis Vest. How cute is he?

These photos were attached to an email from Simone Hulse:

Well it’s been an exciting few days for the build crew at Tech.Ed.  The technology team are very well established as they have been here for over a week.

• The mini-notes are imaging
• The show machines are complete
• CommNet and recharge is set in the foyers ready for APC (editor: Australian partner conference)
• The TLC is well underway (editor: TLC = Technical learning centre)
• Windows 7 Ideas Exchange lighting truss is in – and will be incredibly impressive once the lighting is complete
• The showcase banners and lighting rig is in
• The APC dinner staging is almost complete
• The rig is completely set in the roof, so now we can start building the expo from the ground up.
• Today is all about data cabling and power, tomorrow is about flooring and more cabling and then we build the expo on Friday

For those of you who are not hear yet.  The weather is warm in the day 22-30 degrees, but gets cool at night…. bring layers (and closed shoes!)

Very exciting.  Another edition later in the week.