I finished off the physical design for the MDF and IDFs a while ago and thought I would take the time to post these in case you wanted to build your own 3000 person corporate conference at home. I am only covering off the network side of things below – there is a massive HP + Microsoft private cloud solution going in at the event that will need post(s) in its own right.

In terms of fast Internets

• 1000 mbps of Internet connectivity provided by Telstra Internet Direct. This is a high performance e-line service back to Telstra Internet Direct in the Charlotte St exchange (i.e. it is a wholesale/service-provider quality service, uncontended back to their core).
• An additional 500 mbps of Internet connectivity from Over The Wire as a redundancy option.
• BGP routing under our own AS9984 to make the best use of the combined link.
• IPv4 and IPv6.

In terms of WiFi

• Cisco 5508 WLC
• 50 x Cisco Aironet 1252 access points
• Each AP has 2 x radio interfaces (2.4 and 5GHz) for a total of 100 radios supporting 802.11a, b, g, and n

In terms of physical infrastructure

• 2 x Brocade CER units for core routing
• 2 gbps LACP core trunking between the routers, core switches and wireless LAN controller
• 1 gbps distribution network
• 77 switches deploying 2000+ ports of connectivity

IDF Physical Design

MDF Physical Design

Also available as a PDF export of the original VSD:

Project 3031 – Tech·Ed 2011 Network Design and Implementation (Microsoft Pty Ltd) – Network Design

If you have any questions then post them below.

• Making Wireless Work
• Diagnosing and resolving extremely high RF utilisation
• Resolving a Hunch– Wi-Fi Performance @ GCCEC

Insufficient Traction Error

We had been working extensively with a number of suppliers to get the WLC issues resolved. To be honest, I reached the end of my tether some time ago as every call seemed to revolve around either co-channel interference (so much so that is now the standard butt of jokes on the tech team … car won’t start? co-channel inteference :\), or some discussion of why there were methodological defects in the reports we’d written rather than just focusing on the obvious @#(*&@ issue.

We’d provided really rock solid documented reports showing throughput tests done in isolation with RF scans at the same time to show no interference.

At no time could we get more than 16-20% of the RF throughput we should. We could never saturate a 100mbps edge port even with 300mbps association speed. Despite our best efforts we have really been worn down by the whole experience. At one point I was told on a teleconference (paraphrased) “We would be amazed if there was a defect of this nature … especially so if someone from Queensland found it.” I took offence at that since we’ve had electricity and flushing toilets here since 1997.

Plan B

As mentiond in the linked articles above, we had a plan B all along to remove the APs from the WLC and install autonomous firmware on them all.

David Eagles just finished implementing that plan today (there is a bit of clean up to do tomorrow) … but well whaddaya know Jethro? In between tending to the cows and riding kangaroos, it turns out the country folk from Queensland might have been right all along:

Percentage of Theoretical Throughput, pre and post rectification work

Left is before, middle is after our work, right is theoretical throughput.

Look ma! Weeze yokles from North did good!

500% improvement FTW.

Life would be a lot simpler (well mine would anyway) if people took the time to read and understand the reports you put in front of them and work collaboratively on quality commercial outcomes.

davidc

TechOps (or will be) - This is the room from which we will run all technology related technical activities during APC and tech•ed 2009!

It fills me with a combination of dread and anticipation – not sure which is overriding. Anyway – about the Internets – we have an order in play to get a 500mbps service installed for tech•ed. We have also been installing a private 300mbps circuit between the Gold Coast Convention and Exhibition Centre and Pipe Networks DC2 as a diversity option and Plan B. For all our problems with City Pac and microwave options, it turned out that, upon actual examination of their records, Pipe were surprised to find they had fibre running RIGHT PAST THE FRONT DOOR.

We’re commissioning the Internet services on this link tomorrow (Tuesday the 25th of August) with the helpful guys from Over The Wire (the data centre we’re procuring the Internet connectivity from).

I’ll post some of the facts and figures from the commissioning exercise when we have collected the performance data.

P.S. Macca *possibly* could have made the effort to suggest I relocate. “Yeaaah mate … I heard the water in the pipe gurgling…” @#&^%@#@!

This post briefly explores the sorts of issues we have with DHCP on a large scale temporary network, and the sorts of things that go wrong.

Quick recap

DHCP is an application layer protocol that allows clients to negotiate for an IP address lease. The protocol also allows clients to obtain additional configuration parameters as a part of the lease process (e.g. an embedded device client might obtain the IP address of a server from which it can subsequently download firmware).

You might think that DHCP uses some sort of low-level trickery (hey, it must! You don’t even have an IP address yet!) but in fact it is a very simple UDP protocol that works via broadcasts. So, you are in fact using IP to obtain an address so that you can use IP.

The protocol works roughly as follows:

1. The client sends a DHCPDISCOVER broadcast in order to locate all DHCP servers on the local subnet. The client sends a list of options in this request for items it wants back from relevant DHCP servers.
2. Any and all servers respond to the client with a DHCPOFFER response. At the time of sending the offer, the server(s) will set aside an IP address for the client. The offer also includes additional dhcp options that have been requested by the client.
3. The client receives all of the responses and then chooses one to request as a lease (DHCPREQUEST). Once that is sent, any server that is NOT the server that offered the address that the client chose will return the offered address back to their pool for use by other clients.
4. The server that recognises the response (via a unique transaction ID used throughout the conversation) will respond with a DHCP acknowledgement and ‘lease’ the IP address to the client for the time specified in the policy on the server.

When DHCP Servers go bad

Every tech•ed there will be someone who brings:

1. A dinky little home router with a view to starting their own rogue wireless network
2. An entire enterprise infrastructure in VMs, including DHCP servers and so on, with their network adapter set to bridge the virtual switch with the local network

In our experience, we’ll get at least two to three rogue DHCP servers popping up at each tech•ed.

As you can see from the brief protocol overview above, there isn’t really any authentication or trust between the clients and the DHCP server. Therefore any rogue DHCP server will be treated with the same authority as our own and start handing out IP addresses for a different subnet and with a different router and so on. Once this happens, a large number of clients can aquire an incorrect IP address quite quickly and this creates a massive PITA for the helpdesk guys to tell everyone to do a release and renew to get an IP address that works.

Trusted and authorised servers

You might have noticed that when you use Windows DHCP server there are a few behaviours that might strike a networking traditionalist such as myself as odd:

1. If your DHCP server is a member of a domain, it will not start until someone with administrative credentials ‘authorises’ the server to serve addresses. You do that using the DHCP server MMC snap-in.
2. If your DHCP server is on a standalone server, and it is running, and it sees another DHCP server that is also running and that server is a member of a domain … then your standalone DHCP server will shut down.

I believe that both of these approaches are a little bit silly for a number of reasons, the primary of which being that rogue DHCP servers are rarely Windows Servers. A little busybox machine or home wifi router isn’t going to play in the above scheme and so you’re still exposed to IP addressing integrity issues.

We’re really dealing with a layer 3 network security/integrity issue here and the correct place to solve it is in the network itself, not in the operating system of some select servers that exist on that network.

DHCP Snooping

DHCP Snooping is a catch-all term that covers a number of techniques for ensuring the security and integrity of certain aspects of the edge of your network. You can think of DHCP Snooping as a type of light-weight firewall that sits in every switch on your network.

The main feature we’re interested from the various features in the DHCP Snooping arsenal is where by individual ports can be marked as trusted or untrusted for the purposes of DHCP traffic.

We are deploying a large number of Cisco Catalysts throughout the venue for the event. The standard units in the IDFs will have 24 x 10/100 copper ports, and 1 x fibre GBIC to connect the switch back to the core of the network. The standard configuration we will deploy to these switches will set all 24 10/100 copper ports as untrusted. We will nominate the fibre GBIC as trusted.

The result of this configuration is that packets such as a DHCPOFFER received on one of the untrusted ports are simply dropped. They never enter the network. It does not matter if the device on the edge port is a massive Windows Data Centre Edition server or a $89 home wifi unit … the treatment is at the packet level and the result is the same.

Problem solved!

Since we started insisting on Catalysts in all untrusted/edge scenarios, we’ve never had a repeat of the issues of the tech•eds of many years ago. The solution described above is implemented at the right place in the OSI networking model, and at the right place in terms of the overall topology of your network.

If you have rogue DHCP server problems then it is well worth looking into the benefits of DHCP Snooping. There are a lot of features under that name and I’ve only briefly touched on one that is relevant to the event here.

Further reading:

• http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/snoodhcp.html
• http://tools.ietf.org/html/rfc1531
• http://en.wikipedia.org/wiki/DHCP_Snooping

I recommend the first link as a very comprehensive overview of all of the aspects of DHCP Snooping that you can add to your network security arsenal.

Everyone loves fast Internets. Providing really fast Internets at tech•ed this year has provide to be a complete pain in the proverbial.

Give, and ye shall receive (unfortunately)

The Gold Coast Convention and Exhibition Centre has a permanent 100 meg service to Telstra Internet Direct. We know from last year that we were saturating the 100 meg service at the Sydney Convention Centre (we actually put on more bandwidth during the event) and so with that experience, and 2500 people wanting to download the lastest build of Chrome Vizact we know we’re in for a rough time if that is the only connectivity option available to us.

Our first port of call was obviously upgrading the service in the building. We learned however that that would require an infrastructure upgrade from Southport back to Brisbane. The original estimate for this was … lets just say well and truly out of our price range (we could have bought another 2500 netbooks, but we did manage to get it reduced significantly) however with the GFC and what not there is no way we could budget in the hundreds of thousands of dollars to ramp a 100 meg service up to a 500 meg service.

As luck would have it (or not as you will learn), On The Net has a dedicated DWDM fibre service in the City Pac building at 2 Miama Keys (next door, more or less). In the past few weeks we managed to do the following:

• Gain an agreement from On The Net that we could use the service in the City Pac building and that they would provide a fully managed layer 2 service from there back to Pipe Network’s DC3 or DC2 in Brisbekistan. Once you’re in a Pipe facility your connectivity options are opened up considerably as they’re telco neutral.
• Gain approval from Gold Coast City Council together with an affordable cabling estimate for us to run temporary fibre from the convention centre, across the bridge that connects T.E. Peter’s Drive to the little island to the left of the convention centre, and then hop off that onto a council light pole and into City Pac.
• Over The Wire then organised temporary upgrades with connectivity for the Australian Partner Conference and then a burst up to 300 mbps for the duration of tech•ed. Brent Paddon from Over The Wire had even orgnised some temporary upgrades on the Brisbane to Sydney intercapital link from one of their upstream providers for us.
• We received a verbal agreement from City Pac for access to their MDF and data centre.

“Sounds Great!” I hear you say … “What could possibly go wrong?”

City Pac went into receivership on the 31st of July.

I’ve spent the last week grovelling to the morticians at PPB Receivers who are doing the wind up (nice folk, Receivers … not). We’ve tried everything including offering to rent the space, providing an indemnification from Microsoft. David Haysom even tried to get a call set up between Gianpaolo Carraro (Microsoft’s DPE Director) and the CEO/MD of PPB – no dice. They just want to put the place on the market.

It is an understatement to say that I am annoyed that the team had the entire shooting match organised and we’ve come unstuck at the door to an MDF across the road from the convention centre.

Where to now?

I’ve spent pretty much the last ten days with a phone stuck to my head and talked with everyone with an Internet we can use. A number of options have surfaced, two of which are reasonably firm committments from a supplier that we can do 400mbps microwave from the comms deck at the convention centre to here:

Corporate Centre 1 as seen from the Gold Coast Convention Centre communications deck.

We’d jump onto the roof of that place, then do another microwave link across to Royal Pines. The other plan is we do more or less the same thing to get to Austar’s building. Both of those facilities can provide the fibre back to Brisbane.

There is a final terrestrial option that we are exploring in the next few days and I’d really like to see that come to fruition over using a radio solution. I’ll post an update if we manage to pull that off.

If we do end up with a radio solution to get the extra bandwidth, knowing our luck of late we’ll end up with a freak dust storm/storm/aliens attacking during the show.

The fallback scenario is of course the Telstra Internet Direct service that is already there – and that is not a bad outcome however it might mean congestion at peak break times.

Anyway, we have plans hatching and I’m sure we’ll be able to pull something off.

We’ve not been defeated at a tech•ed yet … though this one is trying my patience.

we shall fight on the comms deck, we shall fight in the IDFs and in the MDFs, we shall fight in the data centres; we shall never surrender.

We left Brisbane bright and early on Thursday the 16th of July to spend a day with the guys from GCCEC to get to the bottom of this latest issue.

Step 1: Knowing what the first step is!

Random reboots, firmware updates, and twiddling of settings does not constitute logical fault finding. If you think there is a problem with a system there is only one place to start and that is at producing a clinical and unambiguous statement of the problem at hand. We almost always work in tricky multi-supplier scenarios at these events and so it is really important to reduce the problem to a clinical document we can all look at so you’re not fingering individuals or companies – but rather working together with all parties to nail the main pain.

With this in mind we set out allocating the first half of our day to an entirely tedious but necessary exercise of conducting a complete access point by access point, radio interface by radio interface survey of the venue (by hand ).  The goal of the survey was not to measure the RF characteristics of the wireless network (been there, done that) but rather the real-world throughput characteristics.

Step 2: Sanity check the fibre/copper network and backhaul before looking the wifi performance

GCCEC’s copper and fibre network is a relatively modern star topology of a single gigE fibre core servicing a number of 10/100 copper edge switches with fibre backhaul. This is good as exhibition venues that have grown over time tend to have a lot of zany and miscellaneous long fibre runs with random Vendor X fibre transceivers all over the place making you unsure of what exactly it is that you’re measuring. The GCCEC network on the other hand is simple to test definitively due to a sane overall architecture.

Additionally, the MDF that houses the core features in-room Foxtel. I could have done with that in the basement of the Sheraton Mirage at Port Douglas for last year’s Australian Partner Conference.

GCCEC MDF - Rather impressive as it is the first machine room I've been into with Foxtel installed, though I couldn't find the bar fridge !

Testing Rig @ MDF/network core

We deployed a high performance x64 machine with 4GB of RAM and IIS7 directly connected to the core switch via 1000BaseTX copper. We deployed several files to this IIS7 server being 1 megabyte, 10 megabytes, 50 megabytes, and 100 megabytes in size.

Consistency in client testing

We used the same clunky Lenovo (waves at the other David ) for all client tests to ensure that we were always using a consistent platform on both ends of the equation. This slowed us down somewhat because it meant we could not split up to make things faster, but I felt strongly that being able to say that all results have as high a degree of consistency as possible was very important. We configured the Lenovo for maximum power wastage performance on all interfaces to ensure that results would not be inconsistent if we were to conduct some tests off battery versus off mains. Wherever possible we used mains power for tests.

Testing procedure

We used a simple script with wget downloading the test files from the testing rig in the MDF three times. We recorded these results together with some other data at each test point (more on that later).

Validating the test rig and core

A Lenovo notebook was directly connected to a port on the core switch via 1000BaseTX copper. We did several transfer tests between the test rig and the Lenovo receiving approximately 800mbps of real world throughput which is pretty good with a notebook as a client any day of the week – and certainly sufficient for our tests.

Validating the edge switch at the first test point

The last step before heading off on the Tedium Express was to test the first edge port that services the first access points we were to test. Again, this was the case of running the test scripts from the Lenovo directly connected to the edge switch on the same VLAN as our private test network. These reported 99.99% of the maximum throughput one would expect from a 100 meg link – so that was certainly very good.

The performance of the copper and fibre network validated as satisfactory.

The next step was to confirm whether or not there was a systemic issue with wifi performance (as per my hunch) or whether there was a simple problem with a loose TNC connector on the one of two APs we saw as problematic during the last exercise.

Anyone who has worked with me knows that I’m a bit like an old dog with a bone when it comes to pushing for a quality outcome and ensuring that things do what they say on the tin. Given that I tend to push damned hard for what I think is the right thing to do – it also means that I was experiencing rather some trepidation at the survey as I’d possibly have some explaining to do regarding billable time if my hunch proved to be false.

Step 3: Conduct the wireless survey

The wireless component of the GCCEC network is comprised of the following elements:

• Cisco 4402 wireless lan controller
• 50 x Cisco 1252 abgn wifi access points
• Each access point has a 2.4GHz and 5GHz radio interface – you can think of these two radios as being akin to having two separate Ethernet ports in your PC … if you’re troubleshooting they BOTH need testing.

Nathan at GCCEC did a wonderful job of building us a mobile testing rig out of a road case, trestle table and ratchet straps + onboard GPOs a 50m power lead and anti-slip mats! Nath – you’re a legend! Christened the Tedium Express, this was our working environment as we tested each AP.

Toot Toot! All aboard the Tedium Express!

We wanted to ensure that we could conduct a single sweep of the entire venue and not have to go back out to reassess it as we only had funding clearance for a day on site. With this in mind we prepared a comprehensive spreadsheet listing:

1. each copper MAC address
2. each base radio MAC address
3. the associated IDF and edge switch port

The process we used was to park the mobile rig under (or as close as possible to) the antennas under test and collect:

1. the advertised radio link speed
2. the current channel allocation
3. three real-world download performance data points
4. Repeat steps 1-3 for 2.5GHz and 5GHz.

This meant 50 access points * 2 radios * 5 data points = 500 collected values.

The first dozen or so radios took nearly an hour and a half to collect the data from, primarily because the Lenovo was taking so long to switch between 2.4GHz and 5GHz radios. We therefore chose to survey the venue twice, once at 2.4GHz and a second time at 5GHz. Despite having to cover twice the distance we ended up doing the collection much more quickly as we weren’t waiting on client WLAN configuration.

Out of interest we chose the Lenovo as the client test rig as the ThinkPad connections software allows you to view a series of access points listed by MAC address and then choose a MAC address to associate to as your preferred access point. Given that we had a pre-prepared list of all of the access points in the venue together with their MAC addresses, this made validation of what we were testing at any point a snap.

Also, David Cormack from CBO was helpfully driving the wlan controller software and printed installation maps to further validate what we were doing. The whole process worked pretty well but the ‘law of big numbers’ meant that it was going to take a while no matter what – there were 100 interfaces to survey and that will never happen in an hour.

The entire survey took from approximately 0830 to 1430 to complete with four people.

I see dead radios

During the course of the survey we found the 5GHz radio in AP-24 to be faulty. It was extremely difficult to obtain an association with the 5GHz interface and when we did the radio link speed was 9mbps and real world download speeds were 3-4 kilobytes (no, that is not a typo) per second. We had to bench it to work out what was going on.

I don’t have a problem with heights per se as I’m fine on the top of a mountain or on a high rise balcony or observation deck. What I do have a problem with is these damned scissor lifts that are rickety and dodgy and wobbly. If that is not bad enough they actually can drive around when you’re 10m in the air.

I edited out the bit where we started percussive maintenance (I’m talking about the AP, not your forehead David Cormack ). As an aside, it is a pretty good view of the hall from up there. During the video you’ll hear a faint rasping sound … that is the GCCEC staff scraping every last skerrick of masking/gaff tape off the raw concrete floor to ensure the venue is spotless for the next show. Keep this view in mind when you’re walking around the nice carpeted expo halls with fancy games and Xboxes and other stuff … there are innumerable venue and staging guys really go to an amazing amount of work to transform that concrete shell into the event you know and love. While we did our survey they blacked out the entire hall to do metered comparisons of a new super-bright ‘green’ compact fluorescent lamp for the hall to ensure they’re a satisfactory brightness and colour temperature compared to the existing metal-halide units. In short – there is just so much stuff that goes on to prepare the minutiae of detail at these events that hours and hours of labour go into light globe selection alone – let alone the what technology team do.

Here’s another view of the area tech•ed will consume from ground level (opposite end of the hall from the previous video):

Step 4: Analyse the results

I digress … where was I? That’s right – the survey.

So after a couple of laps of the venue we came up with this:

GCCEC Wireless Throughput Survey Results

Some of you might see quoted figures of 2 megabytes or 5 megabytes per second and think that is pretty good for wireless, and truth be told if you were getting sustained rates like that out of some dinky home router I’d probably agree. Our expectations are somewhat higher and you have to remember that we’re really the only people at the venue there are only two acceptable performance outcomes:

• saturation of the RF
• saturation of the edge port

The results paint an entirely different picture though. On average, the RF segments of the network are only providing 17.80% of associated RF link speed on 2.4GHz and 16.14% of associated RF link speed on 5GHz.

The other problem with these results is that they are highly inconsistent. Across each of the three runs the individual tests could vary by double or half.

So it turned out that my initial hunch from the last day on site was correct … and we have a serious wifi performance problem on our hands. Ruh Roh!

Step 5: Find the culprit

By this stage we had chewed up most of the day. We had ascertained:

• there was a serious performance problem with wifi and we had quantified it in a clear and clinical way as was our original intention.
• that the problem lay between the access points and the wireless lan controller, or possibly the way the wireless lan controller terminated in the venue core switch.

While this was progress, we really wanted to provide more definitive findings that would further narrow down the source of the problem. I must admit I was very disappointed at the prospect of leaving the venue with only problems identified and no positive prescriptive advice.

Further isolation testing

We had AP-24 in hand so we retreated to the centre of the venue and proceeded to shut down each of the wireless access points. After doing this we patched AP-24 into the wireless lan controller VLAN and reconfigured it in 2.4GHz 802.11n mode (turning off it’s flaky 5GHz radio altogether). We re-ran the same test suite and found performance to be the same as any of the other access points in the venue during our initial survey. This was helpful as it showed us that the problem was not in some way related to load on the wireless controller or the number of access points talking to the controller.

Reflashing AP-24 as autonomous

You will recall from the previous article that we discussed how the access points in the network are running in lightweight mode and so can only run in conjunction with the wireless LAN controller (see this if you’re interested: http://www.cisco.com/en/US/products/hw/wireless/ps430/products_qanda_item09186a00806a4da3.shtml). We decided it would be nothing if not informative to ‘downgrade’ the access point from lightweight mode to autonomous mode.

Autonomous mode firmware gives the access point a more complete IOS feature set and allows it to bridge the 802.11 (wifi) and 802.3 (Ethernet) networks without a controller. After some mucking around we re-flashed AP-24 into autonomous mode and directly connected it to our test VLAN bypassing the controller VLAN all together.

Surprise surprise!

AP-24 Throughput with Autonomous Firmware - Now we're cooking!

With a link speed of 144 mbps you can expect approximately half of that in terms of real-world download speed, which would be 72mbps in this case. Let’s do some sums. Remember that hard drive sizes and network throughput are universally measured using base 10 arithmetic (1 megabyte is 8,000,000 bits), while file sizes on disk and memory consumption and so on are measured using base 2 arithmetic  (1 megabyte is 8,388,608 bits) … I know, I know I could use SI units for kibibytes and kilobytes but most people reading this would have no idea what I was talking about.

Anyway, ignoring framing overhead and so on to make this easier:

1. 72 x 1000 x 1000 = 72,000,000 bps
2. 72,000,000 / 8 / 1024 /1024 = 8.5ish megabytes of files downloaded per second.

Above we see 9.8, 8.9, and 9.1 as the figures from the test of AP-24 in autonomous mode. This is what I would call EXCELLENT!

Conclusion

We made a lot of progress in the day on-site.

1. We proved and documented that there is a systemic problem with wifi performance at the venue.
2. We found and removed a dead AP.
3. We proved that the edge and core networks are functioning as expected.
4. We proved that the poor wifi performance was not related to channel/RF intereference by our first isolation test.
5. We proved that the access points provide excellent performance in autonomous mode.

In short, we turned a hunch into a known and well documented problem.

We provided a written report of our findings back to the venue and they have taken it back to the installers to get the core issue looked at and the installers have basically said that the issue will be resolved ‘no matter what’ – Thank you guys – I love commitment.

We’ll provide a further update when we know definitively what the root cause was and when the matter is corrected to our satisfaction.

Don’t worry, we have a plan B if the above doesn’t get resolved. Fortunately for you this means you will get as good performance as possible at the event. Unfortunately for me it means that damned scissor lift and reflashing 50 access points to autonomous mode with individual configurations before the event and back again afterwards. Hmmm David Eagles; What are you doing this time next month?

Its no secret we’ve been spending a lot of time checking out the venue to ensure we’re getting adequate access to wireless and network facilities. On our last site visit we had the venue network engineers working with codify to figure out ‘black spots’ in the network and work towards a root cause. We were lucky enough to have a loner machine from Nick to test out the profile, the results of which you will see later, but the size factor of the machines make them really easy to cart around the venue, and I generally averaged about 330 Mbps on the network – of course that’s really easy to do when you’re the only person connected….

While the codify guys started getting down to business, we were tasked with another very important job;

Tasting the tech•ed and APC delegate menus….

Some days you have to work very hard, and I guess this was one of those days; leaving the codify guys to continue working on the network, we dragged the jomablue guys down for opinions on the menu. Now I’m not going to go into detail on all the items served but there was an amazing range that was presented to us, such as:

• Beef with Ginger and green peppercorns
• Ocean Trout
• Beef and mushroom Risotto
• Fruit Shots
• Seafood Paella
• Lamb Korma
• Lamb Cutlets

Now bear in mind that this was for both events so we can’t guarantee you’ll see any of them (unless you are going to both..) oh.. there was bunch of wines as well, but best leave you to enjoy that for the event itself. Some shots below…

The team led by “Fletch” at GCCEC did an awesome job of exciting us about what the delegates are going to experience food wise.

Those photos above don’t really do it justice and after we ate till we couldn’t eat no more, we rolled/staggered our way back to find the codify guys still working hard, apparently testing what appeared to be some strange speed results from a couple of the wireless access points.

So as you can see we’re working VERY HARD to give you the best level of service possible for tech•ed and APC…. well at least some of us are

- jorke

We’re going to give the warts and all view of what’s involved in doing this focusing around the pragmatic approach of logistic and planning for this event and those related such as:

• Products and technology used for the event
• How the logistical planning is done
• How to you build a network for such and event and all the prep work.
• Tons of other information in mind numbing detail!

Plus of course lots of statistics and prediction that all go into delivering this kind of technology for the event.

So first of all some introductions for the core delivery team;

Jorke Odolphi – Microsoft – Infrastructure Architecture Evangelist /  tech•ed Technology Guy – responsible for the delivery of the infrastructure technology and ensuring and enabling that Microsoft has the best technology on show.

Other duties include attending long meetings and staying awake, breaking down barriers in delivery, helping the business understand the logistics and food/wine tasting.

(source: flickr.com/photos/heroeshappen)

Codify – International Men of Mystery – These guys actually DO the delivery of all the core infrastructure to tech•ed such as network connectivity, Wireless Networks, building machines for speaker rooms and connecting everyone to the internet. Simply put if you’re going to be putting anything on the network at  tech•ed  these guys are involved in some form. As a result they’re slaves to the network for the event!

To protect their identity (and they’re camera shy) the picture on the right is an artists rendition.. (source wikipedia.org)

Jomablue – International Men of Mystery – again another group of camera shy individuals. These guys are the core of the logistics for the event. When any piece of technology is placed in a location in the event these are the guys bumping it in, plugging it in and making sure the the power is right and in fact everything is right.

To protect their identity (and they’re camera shy) the picture on the left is an artists rendition.. (source wikipedia.org)

Of course we’re only scraping the surface of what we all do, and we can’t forget all the other people we interact with such as the tech•ed core team, Gold Coast Convention and Exhibition Centre staff and if course all the awesome staging staff. We’ll be hearing from all those people leading up to the event, so sit back take it in and tell us what you’d love to hear from the bluelit parts of teched.

- jorke