Author Archive
-
Wifi in the keynote: This is how it is done
This is how it is done.
We had just shy of 1000 concurrent users in the keynote. All the while everyone was tweeting, uploading photos and we were still getting 1-2ms pings to Charlotte Street exchange in Brisbane.
As Borat would say, “Niiiiicee.”
-
TechEd build – Day 2
-
TechEd build – Day 1
-
auteched + apc 2011 Network Design
People often ask me for more information about the network design and implementation for tech•ed and APC.
I finished off the physical design for the MDF and IDFs a while ago and thought I would take the time to post these in case you wanted to build your own 3000 person corporate conference at home. I am only covering off the network side of things below – there is a massive HP + Microsoft private cloud solution going in at the event that will need post(s) in its own right.
In terms of fast Internets
- 1000 mbps of Internet connectivity provided by Telstra Internet Direct. This is a high performance e-line service back to Telstra Internet Direct in the Charlotte St exchange (i.e. it is a wholesale/service-provider quality service, uncontended back to their core).
- An additional 500 mbps of Internet connectivity from Over The Wire as a redundancy option.
- BGP routing under our own AS9984 to make the best use of the combined link.
- IPv4 and IPv6.
In terms of WiFi
- Cisco 5508 WLC
- 50 x Cisco Aironet 1252 access points
- Each AP has 2 x radio interfaces (2.4 and 5GHz) for a total of 100 radios supporting 802.11a, b, g, and n
In terms of physical infrastructure
- 2 x Brocade CER units for core routing
- 2 gbps LACP core trunking between the routers, core switches and wireless LAN controller
- 1 gbps distribution network
- 77 switches deploying 2000+ ports of connectivity
IDF Physical Design
MDF Physical Design
Also available as a PDF export of the original VSD:
Project 3031 – Tech·Ed 2011 Network Design and Implementation (Microsoft Pty Ltd) – Network Design
If you have any questions then post them below.
-
Infrastructure Saturday 2010 Slide Deck
We had a great time at Infrastructure Saturday on the 6th of November. An epic amount of work was put in by one Mr Mark Rhodes to make the event the success that it was.
The slide deck from our presentation is available here:
Infrastructure Saturday tech•ed Backstage 2010
Ben Parker presented quite a few slides on the enterprise software stack on DemoNet – content that we really missed out on delivering at the event.
We hope you enjoyed the talk – No evals through yet but in the wrap up a guy said that he gave us all “1″s because he thought that was the top score (not 5!) Doh!
-
We are presenting at Infrastructure Saturday!
Jorke, Ben, and I will be preseting a re-run of our popular session from TechEd 2010 at Infrastructure Saturday.
Where: Microsoft Brisbane Office (map)
When: November 6th, 2010
Register: http://infrasat.eventbrite.com
Cost: FREE!There are a heap of other great sessions (see more here: http://infrastructuresaturday.org/sessions/) on at the event so be sure to mark the day in your calendar!
-
Behind the scenes of the Giant Twitter/Flickr Wall
TechEd 2010 Community Wall
In case you missed it, there was a giant 6 metre wide plasma wall at TechEd that streamed tweets (from Twitter obviously) and various images from Flickr. We took the following feeds:
- Anything with the #auteched hashtag in it appeared on the tweet slide;
- Anything with #news and #auteched hashtags in it from the user @auteched (the ‘official’ news feed for the event) was used for the blue news slide;
- Anything from the TechEd Australia Flickr set was used for the photo sets.
The app itself was written using .NET 4 + WPF and Visual Studio 2010.
The PC that ran the display (“riceputer”, my blingy gaming rig, named after four-banger rice cars):
- Intel Core i7 920 @ 2.67GHz
- Radeon HD5850 (1GB RAM)
- 3GB of triple channel RAM
The display formed a centre-piece for the community in the expo hall. It was received really well by the delegates and hopefully this will become a permanent fixture at tech•eds of the future. The expo wall was used as backing on the main stage and there were a heap of beanbags and comfy couches around in front of it – the whole area had a really nice community feel. We saw heaps of people sitting around and tweeting to see if their tweets would pop up on the display (come on guys, as if we’d smoke-and-mirrors you – you’re all too smart for that
)Technical Direction Company (the audio visual supplier to the event) provided the display and the necessary matrix hardware. We provided a 1080p DVI feed (1920 x 1080) and they used a device called a Sypder (from Vista Systems) to split the top 729 pixels of that feed across all of the plasmas. You can see the initial calibration of the display in this video (sorry for the end bit, Patrick
).In terms of application development (and it will be no surprise to any of the developers reading this): WPF is a thirsty beast. The application had to run 12 hours a day without a glitch or leak and this proved to be very difficult to achieve with WPF.
The individual character animations on the tweet screen were achieved by:
- Downloading the tweets in a background thread and scrubbing that against a profanity filter;
- Using GetGlyphTypeface() on the TextBlock object to read the metrics for the individual characters in the tweet (see this http://i.msdn.microsoft.com/dynimg/IC26720.png).
- Create a separate TextBlock control for each of the characters in the tweet. We did this while honouring all of the kerning etc in the original font glyphs;
- Animate each of the characters individually to achieve arbitrary layout effects and animations.
In retrospect, this was a LOT of work that probably was lost on people watching the display. If we had our time over we might have just stuck with animating an individual TextBlock or Canvas and transitioning the tweets in in one hit (as was the case on the blue News @ TechEd slide).
We found that using storyboard based animations was very problematic. Each time the tweets appear we bring in four at a time, in four sets for a total of 16 tweets meaning a worst case scenario of ~2500 individual items in the storyboard. We tried a lot of different approaches to interacting with the WPF storyboards but in all cases we found that there were subtle leaks, even for items that were well out of scope. Over time (and remember we’re running this all day and night) memory usage would creep up.
In the end, we resolved to ditch storyboards altogether and manually created the WPF animations and queued them up ourselves. Once we did this, the memory leakage issues disappeared and the app would sit on a steady 111 meg of RAM usage indefinitely.
We found that the static storyboarding in WPF was excellent for prototyping (i.e. just writing up XAML by hand) but as soon as you start modifying those in code you’re in for a world of pain. That isn’t really an isolated experience as far as we can tell – check out the memory usage on Blu, MetroTwit, etc.
For next year, it would be great to do this again using the new Twitter streaming APIs and XNA.
-
Tech•Ed backstage 2010 – VOC208 – The Presentation
We did a talk on the infrastructure behind TechEd 2010 as one of the very last sessions of the event.
The video is available here: http://www.msteched.com/2010/Australia/voc208
The whole talk was very much ‘by the seat of our pants’ as we were insanely busy. Ben Parker, unfortunately, could not make it on stage due to other work commitments at the event. This was a real shame as he is ‘the man’ when it comes to the IT Pro side of the MS Enterprise software stack.
Unbelievably (given the rushed presentation and that most of the slide deck was a re-heated user group presentation I delivered a few months ago because we ran out of time ), we managed to score the #2 spot out of 165 sessions at the event, with an “Overall Satisfaction” of 4.74 out of 5. This is a great result but I cannot help but think there was a lot of other more deserving content at the event that probably should have rated higher.
We had a lot of good laughs – half of them at my expense. The notorious review comment from ARC301 came back to haunt me (2m 20s in … Thanks Jorke
)Evaluation scores follow:
VOC208 Tech•Ed backstage 2010!Friday, August 27 13:45 – 15:00Central ADavid Connors, Ben Parker, Jorke OdolphiAttendance Count:Questions Q10 Overall, I was satisfied with this Session. Q20 The speaker was knowledgeable about the subject matter. Q30 The speaker’s presentation skills helped me better understand the material. Q40 The session was effective in demonstrating the product, technology and/or solution. Q50 The knowledge/skills I gained are relevant to my role. Q60 This session was worth the investment of my time. Q70 I will recommend this session to others. Q80 Please provide additional constructive comments, suggestions, feedback. Overall Results
Evals Submitted Q10 Q20 Q30 Q40 Q50 Q60 Q70 19 4.74 4.74 4.58 4.68 4.53 4.53 4.58 Rollup Results
Question Avg % Strongly Disagree % Disagree % Neutral % Agree % Strongly Agree Num. Submitted Overall, I was satisfied with this Session. 4.74 26.32 73.68 19 The speaker was knowledgeable about the subject matter. 4.74 26.32 73.68 19 The speaker’s presentation skills helped me better understand the material. 4.58 5.26 5.26 15.79 73.68 19 The session was effective in demonstrating the product, technology and/or solution. 4.68 5.26 21.05 73.68 19 The knowledge/skills I gained are relevant to my role. 4.53 10.53 26.32 63.16 19 This session was worth the investment of my time. 4.53 15.79 15.79 68.42 19 I will recommend this session to others. 4.58 15.79 10.53 73.68 19 Comments
Please provide additional constructive comments, suggestions, feedback. Great look at the unsung heroes of etched. Great way to finish the conference. Great session to end Tech Ed on. More of an FYI session than a learning one but still really enjoyed it and found it interesting. interesting to know how the backend stuff is deployed so if i ever need to do something on a similar scale i will have an inkling of what to do Need to use standard template – one used very hard to read. This session was brilliant – I saw just how much work went into Tech.Ed to get it going, made me even more appreciative of you all! Wicked stuff! -
Frankly Speaking Episode 42
Andrew Coates and Michael Kordahi interviewed Jorke and I for their Frankly Speaking podcast. Tune in and listen; hopefully I was making sense with the amount of sleep I did have at that point of the events.
-
tech•ed 2010 / APC 2010 IT Press Round Up
Here is a quick collection of the press articles specifically about the infrastructure. There are a lot of others about Windows Phone 7 etc but we didn’t list those here.
- Microsoft Tech.Ed 2010 network shouldered the load, IT Wire:
http://www.itwire.com/business-it-news/networking/41617-microsoft-teched-2010-network-shouldered-the-load - Tech Ed tames torrent leechers, IT News:
http://www.itnews.com.au/News/229928,tech-ed-tames-torrent-leechers.aspx - Microsoft Tech.Ed 2010 goes IPv6, ZDNet:
http://www.zdnet.com.au/microsoft-tech-ed-2010-goes-ipv6-339305258.htm - How do they do IT: Microsoft’s Tech.Ed goes IPv6, Computer World:
http://www.computerworld.com.au/article/357238/how_do_they_do_it_microsoft_tech_ed_goes_ipv6/
This is a pretty good summary of the event in general:
More write-ups coming … no, really!
- Microsoft Tech.Ed 2010 network shouldered the load, IT Wire:
-
Day 2 Expo Hall Build
I managed to snap this quick video while waiting for some servers to build.
-
We come in peace!
Your friendly technology team
Thank you to everyone for the kind feedback on our recent BIGAU presentation both at and after the event. I couldn’t resist including this picture again as a) I love it, and b) it was the only politically correct on in the slide deck (sorry Princess Parker!).
We’re on site now and work has begun in earnest on provisioning the infrastructure for the event. Jorke and Ben are still in Brisbane busily setting up all of the HP blades for DemoNet and these will ship down to the venue on Wednesday.
Pile of work
We’re going to try and keep the blog up to date as we build out the network infrastructure – but as always, delivery of the event will take priority.
If you have any cool ideas for topics you would like to see covered on the blog, post a comment at the end of this article and we’ll see what we can whip up in response.
-
tech•ed backstage @ BigAU – Tues 13th July 2010!
I will be presenting tech•ed backstage at the Brisbane Infrastructure Group. Guess appearances may include Nathan Wright from GCCEC and Ben Parker from Parker Tech Pty Ltd
Venue
Microsoft, Level 9, 1 Eagle Street, Waterfront Place, Brisbane City.
Time
1700 for Pizza.
Date
Tuesday, 13th of July 2010Further details on #BigAU.
I’ll be covering off stuff that happened last year, stuff that’s happening this year + our wins and losses. Shock/Horror! I actually have some good news re Cisco WLCs this time around too!
See you there!
-
abducted. returned. changed.
Toothless slack-jawed yokels win again
Cisco 4404 Marketing Poster
Long time readers of this blog (and the tech•ed 2009 team who have to put up with my whinging) will know that we had a significant number of challenges getting the wifi at GCCEC up to a scratch. Today I am happy to announce, however, that we have had a win. The 5508 delivers in the areas the 4404 didn’t. Read the rest of this entry »
-
Never gonna give you up!
We mentioned in a previous post (see BitTorrent, traffic shaping and trusting users) that we had a small number of users who were unfairly monopolising network resources in order to download files via BitTorrent. The whole thing was a bit sad for me personally as I took it as a bit of a depressing display of the bad parts of human nature taking advantage of our deliberately liberal and generous policies on network access. We’ve been running the network the same way since 2004 and this is the first time we’ve seen people take advantage of it this way. Read the rest of this entry » -
BitTorrent, traffic shaping and trusting users
It has been a while since we have posted. We had the best intentions of writing a series of wrap up posts (JO has some overdue videos I made for him!) but after living in the convention centre for three weeks we were all really exhausted. The fact that tech•ed 2010 planning is already well underway means we sort of need to wrap up the 2009 blog loose end (that and I recently posted to the ausdotnet mailing list that you have to be careful allocate enough time to a blog so it doesn’t become abandonware – only to have two people mail me and say “What? Like TechEd Backstage??”
)Anyway, I wanted to write a bit about a particular position I advocate each year and how disappointing to was to see that position abused by a very small number of network users to the detriment of other users at peak usage times. Read the rest of this entry »
-
Backstage on ZDNet!
Jorke and I ended up in a ZDNet video for a quick tour of back-of-house. Alas, my carefully laid plan to stay behind the lens for the duration of the event came unstuck!!!
Original article on ZDNet: http://www.zdnet.com.au/insight/hardware/soa/Backstage-tour-at-Tech-Ed/0,139023759,339298840,00.htm
[ I knew we'd get him on camera at one point, btw the guy you whos head you don't see is Nathan from GCCEC - jorke]
-
Now for something completely different – safety and destruction
I managed to grab Patrick Cronin (Jack Morton) on the floor for a few minutes during strike (“set” or “bump in” = building the event, “strike” or “bump out” = pulling everything apart). Patrick is responsible for lots of production management work that does NOT involve computers and networks! He is indefatigable and always smiling – quite a pleasure to walk past and nod at in mutual agreement that neither of us have the foggiest what the other actually does.
-
Connors to Blog. Copy?
We still have a bit of content to post before we wrap up this blog for 2009. Sorry we have been a bit quiet but a LOT went on in the last days of the event and almost everyone in technology has had some sort of challenge in their lives outside of the event.
We’ll wrap things up over the next couple of weeks. There is some pretty interesting content coming so please stay with us.
-
Intel wifi driver issue resolution
We did some further investigation into the wireless issue and came up with a work around. We’ve found that if you set your radio to 802.11a only, the issue will disappear. After doing this, one of the tech team managed to keep a Skype video call back to NetOps going across the entire venue without a hitch, while this same machine BSOD’d yesterday before it even left our office.
The helpful chaps on the Intel stand managed to come up with signed drivers that correct the issue (significantly higher version number than the in-box ones with Win7 RTM) however they’re still trying to get us the okay to release these. Interestingly they suggested that the work around was to disable one of the radios and the escalation technician said to turn off 5GHz – but please don’t do that.
For now, if you have an Intel wifi NIC, I recommend turning off the 2.4GHz radio as there is far greater channel capacity and bandwidth on the 5GHz radios.
- I <3 Helpful People
The Netbooks have a different chipset that are unaffected by this issue however you will likely experience better performance if you turn off 2.4GHz.
-
Day 1
-
Intel wifi driver issues
We’ve had a few reports today from people experiencing the following:
- Wifi becomes unresponsive including being stuck ‘on’ so that hardware switch does not turn off the interface.
- Blue-screen-of-death (only one person on the technology team experienced this).
We did a lot of investigative work and determined that the issue is related to the driver stack in the affected clients. The issue only affects uses who are using Intel wireless NICs. The issue does not affect Broadcom (in the netbooks), Atheros, etc.
We managed to latch on to a very helpful man on the Intel stand in the exhibition who confirmed that this is a known issue where:
- There are a large number of access points.
- There are a large number of clients.
- There is roaming.
Sound familiar?
The underlying issue is a limitation in a table that manages the number of access points in range – when this is exceeded the driver stack will crash. This is exacerbated by roaming.
We turned off CCX4 extensions today and that seemed to helped the issue a small amount by being less agressive in how it manages client roaming.
We’ll turn CCX4 on tomorrow for the keynote and then turn it off for the rest of the event. Hopefully Intel can give us a beta driver that fixes the issue. If we get it, we’ll put it up on the blog.
-
The most important job at the event
-
SCOOP!! MOBY IS DOING THE KEYNOTE!!
The things you find when you take a wrong turn. I was walking to the MDF through back of house and actually stumbled into the keynote rehearsal to find that they’ve flown in MOBY for it.
AWESOME!
It is actually Gianpaolo Carraro, Microsoft’s DPE Director. We did a cook’s tour of the venue for him and his family during set and all enjoyed a nice BBQ that evening as a rare bit of downtime. He’s an extremely good sport for doing this and letting me post it.
-
Random Jorke Vids
Wow. I had some time free this morning and finally got to empty my phone. He are a random Jorke videos from last week (or the week before … I don’t know what day it is any more).
The F12 madness:
Jorke taking some servers out for a spin:
-
End of a sort of day off
We’re at the end of a day off mid way through the events. It was a sort of day off because most of us were working at least part of the day. On a totally non-technical note, I just snapped this pic from the rooftop of Wave Apartments where we are staying. It is kind of nice to stop and reflect on how far we’ve come and somehow posting this pic of GCCEC while having a frostie at a BBQ with some of the team seems appropriate.
-
Diversity is good
Sometimes people think we’re over killing infrastructure asking for two or three more than we need of everything in network operations.
Telstra has a major ongoing outage this morning: http://forums.whirlpool.net.au/forum-replies.cfm?t=1274075
Early reports are that it has taken out Next G, BigPond & Telstra Internet Direct – and certainly the 100mbps fibre service at the venue is dead.
Luckily, we are running APC off our redundant link which is a private circuit back to Pipe DC2 we blogged about earlier. This service is using different intercapital backhaul to Telstra so fortunately APC is completely unaffected (funnily enough we might patch the venue’s network over to ours so that they have Internet access
).Telstra will be in tomorrow to install the 500mbps link but it is good to know we have the diversity up our sleeve in case something goes catastrophically wrong like this. Everyone thought we were bananas for ordering two 7200VXR routers for APC at Port Douglas last year … only to find the first one blew up within an hour of being turned on.
-
Random wireless network commissioning bits
(This post has been in my drafts since last week … we’re post commissioning on wireless now)
One of these things is not like the other one…
We recently mentioned that we completed deleting the WLC from the wireless equation at the venue. In rolling our the custom configs to the 50 autonomous devices we found that one of them returned an error.
One of these things is not like the other ones
-
The imaging begins
-
When I said 16 percent I meant … 16 percent
You will recall our earlier posts regarding the performance problems with the Cisco WLC at the Gold Coast Convention and Exhibition Centre. If you’ve not seen them catch up here:
- Making Wireless Work
- Diagnosing and resolving extremely high RF utilisation
- Resolving a Hunch– Wi-Fi Performance @ GCCEC
Insufficient Traction Error
We had been working extensively with a number of suppliers to get the WLC issues resolved. To be honest, I reached the end of my tether some time ago as every call seemed to revolve around either co-channel interference (so much so that is now the standard butt of jokes on the tech team … car won’t start? co-channel inteference :\), or some discussion of why there were methodological defects in the reports we’d written rather than just focusing on the obvious @#(*&@ issue. Read the rest of this entry »
-
Update on the Internets
Tomorrow is my first day on site at the venue (I don’t get to leave again until the 11th of September). I was supposed to be down there today but some lingering issues with the 2140 imaging procedures and timings delayed my trip to the convention centre by a day. David Haysom sent me an empty e-mail today with the subject of “Techops no turning back” and this image attached:
TechOps (or will be) - This is the room from which we will run all technology related technical activities during APC and tech•ed 2009!
-
Ensuring IP address allocation integrity with DHCP snooping
Nearly every IP network you use allocates IP addresses to clients via DHCP. There is a lot you can do with DHCP and it is a fairly well thought out and extensible successor to BOOTP.
This post briefly explores the sorts of issues we have with DHCP on a large scale temporary network, and the sorts of things that go wrong.
-
Internet Connectivity… plan A to F, plus a few more
I’ve not forgotten to post – I’ve just been busy undebacling a debacle.
Everyone loves fast Internets. Providing really fast Internets at tech•ed this year has provide to be a complete pain in the proverbial.
-
Windows Server 2008 R2 NAT Performance – Guest post by the Windows Product Team!
[ The following performance analysis was submitted to tech•ed backstage by Arpan Gupta and the RRAS team. They own the RRAS/ipnat.sys components of Windows 7/Windows Server 2008 R2. Given that we're using it for address translation at tech•ed 2009, it is rather handy having them on board.
We challenged the RRAS team to validate using RRAS as a NAT solution for 3000 concurrent clients; here is their excellent and helpful response. They did all the hard work with a physical simulation too! -- David ]Read the rest of this entry »
-
Resolving a hunch – wifi performance @ GCCEC
Previously you will recall we talked about the methodology we used to diagnose why the RF utilisation at GCCEC was so stratospherically high in relation to the actual wifi network utilisation and number of associated clients. In the last moments of that day on site we did observe a few anomalies with regards to real-world network performance. Given that the wireless infrastructure is state of the art and was one of the first “enterprise” deployments of 802.11n in Australia 12 months ago – this was odd and definitely warranted further investigation before the event even if to find there wasn’t a problem at all.
We left Brisbane bright and early on Thursday the 16th of July to spend a day with the guys from GCCEC to get to the bottom of this latest issue.
-
What is CCX and why should you care?
Early on in the history of wifi Cisco created some proprietary extensions called CCX (Cisco Compatible Extensions). These are now an open standard that a lot of network card manufacturers implement. If you’re bringing your own laptop to the event you should make sure you have CCX-4 compliant drivers, here’s why …
-
Diagnosing and resolving extremely high RF utilisation
What’s wrong with these pictures?
Per AP TX/RX figures with channel utilisation across the ground floor of GCCEC
Number of Client Associations across the ground floor of GCCEC
I was given logon access to the WCS console at GCCEC at the start of May this year. Shortly (10 minutes) later I started e-mailing “DANGER, WILL ROBINSON!” messages to the venue and the tech•ed technology team.
-
Making wireless work
If you’re interested in tech•ed and have been reading the blogosphere you’ll know that every paying delegate is scoring an awesome HP Mini 2140 as a freebie! This is fantastic news for delegates and a real headache for the propeller heads behind tech•ed. We now know that we’re going to have 2500 laptops, plus delegate’s bringing their own laptops, plus wifi-enabled mobile phones, plus Microsoft staff, speakers, crew, and all of the leechers holiday-makers in Gold Coast apartments across the road to cater for as wireless users.
Read the rest of this entry »
Australian Partner Conference 2009 auteched brain damage cabling CCX chrysophobia cisco codify gccec HP mini-note imaging internet internet connectivity ipv6 jomablue liveid microsoft nat netbooks networking odd issues planning power rras tech.ed teched Tech•Ed telstra wireless wlan
WP Cumulus Flash tag cloud by Roy Tanck requires Flash Player 9 or better.
Archives
- August 2011 (4)
- June 2011 (1)
- April 2011 (1)
- November 2010 (1)
- October 2010 (1)
- September 2010 (4)
- August 2010 (3)
- July 2010 (1)
- March 2010 (2)
- February 2010 (2)
- September 2009 (22)
- August 2009 (11)
- July 2009 (8)