Computer crash takes out most of WDW computers, slowly coming back up now

[Mark Goldhaber]

At approximately 1 p.m., all resort computers at Walt Disney World showed a message that they would be shutting down in two minutes, and to please save all work. The shutdown was reported to cast members as a glitch during the automatic distribution of software updates. However, in light of today's worldwide attack on all Windows 2000-based computers, we are now speculating that this was an actual infection by the worm-rbot.cbq worm.

Front desk, cashier, housekeeping, engineering, reservations and dispatch operations at the resorts were all affected. All run on Windows 2000. Theme park operations, likely on a different update schedule or different operating system, were not affected. External systems related to Disney's Magical Express (both the buses and the luggage operation) were not affected.

The problem affected only the front-end computers. The back-end servers and mainframes holding the actual data were up and running with no problems, and no data was lost.

When the shutdown alert appeared, resort staff rushed to print out as much information as they could, including check-in and check-out lists, but there was much more than could be printed in the time available. Throughout the afternoon, the computers came up for perhaps two minutes at a time before shutting down again. Resorts had to go to their emergency backup kits to manually dispense park passes and process check-ins and check-outs. Staff were positioned throughout the resort to manually let people into their rooms.

Disney has begun trying to push out a patch fix for the worm, and it is slowly propagating out through the system in fits and starts. Technical staff is now going to each machine throughout the resort to manually apply the patch to ensure that it correctly installs.

Systems started coming back online at 5:30 p.m. Resorts are now up with a minimum of one computer, but it will be quite a while before everything is back in full working order. We'll pass along any more information as we get it.

[Darkbeer]

http://today.reuters.com/news/NewsArticle.aspx?type=internetNews&storyID=2005-08-16T232013Z_01_HO683966_RTRIDST_0_NET-VIRUS-DC.XML

Today, at Disneyland and DCA the entire FASTPASS system was down, as was many of the parks cash registers, plus places like ESPN Zone in DtD... Plus other Disney locations were reporting problems..... Most registers are up and running noiw, but I presume they might be on backup systems....

[GusMan]

This worm is a real problem. While it may seem like an abrubt thing for Disney to do, trust me, it was the smartest move. This worm has been a pain in my side since SUNDAY afternoon.

I will most likely not sleep tonight because of it.

[Mark Goldhaber]

According to Lisa Haines, Vice President, Strategic Communications for Walt Disney Parks & Resorts:

"[S]ome of our domestic business units experienced a very brief interruption today due to the virus. There was minimal Guest impact though -- and all systems are now up and running again."

[Darkbeer]

Not at Disneyland, the FP machines were all closed, and while it was better (I presume due to backups), there was some serious problems, as MULTIPLE managers let me know when I was at the park today (DLR).....

[Mark Goldhaber]

I'll see if I can get some follow-up info from some of my contacts inside the parks. Thanks for the tip, Darkbeer.

[TempoNZ]

This worm is a real problem. While it may seem like an abrubt thing for Disney to do, trust me, it was the smartest move. This worm has been a pain in my side since SUNDAY afternoon.

I will most likely not sleep tonight because of it.

I hear ya! Luckily the machines that I'm responsible for (about 1500 plus corporate machines like enrolment details etc) were all patched or run XP so no problems there though one dept had about 300 machine out. We first experianced increaded netowrk traffic (normally first sign f a virus) at 11:21pm NZST) and ended up blacklisting about 500 machines in total. Scary thing is I only applied the patches last thing Friday and was going to wait until Monday as I hadn't really completed my testing to the extent I wanted. Likely we made the big call and went with the gut.

[GusMan]

Problem is that XP is impacted, but it is not as critical. And yes, I am still awake. I have a status call at 5:00 AM CT, then I can sleep for a while. Well, at least I am sucked into meeting heck!

Im tired.... real tired.... but I feel a tootsie roll sugar rush coming on.

[Mark Goldhaber]

Thankfully, we rolled the patch out to all of our servers and desktops over the weekend. We didn't lose a one.

[TowerofTerror]

wow it look like it just impacting major company and not people using windows xp

[WDW heart]

wow it look like it just impacting major company and not people using windows xp

My husband is a computer engineer. All of his clients are small healthcare companies many of which use 2000 and XP. None of his clients have been effected so far.

[SummerinFL]

So did this just affect DL or also WDW? Are they mostly back up and running?

[Andrew]

Disneyland's and Walt Disney World's computer systems are linked. The problem affected the entire system.

[SummerinFL]

Disneyland's and Walt Disney World's computer systems are linked. The problem affected the entire system.


Ah, thank you I wasn't aware. Learn something new everyday!:)

[Opus1guy]

Disneyland's and Walt Disney World's computer systems are linked.

Yep.

Back in the Northeast backstage area of the Magic Kingdom at WDW just East of the old "The Center" building, is one of the most secured, restricted and downright fortified facilities of The Walt Disney Company. It's a large unmarked almost bunker-like building surrounded by barbed wire fences and closed-circuit cameras. Special ID is required to enter even it's parking lot. Most Cast Members aren't even aware of it or what it's purpose is...tucked back in it's little corner of Disney property.

The building is known simply as "The Disc Building" and it's the home of "Disney Worldwide Services, Inc." Within this building are the massive systems that handle most non-attraction computing functions of a financial nature for the entire Company. Hotel, merchandise, purchasing, restaurant systems. Payroll for the entire Company. Accounting systems. Etc, etc.

It's linked to almost every Disney park and facility worldwide.

The Disc Building people are also responsible for maintaining a Backup Disaster Management Facility somewhere in upstate New York, that's staffed 24/7/365, to protect the data and accessibility to it, should anything knock out the Florida facility. It's quite an operation that few (even within Management, unless they're escorted) ever get to see first-hand from the inside.

[Andrew]

The building is known simply as "The Disc Building" and it's the home of "Disney Worldwide Services, Inc." Within this building are the massive systems that handle most non-attraction computing functions of a financial nature for the entire Company. Hotel, merchandise, purchasing, restaurant systems. Payroll for the entire Company. Accounting systems. Etc, etc.
And yet, they apparently run unpatched/unprotected Windows 2000. :eek:

From your description, I think it's this building (http://maps.google.com/maps?q=lake+buena+vista,+fl&ll=28.428345,-81.575192&spn=0.004473,0.006360&t=k&hl=en). Am I right?

[Opus1guy]

And yet, they apparently run unpatched/unprotected Windows 2000. :eek:

From your description, I think it's this building (http://maps.google.com/maps?q=lake+buena+vista,+fl&ll=28.428345,-81.575192&spn=0.004473,0.006360&t=k&hl=en). Am I right?

Yeah, that's it dead center there. Looks like they might have added on a new section at the North end of the building since I visited last (if you zoom in you'll notice the roofing looks different on that upper North-most block).

I really don't know what they run in what systems. Could be the w2000 is being run on the local clients and something else is on the servers in the Disc Building, and only the local clients experienced difficulties? It's all way over my head. But you would have thought they would have got the patch distributed out in time. Maybe they were still testing the patch before distribution when the thing hit?