[DM-MUG] Fwd: Re: [MacLaw] Sidekick Customer Data Lost in the Clouds

[Victoria L. Herring]

>Here's what my source inside the situation is saying.  Pretty sure it's not
>*entirely* MS' fault, even if they're getting the majority of the blame. :
>Danger, purchased by Microsoft, was moved into a Verizon Business datacenter
>in Kent, WA a short while ago. While this had to do with the MS
>assimilation, it was done as a one for one move from Danger to a DC that MS
>uses heavily. (MS didn't re-write, port, migrate to winblows, etc.) The
>backend service uses a variety of hardware, load balancers, firewalls, web
>and application servers, and an EMC SAN (Storage Area Network, think huge
>drive array connected with fiber.)
>
>Well last Tuesday, the EMC SAN took a dump on itself. What I mean by that is
>the backplane let the magic blue smoke out. While usually in the heavy iron
>class of datacenter products like an EMC SAN this means you fail over to the
>redundant backplane and life continues on. Not this time folks. In the
>process of dying, it took out the parity drives. What does that mean? It
>means the fancy RAID lost it's ability to actually be a RAID. How much data
>got eaten by this mega-oops? 800TB. Why wasn't it backed up? It was, to
>offsite tape, like it's supposed to. But when the array is toast, can't just
>start copying shit back.
>
>Apparently EMC has been on site since Tuesday, but didn't actually inform
>Danger/MS that their data is in the crapper until Friday afternoon. On top
>of that, EMC has done nothing to bring in replacement equipment between
>Tuesday and Friday. (In the Enterprise support world, that's fucking
>retarded, multi-million dollar support contracts are that expensive for a
>reason.)
>
>So what's being done? Well the good news is that the complex was slated to
>be migrated into the Verizon Business cloud services (not MS's cloud per se,
>but it's MS's effort.) And as a part of that migration a newer shinier SAN
>array was in process of being implemented. But space isn't ready for it on
>the datacenter floor, and you can't just toss the EMC raid and place this
>one in it's place, it's a different vendor and is 2 racks instead of one.
>This means it's being shoehorned into a different part of the datacenter
>than was originally planned, one that doesn't have the necessary 3 phase
>power installed. So there's a bit of work to be done. Not to mention the
>restoral of 800TB of backup data from offsite tape.
>
>Time to restoral? Looking like Wednesday at the earliest with techs working
>all weekend.
>
>Lessons to be learned?
>*Buy a f'n phone that doesn't store it's address book and your personal data
>somewhere else, and one you personally can backup yourself.
>
>*Don't expect EMC to actually respond to fixing your core business
>application in any reasonable amount of time. They've gotten lazy, consider
>other vendors.
>
>*Just because your phone says T-Mobile on it, and T-Mobile is crediting you
>a month's service, doesn't mean they fucked up.
>
>*Just because Microsoft is involved, doesn't mean MS f**ked up.
>
>*And lastly, it's not always a "server" that f**ks up.
>

-- 
Victoria L. Herring, Des Moines, Iowa. Blogs: 
http://blog.JourneyZing.com  [photography]; 
http://www.herringlaw.com  [civilrights/discrimination]; 
http://victorialherring.typepad.com/serendipity/  [personal].