AAAAAHHH!!!! Shoot me! My server blew up!

Miscellaneous and Personal Stuff I Must Share
1

This is similar to what I posted at http://pub51.ezboard.com/ffathom2frm1 but I’ve updated it some:

You may notice that the FFF (3FMB) site is down. In fact, our whole server is down. UndeadDude just drove to McLean, VA where the co-location company is (where our server lives, physically) to see if he could get it running again…
… and found out that we lost a hard drive.

The server has two (guess why? Heh.) and does backups from one to the other daily… but only for things that have changed in the previous 24 hours. The full backups are to a tape drive in our house, and are done once a week.

Since we have this local tape backup, I didn’t really think I needed to have ANOTHER copy of my site locally, so I only have local copies of stuff I’ve updated in the last couple of months.

Which would be fine and dandy except that when we tried to read the tape backup, it failed. Corrupt somehow. And there isn’t a previous one because the new ones overwrite the old ones (we’re having a long talk about this, btw)

The good news: all the info for the 3FMB and all of my customers’ sites is on the good drive.

The bad news: my business site, several friends’ sites, and fathom.org itself, including my ~1500 page homepage that I’ve been working on since 1995, along with my journals prior to June 2000, is on the bad drive.

More bad news: the money for the new hard drive has to come out of the $1000 we’ve put into the new server fund. We just can’t afford it otherwise. The server fund (thanks everyone, for your donations. We’re up to over $2100. The new server is still in our future!) will drop by whatever the cost of the drive is.

Even more bad news: the cost of data retrieval for the dead hard disk is $250 for the diagnostic, then $1000-4000 for the actual retrieval process. I’m trying to find a way to get this money now. My mom is loaning me $2000 out of a trust fund, which is a start.

Absolute worst case scenario: I lose my entire homepage and a few other sites of mine, and some friends’ sites that I host for free, and the entire server is kaput for … god knows how long, and we lose $4000 trying to retrieve data.

Best case scenario: we manage to get the data off the drive ourselves and onto a new drive tomorrow, and the site gets back up tomorrow, with only a day’s loss (since last daily backup.)

Potential good news: if the data retrieval is on the low end of the price range, and we end up with money left over, we will go ahead and buy the new server at the same time.

Potential bad news: there is a chance that this whole thing might make us have to cancel our trip to Europe this summer, that we’ve been planning as our 10th wedding anniversary celebration (we already bought non-refundable plane tickets)…

So to sum up: Opal isn’t a happy camper.

PS anyone with fff.fathom.org or teemingmillions.zzn.com email can still access their email accounts:

http://fathom.mail.everyone.net
http://teemingmillions.zzn.com
Temporary email for me is emergency@fff.fathom.org as my mail server is on the same server that is currently dead.

2

Opal,

I tried to post this on the ezboards…but it didn’t work. I don’t understand how you can’t just post if it doesn’t require one to not-register and I don’t have the patience to figure it out.

Anyhow, here’s a few late but timely tips for you and all people that run extensive sites, networks, etc.

Have a tape rotation of at least 28 days. Have a test back up every month to ensure the data is being copied over correctly.

There has got to be a Fathomite or Doper that has access to data extraction, somewhere in the Millionings there has to be…for that I appeal to any data recovery person here to help her out. Lend her your services. She has put so much time into what she does and putting up with the likes of me that she deserves it.

I would recommend an FTP backup to your local drive at least every two weeks of the most important stuff. It can take a long while but worth the time if a situation like this happens again. I never think there is too much protection from failure, especially if you can’t afford a RAID system on your server.

Opal, I wish I could help you further, at least know that I am here in hoping for a good outcome for you.
{{{{{OPAL}}}}}}

3

Hi Opal.

Jeez, that’s horrible. I wish you the best luck with the server and with beating away the several angry posters from FFF. Keep us posted, i’m sure there are a lot of people interested on what happens.

4

No wonder fathom was actually loading slower than the SDMB!!

Sorry to hear about the problem, Opal.

5

<wishing my field were data recovery>

:frowning:

{{{{{{Opal}}}}}}

6

{{{fathom world}}}

{{{OpalCat}}}
Opal, get RAID.

7

Mercutio

do you ever think before you post? I doubt that Opalcat will be beating off any angry posters from FFF - why on earth she be?

Opal that’s horrible about your diary. I hope you can recover it.

8

Just bumping this so any fellow Fathomites posting over here this morning get the word.

{{{{{Opal}}}}}

9

That does explain why I couldn’t get there.

That sucks, Opal. I hope you can retrieve your stuff. Losing that much would just be depressing, I would think.

So, does this mean we’ll be getting more things in the auction? I don’t mind more bidding wars. For the good of the server fund, of course.

:smiley:

10

Aw crap, Opal! That sucks!

I remember the days when I used to colocate a server all the way out in the states… and when it blew up… and when we lost a lot of work… and when we began sacrificing tapes to the good old Lord of Backup who incidentally had saved our ass…

Those were the days…

:wink: Good luck with the recovery!

E.

11

I thought the subject said “My servent blew up”. Now that would be a tragedy.

12

I wondered what had happened. I’ve been on your forum for a couple weeks and found it to a very enjoyable place to be. I sent you an email to see if I can help.

13

Opal,
Best of luck with the crisis. You’ve just reminded me to do a restore check of my work backup today. (And buy new tapes!)

techie,
For my own Network Admin knowledge, can you clarify what this means:

Do you mean replace the tapes that often?
Right now on my network, I use 10 backup tapes- 5 one week, 5 the next and replace them quarterly. (I use an Archive Python tape drive 4mm)
Opinion?

Zette

14

{{{Opal}}}

I hope all your best case scenarios work out.

RT, thanks for bumping this. I was wondering why I couldn’t get on.

15

I’m so sorry to hear that all of your work is balanced on the edge of a cliff. Many years ago I had a 20 meg hard drive actually catch fire in my hands. I swapped out the power supply section components from a good drive, repaired some burnt traces and it actually worked without corrupting to data. That is the geek equivalent of getting your broke ass Chevette running with duct tape amd cpat hangers.

Best of luck with your project.

(BTW it makes me all tingly for women to talk all nerdie.)

16

I meant to post a reply but hit the “new thread” button by mistake. http://boards.straightdope.com/sdmb/showthread.php?postid=1247737

17

Actually my client is set up on a 30 day rotation. This way you have a full month’s back up if something goes horribly wrong. Tapes are not always reliable. We do a restore check once a month as well.

It seems like over kill when they are on a RAID 5 set up too – with a hot swap disk sitting there on the server ready for action – but we’ve had problems with tape back ups before.

In addition, the controller backs up her accounting database to CD ROM every night. Actually, it’s not a bad idea for her to do so because that is the only way for us to know if anyone is still logged into the database if they get knocked out of the database it leaves them logged in. Their accounting software is so damn secure she can’t get in to see if anyone’s in.

18

I’m so sorry to hear about your tragedy. You put a lot of yourself into your web site and it shows.

I hope everything get back to normal soon. (Whatever “normal” means :))

19

Opal, Good luck with the data retrieval. I had my fingers crossed all day yesterday for you!

Zette

Thanks for the info, Techie :slight_smile:

20

So that’s why I couldn’t get on the other day!

Best of luck. I have my fingers crossed for you.