How do websites recover from Denial of Service attacks?

General Questions
#1

My publisher’s website has crashed, apparently due to some kind of DOS attack beginning last night at around 5PM. I’m not privy to the tech details, but from a brief message last night, they were receiving 50k packet requests in 50 seconds.

Probably wouldn’t be too much for a major website, but this company doesn’t get that kind of traffic, and as far as I can tell, there were no major news stories or website links toward the company to generate such a traffic spike – like a slashdot effect.

Now I have no idea what the publisher’s webhost is doing to combat this; frankly I’m shocked that it’s lasted this long, but I don’t know how one fights this kind of attack. So that’s my question for you guys: How do hosts handle this kind of thing? How does one ever get back up and running?

(As a sidenote, I gotta say this is the worst possible time of year for this to happen – not that it’s ever a good time, but this company sells romance novels and Valentine’s Day is their biggest sales time of the year, not surprisingly. They’re on a shared server with only a few much smaller sites sharing their space; the evidence is that the attacks are aimed at the publisher. I can’t even imagine why … a competitor? A homophobe (they just released their first same sex novels)? Seems doubtful but who knows…)

#2

You recover by filtering out the bad traffic someplace upstream until it dies down of its own accord. For example …

A particular web server can only support X number of page requests per second. But the firewall immediately in front of it might be able to process 10X requests per second and throw away the 99% which are bad. With the result that the quantity of good traffic getting through to the web server is small enough that it can handle that load; it never sees the rest. The only challenge is some person has to figure out what the bad traffic looks like and tell the firewall what to block. Ideally without blocking much, if any, legit traffic.

For truly giant DOS attacks or attacks on small sites with small bandwidth connections, there might be enough evil traffic to saturate (or even grossly oversaturate) the network connection leading from the internet at large through the server’s ISP to the web server’s firewall. So even though the web server’s firewall can filter out evil traffic it’s still overwhelmed, or the wires leading to it are.

In that case, the server / firewall operator needs to talk to the ISP’s security staff who can put filters in place up at the ISP level which can detect and stop the evil traffic before it clogs the relatively small line leading to the web server’s firewall.

This same tactic can be done at multiple levels if needed. If we imagine some day that Outer Slobovia declares cyberwar on the US, we’d be blocking their traffic at several levels in the internet infrastructure before it got down to the ISPs to prevent the evil traffic from totally gumming up everything.

#3

Ah okay, thanks LSLGuy. That all makes sense. Since this is still going on 20 hours after the problem began, I guess the webhost hasn’t yet been able to identify and filter out the bad traffic via their firewall, OR it’s the second scenario – the attack has them overwhelmed at the ISP level.

It’s really devastating. And I doubt they’ll ever figure out whodunnit.

#4

In the case of a distributed DoS, it can be difficult to filter out the bad traffic because it doesn’t come from any one particular place.

Sometimes it can be possible to recover by using DNS to direct real traffic to a new server on a different network, while to bad guys continue attacking the old server. Most large web sites do this as a matter of course (their web sites are hosted on multiple servers, sometimes in different places).

Shared hosting is often very easy to DOS, and the hosts themselves aren’t always good at dealing with that. This is where the extra cost of high quality web hosting can pay off.

If I were your publisher, I’d have installed a backup of the web site at a different host (or at least a placeholder, if backups were unavailable) and switched the DNS records to point to that. And I would have done that 19.5 hours ago. If the attacker is determined, it may be necessary to use a grid/cluster service.

#5

Some DDOS attacks use requests that don’t complete, so the network stack overloads waiting for completion. Once this happens, you have to reset the stack.

Usually, an attack of this scale is a prelude to a blackmail attempt. Unless the publisher can get a decent action plan in place, the risk of losing business at their busy time of year may make them fold. Gambling sites used to be the favorite target for this sort of thing, but they have wised up.

Si

#6

All of the DoS attacks I’ve seen have been motivated by politics or a simple grudge. None have involved explicit blackmail or extortion. Granted, I’ve only seen a few.

Not saying blackmail doesn’t happen, just that I’m not sure that it’s the usual case.

#7

The last ones that made the news were the attacks performed by the anonymous group against targets they believed were hindering Wikileak’s cause.

At their height, the LOIC DDoS attacks were reported at a maximum strength of 10Gbps against entities like Mastercard, Amazon and PayPal (among others). For websites that large this doesn’t strike me as a particularly powerful attack.

#8

Well, a recent study indicated that it costs from about USD70 at the low end for a 24hour DDOS from a botnet (cite) - more extreme/sophisticated attacks cost more. It costs criminal gangs money to establish and maintain botnets, so there is almost always a financial motivation to the process. The LOIC attacks from Anonymous were political/social in motivation, but only achieved the levels they did due to high media attention and motivation - LOIC is not a technically sophisticated attack tool and traffic/users can be identified easily and countered. This is not generally the case for a botnet attack.

Here is a good example of how a DDOS shakedown occurs, and reports indicate that this problem is on the rise.

Si

#9

Thanks for the responses, guys. Very educational (and scary). I never even heard of the blackmail angle before, but it makes a kind of creepy sense.

The hosts, along with some security firm the publisher hired for a few thousand bucks, finally managed to get the site back up late last night (about 11PM my time), but it’s still rather slow if you ask me. I can’t even fathom how much money the publisher lost with only one day left for Valentine’s purchases.

Yeah, I’ll buy a ticket to that. I’m not sold on the capabilities of the hosting firm this company uses.

They do have a backup website on a different host but it lacks a security certificate, so they couldn’t use it for sales. (Which is something they’d bloody well better rectify ASAP.) Also their webhost said it would take 24 - 48 hours to change over the domain. (Which sounds like BS to me. I know it can take up to 48 hours for propagation, but in my experience, the switchover happens much faster. I’ve purchased domains, switched them from the original host to a new nameserver, and had them work correctly within minutes.) I think the publisher just gambled that his host would fix the problem much sooner than they did. A shame but hopefully a lesson learned.

If it only costs $70 or so to set one of these things up, I’m really wondering about a competitor being involved. Seems hard to believe… they’ve done some tricky things (like using my publisher’s tagline on their own website, copying text, etc.) but there’s a big difference between that and outright illegal activity / sabotage.

(For some real paranoia… what’s to stop a security firm/webhost partnership from using a DDoS as a scam protection racket? The customer would never know whodunnit, right?)