Blog Home For Steve Suehring

[Editor's Note 3, 11:49am central: As some people have now pointed out, it could be that people were wget'ing the site for other reasons. However, that's not how it looked to me. All the requests that I saw were from the same IP and were all for the root / URL.

Unfortunately, I don't have all of the information. I agree that it would be nice to have more. I have what I have and now you have what I have, minus two lines of log file that are virtually the same as the three already shown. If Sys-Con wants to share more, I'll be happy to bring it forward.]

[Editor's Note 2: Even though I thought it was clear below by the statement "In the following example I'm masking some non-essential information," it's obviously not. I've masked the IP below and replaced it with localhost. No need to post the real-world IP from the request. Sorry for the confusion.]

[Editor's Note: I wrote this story late Friday night/Saturday morning but sat on it so I could add additional details and fill in some URLs.]

Yes, I blogged about this yesterday, or what is technically yesterday according to the calendar but it's still the same day for me. I will officially state first that this blog entry is brought to you by Sundrop and Coca-Cola. No, they didn't actually compensate me in any way but their products have helped this writer get through a long week. [Note to Sundrop and Coca-Cola, if you want to send me product, please do!]

I had asked all week for details of the DoS against Sys-Con and the LinuxWorld.com site so that I could bring them to you in a story that would appear on LinuxWorld.com and also, frankly, so I could try to help Sys-Con mitigate the attack. Since I'm no longer with Sys-Con I don't think they'll want my help, but I don't see a reason why I can't share the story that I was working on anyway. [Editor's Note: I've filled the rest of the story in with late-breaking details.]

There is still some doubt over whether the DoS attacks against Sys-Con actually existed or whether they were the result of 'The Slashdot Effect' for lack of a better term. I believe the DoS attacks did exist. I too was initially skeptical but based on e-mail correspondence I now believe them to have happened. In fact, from what I can tell the attacks were distributed, thus making this a DDoS.

I'm not interested in re-hashing the details of the causes for this DoS but rather I'm interested in looking at it from a semi-technical perspective to give the reader a view into what actually happened. Nothing that I'm disclosing in this post was labeled confidential in any way. I am not disclosing confidential correspondence in this post, I don't have any. Yes, I know, half of the readers just left because the story is much less juicy.

Having worked at an Internet provider for the better part of seven years, I've found myself having to mitigate them and clean up after many attacks, including DoS attacks. However, what I saw of the DDoS against Sys-Con makes me think that it is being made to sound much worse than it was.

What I know about the attacks is as follows. On Wednesday, Fuat Kircaali, the publisher of LinuxWorld, said this about the DDoS, "So far we located one of the five originators of these attacks...". It seems to me that five originators isn't much of a DoS but then again, it could be. If those five have fat pipes and they are going against a small pipe, it would be ugly.

From what I've gathered by asking for the logfiles from Sys-Con, the attacks themselves consisted of multiple hits with wget. Not a SYN flood, not any fragmented packets, wget. This means that each packet in the DoS completed the TCP handshake and the attacks were HTTP GET requests, lots of 'em. [Editor's Note: Any technology or non-technology media that picks this up, I'm happy to explain the technical details at more length via e-mail (or visit the links). This article assumes you know what a SYN flood is, what the TCP handshake is, and so on.]

In the following example I'm masking some non-essential information. I apologize in advance for any formatting mishaps. Fuat's e-mail client sends in annoying HTML format and I don't have mutt configured to work with HTML-formatted e-mail. Therefore, I can't always read Fuat's e-mails. These are just the first three lines of the total of five lines of logs that I received.

127.0.0.1 - - [NN/May/2005:NN:20:44 -0400] "GET / HTTP/1.0"
200 49107 "-" "Wget/1.9.1" eudora="autourl"

127.0.0.1 - - [NN/May/2005:NN:20:45 -0400] "GET / HTTP/1.0"
200 43219 "-" "Wget/1.9.1"

127.0.0.1 - - [NN/May/2005:NN:20:45 -0400] "GET / HTTP/1.0"
200 43219 "-" "Wget/1.9.1"

So, adding up a bunch of hits from a handful of well-connected hosts would definitely lead to a DoS. But it certainly doesn't seem like the DoS that I or others were led to believe. I'll fully acknowledge that there may be other details that I haven't seen about this DoS. I'd be more than happy to provide an outlet for those details should anyone at Sys-Con want to share them.

I suggested three tactics to Sys-Con when I finally got details on Friday. First, look for a pattern. Are they all wgets? Are they all coming from the same IP range, and so on. Next, block the handful of IPs that are causing the DoS at the screening router, TCP port 80 minimally, but all protocols would be better. If that's not feasible, use Apache mod_security to block the wget User-Agent for the time being (Yes, turn it back on eventually). Finally, and most ugly, put a rate limit in for TCP port 80 per IP. Granted this might block some requests from valid users but it should also help the DoS. There are other ways but those came to me within a few seconds as being plausible and easiest to implement quickly.

My opinion is that the DoS was a combination of a few factors. First, the sites were Slashdotted twice this week. I distinctly recall conversations with Sys-Con in the past when the sites got Slashdotted and then became unavailable or severely degraded. So, in effect, a Slashdotting could hamper the performance of the Sys-Con sites anyway. Not only did the sites get Slashdotted, but other news outlets picked up on the stories as well thus generating even more traffic. Finally, the proverbial straw that broken the camel's back were the extra requests generated by the HTTP GET requests.