bruce
06-26-2007, 10:16 AM
As most of you well know, DiscountASP.NET's network came under a massive DDoS attack (http://en.wikipedia.org/wiki/Denial-of-service_attack) last week.
In this post, I want to give you a overview of what actually happened.
When the DDoS hit, it completely saturated our gigabit link into our data center. To give you some perspective, that is equivalent to 660 T1 lines transmitting at full capacity.
With the help of our upstream provider, we were able to figure out the traffic was targeting our name servers. The engineer at our upstream provider was able to perform a brief network sniff to determine the traffic pattern (a side note: the engineer setup a span port on their core route and sniff some of the traffic with his lap top, his lap top crashed after 60 seconds of sniffing).
In any case, he was able to retrieve some important information on what type of attack was that and which server it was targeting. The attack was simply a lot of UDP packets to port 80 on the server. More precisely, over 500,000 packets per second. More research indicated that the attack was likely generated by computers infected by malware (http://isc.sans.org/diary.html?storyid=1901) which is capable of sending 100udp packets per second. That means we were attacked by at least 5,000 infected computers.
We thought, well, that's cool. Now that we have the target server (ns1 & ns2), we should be able to block it from their router. To our dismay, the network engineer was not allowed to do that. Eventually, after serious wrangling with their management, we were able to convince them to put a filter on their network to block UDP 80 traffic targeting our server.
We thought that would resolve the problem but it did not. About 30 minutes after our upstream provider installed the filter, they informed us that they had to remove the filter because the attack traffic was so massive that it was causing their core router to experience latency, which slowed down connections to the entire data center. For those of you who are not familiar with networking, any Access Control List (ACL) you install on the router increases the CPU usage. Our upstream provider is a Tier One international provider and our data center neighbors (which I am not allowed to name) are several major search engines and banks.
So with our upstream provider basically telling us there was nothing they could do, we thought, now what?! We came up with the idea of migrating our DNS to another Tier 1 provider. So we started calling all the big name providers around the country. Most of these guys were happy to get a new customer UNTIL we told them about the size of the attack.
The sales engineer for one of the major providers that we talked to told us they have "a very kick ass anti-DDoS solution." When we informed him of the scale of the attack, he just repeated, "Oh my god" several times. He went offline and talked to their backbone engineer and 30 minutes later, we were told that we were not "a good fit" as a customer and that their DDoS protection can only handle up to 1 gigabit per second.
At this dire point in time, we found a company, Prolexic, that specializes in DDoS mitigation. Prolexic has a very unique solution to mitigate attacks. In general, most industry experts will tell you that you cannot prevent or resolve a DDoS situation. You can only mitigate the effect until the attack dies down. The only mitigation method is to have a BIGGER pipe than the attackers. In most cases, no hosting provider - even most national ISPs - have a pipe that is larger than 1 gigabit.
The Prolexic idea is actually very simple; basically we renumbered our DNS server's IP to their service and they balance the attack traffic to a group of geographically dispersed data centers, using their filtering solution to cleanse the traffic. Once the traffic is cleaned, the legitimate traffic is sent back to us.
The only problem with renumbering the IP addresses of the name servers was that it took a little time to propagate. Propagation times are much less than they were in the past, but most of you no doubt noticed an outage of two or three hours.
I have seen many posts about this attack in our forum and other public forums. Here are some of the common questions that I want to address:
Q: Why did your intrusion prevention equipment (Tipping Point) not stop the attack?
A: The Tipping Point hardware resides on our network. The attack was so massive that it saturated our gigabit connection and bogged down the traffic before it even reached Tipping Point.
Q: Why was DiscountASP.NET unprepared for this type of attack?
A: It was not possible for us to prepare for attack of this scale. Remember that our upstream provider and various other tier 1 data centers around the country were not even able to (or afraid to try to) handle the attack.
Q: Why did it take so long to resolve the problem?
A: See above!
Q: Can you guarantee that this won't happen again?
A: Yes and no. With the Prolexic solution now in place and the Tipping Point hardware inside our network (Tipping Point protects against more than just DoS attacks, it also does malicious file scans and other traffic analysis) we are pretty confident that we have much more extensive protection in place than any other host that is comparable to us in size (and many much larger hosts). Any attacks on the same scale that we just experienced should be effectively mitigated.
But DoS and DDoS attackers evolve and adapt, and it is impossible to predict what methods may be used in the future to carry out the attacks (bot networks are already being abandoned for peer-to-peer network based attacks). Any host that claims they can protect you from any conceivable attack is not being honest! But we believe that we have an advantage now, and will be able to deal with any future malicious attacks much more effectively, and without the serious service disruptions that we experienced last week.
Finally, I would like to thank those of you who sent us your support and encouragement during and after the attack, and thank all of you for hanging in there with us while we rode this out. We learned a lot from the experience and feel that our service has now improved in many different ways as a result.
Post Edited By Moderator (mjp) : 6/27/2007 12:28:07 AM GMT
In this post, I want to give you a overview of what actually happened.
When the DDoS hit, it completely saturated our gigabit link into our data center. To give you some perspective, that is equivalent to 660 T1 lines transmitting at full capacity.
With the help of our upstream provider, we were able to figure out the traffic was targeting our name servers. The engineer at our upstream provider was able to perform a brief network sniff to determine the traffic pattern (a side note: the engineer setup a span port on their core route and sniff some of the traffic with his lap top, his lap top crashed after 60 seconds of sniffing).
In any case, he was able to retrieve some important information on what type of attack was that and which server it was targeting. The attack was simply a lot of UDP packets to port 80 on the server. More precisely, over 500,000 packets per second. More research indicated that the attack was likely generated by computers infected by malware (http://isc.sans.org/diary.html?storyid=1901) which is capable of sending 100udp packets per second. That means we were attacked by at least 5,000 infected computers.
We thought, well, that's cool. Now that we have the target server (ns1 & ns2), we should be able to block it from their router. To our dismay, the network engineer was not allowed to do that. Eventually, after serious wrangling with their management, we were able to convince them to put a filter on their network to block UDP 80 traffic targeting our server.
We thought that would resolve the problem but it did not. About 30 minutes after our upstream provider installed the filter, they informed us that they had to remove the filter because the attack traffic was so massive that it was causing their core router to experience latency, which slowed down connections to the entire data center. For those of you who are not familiar with networking, any Access Control List (ACL) you install on the router increases the CPU usage. Our upstream provider is a Tier One international provider and our data center neighbors (which I am not allowed to name) are several major search engines and banks.
So with our upstream provider basically telling us there was nothing they could do, we thought, now what?! We came up with the idea of migrating our DNS to another Tier 1 provider. So we started calling all the big name providers around the country. Most of these guys were happy to get a new customer UNTIL we told them about the size of the attack.
The sales engineer for one of the major providers that we talked to told us they have "a very kick ass anti-DDoS solution." When we informed him of the scale of the attack, he just repeated, "Oh my god" several times. He went offline and talked to their backbone engineer and 30 minutes later, we were told that we were not "a good fit" as a customer and that their DDoS protection can only handle up to 1 gigabit per second.
At this dire point in time, we found a company, Prolexic, that specializes in DDoS mitigation. Prolexic has a very unique solution to mitigate attacks. In general, most industry experts will tell you that you cannot prevent or resolve a DDoS situation. You can only mitigate the effect until the attack dies down. The only mitigation method is to have a BIGGER pipe than the attackers. In most cases, no hosting provider - even most national ISPs - have a pipe that is larger than 1 gigabit.
The Prolexic idea is actually very simple; basically we renumbered our DNS server's IP to their service and they balance the attack traffic to a group of geographically dispersed data centers, using their filtering solution to cleanse the traffic. Once the traffic is cleaned, the legitimate traffic is sent back to us.
The only problem with renumbering the IP addresses of the name servers was that it took a little time to propagate. Propagation times are much less than they were in the past, but most of you no doubt noticed an outage of two or three hours.
I have seen many posts about this attack in our forum and other public forums. Here are some of the common questions that I want to address:
Q: Why did your intrusion prevention equipment (Tipping Point) not stop the attack?
A: The Tipping Point hardware resides on our network. The attack was so massive that it saturated our gigabit connection and bogged down the traffic before it even reached Tipping Point.
Q: Why was DiscountASP.NET unprepared for this type of attack?
A: It was not possible for us to prepare for attack of this scale. Remember that our upstream provider and various other tier 1 data centers around the country were not even able to (or afraid to try to) handle the attack.
Q: Why did it take so long to resolve the problem?
A: See above!
Q: Can you guarantee that this won't happen again?
A: Yes and no. With the Prolexic solution now in place and the Tipping Point hardware inside our network (Tipping Point protects against more than just DoS attacks, it also does malicious file scans and other traffic analysis) we are pretty confident that we have much more extensive protection in place than any other host that is comparable to us in size (and many much larger hosts). Any attacks on the same scale that we just experienced should be effectively mitigated.
But DoS and DDoS attackers evolve and adapt, and it is impossible to predict what methods may be used in the future to carry out the attacks (bot networks are already being abandoned for peer-to-peer network based attacks). Any host that claims they can protect you from any conceivable attack is not being honest! But we believe that we have an advantage now, and will be able to deal with any future malicious attacks much more effectively, and without the serious service disruptions that we experienced last week.
Finally, I would like to thank those of you who sent us your support and encouragement during and after the attack, and thank all of you for hanging in there with us while we rode this out. We learned a lot from the experience and feel that our service has now improved in many different ways as a result.
Post Edited By Moderator (mjp) : 6/27/2007 12:28:07 AM GMT