Outages are crippling my business

While I apreciate your efforts, they are falling short. We are still experiencing severe problems, delays, emails not getting out, or not getting in. I have lost severalthousand dollars in the last two days and I am actively looking for a new host. If I actually had a # to call to get info, I might stay, but I don't. It's a little tough to figure what is going on when I can't even get to this forum. Our system configurators are online, so we can't even give our customers a quote if they call in.

Finally, I was able to make this post. SO, what's the latest, clearly it's not fixed yet as of 12PM EST on Friday.

With a problem this serious it should be on your home page.

 

I apprecaite your replay, but you must understand that we cannot operate like this.Fixing the barn door after the horse has left does not work for us.

 

We're also suffering because of that, as our business relies entirely on our site to generate sales. But, I do understand that this can happen and it does happen to any hosting company.


We stopped all PPC advertising, till after the weekend, in order not to lose the cost for those clicks.


Now, we're re-thinking our stratergy, to have another site (e.g. oursite.com and oursite.net), and run different PPC campaigns. But this comes at a cost (hosting 2 sites, maintaing 2 seperate databases).


What are other guys doing to prevent this from affecting your business?

 

Discountasp.net, our company's confidence is almost at 0 with you now.

We are likely moving hosts because of the lost business we have had to endure over the past 2 days.

Sure you can't prevent the attack and blah blah blah.. but that is YOUR problem. Your customers want solutions, not "we are working on it".



This is your second major DDOS attack in 2 years? Who have you been pissing off that they want to bring this company to its knees???

Post Edited (kleinma) : 6/22/2007 8:17:23 PM GMT

 

kleinma said...

This is your second major DDOS attack in 2 years? Who have you been pissing off that they want to bring this company to its knees???

Every host that gets to a certain size deals with DoS and DDoS all the time. I have worked for three other hosts, one larger than DisccountASP.NET and two smaller, and DoS was an issue at every one.

You don't notice most of the DoS attacks that happen here because Tipping Point nullifies them (more on that later today). The one we have been dealing with over the past 36 hours is a different beast all together, and involves staggering amounts of traffic.

 

Hi,


We have many customers using DASP at our suggestion (insistence, sometimes!), and while they have been sympathetic for the most part, at the end of the day most feel it is not their problem. So we will probably look at splitting root nameservers among two ISPs to see if that helps. Most of our customers have brochure-style websites, so being without this for hasn't hurt, but DNS and email service has been the killer for us.


Which makes me wonder why DASP has only one upstream provider. Is this true? I realize having two or more would not stop the attack, but you would have had more support or competing teams working to solve the problem. Complete shutdown may have been avoidable.Is your location "owned" by just one provider?


And, of course, the question always asked is who do they (DASP) do business with that would resort to this criminal activity? Given the size of the attack forces, someone had to know how much it would take. So my guess it is a disgruntled (if that is a strong enough word) ex employee or business partner. In other words, an inside job.


Michael

 

We aren't going to publicly speculate on who launched the attack. But I can tell you there are no disgruntled ex-employees (no one has ever been fired), and this was more than likely targeting a site that we host.

You might ask, 'Doesn't DiscountASP.NET monitor the sites coming in?' Yes, we do extensive fraud checking, and that catches the majority of sites that might become targets. But fraud screening doesn't catch accounts set up legitimately, and legitimate accounts can have enemies. There is no way to prevent or predict that.

mjp

DiscountASP.NET - Microsoft Gold Certified Partner
- asp.netPRO Magazine 2007 Readers' Choice Award for Best Hosting Service
- Visual Studio Magazine 2007 Readers' Choice Award for Best Hosting Service

 

'While I appreciate your efforts, they are falling short.'

I totally agree.

Now, days following this attack, while most services appear to be back online, there still seems to be significant problems at DiscountASP. Just like most of the customers in this thread, our business has suffered too. 100% of our income is from our site, and our business is 100% service orientated - if we aren't replying to our customers, and providing fast, efficient service of our product, we loose customers. In several cases, we have been forced to issue gift certificates, discounts and refunds to many of our customers that have complained. This is costing us huge amounts of $$$.

Has anyone on reading this tread contacted DiscountASP for a refund or compensation? While, as far as I can tell, they do not provide any up-time guarantee or have a policy of issuing refunds for issues that our 'outside of their control'. You would think in this case, in the interest of maintaining some level of customer satisfaction, DiscountASP.net would provide some kind of small gesture to show that the are sorry and that they care.

It could be a refund of a month's billing or if cash flow is an issue, they could extend everyone's hosting by a month. Something small to show that they care - the blaa blaa blaa of the notices and status updates and the constant explanations of what a DDOS attack is is bordering on insulting at this point. I know what a DDOS attack is, especially now after your multiple explanations of it. While I appreciate the status updates, now that the issue has progressed, DASP need to rethink their service strategy. Just like how DiscountASP.net don't feel this problem was their fault, I certainly don't either when discussing this issue with my customers... but I step up in the interest of customer service, and offer something to my customers to keep them. DiscountASP needs to do the same if they even care.

I understand that the DDOS attack was not their doing, or fault and that they themselves are also victims. But I feel that they were extremely negligent in not considering and planning for a worse case scenario such as this and I feel that it ultimately lead to the significant delay in thwarting this attack. This is a fact that even DiscountASP.net have come very close to admitting to in communications with their customers.

At the beginning of this post I mentioned that there are still significant problems at DiscountASP... I am not sure if anyone else has noticed this, but e-mails are being held up for 10-14 hours on the DiscountASP mail servers. I have had a support ticket open for about a day now on this issue, and I have only received one reply from a support agent on it. And the reply was as if they had not read my initial report of the problem.

Email messages to my account appear to be arriving into my POP mailbox without any problem, as far as I can tell there are no bounce backs. However, the mail appears to be sitting on the DiscountASP.net mail server for extended periods of time before being release to me. The way I can tell is that if you log into webmail, you can see the new messages sitting there. However, if you use a mail client to connect to the POP account, the client will tell you that there are no new messages in your account! If you think you are not getting mail, or your messages are delayed, check webmail and see if they are sitting there.

Note that the connection to either the POP server or WebMail is not slow or interrupted, so that's a good thing.

But after a long period of time and regular checks for new mail, suddenly a new message comes into my mail client and the 'received' date and time reflect the time from 10-14 hours ago when it landed in WebMail.

Like I said, still no answer from DiscountASP.net on this issue.

Tek

 

I just wanted to post a vote of confidencefor DASP. People on this thread are understandably upset but they have no idea what a DDOS attack is. This type of attack gets more intense and sophisticated every year. There is no way to prepare for the next generation of attack. Jumping ship is just the kind of thing these criminals want. Be happy your hosting company is working so hard to remedy this problem ... many wouldn't even care. Also, like the moderator said ... it was probably directed at a site they host. Do you really think there is a host out there that doesn't host sites that are targets for these people?

 

The DDOS attack has spawned another issue with the response of DASP and their communication to their customers. I sat and 4 hours before I received a response or any information on the problem. I understand what a DDOS attack is and it's not my complaint. It's the communication from DASP that I'm a discouraged about. Anything that I can do to improve their communication on these types of "Emergency" service degradation issue, I will do. If that means beating up the message board until the point is driven home, that's what I will do. Are you satisfied with their communication on this "Emergency" service degradation?

 

I have been watching this post with some interest because my site was down all weekend. While I am willing to give DASP a pass because I know how hard this is to combat, the last post does bring up a good point. My one complaint is that I don't recall ever seeing the status page before this happened. I think if DASP would make it priority to inform customers (of this URL)when they sign up it may eliminate the frustration. That said, I didn't have a big problem with the communication. Their name is "Discount" ASP. While I expect good service, I also understand that providing discount services does mean you are not staffed to the hilt.

 

What is done is done. My concern, going forward, is whether or not DASP will takes additional steps to help mitigate DDoS attacks in the future. I'm not saying they need to prevent them, but I think they need to explore what options are available to make these attacks less painful in the future:

1) Can their SLA with their upstream provider be upgraded to guarantee faster response times.
2) Are there additional technical solutions that can allow the problem to be corrected more quickly.
3) Can additional preparedness lead to faster fixes in the future.
4) Is it possible to compartmentalize things so that all their eggs aren't in one basket?

The problem took a long time to correct (days rather than hours). The communications that were sent out suggested that the technical team wasn't really sure what to do (we are trying this, we are trying that) which implies that perhaps they weren't as prepared for this as they ought to be.

There are lots of juicy targets out there (Microsoft, the White House, political candidate websites, etc.) that must be under attack on a regular basis, yet they manage to keep the sites up and running with minimal outages.

 

My thoughts:

1) My business, reputation and my customers were negatively impacted by the outage.
2) I can't reasonably expect any provider tofoil all attacks. However, I do expect DASP to be prepared to defend against them.
3) I have full responsibility to make sure the products I sell gracefully handle network outages, and as a result have ordered service from another provider as a backup.
4) It will cost me time and money to do this.
5)DASP has a responsibility to keep me informed of conditions which could impact their service to me and they could have done a better job to keep me informed during this event.
6) The real bad guy(s) (is)are the one(s) behind the attack.
6) I hope DASP looks for and finds who was behind it, prosecutes to the fullest extent possible, reports on their progress in that regard, and provided they uncover the attacker(s), gives me access tothat information so I can opt to take my own legal action.

Best,
Dan

Post Edited (Dan Quigley) : 6/25/2007 10:06:32 PM GMT

 

Those sites also cost a bit more than $10 a month to keep up. I hope you don't expect us to have the same resources at our disposal as the Federal Government or Microsoft.

It took days rather than hours for a lot of reasons. Mainly because it was not a sustained attack. DDoS rarely are. They start hard and taper off, then usually stop.

This one started hard, tapered off, stopped for about 6 hours, then started just as hard as the first time. So, if the usual pattern is one strike and that's it, that's what you expect you're dealing with.

If you heard hoofbeats in the distance, what would you think was coming, horses or zebras? We expected horses, which is realistic, but we got zebras. Lots and lots of zebras.

We are preparing a long post to answer many of the questions here, we aren't ignoring you.

 

After reading what many others have lost ($$$) due to this attack I do not feel as bad that I went from page 2 in google to "gone", as i just launched my site a few weeks ago. I think this sad situation will make Discountasp.net a better company and that they will take to heart (and wallet) the many concerns aired here. I have been very happy with them thus far and am happy I am NOT financially dependant (yet) on internet sales.

 

If your website is mission critical, then you're a tightwad for using shared hosting. Pry open the wallet a bit and pay for hosting designed for mission critical websites.

 

That's an interesting statement. Forgive my ignorance, as I'm a relative noob to web hosting, but what are the primary differences between a web hosting service that costs $10 per month vs. $100 a month vs. $1000 a month? I guess I had always assumed that the implementation was about the same, but that economies of scale was what kept the price down.

What sort of things do services that host "mission critical" websites do differently than a normal web hosting service?

 

There are many differences in a site hosted here and a site like microsoft.com or whitehouse.gov. This is shared hosting, many sites on each server. Our uptime is good, but to guarantee close to 100% uptime many redundant systems must be in place, and the expense is considerable. For $100 a month you may find somewhat of an uptime increase, but for $1000 you can buy a lot of redundancy and geographical mirroring that can go a long way in keeping things up and running no matter what.

 

Another thumbs up here for DiscountASP.

I understand the frustration of losing customers through perceived bad service; it's happened to me too.But reading some of these comments,I do wonder if some people really understand just how difficult defeating a well organised DDOS attack is. Yes, we all know what 'DDOS' means, but do you really, really understand how, if the attack is co-ordinated in the right way, it is practically impossible to defend against? If you believe otherwise then I think you need to learn more about it.

Until I struck out on my own last year, I was an IT Security professional working for a major multi-national corporate. We ran sites with millions of hits a day and had to defend against thousands of attacks. And despite our budgets of many millions of dollars for security, none of us ever believed that it is possible to be 100% protected against a well organised DDOS attack. While the Internet remains a network of multiple ownership, it is simply impossible to be prepared for all eventualities. Indeed, we sometimes needed the cooperation of several Tier 1 carriers to mitigate certain attacks. And we only had access to this kind of cooperation because we were such a carrier ourselves.

For the price, the DiscountASP service is unbelievably good value. If you want a serious increase in network redundancy, you'll have to increase you spend by two or more orders of magnitude. And even then, if any providerimplies they can defend against all DDOS attacks thenfrankly, they are lying to you.

I am not a DiscountASP employee. I am just somebody who perceived that the company was being done an injustice by this seemingly one sided thread. I just wanted to restore some balance based onyears of experience in dealing with precisely this sort of problem.

 

Not to shill for DASP, as I am also not an employee, but Holf's post hit the mark. If you want five nines uptime, then you gotta pay for it. If you don't know what five nines means, then you should educated yourself.

You get what you pay for. DASP does a great job of providing a decent service for very minimal price. You want reliability for your mission-critical apps, go pay the 1k+ per month for it. You fly-by-night internet biz guys really don't have a clue as to what it takes to run a business in the real world. My company houses all their stuff in-house and pays MILLIONS per month to manage it. I use DASP for my personal siteand asocial organizationsite that isn't mission-critical. Call me crazy but I think I can live without the use of my site while a DDOS is going on and I don't plan on complaining too loudly at the DASP guys who quite honestly did a much better job of recovering from it than I expected.