Thread subject: Whaler Central - Boston Whaler Boat Information and Photos :: Site Outages

Posted by Joe Kriz on 10/24/07 - 11:57 AM
#1

Just wanted to give everyone a little information about all of our site outages the past 6 days.

Our Hosting service has been upgrading their hardware and software.
Unfortunately, this has NOT gone well.

We were down again yesterday, Tuesday, from 3 PM to about 4 AM this morning...
NOT good.

Our Hosting service supports 125,000 customers and I assume all of them were just as upset as we are.

It's not an easy thing switching hosting services as all the files and databases have to be transferred to the new site plus the domain name takes a day or two to propagate over the Internet. We really hope we don't need to change hosting services but if the outages continue, we will have no choice as this much down time is unacceptable........

Lets hope that they have these growth and upgrade problems under control and there are NO MORE outages.

Joe Kriz

Posted by sraab928 on 10/24/07 - 4:25 PM
#2

Sometimes you just have to deal with it. Its a reality with some hosting services. I'm sure its tough on their end too. Hopefully it will be all over soon an it will get back to normal.

Posted by kamie on 10/24/07 - 4:47 PM
#3

I am assuming that with all the outages this month is free and probably next month as well? I would have my service rep and their boss on a conference call, ASAP!

Yes, I have been on those calls, not pretty

Posted by Joe Kriz on 10/24/07 - 5:04 PM
#4

I would imagine that "some heads are going to roll" at there company.

Yes, they gave us 2 months free service... However, if it is always down, what good is 2 months of NO service.......???

What is strange to me is that they are supposed to have a built-in redundancy....
This to me means if one set of servers or routers go down, then the second set keeps the customers up and running.
However, this was not the case as we were down completely, so where was the redundancy????

Beside the outages, we were happy with their service.
Lots of room to grow and a lot of nice features available....
I have shell shock now and any time there is a moments glitch, I get a little nervous....

Time will tell....

Posted by Phil T on 10/24/07 - 5:21 PM
#5

Prior to being doing daddy-daycare, I was a Project Manager in IT (10+ yrs) and can tell you there are always problems in any "redundant" system. Some big and some huge.

Upgrades or migrations for the web and back office are ALWAYS bumpy. It is only a question of the degree.

I presume their "plan" has fallen apart due to unforseen issues. Now it is just a question of how long it will take them to get the process finished. If this is the last outage, I take that as a good indicator that they have good techs. Could be a third party (Fiber provider, hardware manufacturer etc.) and out of their direct control. Seen many projects have much worse issues.

If the company has listened, expressed appreciation and apologized, that is a good sign.

Think hard before you bail. A known foe is better than a stranger.:D

Edited by Phil T on 10/24/07 - 5:21 PM

Posted by cdnwhaler on 10/24/07 - 5:36 PM
#6

While I'm as peeved as most are I have to admit Casco Bay Outrage's comments do make some sense.

"Think hard before you bail. A known foe is better than a stranger"

And I'm typing as person who has very little patience these days:@

Pete

Posted by Joe Kriz on 10/24/07 - 5:37 PM
#7

We are hanging in there for now...

When I get frustrated and fed up, it makes me think of the 2 Vultures sitting in the tree top.
One vulture said to the other: Be patient, something will come along.
The other vulture replied: Patient my rear, I want to kill something..... :D

I'm sure it can't be easy taking care of all the equipment and hearing from 125,000 customers who don't have any patience with outages. I know I wouldn't want to have their job(s)...

Posted by arthureld on 10/24/07 - 5:48 PM
#8

It's probably a good time to make a backup. I need to follow my own advice. :o
I switched hosting services once and it wasn't a big deal for me, but my site wasn't being used by anyone but me at the time.
If you do switch, it would probably be best to have the site up on the old and the new hosting service until you are sure everything is working.

By the way, I'm not an IT guy. I'm just a nut that likes to mess with web development. I also teach a online CADCAM class that I put together with Moodle.

Posted by kamie on 10/24/07 - 6:47 PM
#9

Joe Kriz wrote:
What is strange to me is that they are supposed to have a built-in redundancy....
This to me means if one set of servers or routers go down, then the second set keeps the customers up and running.
However, this was not the case as we were down completely, so where was the redundancy????


Joe,
your right 2 free months would be nothing if you were down for those entire months. But if your down for 12 hours out of 1440 it's not so bad a percentage (99.16%) depending on their stated guarantee. I would also have them go over the root cause with you and yes they should be doing one to see what happened. Ask them exactly what components are redundant in their infrastructure and which are not. I have seen systems fail, I have seen a software upgrade to one switch of a redundant pair take both down, I have seen a database on the primary server have issues, only to fail over to DR and have different issues on the DR side. You can do belt and suspenders but you still have the risk that both will fail. You can do two belts and two suspenders and your risk of failure is lower but can you afford the cost for that lower risk, and the more redundancy you have the bigger the price tag.