SanMar's Apparent Meltdown

[levelbest]

Anyone else besides me caught up in the middle of the apparent meldown of SanMar's ordering systems? They've been offline for at least 2 full business days now if not more.

Stuff I ordered at the beginning of the week isn't here yet and their website has the following message:

"Update 3 p.m. ET/ 12 p.m. PT:

We continue to experience unforeseen system issues. We now anticipate processing orders beginning tomorrow, Thursday, May 20. We will begin processing orders by the date they were received, starting with orders taken Monday, May 17.

We have been working around the clock since this system issue occurred, and we will continue to update you as we have more information.

Notice: all warehouse will calls are closed until further notice."


It's interesting as I got emails from Broder/NES with the headline - WE TAKE RELIABILITY SERIOUSLY 365 DAYS A YEAR (obviously referring to the mess at SanMar) and the following message...

"At NES, we know that customers rely on fast, dependable delivery throughout the year.
Here's what NES does to ensure reliable service every day.

• Investment of over $10 million in hardware, software, and network infrastructure

• Redundancy - full backup of all systems and databases and an offsite backup location

• Experienced in-house team of network engineers, programmers, and database administrators
NES continues to invest in these areas every day.

Why? We know that if our business is interrupted, your business is interrupted, too.
"

Crazy stuff going on in the biz right now......

[Knehmer]

I, personally, think it's a crying shame that these other businesses are piling it on...SanMar takes their business very seriously as well and had a dreadfully sad incident occur. Quite frankly, I thing the nasty measures the other companies are taking probably illustrate their "fear" of the fabulous business SanMar does. It is competition that keeps America going (even though I pray no competitors to my business set up shop nearby [img]smile.gif[/img] ...

[Steve]

I'm on both companies email lists and get specials weekly (sometimes more than once a week.) I have not seen anything about the outages. hmmmmmmm

[Maggie21]

Yes, we placed an order yesterday and we got that message today. I guess we won't get our garments tomorrow either.

[Steve]

I got an email from Sanmar last night.


May 19, 2010


Valued partner –

Monday afternoon we experienced a software failure in our central ordering system. We have been working around the clock to fix the situation and I am confident we will have this resolved shortly. Though I can’t give you a specific time, we will update you when the problems are resolved. Once the problems are resolved we will work tirelessly to get your orders shipped. SanMar is fully committed to servicing our customers and we are putting every resource possible to fixing the problem. We understand the challenge this has posed for many of our customers and fixing this problem and making sure it never happens again is our number one priority.

I appreciate your patience and understanding. You have my most sincere apologies and commitment to making this right.

Sincerely,


Marty Lott, President

[levelbest]

Latest:

Update 10 a.m. ET/ 7 a.m. PT: We have resolved our system issues, and are currently processing our backlog of orders. At this time our resources are focused on processing these orders and we cannot yet provide specific information on order status. We will have this information shortly and will provide another update when our customer information systems are fully available. We are currently processing orders based upon the date they were received, starting with orders taken Monday, May 17.

That essentially means they were down for three whole days and that someone in the IT department is going to get fired...

[signman]

Must be a major meltdown...my son is IT for a local bank chain and they've had program crashes-and they don't get a break for anything until the system is up and running-no matter how late into the next morning they have to work or how it can screw up a weekend! The overtime pay is great for him but the stress level is a killer...I don't envy the IT at Sanmar but I bet SOMEONE is going to recommend a backup system or alternative solution and get a raise...
Roland

[signman]

It will all trace back to a little mouse that climbed in the main computer and peed on the motherboard...LOL...

[Steve]

John,
Depends what the issue is.
As a computer consultant, I have seen all kinds of whaky stuff. Many companies choose not to implement stuff until its too late. Just last week, had a customer that wouldnt spend money on a backup and the server crashed. We suggested they get one many times. Now they have a SAN in place...most companies are reactive and not proactive.

[levelbest]

I hear you but three days down, millions of dollars lost - someone is getting canned. A company this size simply has to have a backup solution in place or an alternative way of doing business. I can't wait to hear what actually caused all this.

I think smaller scale wise, many medium to small businesses are indeed reactive - but when you get to someone this big, I'm very suprised. We order most of our stuff from SanMar and this has been a royal pain this week. Plus they didn't let us know until very late in the outage that the stuff we ordered early in the week wasn't shipping. Fact is, maybe they didn't know since the computers were down.

[signman]

Didn't someone this week post that they had a power spike and lost communication with DS on their system? And how many times have John, Rod, Ed, etc. etc. recommended power packup (APC) systems?
Yes, it's an expense...but spending $300 for a battery backup/spike prevention system is a lot less money than the business you lose not being able to sew for a couple of days. We are not a big shop, just Sue and I, no volume orders...all small stuff-but we have over half a dozen APC/UPS power units, one on every computer, the ethernet system, even one that is on the Amaya-and YES, it does power it up long enough to shut it down without losing anything! Just can't say it often enough...power backup units SAVE you money.
Roland

[Steve]

Backup-UPS's are required here in Florida. I have several as well. Just yesterday the power flashed several times. My UPS on my embroidery machine can go 75 minutes on a computer but I'm sure its long enough to power the machine long enough to finish a 10,000 stitch design [img]smile.gif[/img]

[Body Cover]

I just received this minutes ago.
When we talked to our inside sales rep. they told us that all the orders, were never lost, they were backed up and still in the system, but that the program to process the orders was not working,
but nice to know they are back up and working. And seem to be concerned that it happened.

http://view.cs.sanmar.com/?j=fe64157...7c761676&WT.mc _id=

[Body Cover]

Hi all, for those who haven't received this I thought I would post a copy, as John Said before, I'll bet some IT guy or two lost there jobs over the new software install. but I think it says a lot about San Mar in the way they handled this nightmare
especially thinking how much money they must have lost and customers that won't stand for the inconvenience... here it is. Ron

"May 25, 2010

Valued customer,

I’m pleased to report that we’ve recovered from last week’s software failure. After working through the weekend and adding extra shifts, we expect to be fully caught up with the order backlog in all warehouses by tomorrow.

Our team of system engineers worked to precisely trace the cause of our system issue. We determined that a third-party software application caused resource conflicts, which resulted in a system shutdown. In response, we have now completely erased this software from our system, so this issue will not happen again. Before loading this new software, we went over ten years without our system being down for a day, and we’re committed to preserving that good record. We understand the impact our system has on your business, so maintaining an operating system that provides consistent, reliable service is absolutely our top priority.

Working towards that goal, last Thursday we began running an internal campaign, “Make it Right.” Because this issue affected many of your businesses, I’ve authorized our Customer Care Center to make it right. I know there are some things we can’t fix, but we’re doing everything in our power to correct what we can.

If you feel you were wronged, and we haven’t already fixed the problem, please let us know what we can do to make it right. E-mail me at [email protected]. I may not be able to personally answer every e-mail, but a dedicated team will be promptly responding.

Sincerely,

Marty Lott, President"

[Steve]

I didn't place an order last week with them so the melt-down doesn't affect me. I have always worked at places that Beta-test stuff on a spare server and there is no reason why it shouldn't have been done in this case as well. Especially, when if something even has a .001% chance of bringing down the whole business. This is a complete failure of the Network Engineer and I agree he/she should be fired. In a large corporate environment, these things should be tested before being rolled out. No excuse!