In the middle of a major technical problem that disrupted about 5% of its merchants for 5 days, BigCommerce’s Head of Operations and Site Reliability Scott Baker had harsh words for IBM.
Bigcommerce is a store-hosting platform. In a status update on the Bigcommerce website on Friday evening, Baker claimed IBM Softlayer “intentionally let their Object Storage cluster fall into disrepair and chose not to scale it,” but said he only learned of this 3 days ago.
Bigcommerce spokesperson John Yarbrough told EcommerceBytes Monday that the service disruption impacted approximately 5% of Bigcommerce stores for a duration spanning 10pm CDT Wednesday, April 1 until 1am CDT Sunday, April 5.
Yarbrough explained that Bigcommerce relies on third-party service providers for some components of its infrastructure – including IBM SoftLayer object storage. “On late Wednesday evening, April 1, the storage cluster started exhibiting degraded performance,” he said. “This issue caused slower site performance and inadequate store rendering for a subset of Bigcommerce clients.”
Baker was less sanguine, writing in the status update, “Our engineers placed too much trust in IBM Softlayer and that’s on us. However, the catastrophic failures to see metrics and rapidly scale capacity, the decisions to let hard drives sit at 90% utilization for weeks and months, the cascading failures of an undersized cluster of 52 nodes for the busiest data center in their business speaks to IBM Softlayer’s lack of concern for their customers. We found this out 3 days ago.”
He told users Bigcommerce was planning to move off Softlayer Object Storage and better plan for a single vendor failing.
In addition to providing updates on Status.Bigcommerce.com, Bigcommerce responded to questions from merchants on this thread, where sellers discussed the impact on their stores.
Some merchants who were impacted by the technical outage criticized the way Bigcommerce communicated with them, saying they heard about the problem from customers.
“We’ve heard that feedback as well, and we are working to ensure all clients are notified and regularly updated,” Yarbrough said. “Since the issue was discovered, we have been regularly providing updates at status.bigcommerce.com, which we use to communicate updates for any service disruptions.”
We asked if Bigcommerce would be providing any credits as a result of the disruption. He replied:
We encourage any clients that have questions about their accounts to contact Bigcommerce Support. Our clients and their business are our top priority, and we are contacting all impacted stores. The relationship we have with our clients is based on trust, and the severity and duration of this incident broke the promise we make to support their businesses without interruption. We sincerely apologize to all clients that were impacted.
He also pointed to a 99.99% or better uptime for Bigcommerce merchants over the past 6 months, including 100% uptime during the busiest online shopping period in history from Black Friday through Cyber Monday. “We will continue to improve platform stability and performance including taking proactive steps to ensure our systems, as well as those of our third-party providers, incorporate the utmost levels of resiliency and redundancy to prevent issues such as these from happening again.”
