Workday’s outage began with its own backup test, and workers couldn’t clock in

Workday, the software many large employers use for payroll, clock-ins and time off, went dark for its customers after a test of its own emergency backup went wrong. While checking its disaster-recovery systems, the company found that backup services had unexpectedly connected to the live databases its customers work in, leaving records in an inconsistent state. Workday pulled the affected systems offline to repair them.

For employees, the outage meant a login page that read “Workday is currently unavailable.” Staff at affected companies could not clock in or out, check pay information, request time off or finish benefits enrollment, and scheduled jobs that feed other company systems could fail. The safety net built to keep the service running through a disaster is what knocked it over.

The scale is why this reaches beyond HR departments. Workday says more than 11,500 organizations use its software, and its public client list includes Walmart, Target, Netflix, Chipotle, Southwest Airlines, the US Department of Energy and NASA’s Jet Propulsion Laboratory. The company has not said how many of them were hit, describing the impact only as “some of our customers,” so viral posts treating every one of those employers as down go further than the facts. Outage trackers logged reports from the US and India, according to UC Today.

Workday’s own explanation, posted on its customer status page and quoted by Washington University’s IT team, is unusually blunt. “While validating the weekly Maintenance Window for Disaster Recovery tenants, we have identified that tenanted services within the DR Site unexpectedly connected to Production databases,” the notice said, adding that the data was left “in an inconsistent state” that could cause errors when people opened their accounts or make tasks fail. The company then put every production environment in the customer group it calls WD1 into maintenance and promised updates every 30 minutes.

That explanation leaves the biggest question open. A disaster-recovery copy exists so that trouble in one place cannot spread to the other, and here the test reached the real thing. Workday has not said how that path existed or whether any records were lost rather than just mismatched. Its messaging drew complaints as well. Administrators on Reddit’s sysadmin and Workday forums said the problem started hours before the first customer alert, a timeline Workday has not confirmed, and one customer email shared there admitted there was no estimate for restoration and told companies with critical deadlines to switch to their business continuity plans.

For workers, the practical damage depends on timing. Hourly staff who could not punch in need their hours corrected by hand, and anyone racing a benefits deadline found the door shut. Some employers moved the goalposts: Emerson College said it extended its open enrollment window because of the outage. The incident also landed at the end of a bad week for workplace software, after a Microsoft 365 failure left Outlook and Exchange email unreliable into a second day.

The timeline so far: Emerson College said Workday took the system offline at about 5:15 p.m. Eastern on Friday, November 13, and LiveNOW from FOX reported that Downdetector logged nearly 3,000 problem reports shortly after 5 p.m. A Workday spokesperson said the company was “working urgently to investigate and resolve the issue” and apologized to affected customers. By Saturday morning, Workday told customers that reindexing and releasing their environments was expected to finish for the majority of them by 11 a.m. Eastern on Saturday, November 14, with some taking longer; those customers would be contacted through a support case. Emerson’s extended enrollment now runs until noon Eastern on Monday, November 16.

Service coming back will not settle the harder question that employers are now asking their vendor. They will want to know why a test designed to prove Workday can survive a disaster had a route into the data it was meant to protect.

Tags: cloud services, disaster recovery, HR software, IT outages, payroll, Workday