One command erased GitLab's production database. What to do the morning after

On 31 January 2017, GitLab lost about six hours of production data. An engineer wiped the wrong database server, and the backups turned out to be broken. The morning after a developer breaks production, run a systems review. Protect the person involved. Publish the timeline. Fix the recovery path that let one command do so much damage.

Last updated: 2026-11-12

Key facts

In this article

• What happened at GitLab on 31 January 2017?
• Why did one command take out the GitLab database, and why did five backups fail?
• What should an operator do in the first 24 hours?
• How do you run a blameless review on a remote team?
• Does blameless practice work, or is it just kindness?
• When is it a person problem after all?
• Which three fixes stop a repeat?
• Where RocketDevs fits
• Conclusion
• Frequently asked questions

What happened at GitLab on 31 January 2017?

An engineer was repairing database replication. He wiped the data directory on GitLab's production primary instead of its secondary, and about 300 GB was gone within a second or two.

GitLab is an all-remote company, so this happened during one person's late evening, not in a shared office. Here is the night in order, using GitLab's own times (all in UTC).

The setup

At 17:20, the engineer took a manual snapshot of the production database and loaded it into staging. He wanted a fresher copy to test a load-balancing tool. The automatic copy only runs once every 24 hours. Remember this detail, because it is the reason GitLab had anything left to restore.

Around 19:00, spam pushed the database load up. By about 23:00, replication to the secondary had failed. The secondary needed a full re-sync. GitLab was not using WAL archiving, so the files it needed had already been deleted.

The repair that went wrong

The engineer wiped the secondary's data directory and ran pg_basebackup. The tool hung with no output. He raised max_wal_senders from 3 to 32. PostgreSQL then refused to restart because max_connections was set to 8,000, so he cut it to 2,000. The tool still did nothing.

GitLab's first account says he had mentioned earlier that he was about to sign off. It was around 23:00 local time, and the replication trouble kept him there. The Register described him as a tired sysadmin working late in the Netherlands. "Tired" is the reporter's word, though. The company's own record says only that it was late.

The mistake

He suspected that leftover files were blocking the tool, so he wiped the data directory again. He believed he was on the secondary. He was on the primary. He stopped the command a second or two after noticing, but only about 4.5 GB of the 300 GB was left.

What the team did next

The engineers asked for help on Slack and went looking for backups. Before they restored anything, they copied what remained of production onto a spare server. That kept the evidence safe during the recovery.

Then GitLab went public. It kept notes in a publicly visible Google document. It also streamed the recovery on YouTube, and the stream peaked at about 5,000 viewers by GitLab's own count.

Service returned at 18:14 on 1 February. It was restored from a backup that was six hours old. Data from 17:20 to about 23:25 was gone. GitLab estimated the loss at roughly 5,000 projects, 5,000 comments and 700 new user accounts. Git repositories and wikis were unavailable for a time, but they were not lost.

GitLab published its full postmortem on 10 February 2017. That is where the real lesson sits. The delete took a second. It hurt the company for 18 hours because nothing was ready to restore from.

Why did one command take out the GitLab database, and why did five backups fail?

The command started the incident, but an untested restore path decided how bad it got. GitLab's live notes said that none of its five backup and replication methods was working reliably, and some had never been set up at all.

Precision matters here, because the popular version says "five backups failed." The postmortem actually lists four procedures. The live notes reached five by counting the S3 uploads as their own method. One of the four, the manual snapshot, did work. It had never been meant for disaster recovery.

Each of these failures was ordinary. Here they are one at a time.

The daily dump

GitLab's package picks the PostgreSQL version from the data directory. The backup ran on an application server with no data directory, so it defaulted to version 9.2. PostgreSQL's documentation says pg_dumpwill refuse to even try against a server with a newer major version. The job failed, and it failed quietly.

The missing alert

The failure emails from the cron job went to a receiver that rejected them for not being DMARC-signed. InfoQ summed it up plainly: "nobody knew it was failing." A backup job that reports only by email is one that nobody can be sure is running.

The unused snapshots

The snapshots that would have helped were never switched on for the database servers. The team assumed its other backups were enough. In effect, each system covered for the others, and none had an owner. GitLab's own "five whys" analysis ends on exactly that point: "nobody was responsible for testing this procedure."

This is worth a pause for anyone running a team. A backup with no owner is a hope. A restore you have never run is a hypothesis, not a backup. NIST says the same thing in dull language: backup media should be tested regularly, so that files can be retrieved without errors or lost data.

The slow recovery

Restoring from staging meant copying data across throttled network disks of around 60 Mbps. The copy alone took about 18 hours. Cheap disks were a sensible saving on a test server. These were a poor choice for the only recovery source.

The knowledge gap

Small teams should note one more detail. The pg_basebackup tool waits silently until the primary starts sending data. This is normal behaviour, and an engineer who was not around knew it. It was missing from the runbooks. If knowledge sits in one person's head, you may be reading a bus factor problem as a database problem.

A fair objection

GitLab's five whys stop at "no ownership." John Allspaw has argued that complex systems have no single root cause. He has a point. Spam, an account scheduled for removal, a hung tool, a wrong host and an empty bucket each had to happen. Use the five whys to decide where to put fixes, not to find a villain.

What should an operator do in the first 24 hours?

Run it as a systems review, and protect the person from the first hour. GitLab's handbook now expects a public root-cause analysis within 7 days for severity 1 incidents. That means the first day is about stabilising the system, preserving evidence and not making the human problem worse.

This runbook is our own judgement, built on the GitLab record and the policies mentioned alongside it. Adjust the owners to suit the size of your team.

Three parts of this table need more explanation.

Protect the person

We looked for the line where GitLab promised not to punish the engineer. It is not in the record that we could read. What the record shows is different, and is arguably stronger. The postmortem names no one. GitLab's CEO apologised personally to affected users. On Hacker News, a poster writing as sytse agreed that the CEO was responsible. He wrote that "the blame is with all of us, starting with me. Not with the person in the arena."

The engineer's own reply was calm. He wrote that many problems had come together into one. He also said that someone else would eventually have made the same mistake. GitLab said he had chosen to be named in the live notes. It added that it would redact names in future, because other engineers might not be comfortable with that.

It is unclear what happened to the engineer's job after the incident. Atlassian tells a similar story about a config mistake that took its whole company down for 45 minutes. It says the engineer still works there. That is Atlassian describing itself, so weigh it accordingly.

Publish the timeline

Publishing earned GitLab more attention than shame. The Register reported that online opinion mixed admiration for the open report with condemnation of the untested backups.

It also reported that GitLab's PR staff called the lost data peripheral metadata and said that under 1% of users were affected. The company's own log counted 707 users who lost data. Do not shrink the number. The people who lost data know what it is.

Check the remote angle

Remote work is mainstream. In the 2025 Stack Overflow survey, which drew over 49,000 responses, 32.4% of respondents said they work remotely. Your developer probably made this mistake at the end of their day, alone, in a time zone where nobody senior was awake. That is not unique to remote teams, but it is sharper there.

If your handovers depend on someone being online, think about how remote teams lose time waiting for answers. Decide now who the second pair of hands is when it is midnight for the one person who knows the system.

How do you run a blameless review on a remote team?

Run it as a written review first and a short live call second. Keep it blameless in the strict sense that Google's SRE book gives. The review must identify contributing causes without indicting any individual or team. Writing first suits a distributed team, because nobody has to be sharp at an awkward hour to tell their side.

Here is an agenda you can paste into a calendar invite.

Notes on running it well

Let the person at the centre write first, if they want to. GitLab's engineer put his own name in the live notes. Offer the choice, but never make it an obligation.

Rotate the call time so that it is not always someone's evening. Record the call. Let people add comments for a day afterwards.

The "what went right" step is not just politeness. In GitLab's case, a snapshot taken by hand for an unrelated test limited the loss to six hours instead of almost 24. Write down what luck did, then turn that luck into a rule. Hourly snapshots, which GitLab later listed, are that kind of rule.

Measure the culture too

Google's team survey used a single item to measure psychological safety: "If I make a mistake on our team, it is not held against me." Ask it anonymously before and after the review. If the score falls afterwards, the review was not blameless, whatever the agenda said.

Does blameless practice work, or is it just kindness?

It is a measured practice. In 1999, Amy Edmondson studied 51 work teams in a manufacturing company. She found that team psychological safety was associated with learning behavior. That learning behavior, in turn, connected safety to team performance. The paper has been cited 14,271 times, according to OpenAlex.

Edmondson defined psychological safety as a shared belief that the team is safe for interpersonal risk taking. A later meta-analysis of 136 independent samples, covering over 22,000 individuals, tested what causes it and what it leads to. Edmondson's earlier hospital work found that groups differed in more than how often errors occurred. They also differed in how likely those errors were to be detected and learned from.

Evidence closer to software

Ron Westrum's review found that information culture is associated with error reporting and with performance. He also said the link needed more exploration before it could be called definitive. DORA's research draws on his work. It reports that high-trust cultures that emphasise the flow of information predict delivery performance.

Why it works

James Reason's framework explains the mechanism. The person approach blames individuals for forgetfulness, inattention or moral weakness. The system approach looks at the conditions people work in. Reason also observed that the best people often make the worst mistakes. He noted that the same circumstances provoke the same errors, whoever is present.

GitLab's CEO made the same point about his engineer on the night, in plainer words. He said "it happens to the person that is working the hardest."

These papers are widely read. OpenAlex counts 6,062 citing works for Reason's paper and 1,668 for the meta-analysis. It counts 961 for Edmondson's hospital study and 309 for Westrum's typology.

The cost of blame is silence

The SRE book says that where finger-pointing prevails, people will not bring issues to light for fear of punishment. PagerDuty makes the same argument. It adds that silence raises the time to acknowledge and the time to resolve an incident.

Interest in the idea has grown too. English Wikipedia's article on psychological safety drew 810 views in October 2015 and 5,653 in October 2020.

How far the evidence goes

Be careful about how far this evidence stretches. Edmondson's teams made things in a factory, not software. Westrum says his link is not yet definitive. Google's Project Aristotle is Google's own research, and the source mentions no independent replication. We found no published study of blameless review in remote developer teams. The research supports the direction. It does not tell you how much you need.

When is it a person problem after all?

Sometimes, but rarely. The test is behaviour, not the size of the damage. A just culture is not the same as a no-blame culture, because people can still be held accountable for misconduct or negligence.

Reason wrote that some unsafe acts are egregious, but the vast majority are not.

This answer has a cost, so I will say it plainly. Some people should not stay on your team, and that includes some developers we have placed. Blameless does not mean consequence-free. It means you decide on conduct, with the facts in front of you, after the systems review. You do not decide at nine in the morning with a Slack thread open.

Five questions to ask, in this order

1. Did they follow the process as written? If the process was missing or unclear, the person is not the finding. GitLab's runbook was silent on pg_basebackup, for example.
2. Did the system make the wrong action easy? Two hosts with similar prompts and a destructive command with no safety check is a design flaw.
3. Would a peer in the same position have done the same? GitLab's engineer thought so, and said so in public.
4. Did they hide it? A mistake reported late but honestly is very different from one that was concealed or misreported.
5. Is it a pattern after the fixes? One incident is data. The same incident after the guardrails are in place calls for a conversation.

A yes to question 4 or question 5 moves the issue from the system to the person. It then becomes a matter of performance and trust, and you handle it as you would for any engineer.

Our piece on when a new engineer is not working out sets out the 30-day version of that decision. If you do reach that point, also read why good developers leave small teams.

Which three fixes stop a repeat?

Fix the place where recovery fails, not the place where the finger points. GitLab's published list included clearer host prompts, backup monitoring and hourly snapshots. It also listed automated testing of database restores and an owner for data durability.

GitLab said its main focus was disaster recovery and making the host more obvious. It did not focus on stopping engineers from running commands. An alias, it argued, would only protect one person against one command. From that list and the research, three fixes matter most.

1. Schedule restore tests, with a named owner

NIST says recovery capabilities "shall be tested annually." Google Cloud advises testing backups by restoring them in a non-production environment. GitLab's later tracker shows an owner assigned for data durability. Without an owner, the test never happens.

2. Make failure loud

The backup died silently because its alert was an email that nobody received. Send backup status to a channel that a human reads. Alert on the absence of a good backup, and not only on errors. GitLab listed Prometheus monitoring for backups for this reason.

3. Make the wrong host obvious, and make recovery cheap

GitLab planned prompts that tell hosts and environments apart. It also moved to continuous archiving. Its tracker records this as streaming to S3 and Azure with WAL-E. pgBackRest is another established route, with about 4.5k stars on GitHub. Both are options, not endorsements.

The other side of the lesson

AWS shows the same lesson from a different angle. Four weeks after GitLab, an Amazon S3 outage began when one input to a command was entered incorrectly. AWS then changed the tool so that it removes capacity more slowly. It also added safeguards. Guardrails and recovery work as partners. Neither replaces the other.

Where RocketDevs fits

Our part is small, and it is not a pitch. Every RocketDevs developer goes through 6-8 hours of vetting. Only the top 2% are accepted.

No amount of vetting makes a person immune to a wrong command at midnight. GitLab's engineer was working on a hard problem late in the evening, and the system let one command do far too much. Hiring well lowers the odds of a mistake, but it cannot remove them. That is why the fixes in this piece sit in the system and not in the person.

What vetting does buy

What vetting buys is judgment you can see. It shows in how a developer reads a problem, how they ask for help and how they behave when something goes wrong.

If a new developer is still in the 14-day risk-free trial, the morning after an incident is a useful test. You are not testing whether they ever make a mistake. You are watching what they do next. Do they report it quickly and honestly? Do they help build the timeline? Do they ask what the system should have done differently? These are the same behaviours that made GitLab's response work.

Fix the system before you swap the person

If the review points at a system, fix the system first. Use the five questions from the previous section before you decide anything about the person. A missing runbook, a confusing prompt or an untested backup will catch the next developer in the same way. Replacing the person without fixing the cause only resets the clock.

The outside-in version of this morning

This piece covers an incident that starts inside your team. If you want the external version, where someone else discloses a vulnerability in your systems, see our guide on critical vulnerability response for a small team.

Conclusion

The delete at GitLab took a second or two. The recovery took 18 hours. The gap between those two numbers is the real story.

One engineer did not create that gap. A backup failed quietly, and nobody was responsible for testing the restore. The engineer was simply the last person to touch the problem.

Every team has a command like that somewhere. Someone will run it late at night, on the wrong host, and believe they are safe. You cannot promise that this never happens. You can only decide what happens next.

On the morning after, you have two questions to choose from. "Who did this?" gets you silence and a name. "What made this possible?" gets you a timeline and a fix. GitLab asked the second question. It kept the engineer out of the postmortem and published the whole story. The Register found that openness drew admiration alongside the criticism.

Blameless does not mean consequence-free. It means you look at the system first, and you judge the person only on conduct. That is a harder standard than blame, and a far more useful one.

So here is the test for your own team. Pick one backup and restore it this week. If the restore works, you have a backup. If it does not, you have just found your own 31 January, and you found it on a quiet day.

Prepare for the worst morning before it arrives. The command will still matter, but it will no longer decide how the day ends.

Frequently asked questions

Should you fire a developer who broke production?

Usually not, and never before the systems review. A just culture holds people accountable for misconduct or negligence, not for honest mistakes. Reason's view is that the vast majority of unsafe acts are not egregious.

Fire someone for concealment, or for repeating the mistake after the fixes are in place. Treat one wrong command as a design finding.

What is a blameless postmortem?

It is a written review of an incident. It identifies contributing causes without indicting any individual or team, as Google's SRE book describes it. It assumes that everyone acted in good faith on the information they had.

Atlassian and PagerDuty describe the same practice. GitLab's handbook commits to it for incident reviews.

How do you write a postmortem?

Start with a timeline in one time zone. Then list the contributing factors and what went right. End with actions, each with an owner and a date. Add the impact in plain numbers.

GitLab's own postmortem followed this shape and ended with a public list of fixes. Keep it factual. Do not name the person unless they ask you to.

What happened in the GitLab database incident?

On 31 January 2017, an engineer was repairing replication. He wiped the data directory on GitLab.com's primary database server instead of the secondary. About 300 GB was lost.

GitLab restored from a snapshot taken six hours earlier. The service was down for about 18 hours. Six hours of issues, merge requests, comments and users were lost.

How often should you test backups?

NIST says backup media should be tested regularly. It says recovery capabilities should be tested annually. For a database you cannot afford to lose, test a full restore on a schedule that someone owns. Test again after any change to the backup job.

A backup is only real once you have restored from it and the application works on the result.

James Hitch, COO at RocketDevs.LinkedIn

Sources

• GitLab, Postmortem of database outage of January 31, 10 February 2017, and GitLab.com database incident, 1 February 2017
• GitLab, infrastructure issue 1684 tracking the fixes, and Handbook: all-remote and incident review
• Simon Sharwood, The Register, GitLab.com melts down after wrong directory deleted, backups fail and GitLab.com luckily found lost data on a staging server
• David Iffland, InfoQ, GitLab.com Postmortem Digs into Root Causes of 18 Hour Outage, 21 February 2017
• Hacker News, GitLab Database Incident, Live Report, CEO's comment, second comment and engineer's comment, 1 February 2017
• Wikimedia, Pageviews API for GitLab and Psychological safety
• Google, John Lunney and Sue Lueder, SRE book, Postmortem Culture; Google re:Work, Understand team effectiveness; Google Cloud, Perform testing for recovery from data loss; DORA, Generative organizational culture
• Amy Edmondson, Psychological Safety and Learning Behavior in Work Teams, Administrative Science Quarterly, 1999, and Learning from Mistakes Is Easier Said Than Done, 1996
• Frazier and colleagues, Psychological Safety: A Meta-Analytic Review and Extension, Personnel Psychology; OpenAlex, records for Edmondson 1999, Frazier, Edmondson 1996, Westrum and Reason
• R. Westrum, A typology of organisational cultures, 2004; James Reason, Human error: models and management, BMJ, 2000
• John Allspaw, Each necessary, but only jointly sufficient, 2012; Wikipedia, Just culture
• PagerDuty, blameless culture and accountability; Atlassian, How to run a blameless postmortem
• NIST, SP 800-34 Rev. 1; PostgreSQL, pg_dump and pg_basebackup documentation
• Amazon Web Services, Summary of the S3 service disruption; GitHub, wal-e and pgbackrest
• Stack Overflow, 2025 Developer Survey, Work