An example of how not taking proper procedures and acting with due care can cost a man his whole enterprise, Marco Marsala seemingly removed his entire company by running one line of code in a script. Running a web hosting company can be difficult work, you have to maintain backups of all customer’s data in the event of a hardware failure, preferably in a separate location in the event of a physical catastrophe. Things don’t always go as planned however, especially in this case for the man behind the interesting post made earlier this week.
The man wrote in the ServerFault forum on StackExchange asking how to recover from the mistake, where he included the ‘rm -rf’ command in a shell script that he was running to automate backups for all 1535 of his customers. Instead, this function deleted all files from all volumes he had mounted, deleting both customer websites and their backups. This would not usually happen because of how the ‘rm’ command works in unix, however he was running an automated backup script, which had set certain parameters allowing the script to continue regardless of errors, causing the catastrophe.
The issue with this was that this shell script was responsible for backups, and consequently the backup volumes were mounted. This caused both the live customer’s websites and the backup data to be wiped at the behest of the ‘rm -rf’ command, which can cause nightmares for data recovery as the drives may be paging, writing temp files and logging from in-memory applications after the files are deleted.
Human Mistakes often the Biggest
Problems like this happen all the time, with even Microsoft causing admitting to some major screw-ups where they accidentally lost over a hundred root certificates for their systems.
He has since recovered from the issue, likely using data recovery tools that attempt to re-index drives with deleted data – however this does show that even when precautions are taken to mitigate against failure, you should always expect the worst – it might just be you that causes it!