Hitchwiki:Server crash 2013

From Hitchwiki
Jump to navigation Jump to search

On Thursday, 22nd August 2013 one of the Hitchwiki servers file system died. This lead to the site being down for a while. Unfortunately we realized that our backups were not complete. Guaka, MrTweek and Mikael managed to restore most of the content, but unfortunately not everything. Some backups were current, some where from April 2013 so that we lost only changes between April and now.

  • Completely restored: Wikis in these languages: en, pt, hr, ro, tr, de, lt, it, Maps and Translation tool.
  • Reset to state from April 2013: Wiki: fi, es, fr, he, zh, ru, pl, bg and the country ratings
  • Reset to state from 31st July 2013: Community, Blogs
  • Unknown: Images. A lot of them are lost, unfortunately.
  • Lost: Dutch Hitchwiki. A lot seems to be still in Google's cache, so please help us restoring it if you can.

Also some less important things like web statistics and Langiro, the bot are lost.

Most things are up and running again, but we are still working on it. If you find anything that does not work as expected, please let us know!

We are sorry for the inconvenience caused by this. We promise we learned our lessons in making hitchwiki safe, fast and stable. :)

Technical explanation

A scheduled system upgrade led to minor technical difficulties. While fixing these, I had some trouble mounting one of the partitions, depending on the used Kernel. Naturally solving these issues consisted of rebooting, mounting and unmounting a couple of times which at one point resulted in a corrupt file system. Unfortunately, the file system was xfs which (as I learned then) when it's corrupt, it's completely destroyed - there is no proper repair, recover or undelete tool for xfs.

After all, all of my /home and /var was gone. The second contained the web root and mysql databases, so all of hitchwiki. And since we are doing this as a hobby, we didn't realize that our backup scripts were relatively incomplete.