The time I lost around 100 GB of family photos

Backup your files, kids

Cet article est entièrement en anglais.

Aha! I told you I'd eventually make a blog post on system administration!

While it's a thing that feels difficult for me to grasp, as well as for literally anyone else at the start of the Dunning-Kruger curve, making mistakes IS the way to get better. Every error is a way to make you understand that you know nothing at all, and this makes you more modest, more more careful, and less likely to screw up.

I say this because I'm very much still learning about system administration, and this is what I feel, and what I'm going through. I'm still learning about all the faces of system administration - and the experience feels like this:

This XKCD comic is under CC BY-NC 2.5 .

Anyway, I started with this whole "speech" because that is the idea behind why I feel the need to "come clean" about my previous mistakes. I can also show I've learned, and hopefully so others can learn too!

This happened over a year or two ago.

Precursor

My old family computer had a ton of images in it. I could not care for my old pictures - but I felt a need to archive the whole family. Historically speaking, I don't have much from my ancestry, so I wanted to keep stuff for people who will live after me.

The setup

I was running Arch Linux and had around a combined 1TB of space spread across 3 or 4 hard drives running in RAID and LVM, with one exterior hard drive that had the OS in it. At this time, I had a personal server that didn't have much on it other than some niche projects, I wasn't self-hosting any software.

I copied the family computer hard drive to this array of hard drives and wiped the family computer hard drive.

What happened

One faiteful morning, I thought of cleaning my computer physically. That included me unplugging the hard drives. After cleaning, I plugged them back in, but not in the same order. Unbeknownst to me at the time, I had done a fatal mistake - that was not that fatal had I not panicked.

What I did

In a state of total panic, once I plugged them back in, I was getting mounting errors. I didn't know what this meant, and when you're panicking, you can't search very well. I knew it was not the idea of the century to go crytyping on forums, trying to get an answer. I immediately assumed the data on the hard drives were gone, and using TestDisk yielded no help.

I tried using the same program on the formatted family computer hard drive, which I had not zeroed out, and managed to recover around 10 GB of pictures... ...and 50 GB of garbage, sprites and unrelated data.

What I should have done

I've been working on myself to stop panicking if I make mistakes - and I know how hard it is.

If you screw up, DO NOT PANIC. Easier said than done, but the consequences are avoidable as long as you keep a clear mind. It is best to work on this before you screw up again, so that you can keep cool. I don't have any recommendations on how to work on yourself - everyone is different. I welcome you to look up how to do that, and to try.

Understand what you are doing before doing it. Test around, take your time. Really. It's insanely easy to assume you know everything, and that it is so easy, but this way of thinking will bite you in the arse. Caution and testing - these are the good ways of a system administrator. You are at the bottom of every chain. No test will run on a broken server. No program will compile or run. System administrators are at the bottom of everything.

Everything was still recoverable had I not relied on my churlish and ignorant confidence. I just had to plug everything back in, check the UUIDS of the hard drives and compare them to what was on /etc/fdisk in order to get everything back in the right place.

Conclusion

Modesty is your best friend. We're all perpetually students and teachers. The speed at which we learn is different and the importance of it all is to not be afraid. It is by making mistakes that you know mistakes are to be made in the first place.

I know so many people who I look up to in terms of coding, and I am sure even they have, at some point, run rm -rf / --no-preserve-root by accident.

I'm surely wrong in parts of this blog post too. I invite you to correct me. It can only make both of us stronger.

If I sound overzealous and pompous, braggadocious, etc, I don't want to put myself on a pedestal. I'm no better than anyone else and that is not the tone I want to convey.

My email's miaoutage@42l.fr if you want to talk.