tio

2023-10-20 03:00:41

NEW BLOG POST:

The best backup tool?

Read it on my website (recommended):

https://www.tiotrom.com/2023/10/the-best-backup-tool/

Or read it here:

Disclaimer: I only have used this for less than a week but I plan to use this for years to come. My excitement compels me to write an article about it.

Backups. So useless 99% of the time, but when you need them, that 1% of the time, they are crucial. I lost TB of files, twice, by mistake, and I had no backups, so that made me be extremely cautions about securing the survival of my digital files. And the files of my family.

The challenge: 4 laptops, around 3TB of files, and hundreds of thousands if not millions of them. From tiny files to tens of GB in size. I need them to be backed-up daily in at least 1 safe place. Sounds easy? Well…may not be.

Local backups plus file sync.

I used to backup all of my files on external HDD’s that I kept in a shoebox attached with a lot of umbilical cords to my laptop.

I was syncing them with FreeFileSync since I found it easy to use. You have a left side where you can select directories you want to back up, and a right side where you can select where to sync.

But ofc you had to go through some settings and put on some rules about what to do with the deleted files and so forth. Plus you had to create a CRON job to automate the process.

Took me some time to figure that thing out. On top of this some files were not properly copied. This was hardly a backup solution. It was sketchy.

Setting this up for the other laptops was not feasible since you would need the FreeFileSync to copy files over the internet from their laptops to my HDD. Ttherefore I used Syncthing to basically sync the files from the other laptops (my family) to my external HDDs. When you need to sync hundreds of thousands of files like that, it is a pain in the ass. Slow, unreliable.

So basically it was difficult, slow, and I was entirely attached to my desk and the drives. No good!

I also tried DejaDup Backups and although super easy to use, it was never that reliable and too simplistic.

Borg, Vorta and Pika.

Before using these I tried a traditional backup method for those who wanna “pay and backup”. Simple as that. Mega. I was paying around 20 € a month for 8TB. Fuckers only provide 2TB or 8TB but nothing in between. I needed only 4TB or so…

Anyway, with Mega it is easy to backup, files were encrypted, they have a good Linux desktop client. I was doing that in parallel with the local backups. But it was not as reliable and would always backup, or sync to Mega. Not every X hours or daily, but ALWAYS. Plus I was relying on them and them only, and their website to go through my files and restore any. And took ages to decrypt my files in the browser to then access them….I felt uncomfortable to use Mega and it was a bit expensive. Plus I could not easily backup the other laptops to the same Mega account.

In all, it was not that great.

Borg. A new way to do backups.

Mega is designed for filesharing, but Borg is for backups. They must be good! But the truth is, a bit complicated. Now, to backup with Borg locally you need either Vorta or Pika, since I am not aware of any other tool for the job that is easy to use.

Pause!

I know there are many people passionate about computers who rsync and restic into ssh sources and daemon-kickstart their backups. I do not want that! I want an app with an interface that I can easily setup for me and my family. I do not do backups because is fun, but because I want to easily backup my files and restore them.

So with Vorta you open it and have to understand how and why they use “repositories”, else you are lost. Repository means….well…hard to explain but you have to select this:

Then choose a folder/drive on your computer where Vorta will create this repository, which is a bunch of files with rules, containing data about your backups. Once you understand this concept is easy. Select a local path then navigate the oldish Vorta interface to select what files you want to backup. Vorta also advertises their borgbase.com, their paid service:

And that does not sit well with me.

They also use this “Prune” notion to delete old backups. In all Vorta is quite messy I would say hard to navigate. For me at least.

But it gets the job done. Locally, from what I remember, it took a few days to backup my 2TB files because it is also encrypting them and compressing them. So good that 2TB may transform in less than 1TB in the repository. That’s fantastic!

The incremental backups are also fantastic and take only a few minutes. Therefore only the first backup is slow. To access your backups you have to go to “archives”, select one, then either “extract” it – which means you’ll be able to download all of it, or parts, somewhere – or “mount” it which asks you to select a folder on your computer where it will mount it all.

The “mount” part is what Borg introduced me to, and it is fabulous. It basically creates a folder where you can access ALL of your backed-up folders and files. These are NOT on my computer, they are “streamed” from the backups drive into a mounted directory:

Which means I can access how my computer files were 6 months ago, if I have backups for those, and navigate them and copy from it to my computer anything that I want, while being able to preview any file.

Stop for a moment to think how useful this is. It is like going back in time, via your file manager, and see your files. Resurrect the dead 2.0!

This is even more amazing when you backup to an online location.

Borg plus Online = Hard.

The same concept and gymnastics apply when you backup to an online location. Remember, I need to backup 4 laptops not just mine. So I decided to rely mostly, if not only, on online backups.

Online backups.

Look…I’ve heard that people say that you have to backup in 2 different locations and one should be local. But let’s be honest, most people do not do any sort of backups and they are fine most of the time. And backing up to an online location is damn secure nowadays. What can happen? The datacenter may be destroyed? Most datacenters have backups anyway so they can restore immediately, and overall it is so rare for this to happen. Almost never.

Second potential problem: these online providers may close your account for some reasons. But super unlikely. If you do encrypted backups and no one knows what you are storing, why would they? Yes they can close your account if you use their storage locations to store “illegal” files that you publicly share….but else…

So I decided to rely on online backups and also finally detach my laptop form the umbilical cords.

Problem numero 1.

For Borg to work you need to find a hosting that provides Borg backups. The server itself NEEDS to have Borg installed on their side too. Right now I only know 2-3 such provides. So from the get-go this is already very limiting. I chose BorgBase since they seem to be the cheapest and most reliable. But for some 4TB of storage you have to pay around 23 € a month. Mega is twice as cheap.

Problem numero 2.

Fucking SSH keys…

I won’t rant again about them, but it was always a pain to create one on my machine, then add to BorgBase, then add to the repo, then connect the two. At times it won’t connect. In short, it was always uncomfortable for me with the SSH keys and 4 laptops. I am still confused about them. I hate them!

BorgBase!

So I ended up setting repositories with BorgBase for everyone and backing up there, daily. Despite being a lot more expensive than Mega, it was a lot more reliable and I felt more comfortable using it.

But I cannot tell my girlfriend, sister, or parents to use Vorta in case they may want to restore something. Because it is too complicated. So I thought to try Pika.

Pika – great and unreliable.

This one is light years better than Vorta in terms of the interface.

Easy! Files to backup, files to exclude, Archives…

Even the schedule is sane:

I can give instructions to my mother over the phone and she can manage it no problem. I cannot say that about Vorta….

However…to connect to BorgBase is not something my mother can do.

So I have to create a new repository on BorgBase.com for my mother, copy the URL for it, add to Pika. Open the terminal on her laptop and figure out what’s the command for creating an SSH key. Create! Find out the command to display it. Copy the key. Go to BorgBase and add it to the Keys section. Then go to the repository I created for my mother on BorgBase and attach the key to it.

Only after that connect to the repository via Pika. I don’t like it, but I did it for everyone.

And guess what, at times this SSH key needs to be “unlocked” with a master password on your XFCE desktop and if you are not prompted for it when you reboot, then Pika is like “I cannot connect there is no SSH key I can use”. So for example on my sister’s laptop the backups with Pika failed multiple times because of this.

Am telling you the SSH keys suck balls 😀

The main issue with Pika. It is slow.

This is something that I reported to the developers. Pika was several times slower than Vorta to backup. So slow I had to use Vorta to do the first backup for everyone, before using Pika to pick it up on the incremental backups. I remember backing up a few hundreds of GB for my sister’s laptop and it was like 10% after several days.

If Pika wasn’t so slow, it would definitely be worth using it. But if you do not have to backup a lot of files, it can be ok. It is a reason why I added Pika as the default backup tool in TROMjaro. It is great overall!

Pika can also mount archives like Vorta and it can even backup to FTP/SFTP sources. And that’s the key for finding out the best backup tool!

Kopia – the best!

We have used Pika and Vorta for the past 2 or so years. Pika was unreliable at times for a few of us. And Borg overall limited in terms of hosting choices and a pain to setup. Plus expensive.

A backup tool needs to have a simple interface; to allow you to easily control what to backup and what to exclude; to allow you to manually delete old backups; to make it easy to schedule and to restore. Plus to support different locations: SFTP/FTP, WebDav, S3, local…

Kopia is an app I found a few days ago while I was trying to find out how the hell to move from BorgBase to a Hetzner Box which supports Borg.

Hetzner provides some really cheap storage boxes. Look:

For 13 € a month you can get a 5TB storage, half as cheap as BorgBase. Mega was 20 € for 8TB, Hetzner is 26 € for 10 TB. So Hetzner is almost as cheap as Mega. Plus look how many protocols it supports:

I tried for several days to make Pika/Vorta borg backups work with Hetzner, but because of multiple SSH keys failures I gave up. No more Borg, that’s it! Fuck those SSH keys!

So I tried Pika via SFTP to Hetzner. Connecting via SFTP is 100 times easier. You get an URL, a username, and a password from Hetzner. Add that to Pika and voila. Connected!

Problem is when I tried to backup my 2TB with Pika to Hetzner it took almost 2 days to backup 3%. Terrible. I had to try something else.

Ok, so what’s this Kopia?

Well it is open source, does encrypted backups by default and with compression, does not promote any hosting so it is independent, and it looks a bit oldish or Windows-ish. Those were my first impressions. I quickly found it in TROMjaro Add/Remove Software as KopiaUI. And it looks like this:

Not bad…And all of those are the protocols that it supports. Yes we see Google there and Azure, but well in case people wanna use these to backup….

So I like that it is so straight forward from the get go. I go and select WebDav since Hetzner supports it.

3 fields. Completed them and connected in less than 30 seconds to Hetzner! See that Borg? See that SSH key nonsense? 😀

And this was the first WOW, fuck I love it! moment. No more BS SSH keys. That’s it. Connected.

You see they also use repositories and now I understand why this is great. Once I connected to my Hetzner WebDav, Kopia created some config files there and from that moment on it will update those files based on the changes I make via Kopia. I will explain later why this is amazing!

Ok, next step. They call backups as “snapshots”. Click and see this:

Select the folder/directory you want to backup. I chose my entire Home folder. From here on can be a bit more complicated but it is totally worth it!

Policies!

If they called these “rules” would have been better, because that’s what they are. So in Policies click edit on your snapshot that you have created.

Here you will see a lot of options but here are the important ones.

Snapshot Retention (how many backups to keep):

This can be confusing but most backup apps use this approach. I want mine to be kept for 30 days, like imagine the past 30 days I have a backup a day. Then for each week to have 20. So for the past 20 weeks, that can go back some 5 months, a backup a week. 12 a month means the same, basically I could have a backup from june, one from mach, or january, and go back 12 months. Took me a bit to understand what these mean….

Then Files is meant to be about ignoring certain files or directories:

I am still unsure if you should add the path with / or without, so I added both. You have to add this manually. Pika is far superior in terms of UI.

Next in Compression:

I selected the one that you see there. You have lots of options. Not for casual users, but well…make sure you select a compression so that the size of your backups is reduced.

Lastly: Scheduling. To schedule how often to take a snapshot.

Bellow all of those is a save button. Save it!

This was the hardest part. Once you learn to navigate that part a bit, that’s it. No more headaches. Trust me this is better than setting up Borg with SSH keys :D.

Now simply go to Snapshots and click: Snapshot Now to create your snapshot.

Keep in mind this app is actually quite simple and efficient. You only have Snapshots, where your backups are. Policies to edit the rules of how your snapshots will be taken, and the rest 2 are for Tasks to see what tasks have been made or, Preferences for a few settings.

It really is not much to it. And that’s fantastic! Except setting up those policies, all is easy.

Fast as hell!

2TB of data backed-up in less than 20 hours, and compressed to 1.4TB. Remember, same files, same server, took Pika 2 days to backup 3%. Would take Vorta a few days to backup to BorgBase.

But I was shocked at how fast it is. And it never crashed.

Easy to restore.

Click the Snapshots, then the “path” – basically your snapshot, and you’ll see them all:

Very easy! I also like the retention tags, the colorful ones.

Now say I wanna restore from the 17th of October. Click it and you’ll see a list of your directories and files:

What I suggest now is to click the top Mount as Local Filesystem button. And then browse:

And this does what Vorta and Pika do, mounting the backup on your system. Again, this is NOT on my computer:

This is from Hetzner but I can browse it as if it is on my computer. How fucking cool! And Kopia is much faster when it mounts them than Vorta for example. Actually Kopia is instant for me.

Icon tray that’s sane!

So simple. Choose to start it on boot, or connect to another repository.

Move it anywhere!

So after I backed-up 2TB of data to Hetzner via WebDav with Kopia, I removed all of the Kopia settings files from my laptop, almost like starting Kopia as brand new. Then I moved ALL of the repository files and backups (1.4TB) from Hetzner to another location on the same Hetzner box. So almost like being on another server. Opened Kopia and it was as if I opened it for the first time. No more backups or anything.

Added the WebDav credentials that would point to the new Hetzner location where the repository was, and in less than a minute Kopia brought back all of my backups and settings.

From this:

To this:

In seconds.

How cool!

This means that I can move the entire Kopia repository anywhere, including locally on a drive, and simply point Kopia to it and we are back on track in minutes! Tell me this isn’t amazing!

Imagine if I find a better alternative to Hetzner. I can move the repository from Hetzner to the new location and then point Kopia to it, and voila. All back!

I cannot do this with Borg. At least not easy at all.

Kopia + Hetzner

I will give you more info about how I set it all up. Remember, 4 laptops need to be backed-up.

In the Hetzner Storage Box panel you can create sub-accounts. These accounts have access to only the folder you give them access too. Like in the example bellow, Sasha only has access to the folder Backups/Sasha and cannot access the other folders:

When you create such a user Hetzner gives you a username and password. Save them. For example when I created the user for Sasha I saved the username (different from Sasha, unique to this storage box), and password. She needs those plus an URL like: https://her-username.your-storagebox.de.

That’s all. She can connect to it via Kopia WebDav in seconds.

Side-note: you also have a MAIN admin user for the entire Hetzner box that you can find in the panel. That user has access to all of the folders for all of the users. I used that admin user to connect in the same way to my Nextcloud account via TROM Files. Simply go to Personal Settings and then External Storage and add a WebDav connection:

Now I have extended my Nextcloud storage to 5TB :D. And I can access the entire box. That’s how I created the folders for the other users. So yeah, you can use the Hetzner Storage Boxes to expand your TROM Files storage to massive amounts.

So yeah, basically you can easily create new sub-accounts on Hetzner and give WebDav access to whoever you want. That’s how I now do the backups.

Hetzner also allows for automated snapshots of the entire storage box:

And these snapshots take very little storage space:

Imagine if by mistake you delete your backups from Kopia or who knows how. You can restore them via the Hetzner snapshots. Neat! Double the safety!

Conclusions.

13 € a month for 5TB, 4 different users with access to WebDav, plus Kopia on individual laptops.

5TB should be enough for many years to come, so it is totally worth it.

Kopia is not as nice looking as Pika, and their “Policies” can be a bit confusing, but else it is so to-the-point, fast, flexible, and reliable. And it is way more than what I presented. See here.

I feel much better now when I know that I can move my backups anywhere and still have them work with Kopia in less than a minute.

Plus doing all of my backups online makes me more free to use my laptop as a laptop. Fuck those cables, I got sick of using them for so many years. I need to keep a balance between security/safety and comfort/mobility.

FreeFileSync

Download FreeFileSync 13.0. FreeFileSync is a free open source data backup software that helps you synchronize files and folders on Windows, Linux and macOS.

^{FreeFileSync.org}