VioletPixel

đź’ľ Backblaze, You Had One Job

Turns out Backblaze no longer backs up cloud files—not even cloud files that are already downloaded and stored locally on your computer. Worse, they quietly shipped this change in a minor point update and hoped no one would notice:

Release Version 9.2.2.877

[...]

  • The Backup Client now excludes popular cloud storage providers from backup, including both mount points and cache directories. This prevents performance issues, excessive data usage, and unintended uploads from services like OneDrive, Google Drive, Dropbox, Box, iDrive, and others. This change aligns with Backblaze’s policy to back up only local and directly connected storage.

That "others" includes iCloud Drive, which is where I happen to keep most of my stuff. My important stuff. The stuff I trusted Backblaze to back up.

Backblaze also has a new banner on their old instructions that explain how to back up cloud files before this change, like the one on their instructions for iCloud Drive:

NOTE:
iCloud's most recent update prevents Backblaze from backing up files that iCloud synced.

To back up these files, download them to another local location where Backblaze can read them.

First, that's a baldfaced lie. There's no iCloud update that prevents Backblaze from backing up iCloud files. There's similar wording on their page for Google Drive:

Google Drive's most recent update prevents Backblaze from backing up files that Google Drive synced.

And their page for OneDrive:

OneDrive's most recent update prevents Backblaze from backing up files that OneDrive synced.

Bullshit. Apple, Google, and Microsoft didn't release updates at the same time across multiple operating systems that stopped Backblaze from backing up their cloud files. Backblaze made the choice to not back these files up. Backblaze made the choice to not tell their users about it outside the third bullet point in a minor point release. Backblaze made the choice to put these banners on their support articles and knew they were lying through their teeth by doing so. Other backup software has the ability to back up iCloud Drive and other cloud files just fine; there's nothing preventing Backblaze from doing the same.

Second, the content of the original instructions remains below these warnings, but none of it applies anymore. It would be easy for someone to skip over one of these warnings, read the instructions below, and think their cloud files are backed up when they're not. That's shameful and irresponsible.

When I learned about all this I checked Backblaze's file restore tool and confirmed all of my iCloud Drive files were no longer available to restore even though I explicitly configured iCloud Drive on my Mac to keep a full local copy of every file.

It's really important to have my iCloud Drive files backed up because sync is not a backup, as Backblaze's own blog explains:

While cloud sync is great for giving you 24/7 access to your files and enabling collaboration, it is not a trustworthy backup solution. Here’s why:

  1. Tiered pricing discourages usage: Typically, these services have tiered pricing, meaning you pay for the amount of data you store with the service, or for tiers of data that you are allowed to use. Odds are, if you are using the free tier of a syncing service, you have a lot of data on your computer that’s not syncing because you’re trying to manage your usage to avoid paying more.
  2. Data outside the service is vulnerable: Only the files, folders, or directories you put into the sync service are synced. The rest of the data on the computer is not.
  3. Data within the service can be deleted and lost forever: If there is data loss (let’s say you share a file with someone and they simply delete it), it may be lost forever. Sometimes these services have a version history feature, meaning you’re able to recover an earlier version of your work (before your friend or coworker deleted it).
  4. Data in sync services is vulnerable to ransomware and malware: If your computer is attacked by a bad actor and your sync service automatically synchronizes after the attack, your synced files are also corrupted. With a backup solution (discussed below) with longer version history, you can simply roll back to an earlier backup before the attack occurred.

We often hear people say, “I don’t need backup. I use Dropbox.” But, that’s an assumption to leave in the past. Where sync services ensure that a certain set of data is the same across multiple devices, backup ensures that all or most of the data on one device is backed up elsewhere.

It's a huge problem when the backup service you trust to back up your files stops backing up your files. It's an even bigger problem when they make that change as quietly as possible, without letting you know, and lie about the reason why.

These changes—both the decision making process that led to it and the way it was implemented—have completely shattered my trust in Backblaze. Beyond their computer backup, we also used their B2 storage service to backup our NAS, but now I wouldn't trust them or any of their products or services with a single byte of my data or anyone else's.

I just finished removing Backblaze from all of our devices and deleting our Backblaze accounts. If you use Backblaze I strongly recommend you seek an alternative.

Our new backup solution

Our household has the following devices that need to be backed up:

The iPhones and iPads use the only practical backup solution for iOS and iPadOS devices: iCloud Backup.

Our five Macs back themselves up to our Synology NAS using Time Machine for local backups. After some extensive research, I decided on Arq backup software for our Macs and rsync.net for offsite backups.

Note that this approach isn't as easy to set up or maintain as Backblaze. If you're looking for a more-or-less drop in Backblaze replacement, I recommend you give the Arq Premium trial a try. I'm not affiliated with them in any way, but I used the trial quite a bit during my research and found it to be a solid option when I tested it.

Arq is available in two flavors. You can pay once for the backup software itself and use it with the cloud provider/server of your choice (including a a five-computer family pack license), or you can choose Arq Premium: a subscription service that includes the Arq software and Arq's own cloud backup storage bundled together (at time of writing Arq uses Google Cloud Storage behind the scenes and offers 1TB of storage for up to five computers).

Arq Premium was tempting, but ultimately I wanted a unified and more flexible offsite backup destination than Arq Premium or S3-compatible storage providers could offer (shout out to S3compare.io as a very useful tool during my research, though!). If I went with Arq Premium I wouldn't be able to backup our Synology to it, and would have had to find a separate offsite storage provider for it. That's more complexity and accounts than I want to juggle.

Regarding cloud files (the ones Backblaze says they can't back up), Arq backs them up just fine. On a Mac, cloud files that haven't been downloaded to your computer show up with a cloud icon next to them in the Finder. If you try to open or access a cloud file, the OS automatically downloads the file and, when the download is complete, the cloud icon is removed and the file opens normally.

Arq lets you decide how cloud files should be handled on each computer with a, 'When a dataless ("cloud-only") file is encountered,' setting that has three options:

The first two are self-explanatory. The third, "materialize," tells Arq to trigger a download of any cloud file it comes across so it can then back it up.

I, personally, have two Macs signed in to my iCloud account:

I keep a local copy of everything in iCloud Drive on my MacBook Air by keeping the "Optimize Mac Storage" setting in iCloud Drive's settings turned off. I also keep my entire photo library stored locally by setting the "Download Originals to this Mac" option enabled in Photos. Finally, Arq's "cloud-only" setting is set to "materialize" on my MacBook Air to make sure all of my cloud files are present and available for offsite backups.

On my MacBook Neo I have the opposite setup. That Mac has a smaller internal drive and is often offline, so I have iCloud Drive and Photos set to optimize my Neo's storage, which means things will only be downloaded locally when I need them. Likewise, in Arq, I have the "cloud-only" setting set to "ignore" because I know those cloud files are being backed up via my MacBook Air, so I don't need to worry about them on my Neo.

Beyond Macs, our Synology used to back up to Backblaze B2, but it now uses Hyper Backup to back up to our rsync.net server. One annoying thing about Hyper Backup when using it with an rsync server is that it doesn't support SSH key authentication. I was hoping to turn password authentication off on our rsync.net server for security reasons, but until Hyper Backup gets its act together I'm forced to leave it on (with an absurdly long password).

I chose rsync.net because they have an experts-only plan that works well for our needs. In exchange for cheaper storage, they provide a fully-functional rsync.net server with the following caveats:

That's all fine by me, but anyone who isn't technically inclined will probably consider some or all of that a dealbreaker.

One other important consideration when choosing our cloud backup provider was the physical location of the data. There were a few options that looked promising until I realized the only practical server location was in Seattle. We live in Portland, and if the big one hits I want our data to be a good distance away from the Pacific Northwest. rsync.net's Denver location is an excellent balance between geographical distance/safety and latency.

Another important thing to note about backup solutions in general is how they handle backup history and retention. For example, if a file accidentally gets deleted today, but you don't notice it's missing until three months from now, you want the ability to go back to a backup from three months ago and retrieve the file. That's one of the big reasons sync is not a backup; changes and deletions are synced immediately. Some cloud sync providers offer version history, but it's usually limited or restricted (e.g., only 30 days, costs extra, etc.).

With a normal rsync.net account, the idea is that you get the files you want backed up onto your rsync.net server and backup history is handled by ZFS snapshots. However, with an experts-only account you don't get free ZFS snapshots, which means you either need to configure ZFS snapshots yourself or make sure historical backups and retention are handled in another way.

I chose the latter approach. Both Arq on the Macs and Hyper Backup on our Synology can be configured to keep historical backups and prune old versions as they age. This provides fine-grained client-side control instead of a one-size-fits-all solution on the server, which works best for us. We can, for example, keep more granular history for our Macs but less granular history for our Synology. Thus, I have not configured ZFS snapshots on our rsync.net server and don't plan to in the future.

That backup history distinction is important to keep in mind if you get an experts-only rsync.net account, because most of the tutorials and guides for rsync.net assume you have snapshots enabled. For example, rsync.net provides instructions for backing up Synology devices using Hyper Backup, and tell you to use the "rsync copy (single-version)" option, assuming the ZFS snapshots will handle historical backup needs. Without snapshots you should choose the option that keeps multiple versions and configure retention for those versions in Hyper Backup.

I would also be remiss if I didn't mention encryption, especially for offsite backups stored on infrastructure you don't own. Arq and Hyper Backup allow you encrypt your data before it leaves your device, so all of the data on rsync.net's server is fully encrypted and we're the only ones with the keys. We also have our Macs encrypted with FileVault, and our Time Machine backups to the Synology are encrypted as well. Your specific encryption needs may vary, but encrypting your data when possible is generally recommended.

One last thing: keep the 3-2-1 rule in mind:

The idea that a minimal backup solution should involve three copies of the data (one primary copy and two backup copies), where two different media types are involved in storing the copies, and one of the copies is stored offsite in a remote location.

Unfortunately, following the 3-2-1 backup rule is inconvenient for iOS and iPadOS devices. You can do it, it just requires you to plug your iPhone or iPad into a computer and make a local backup which, combined with iCloud Backup, gives you three copies across two media and one offsite. What you end up doing is up to you and how valuable the data on your iOS and iPadOS devices is, but I really wish Apple had a local Time Machine-like solution for iPhones and iPads that allowed wireless backups to a NAS.

For our Macs we're set: three copies (the Macs themselves, Time Machine, and rsync.net) across two media (SSDs in the Macs and magnetic hard drives in the Synology and, I assume, at rsync.net) with one offsite (rsync.net).

For our Synology, we keep less vital data on it, so we only have the one local copy and the copy on rsync.net. That's an intentional choice based on a considered risk analysis.

That's the important thing to keep in mind: the 3-2-1 rule is called a rule, but it's more of a guideline. More than three is better, for example. Sometimes less than three is okay. The real rule is to use the 3-2-1 approach as a starting point and make thoughtful choices about your own situation and your data security needs. Mostly this boils down to asking yourself questions about what data would be lost forever in different scenarios:

Having a good backup strategy means having good answers to those questions that you're comfortable with. The answers and comfort levels are different for everyone, and that's fine. The important part is giving this some thought and putting together an intentional plan to keep your data safe.

In other words, do the opposite of what Backblaze chose to do.