Out of balance - Out of space

I run a small self-hosted system with a bcachefs filesystem built from 3 drives (two HDD and one SSD). Globally the filesystem is set to two replicas (data_replicas=2) but I do have low-value (essentially cache) directories that are manually set to use only one replica.

My system started reporting that it was running out of space, despite it appearing to have space available across the drives. This happened because the filesystem was out of balance resulting in one drive being nearly full.

Had I started with two drives initially I believe that bcachefs would have balanced the single-replica files between the drives when writing them. In my case I added the second drive and set data_replicas=2 later so the single-replica files exist mostly on the first drive.

An initial view of my bcachefs fs usage -h looked like:

Free:             5.99T  5.02T  writable now
  at 2 replicas:  1.83T   368G
  at 3 replicas:   686G      0

Device label          Device     State   Size   Used  Use%
hdd.hdd1 (device 1):  sdb        rw     16.4T  14.8T   90%
hdd.hdd2 (device 3):  sda        rw     16.4T  9.90T   61%
ssd.ssd1 (device 0):  nvme0n1p3  rw      886G   758G   92%

As you can see hdd1 is much more full than hdd2 and only 1.83T / 368G are available at 2 replicas. I'm not sure if bcachefs would move files when this ran out. Even if it did, writes would likely slow down at that point.

Manually balancing low-value data

For this example I'll use /cache as the path for this data. I chose some subdirectories under /cache that, if moved to hdd2 would better balance the data. Some of my cache directories were previously copied with cp --reflink=always, those were specifically excluded from consideration as I don't know if that would have duplicated the data or not.

For each of the candidate paths I ran:

sudo bcachefs set-file-option --background-target=hdd.hdd2 /cache/PATH

This forces all files and directories under those paths to be moved to hdd2. This happens in the background and can take a while. Progress can be monitored by watching the target: value in:

bcachefs reconcile status /

Or by watching Leaving in:

bcachefs fs usage -h

For example while running I saw the reconcile status show:

                     data  metadata
target:             2.50T      449M     <- mid-flight

Cleanup

After the data was moved the --background-target should probably be removed from the directories. For each of the modified directories I ran:

sudo bcachefs set-file-option --background-target=- /cache/PATH

Results

Here are the (trimmed) results from before and after the move:

Before:

Free:             5.99T  5.02T
  at 2 replicas:  1.83T   368G
hdd.hdd1  90%
hdd.hdd2  61%

After:

Free:             6.02T  5.03T
  at 2 replicas:  6.02T  4.69T
hdd.hdd1  75%
hdd.hdd2  77%

The drives are much closer to balanced now, and a lot more space is available.