Starting the new year with :
No space left on device
I looked to the usual suspects (/tmp, /data, …) but nothing excessive … It was my sanoid setup, woops! A gazillion snapshots ! For some reason one of my containers changes around ~300 Gb a day, and guess who keeps 60 days of snapshots, on top of 6 months snapshots ? –> you guessed right.
Its annoying ZFS does not warn you, or that sanoid doesn’t take care, not to fill it up 100% . Not that’s its there task, but still, it was holidays for everyone guys ! So how to fix this ? Simply remove excessive snapshots (those can be deleted)
zfs destroy jbod1/users@autosnap_2016-12-09_14:00:01_hourly
Note that during a 100% full you can’t remove data due to Copy On Write (COW) so rm`ing a big file would result in :
# rm /that/big/file Can't remove /that/big/file. No space left on the device.
What you can do (if you can’t remove snapshots, or you just filled it up the old fashion way)
echo > /that/big/file
Credits for this great solution go to this guy.
Best wishes for 2017 🙂