full ZFS filesystem : No space left on device

Starting the new year with :

No space left on device

I looked to the usual suspects (/tmp, /data, …)  but nothing excessive … It was my sanoid setup, woops! A gazillion snapshots ! For some reason one of my containers changes around ~300 Gb a day, and guess who keeps 60 days of snapshots, on top of 6 months snapshots ? –> you guessed right.

Its annoying ZFS does not warn you, or that sanoid doesn’t take care, not to fill it up 100% . Not that’s its there task, but still, it was holidays for everyone guys ! So how to fix this ? Simply remove excessive snapshots (those can be deleted)

zfs destroy jbod1/[email protected]_2016-12-09_14:00:01_hourly

Note that during a 100% full you can’t remove data due to Copy On Write (COW) so rm`ing a big file would result in :

# rm /that/big/file
Can't remove /that/big/file. No space left on the device.

What you can do (if you can’t remove snapshots, or you just filled it up the old fashion way)

echo > /that/big/file

Credits for this great solution go to this guy.

Best wishes for 2017 🙂