ZFSonLinux (ZOL) is a great project that creates a Linux kernel port of the ZFS filesystem. However when the kernel updates, it always makes problems with ZFS kernel module 🙁 I have not found a stable solution, only a very dirty “windows alike method”. I will share it as a future reference for my colleagues and -primary- myself.
Failed to load ZFS module stack.
In essence, a new kernel is installed, it will “weak” link the ZFS modules, for some reason ZFS doesn’t like that and gets partially updated. Both the new and the old kernel will not be able to load the ZFS data, for people who are now in full panic mode (like myself every-time this happens) : Your data is not lost.
# find the version of spl and zfs dkms status # both remove them dkms remove -m zfs -v 0.6.3 --all dkms remove -m spl -v 0.6.3 --all # install the headers for the new kernel # ubuntu/debian apt-get install linux-headers-$(uname -r) # centos yum install kernel-headers # reinstall zfs yum reinstall zfs #add & build them again dkms add -m spl -v 0.6.3 dkms add -m zfs -v 0.6.3 dkms install -m spl -v 0.6.3 dkms install -m zfs -v 0.6.3 # try loading in : modprobe zfs # if you can load zfs again now, you can skip this step # I can't so I had to reboot my machine. (I know its crazy) # find the data zpool import # my poolname is tank zpool import tank
And that is how I saved myself! (for now.)
Some notes :
- Reinstalling doesn’t always work, sometimes you just need to remove zfs
yum remove zfsafter that its a good idea to clean up dkms manually. The command below is floating around on the web; It comes down to removing the modules from
/lib/modules/$(kernel_version)/extra/I removed them from all the kernels, as I only wanted to use the newest kernel anyway.
find /lib/modules/$(uname -r)/extra -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f find /lib/modules/$(uname -r)/weak-updates -name "splat.ko" -or -name "zcommon.ko" -or -name "zpios.ko" -or -name "spl.ko" -or -name "zavl.ko" -or -name "zfs.ko" -or -name "znvpair.ko" -or -name "zunicode.ko" | xargs rm -f
- Update 11/01/2016
- The same problem happened today, a Centos 6.X server crashed due to a raidcontoller blocking. This forced a reboot, for some reason this booted in the not-latest-installed kernel, so zfs was installed in a newer kernel and weak linked to the “older” kernel. Rebooting in this case is the thing you should try first.
- Update 14/12/2016
- Its nice to see more people facing the same issues, here is a similar solution to solve it on CentOS 7