The issue

So QNAP QTS suffered from a data corruption bug when using the NAS ( well, not only one, but that just tells you to migrate your NAS to a better OS if you can, all qnaps with HDMI output should be able to install a proper truenas/unraid/purelinux whatever that is better than their stuff ). Check that https://www.qnap.com/en-au/technical-advisory.

I got hit by the cache ‘data corruption bug’, one day finding all my pools R/O and after a reboot not even able to be mounted. Like a good citizen, called out on qnap support thinking given the extent of the problem would have a way to get back from it. Couple days later, a technician plan a teamviewer session ( thank you throw away vm, I don’t run windows here…. ) and spend a couple hours running fsck many times to the same result => i/o error while writing, then trying to restore snapshots that fails. After telling me that all data is lost and that they can;’t do anything, I decide to investigate further, after all it does not make sense that all my data is gone, it was there before the dreaded reboot.

Yes, I know, data should be backed up, but in this case, snapshots used to backup were also corrupted and problematic. yeah, no chocolate…. Proper copy next time, I know…

First, let’s try to understand how it is all setup

I will pass you the steps of looking into how QNAP setup the system. It is layers upon layers.

Basically, Qnap creates a mdadm raid with your disks, then put drbd with a bogus remote peer on top, then LVM thin provisionning on the drbd device, with lvm subvolumes representing your pools, and then a weird cache system on top of the LVM devices. so in my case: MDADM RAID5 /dev/md1 <=> DRBD /dev/drbd1 <=> LVM thin provision pool /dev/mapper/vg1-t* <=> LVM subvolume of the pool per ‘storage pool on the UI’ /dev/mapper/vg1-lvX <=> cache device ( without a fstab ) /dev/mapper/cachedevX <=> ext4

so yeah, lots of layers. While looking into dmesg, I find that LVM metadata has issues and the LVM pool is mounted read only, you can’t correct the errors if it’s r/o….

The relevant logs:

[   60.177484] device-mapper: thin: 253:4: switching pool to read-only mode
[   69.510031] EXT4-fs (dm-9): INFO: recovery required on readonly filesystem
[   76.867155] EXT4-fs (dm-16): INFO: recovery required on readonly filesystem

For those that wants to go further, /sbin/storage_util is a good tool to use, especially the –sys_startup_p2 and the –stop_storage options. Another tool of interest if you want to mount manually their stack ( mdadm assemble , drbd up, lvscan ) for the last cachedevX, you can use /sbin/SSDcacheutil to mount the /dev/mapper/vgX-lvY device to the /dev/mapper/cachedevX entry that Qnap uses to mount the device. This is not necessary though, since /dev/mapper/vgX-lvY contains your filesystem without that added layer ( and I recommend bypassing it during a recovery anyway, since more layers => more troubles ).

The actual recovery steps:

You do need an external hard drive large enough to hold your data.

First, let’s try to dd /dev/mapper/vg1-lv1 into an external hard drive, since the cachedev is not necessary, we are not gonna do any kind of caching while recovering. Well, it fails, because I/O error ( probably the famous corruption ? ). ddrescue is unavailable, but you can always static compile it from source on any os, and copy it to the NAS, so that gives you an image of your pool to work with should something go wrong. That being done, for the fun of it, a fsck on the image works perfectly fine and the image is mountable without issues. That tells me that the main problem I faced was on the LVM thin provisionning metadata layer ( I did not choose thin provisionning in their UI if you ask ). Issue there, you hard-drive needs to be as big as your pool, not just the dataset, since the image resulting will be as big as the pool data you used. In my case, not possible, my HD is 8Tb, my pools are 10.2Tb.

There’s another way we can recover all our files without going through the pain of imaging, and simply using an external hard drive large enough to hold the actual data you had on the pool ( in my case roughly 4Tb ). Enter debugfs !

debugfs -c /dev/mapper/vg1-lv1
rdump / /path/to/externalhd/folderonexternalhd

That launch debugfs in catastrophic mode, bypassing reading inodes, and the rdump debugfs command will dump all files on the / directory of the filesystem being debugged to the /path/to/externalhd/folderonexternalhd on the host filesystem.

Basically, all files debugfs can find on the filesystem will be copied to your external drive. That’s it, that’s the magic. After a couple days of copy, the data is backed-up, and ready to be checked out if there’s anything missing.

The future:

While I understand how QNAP decided to leverage each layer they put on the system ( mdadm raid, a classic linux, drbd to handle livestreaming the replica to another enclosure, lvm to allow easy resize and adding space, ext4 for no fuss filesystem ), this is clearly not optimized, and having lost confidence in their OS with the amount of security vulnerability and the absence of proper support should one of their issues hit you, it’s time to migrate away from their ‘proprietary’ OS.

I am currently evaluating truenas and a custom built solution to continue using my NAS enclosure and forget about the hot mess QTS is. Next NAS I build, I will be buying mini ITX components and do it myself.

Those dedicated soho NAS solutions are not worth the money anymore, and we can get way better specs, an opensource OS in a small form factor for less money. We can even open-air hardware in a closet without trouble.