Recovering lost space on thinly provisioned sans when using LVM

So you think you’re saving money by using thin provisioning? You probably are. However if you use LVM and not say a LUN per VM on a hugely multitenanted platform you’re going to hit a world of pain in the future.

Or more accurately you’re probably going to have to spend a load of cash to keep the storage upgraded. This is inevitable, but maybe I can stall the upgrade time for you. I’m not going to give you a magic bullet here but I can give you some tips to keep the ‘free space’ in your system free.

The issue at hand is kind of hard to understand but under the hood it’s simple enough. You create LVs on your VGs. This is normal. Months pass, possibly years and you’ve got possibly 1000s of VMs and you’ve also probably culled a number over this time. The space used by the deleted VM disks is never recovered. Why? Well in a thinly provisioned system the SAN will write new blocks to empty space because it assumes your used blocks are still used. Oops. The SAN is not getting ‘discard’ commands from the file system on the hypervisor or from within the VM to tell it the blocks are recoverable when you delete a volume. If files are deleted in Windows 2008R2 or 2012 or most recently linux distros that support ‘Discard’ those files will be recovered. So any garbage collection system will get that space back.

The trick to recover the space is simple enough. it does require you to play around a small bit and to be careful.

On a box connected to your SAN and subscribed to all your LUNs do:

# pvscan

Output might look like:

# pvscan
PV /dev/mapper/randomthingy VG Vg-Name lvm2 [1.95 TiB / 619.95 GiB free]

So we create an LV to consume all the free space on this LUN.

# lvcreate -L 619GiB -n SpaceRecoveryVol Vg-Name

It’ll say LV Created Successfully or similar.

Then simply format it with ext4, now you might need to force the discard option using -E discard but this will depend largely on your OS of choice and the version of lvm utils installed.

e.g.

# mkfs.ext4 -E discard /dev/Vg-Name/SpaceRecoveryVol

Depending on the performance of your SAN you may or may not see it discarding the blocks.

When you’re done, remove the LV and all will be well again.

# lvremove Vg-Name/SpaceRecoveryVol

Leave a Reply

Your email address will not be published. Required fields are marked *