Recovering lost space on thinly provisioned sans when using LVM

So you think you’re saving money by using thin provisioning? You probably are. However if you use LVM and not say a LUN per VM on a hugely multitenanted platform you’re going to hit a world of pain in the future.

Or more accurately you’re probably going to have to spend a load of cash to keep the storage upgraded. This is inevitable, but maybe I can stall the upgrade time for you. I’m not going to give you a magic bullet here but I can give you some tips to keep the ‘free space’ in your system free.

The issue at hand is kind of hard to understand but under the hood it’s simple enough. You create LVs on your VGs. This is normal. Months pass, possibly years and you’ve got possibly 1000s of VMs and you’ve also probably culled a number over this time. The space used by the deleted VM disks is never recovered. Why? Well in a thinly provisioned system the SAN will write new blocks to empty space because it assumes your used blocks are still used. Oops. The SAN is not getting ‘discard’ commands from the file system on the hypervisor or from within the VM to tell it the blocks are recoverable when you delete a volume. If files are deleted in Windows 2008R2 or 2012 or most recently linux distros that support ‘Discard’ those files will be recovered. So any garbage collection system will get that space back.

The trick to recover the space is simple enough. it does require you to play around a small bit and to be careful.

On a box connected to your SAN and subscribed to all your LUNs do:

# pvscan

Output might look like:

# pvscan
PV /dev/mapper/randomthingy VG Vg-Name lvm2 [1.95 TiB / 619.95 GiB free]

So we create an LV to consume all the free space on this LUN.

# lvcreate -L 619GiB -n SpaceRecoveryVol Vg-Name

It’ll say LV Created Successfully or similar.

Then simply format it with ext4, now you might need to force the discard option using -E discard but this will depend largely on your OS of choice and the version of lvm utils installed.


# mkfs.ext4 -E discard /dev/Vg-Name/SpaceRecoveryVol

Depending on the performance of your SAN you may or may not see it discarding the blocks.

When you’re done, remove the LV and all will be well again.

# lvremove Vg-Name/SpaceRecoveryVol

Ember coffee mug

I’ve always been a big proponent of technology. In our every day lives we use technology to help us out and get by. In December 2015 I backed the Ember coffee mug on IndieGoGo

It has taken A LONG time for this product to become a reality. But it finally arrived, after much coaxing.

Essentially like all kickstarter or indiegogo projects the founders had set rather unrealistic goals and deadlines due to their lack of manufacturing, logistics and compliance knowledge. Each of these areas are extremely complex areas when you’re dealing with a product that you aim to ship internationally. I guess they just didn’t know. And to be honest this seems rather typical of everything off kickstarter.

Now the review.

The product arrived in very “applesque” minimal packaging, a big plus in my book, here are some photos:

ember mug packaging

the mug

Whats it like to live with?

Well here’s what it looks like in the back of a BMW 5 series, snazzy eh?

Ember mug, in the back of a BMW

The lid, the iphone app, it’s ergonomic design etc are all exquisite. i.e. it’s rather well made and this is quite surprising as much stuff I’ve backed has ended up being cheap tat. But this is quite the opposite.

The capacitive touch logo is nice, it has a number of functions which differ just by the mode it’s in and whether it is on or off. You can touch it 3 times and it’ll tell you it’s name. In this case I called it the ‘CAFFINATOR’, you can see battery life, current temperature, whether it’s cooling or heating for example and turning it on and off.

All in all it has been a very good device from a functionality and ease of use point of view. The lid is very well designed with a push-push open/close function. Your hot beverage comes out quite nicely and it does not leak. But take care to not overfill it as it will leak in this instance.

The last few pictures and a very quick video of it telling the world it’s name!

ember battery charging logo

The Caffinator:

The only downside so far has been the battery life. I thought I’d read somewhere that it could keep a beverage at temperature for 8 hours but this is rather unrealistic. At best you’ll get about 90 minutes out of a charge. This is not an issue if you use it in the same place all the time and you can pop it on it’s charger. But for me this is great device. Well worth the 129 USD price.