Recovering lost space on thinly provisioned sans when using LVM

So you think you’re saving money by using thin provisioning? You probably are. However if you use LVM and not say a LUN per VM on a hugely multitenanted platform you’re going to hit a world of pain in the future.

Or more accurately you’re probably going to have to spend a load of cash to keep the storage upgraded. This is inevitable, but maybe I can stall the upgrade time for you. I’m not going to give you a magic bullet here but I can give you some tips to keep the ‘free space’ in your system free.

The issue at hand is kind of hard to understand but under the hood it’s simple enough. You create LVs on your VGs. This is normal. Months pass, possibly years and you’ve got possibly 1000s of VMs and you’ve also probably culled a number over this time. The space used by the deleted VM disks is never recovered. Why? Well in a thinly provisioned system the SAN will write new blocks to empty space because it assumes your used blocks are still used. Oops. The SAN is not getting ‘discard’ commands from the file system on the hypervisor or from within the VM to tell it the blocks are recoverable when you delete a volume. If files are deleted in Windows 2008R2 or 2012 or most recently linux distros that support ‘Discard’ those files will be recovered. So any garbage collection system will get that space back.

The trick to recover the space is simple enough. it does require you to play around a small bit and to be careful.

On a box connected to your SAN and subscribed to all your LUNs do:

# pvscan

Output might look like:

# pvscan
PV /dev/mapper/randomthingy VG Vg-Name lvm2 [1.95 TiB / 619.95 GiB free]

So we create an LV to consume all the free space on this LUN.

# lvcreate -L 619GiB -n SpaceRecoveryVol Vg-Name

It’ll say LV Created Successfully or similar.

Then simply format it with ext4, now you might need to force the discard option using -E discard but this will depend largely on your OS of choice and the version of lvm utils installed.


# mkfs.ext4 -E discard /dev/Vg-Name/SpaceRecoveryVol

Depending on the performance of your SAN you may or may not see it discarding the blocks.

When you’re done, remove the LV and all will be well again.

# lvremove Vg-Name/SpaceRecoveryVol

Ember coffee mug

I’ve always been a big proponent of technology. In our every day lives we use technology to help us out and get by. In December 2015 I backed the Ember coffee mug on IndieGoGo

It has taken A LONG time for this product to become a reality. But it finally arrived, after much coaxing.

Essentially like all kickstarter or indiegogo projects the founders had set rather unrealistic goals and deadlines due to their lack of manufacturing, logistics and compliance knowledge. Each of these areas are extremely complex areas when you’re dealing with a product that you aim to ship internationally. I guess they just didn’t know. And to be honest this seems rather typical of everything off kickstarter.

Now the review.

The product arrived in very “applesque” minimal packaging, a big plus in my book, here are some photos:

ember mug packaging

the mug

Whats it like to live with?

Well here’s what it looks like in the back of a BMW 5 series, snazzy eh?

Ember mug, in the back of a BMW

The lid, the iphone app, it’s ergonomic design etc are all exquisite. i.e. it’s rather well made and this is quite surprising as much stuff I’ve backed has ended up being cheap tat. But this is quite the opposite.

The capacitive touch logo is nice, it has a number of functions which differ just by the mode it’s in and whether it is on or off. You can touch it 3 times and it’ll tell you it’s name. In this case I called it the ‘CAFFINATOR’, you can see battery life, current temperature, whether it’s cooling or heating for example and turning it on and off.

All in all it has been a very good device from a functionality and ease of use point of view. The lid is very well designed with a push-push open/close function. Your hot beverage comes out quite nicely and it does not leak. But take care to not overfill it as it will leak in this instance.

The last few pictures and a very quick video of it telling the world it’s name!

ember battery charging logo

The Caffinator:

The only downside so far has been the battery life. I thought I’d read somewhere that it could keep a beverage at temperature for 8 hours but this is rather unrealistic. At best you’ll get about 90 minutes out of a charge. This is not an issue if you use it in the same place all the time and you can pop it on it’s charger. But for me this is great device. Well worth the 129 USD price.


I’ve been inactive for some time. Mainly due to life, the universe and everything. I now have 2 kids, Ethan who is almost 3 and Alex who is just 7 weeks old. It’s been fun for the last few years 🙂

CentOS 6 Xen VM kernel panic after kernel update / 6.1 update

It looks as if the Fedora guys made a decision to update the kernel device detection in RHEL 6.1 when it was being made. I found this out the hard way when I went to build a bunch of new VMs for a customer and get them up to date before deploying them.

You have two choices here:

1) update /etc/fstab and /boot/grub/menu.lst to have the new device name which _will_ be /dev/xvde1 or /dev/xvdeX where X is the device number. This is usually only the case if the Xen 3 config file has the device name as sda1 or sdaX.
2) if you are running Xen and you are using /dev/sdaX as the device name simply change this to be /dev/xvdaX and the Guest kernel won’t have any issues booting / doing kernel upgrades.

Qmail delivery problems, symtoms and a “hacky” fix

We’re a PA partner and have clustered Qmail with 4 servers + NFS storage. We’ve seen an issue where Qmail throws the following error:

failure: Sorry,_I_could_not_find_a_mail_exchanger_or_IP_address._(#5.4.4)/

This is a permanent error and means Qmail will not retry, the sender will receive an NDR (Non delivery reply / bounce back).

I’ve seen 3 causes so far that cause this error to appear with Qmail. Other MTAs like Postfix and Exim don’t have this issue.

1) CNAMEs as MX records and or there being no A record for the $domain.tld in the absence of an MX record.
2) smtp fixup is enabled on the cisco pix/asa firewall where the MX record resides.
3) The size of the returned DNS packet from the nameservers of the domain causes Qmail issues.

Our experience isn’t limited to these scenarios but they are the most common that we’ve seen.

As we manage quite a number of Cisco ASA firewalls and we have mixed MTAs behind them (qmail, exchange, postfix, exim etc) we always disable Ciscos smtp fixup. I’ve seen too many problems caused by it and it serves no use.

Parallels should take a leaf out of Postfix’s book as it actually detects this and performs a work around on the fly:

Jul 8 09:46:00 bk1-relay relay16/smtp[8773]: 21EDC39803E: enabling PIX workarounds: disable_esmtp delay_dotcrlf for hostname.domain.tld[]:25

How do you detect if smtp fixup is enabled?

Easy telnet to the MX record on port 25 and if the 220 banner returned looks like:

Escape character is ‘^]’.
220 ***********************

then it is enabled. I’d advise the end user in question to disable this setting on their firewall(s).

We’re using the smtproutes fix for domains that refuse to fix their end and it works fine. This is normally located in:

/usr/local/qmail/shared/control/ (typically in clustered configurations)

If the file “smtproutes” doesn’t exist simply create it and you add the route like this:

$domain.tld:relay.domain.tld where relay.domain.tld is your non qmail relay server.

Tweetdeck breaks when you dist upgrade from ubuntu 8.04 to 9.04

If like me you’ve been using LTS 8.04 for a while in work, at home or for play you may have been doing so for stability reasons.

This morning I got into the office and I was annoyed at some of the things that 8.04 was falling short on so I decided to dist-upgrade.

The main breakage post upgrade was tweetdeck, it kept saying that my ELS was corrupt and I’d need to re-enter my details. However then it would say that “A small number of people have problems with tweetdeck, we’re working on this”. I was stumped.

Easy fix was to remove Adobe Air, remove the .appdata/ folder in my users home directory and do:

# sudo dpkg -r tweetdeckfast.f9107117265db7542c1a806c8db837742ce14c21.1

This will remove tweetdeck from your system.

Now re-install Adobe Air and Tweetdeck and you should be back in business.

My version of chicken marsala

The background behind this post is important I guess. I went on holidays with Tara and her family earlier this year (for her sisters wedding) to the USA. We booked various hotels in different areas for the 15 days or so that we were going for. Tara is my girlfriend whom I’m going out with now for over a year.

While in the US we visited some of her relatives and we spent quite a bit of time in her aunt Tereasa’ house. Now Taz had given her fair warning that I’m fairly picky when it comes to food (those of you that know me well, know how bad I am!!) so she was all set for meat and chips. However while there she made this absolutely stunning dish called “Chicken Marsala” and I had seriously never tasted chicken as good before.

So when we came back I kept thinking about this great meal I had while I was away (I had many other great meals in her house, but this one sticks out the most) so I had to go about trying to make it.

First off, my only issue was really finding Marsala Wine. Thankfully O’Briens in Carlow had it stocked, a bottle cost me 12.99 which to me was a bit steep. But I got it and off I went.

I’ve tried it 4 times now and I hadn’t been able to successfully “make it the right way”, that was until today. I’ve got the correct measures down (be they a bit unaccurate for sharing) and I had the most beautiful chicken marsala again tonight. Tara concurs that it was excellent (not just said so, she ate the lot!).

The following receipe was what makes an excellent chicken Marsala (for 2 or 3, depending on what you serve with it)

  1. 3-4 chicken breasts, thinly sliced.
  2. real butter (10gramms) and olive oil (smallest amount will do)
  3. flour
  4. mushrooms
  5. chicken stock cubes, knorr seem good
  6. half a cup of Marsala Wine *
  7. 1 cup of chicken stock *
  8. A wide deep frying pan (*) or skillet

The process:

  • Put 1.5 chicken stock cubes into a cup, fill with boiling water and stir, this is now your chicken stock.
  • Cover a baking tray with flour, add salt and pepper and then cover all the pieces of chicken in flour. Heat the pan, after putting some of the butter and olive oil in, to the point just before the butter starts smoking (this is crucial) and then put the chicken on to it for approx 2-3 mins either side
  • Take the chicken off and take the pan off of the heat for a minute. Put the chicken somewhere to keep it warm and where you can gather the juices from for later.
  • Put the pan back on the heat and place the mushrooms onto it (obviously you can chop them what ever way you prefer) and cook them for 5-6 minutes or until their own moisture/juice starts to show.
  • Immediately after the mushrooms own juices starting to show add the chicken stock. You’ll now cook the mushrooms for around 5 minutes on a medium heat in the stock.
  • After 5 or so minutes have elapsed turn the heat down and now put the chicken back in, let the whole lot simmer for around 2-3 minutes and then add in the half a cup of marsala wine.
  • Let this now simmer for 10 minutes on a low heat

* the measurements for the stock and wine I’ve used are because my frying pan is wide but the sides are not very high and it can’t support a lot of sauce/juices. Ideally use a pan with high sides and double up on these measurements so that you end up with quite a bit of the juices to place over your veg, potatoes or rice.

You can now serve the dinner to 2-3 people.

Enjoy this, I’d like to welcome comments on this post if anyone tries it out 🙂

Blacknight launch new shared hosting system

Blacknight are going to launch their new shared hosting platform tomorrow August 7th 2008. We’ve been waiting to do this for close to 10 months now when we put development into overdrive.

What to expect?

  1. More of everything
  2. The basic package has 120 databases (30 Mysql4, 30 Mysql5, 30 Ms SQL 2005 and 30 Postgresql)
  3. You can use MS SQL 2005 on linux now, wow?
  4. The system is partially virtualised, we don’t need to install servers anymore. We simply tell our provisioning system to create a new VE (virtual environment) and apply a set of templates to it that match the shared hosting system.
  5. PHP 4 and PHP 5 are available on both linux and windows
  6. Complete DNS control, complete control over your domain and it’s name server settings
  7. Manage your own spam and antivirus settings
  8. Blacknight application vault with loads of apps you all have looked for 🙂
  9. Single login gets you access to everything. i.e. once your in, you can manage your plesk vps, your shared hosting account, register domains, manage your payments, add a new credit card etc.
  10. You get access to our new site builder that comes with over 250 templates and 100s of new features.
  11. You can see the system defined mime types, you can also setup your own, awesome eh?
  12. CGI sub section of the control panel tells you the paths to perl, python and sendmail and also what permissions to set on your files.
  13. Theres a log manager where you can login and view your logs via ftp
  14. Manage multiple ftp users without issue
  15. @Mail webmail system
  16. Advanced backup system that lets the user manage their backups.

It’s a complete suite of tools that anyone who has used any other control panel will fall in love with immediately.Where is this new system, well to get a look at it you’ll need to head over to and purchase a domain or a hosting package.

Our entry level plan has 10GB of disk space and 100GB of bandwidth, thats more than our existing plans put together almost!

last but not last a bit of information on the technology. We’ve deployed clustered Qmail + LDAP for e-mail. In front of this we’ve got load balancers that’ll send you to the least busy mail server when you use pop or smtp. We’re fully support pops, smtps and imaps. Behind the mail servers we’re using Solaris nfs servers attached directly to an EMC Clarion CX300 SAN unit providing us with the ultimate in enterprise storage and performance. Each linux web server will be a VPS. The benifit of this is ease of deployment, we can move VPS servers to less busy hardware nodes seemlessly and with minimal downtime (soon with no downtime!!). The back end provisioning system is comprised of two Parallels products, one is POA and the other is PBA, Parallels Operations Automation and Parallels Business Automation. Thats a split 40/60 for provisioning and billing. The billing end manages domain provisioning and a few other things.

Using Firefox3 in ubuntu and can’t get Sun JRE working??

Right this was a bit of a doozy.

  1. sudo apt-get install sun-java6-plugin
  2. cd .mozilla/plugins/
  3. ln -s /usr/lib/jvm/java-6-sun-
  4. restart firefox

It still doesn’t work? Ok. Type the following into your address bar about:plugins – if you see anything in relation to icedtea, remove it:

  1. apt-get remove icedtea-gcjwebplugin
  2. restart firefox

You should now have a functional JRE installed in Firefox3. This has been bugging me for a while as I’ve a few apps that use a JRE that I’ve not been able to use for a while.

icq broke in pidgin – patch available

I’ve been using ICQ for near enough 10 years. Yesterday it looks as if ICQ pushed out a network update that changed versions around. This broke ICQ for me on ubuntu hardy. I’ve been waiting since last night for a patched install but alas it hasn’t come and because time is money in business I needed ICQ working so I can liase with our Russian friends.

I’ve built a patched version which is available here:

Just download the 2 files and do:

sudo dpkg -i pidgin_2.4.1-1ubuntu3_i386.deb libpurple0_2.4.1-1ubuntu3_i386.deb

Restart pidgin and all will be well.