Recovering lost space on thinly provisioned sans when using LVM

So you think you’re saving money by using thin provisioning? You probably are. However if you use LVM and not say a LUN per VM on a hugely multitenanted platform you’re going to hit a world of pain in the future.

Or more accurately you’re probably going to have to spend a load of cash to keep the storage upgraded. This is inevitable, but maybe I can stall the upgrade time for you. I’m not going to give you a magic bullet here but I can give you some tips to keep the ‘free space’ in your system free.

The issue at hand is kind of hard to understand but under the hood it’s simple enough. You create LVs on your VGs. This is normal. Months pass, possibly years and you’ve got possibly 1000s of VMs and you’ve also probably culled a number over this time. The space used by the deleted VM disks is never recovered. Why? Well in a thinly provisioned system the SAN will write new blocks to empty space because it assumes your used blocks are still used. Oops. The SAN is not getting ‘discard’ commands from the file system on the hypervisor or from within the VM to tell it the blocks are recoverable when you delete a volume. If files are deleted in Windows 2008R2 or 2012 or most recently linux distros that support ‘Discard’ those files will be recovered. So any garbage collection system will get that space back.

The trick to recover the space is simple enough. it does require you to play around a small bit and to be careful.

On a box connected to your SAN and subscribed to all your LUNs do:

# pvscan

Output might look like:

# pvscan
PV /dev/mapper/randomthingy VG Vg-Name lvm2 [1.95 TiB / 619.95 GiB free]

So we create an LV to consume all the free space on this LUN.

# lvcreate -L 619GiB -n SpaceRecoveryVol Vg-Name

It’ll say LV Created Successfully or similar.

Then simply format it with ext4, now you might need to force the discard option using -E discard but this will depend largely on your OS of choice and the version of lvm utils installed.


# mkfs.ext4 -E discard /dev/Vg-Name/SpaceRecoveryVol

Depending on the performance of your SAN you may or may not see it discarding the blocks.

When you’re done, remove the LV and all will be well again.

# lvremove Vg-Name/SpaceRecoveryVol

Ember coffee mug

I’ve always been a big proponent of technology. In our every day lives we use technology to help us out and get by. In December 2015 I backed the Ember coffee mug on IndieGoGo

It has taken A LONG time for this product to become a reality. But it finally arrived, after much coaxing.

Essentially like all kickstarter or indiegogo projects the founders had set rather unrealistic goals and deadlines due to their lack of manufacturing, logistics and compliance knowledge. Each of these areas are extremely complex areas when you’re dealing with a product that you aim to ship internationally. I guess they just didn’t know. And to be honest this seems rather typical of everything off kickstarter.

Now the review.

The product arrived in very “applesque” minimal packaging, a big plus in my book, here are some photos:

ember mug packaging

the mug

Whats it like to live with?

Well here’s what it looks like in the back of a BMW 5 series, snazzy eh?

Ember mug, in the back of a BMW

The lid, the iphone app, it’s ergonomic design etc are all exquisite. i.e. it’s rather well made and this is quite surprising as much stuff I’ve backed has ended up being cheap tat. But this is quite the opposite.

The capacitive touch logo is nice, it has a number of functions which differ just by the mode it’s in and whether it is on or off. You can touch it 3 times and it’ll tell you it’s name. In this case I called it the ‘CAFFINATOR’, you can see battery life, current temperature, whether it’s cooling or heating for example and turning it on and off.

All in all it has been a very good device from a functionality and ease of use point of view. The lid is very well designed with a push-push open/close function. Your hot beverage comes out quite nicely and it does not leak. But take care to not overfill it as it will leak in this instance.

The last few pictures and a very quick video of it telling the world it’s name!

ember battery charging logo

The Caffinator:

The only downside so far has been the battery life. I thought I’d read somewhere that it could keep a beverage at temperature for 8 hours but this is rather unrealistic. At best you’ll get about 90 minutes out of a charge. This is not an issue if you use it in the same place all the time and you can pop it on it’s charger. But for me this is great device. Well worth the 129 USD price.

CentOS 6 Xen VM kernel panic after kernel update / 6.1 update

It looks as if the Fedora guys made a decision to update the kernel device detection in RHEL 6.1 when it was being made. I found this out the hard way when I went to build a bunch of new VMs for a customer and get them up to date before deploying them.

You have two choices here:

1) update /etc/fstab and /boot/grub/menu.lst to have the new device name which _will_ be /dev/xvde1 or /dev/xvdeX where X is the device number. This is usually only the case if the Xen 3 config file has the device name as sda1 or sdaX.
2) if you are running Xen and you are using /dev/sdaX as the device name simply change this to be /dev/xvdaX and the Guest kernel won’t have any issues booting / doing kernel upgrades.

Qmail delivery problems, symtoms and a “hacky” fix

We’re a PA partner and have clustered Qmail with 4 servers + NFS storage. We’ve seen an issue where Qmail throws the following error:

failure: Sorry,_I_could_not_find_a_mail_exchanger_or_IP_address._(#5.4.4)/

This is a permanent error and means Qmail will not retry, the sender will receive an NDR (Non delivery reply / bounce back).

I’ve seen 3 causes so far that cause this error to appear with Qmail. Other MTAs like Postfix and Exim don’t have this issue.

1) CNAMEs as MX records and or there being no A record for the $domain.tld in the absence of an MX record.
2) smtp fixup is enabled on the cisco pix/asa firewall where the MX record resides.
3) The size of the returned DNS packet from the nameservers of the domain causes Qmail issues.

Our experience isn’t limited to these scenarios but they are the most common that we’ve seen.

As we manage quite a number of Cisco ASA firewalls and we have mixed MTAs behind them (qmail, exchange, postfix, exim etc) we always disable Ciscos smtp fixup. I’ve seen too many problems caused by it and it serves no use.

Parallels should take a leaf out of Postfix’s book as it actually detects this and performs a work around on the fly:

Jul 8 09:46:00 bk1-relay relay16/smtp[8773]: 21EDC39803E: enabling PIX workarounds: disable_esmtp delay_dotcrlf for hostname.domain.tld[]:25

How do you detect if smtp fixup is enabled?

Easy telnet to the MX record on port 25 and if the 220 banner returned looks like:

Escape character is ‘^]’.
220 ***********************

then it is enabled. I’d advise the end user in question to disable this setting on their firewall(s).

We’re using the smtproutes fix for domains that refuse to fix their end and it works fine. This is normally located in:

/usr/local/qmail/shared/control/ (typically in clustered configurations)

If the file “smtproutes” doesn’t exist simply create it and you add the route like this:

$domain.tld:relay.domain.tld where relay.domain.tld is your non qmail relay server.

Nokia N95 is the winner for best phone since sliced bread!

It’s official. Well unofficially official. Myself and Michele went to Vodafone and got us some Nokia N95’s. I’d been fairly happy with my N91, but me having thick thumbs… I found the keypad a bit of a chore to use. Also the N91 was sooo fricken heavy, mostly because of its 4GB microdrive but the N95 weighs in at 120gramms.

Good things about the N95:

a) Web browser rocks, it’s a version of Safari apparently and it renders our site, crazy DHTML menus and all, very well. I used it a fair bit over the weekend and I like it. Most sites look as one would expect.

b) 5Mega Pixel camera, wow. That’s about all I can say. This small little device that weighs a lot less and is physically smaller than the N91 has one of the best cameras in the phone market. Not only does it have a good resolution, it is also designed so that it can be used as a normal camera. The up/down volume bottons are for zooming, then there’s a gallery button and the all important photo snap button and you can do this while the phone is on it’s side which allows you to make use of the large 2.6″ (240 x 320) screen.

c) Other misc features: Wifi which works very well, Bluetooth which seems quite compatibable with tonnes of devices including heaps of car kits etc, infra red which now is mostly not used, but hey it has it, 3G support which works very nicely, MMS and txt message support etc.
d) Digital music player – supports MP3/AAC/AAC+/eAAC+/WMA/M4A with playlists and equalizer.

e) It’s mighty pretty, I like it a lot.

f) It takes those new mini-SD cards that are super small

g) GPS, which works well. Friend mentioned getting route 66 software for it which I’m going to look into this evening.


a) Battery life

b) Battery life

c) Battery life

Yes, overall this is a fantastic phone. The battery life is pretty poor. Just incase anyone asks, I’ve 3G off most of the time along with bluetooth, wifi scanning etc. About 1 day of use is all I’m getting. This makes me not want it. But all the other cool features save it for me.

It’s a deadly phone, get one.


So I noticed on an IRC channel today a bit of buzz around PHP4 going EOL. Bye bye php4, with that announcement in mind, we’re going to discontinue PHP4 support at the end of the year also. In the mean time we’ll simply encourage our customers to make sure their applications work in PHP5. At this stage in the game there should be little or no applications that are still maintained that won’t work in PHP5.

The news was also posted on

3D secure teething issues

We’ve spent about the last month or so writing code to help prevent against fraudulent orders. This has meant a huge re-write of parts of Modernbill previously thought to be “demented and wrong”. Along the way we’ve tested it pretty much completely. Today we put the code (along with tonnes of other cool stuff) live and we sat back and watched payments coming in.

All works perfect, as expected. However I noted certain Visa card payments going through detailing that the user was not enrolled. This is normal enough. However I have a personal BOI visa card that I use online a fair bit to buy various bits ‘n’ bobs. I went through our order process and came to paying and I was not presented with anything asking about 3D secure. I thought this a bit odd, as my work Visa card is enrolled and I get presented with the “Verified by Visa” where it asks for my password.

So to find out what was going on I ring BOI. I explain to the nice lady what I’m talking about and she replies “I haven’t heard of that, it isn’t an option on our cards.”. I go into a bit more detail and she goes off to ask a supervisor. She comes back saying “He hasn’t heard of 3D secure either, but we googled for it and printed off all the details and have sent them onto our security department.” The mind boggles. So BOI have not heard about or implemented 3D secure?? WTF??? It really is apparent to me now that AIB are years ahead of the other banks in Ireland. I’m going to keep a close eye on other Banks cards over the coming few days and see which banks have actually signed up for 3D secure and which haven’t. I’ll make a list of banks to avoid getting credit cards from then.

Simple annoying e-mail related problems

One thing that always causes hassle for us as an e-mail services provider is the use of double extension file names attached to e-mails. e.g. annualreport.doc.pdf – this looks harmless, however in file naming terms it’s wrong. A filename should be comprised of a meaningfull name and it’s extension. I don’t know if this is a user issue or an application issue, but I suspect a bit of both.

If you insist on using double file extensions because your application makes them that way, then change the application. Or even better rename the file before sending. You have to save it anyway, correct? Well then, save the file with the proper file extension, then e-mail that file to the intended recipient.

The main reason for this slightly ranty post is that our mail scanning software catches all double extension attachments and treats them as an attempt to send a virus. Now it doesn’t delete them, or do anything funky with them, it sends the original message with the attachment replaced with a warning text file letting us know what server and the location to the e-mail in the quarantine so we can release it for you. It does this as a lot of traditional viruses used to send themselves from an infected users pc to everyone in their address book as an attachment with a name of something like or .doc.exe and it would infect the end users pc.

I’ve considered removing this restriction a few times. But the pedant in me doesn’t let me. Please just use common sense and use proper file extension naming, for the sake of everyone.

Promise Raid cards?

Yes the saga continues, only 4 years later. A few years ago we dabbled with promise cards. A) they were cost effective, B) they apparently had a good feature set and C) Our vendor at the time recommended them.  Long story short, they sucked. We used the TX2300 I think then. Over the past while we’ve been using another vendor that is local to us instead of just using Dell and HP so we can diversify. They were raving about the new Ex series of cards so we got an Ex8350 card off them and put it into a quad core server with plenty of ram etc.

Initial findings:

Redhat EL 4U4 supports the card. Yaay. However the kernel on the install cd didn’t. Promise had no driver for U4. We contact redhat, follow instructions, nothing. Redhat come back at 09:06 the following morning with a new driver disk that is untested, but hey it worked! Horray. Installed RHEL 4, upgraded to latest redhat kernel and all looked great.

The card performed quite well during initial testing. Compile times were up on other systems and uncompressing tar.gz files etc was very quick. I wasn’t sure if this was the quad core or the raid card/disk performance but I gave the raid card the benefit of the doubt.

Things started to get a bit hairy after 2 weeks or so. The server was our latest shared hosting box, which is a beast in comparison to some of the older machines with 4 cores on the cpu. It started showing high load averages even when it wasn’t particularly busy. A closer look and see a huge amount of IO wait. I think nothing off it but keep an eye on the server. Last friday it had a serious brain fart, a simple untar job of a 200mb file brought the load to over 200 and poor Niall had to look after it.

Another week has passed and it has another brain fart, this time input/output errors for almost all binaries on the file system. Now this brings back vivid memories of our first ever shared server from years ago with the card in it that I mention above. So Promise cards suck. Redhat support is second to none. Infact Redhat are bloody brilliant, never has a vendor been so usefull or quick to help me out on an issue before.

Blacknight support ASP.NET AJAX extensions

We’re proud to announce that we now support the new ASP.NET Ajax extensions. We’ve had Robert from Microsoft testing it out to make sure it all looks well. And it does! He had a lot of fun trying it out and playing with a hosted environment which I don’t think he had seen or used before.

Anyway all new Windows sign-ups will have this support. I’ll be rolling it out on the other servers to cater for existing customers in the coming week. Just after I do a load of patching of php 4 on both our linux and windows platforms 🙁

We have a test site setup where Robert put up his test application that utilises this new extension to .NET and it looks pretty damn cool.