Recovering lost space on thinly provisioned sans when using LVM

So you think you’re saving money by using thin provisioning? You probably are. However if you use LVM and not say a LUN per VM on a hugely multitenanted platform you’re going to hit a world of pain in the future.

Or more accurately you’re probably going to have to spend a load of cash to keep the storage upgraded. This is inevitable, but maybe I can stall the upgrade time for you. I’m not going to give you a magic bullet here but I can give you some tips to keep the ‘free space’ in your system free.

The issue at hand is kind of hard to understand but under the hood it’s simple enough. You create LVs on your VGs. This is normal. Months pass, possibly years and you’ve got possibly 1000s of VMs and you’ve also probably culled a number over this time. The space used by the deleted VM disks is never recovered. Why? Well in a thinly provisioned system the SAN will write new blocks to empty space because it assumes your used blocks are still used. Oops. The SAN is not getting ‘discard’ commands from the file system on the hypervisor or from within the VM to tell it the blocks are recoverable when you delete a volume. If files are deleted in Windows 2008R2 or 2012 or most recently linux distros that support ‘Discard’ those files will be recovered. So any garbage collection system will get that space back.

The trick to recover the space is simple enough. it does require you to play around a small bit and to be careful.

On a box connected to your SAN and subscribed to all your LUNs do:

# pvscan

Output might look like:

# pvscan
PV /dev/mapper/randomthingy VG Vg-Name lvm2 [1.95 TiB / 619.95 GiB free]

So we create an LV to consume all the free space on this LUN.

# lvcreate -L 619GiB -n SpaceRecoveryVol Vg-Name

It’ll say LV Created Successfully or similar.

Then simply format it with ext4, now you might need to force the discard option using -E discard but this will depend largely on your OS of choice and the version of lvm utils installed.


# mkfs.ext4 -E discard /dev/Vg-Name/SpaceRecoveryVol

Depending on the performance of your SAN you may or may not see it discarding the blocks.

When you’re done, remove the LV and all will be well again.

# lvremove Vg-Name/SpaceRecoveryVol

CentOS 6 Xen VM kernel panic after kernel update / 6.1 update

It looks as if the Fedora guys made a decision to update the kernel device detection in RHEL 6.1 when it was being made. I found this out the hard way when I went to build a bunch of new VMs for a customer and get them up to date before deploying them.

You have two choices here:

1) update /etc/fstab and /boot/grub/menu.lst to have the new device name which _will_ be /dev/xvde1 or /dev/xvdeX where X is the device number. This is usually only the case if the Xen 3 config file has the device name as sda1 or sdaX.
2) if you are running Xen and you are using /dev/sdaX as the device name simply change this to be /dev/xvdaX and the Guest kernel won’t have any issues booting / doing kernel upgrades.

Qmail delivery problems, symtoms and a “hacky” fix

We’re a PA partner and have clustered Qmail with 4 servers + NFS storage. We’ve seen an issue where Qmail throws the following error:

failure: Sorry,_I_could_not_find_a_mail_exchanger_or_IP_address._(#5.4.4)/

This is a permanent error and means Qmail will not retry, the sender will receive an NDR (Non delivery reply / bounce back).

I’ve seen 3 causes so far that cause this error to appear with Qmail. Other MTAs like Postfix and Exim don’t have this issue.

1) CNAMEs as MX records and or there being no A record for the $domain.tld in the absence of an MX record.
2) smtp fixup is enabled on the cisco pix/asa firewall where the MX record resides.
3) The size of the returned DNS packet from the nameservers of the domain causes Qmail issues.

Our experience isn’t limited to these scenarios but they are the most common that we’ve seen.

As we manage quite a number of Cisco ASA firewalls and we have mixed MTAs behind them (qmail, exchange, postfix, exim etc) we always disable Ciscos smtp fixup. I’ve seen too many problems caused by it and it serves no use.

Parallels should take a leaf out of Postfix’s book as it actually detects this and performs a work around on the fly:

Jul 8 09:46:00 bk1-relay relay16/smtp[8773]: 21EDC39803E: enabling PIX workarounds: disable_esmtp delay_dotcrlf for hostname.domain.tld[]:25

How do you detect if smtp fixup is enabled?

Easy telnet to the MX record on port 25 and if the 220 banner returned looks like:

Escape character is ‘^]’.
220 ***********************

then it is enabled. I’d advise the end user in question to disable this setting on their firewall(s).

We’re using the smtproutes fix for domains that refuse to fix their end and it works fine. This is normally located in:

/usr/local/qmail/shared/control/ (typically in clustered configurations)

If the file “smtproutes” doesn’t exist simply create it and you add the route like this:

$domain.tld:relay.domain.tld where relay.domain.tld is your non qmail relay server.

Tweetdeck breaks when you dist upgrade from ubuntu 8.04 to 9.04

If like me you’ve been using LTS 8.04 for a while in work, at home or for play you may have been doing so for stability reasons.

This morning I got into the office and I was annoyed at some of the things that 8.04 was falling short on so I decided to dist-upgrade.

The main breakage post upgrade was tweetdeck, it kept saying that my ELS was corrupt and I’d need to re-enter my details. However then it would say that “A small number of people have problems with tweetdeck, we’re working on this”. I was stumped.

Easy fix was to remove Adobe Air, remove the .appdata/ folder in my users home directory and do:

# sudo dpkg -r tweetdeckfast.f9107117265db7542c1a806c8db837742ce14c21.1

This will remove tweetdeck from your system.

Now re-install Adobe Air and Tweetdeck and you should be back in business.

Blacknight launch new shared hosting system

Blacknight are going to launch their new shared hosting platform tomorrow August 7th 2008. We’ve been waiting to do this for close to 10 months now when we put development into overdrive.

What to expect?

  1. More of everything
  2. The basic package has 120 databases (30 Mysql4, 30 Mysql5, 30 Ms SQL 2005 and 30 Postgresql)
  3. You can use MS SQL 2005 on linux now, wow?
  4. The system is partially virtualised, we don’t need to install servers anymore. We simply tell our provisioning system to create a new VE (virtual environment) and apply a set of templates to it that match the shared hosting system.
  5. PHP 4 and PHP 5 are available on both linux and windows
  6. Complete DNS control, complete control over your domain and it’s name server settings
  7. Manage your own spam and antivirus settings
  8. Blacknight application vault with loads of apps you all have looked for 🙂
  9. Single login gets you access to everything. i.e. once your in, you can manage your plesk vps, your shared hosting account, register domains, manage your payments, add a new credit card etc.
  10. You get access to our new site builder that comes with over 250 templates and 100s of new features.
  11. You can see the system defined mime types, you can also setup your own, awesome eh?
  12. CGI sub section of the control panel tells you the paths to perl, python and sendmail and also what permissions to set on your files.
  13. Theres a log manager where you can login and view your logs via ftp
  14. Manage multiple ftp users without issue
  15. @Mail webmail system
  16. Advanced backup system that lets the user manage their backups.

It’s a complete suite of tools that anyone who has used any other control panel will fall in love with immediately.Where is this new system, well to get a look at it you’ll need to head over to and purchase a domain or a hosting package.

Our entry level plan has 10GB of disk space and 100GB of bandwidth, thats more than our existing plans put together almost!

last but not last a bit of information on the technology. We’ve deployed clustered Qmail + LDAP for e-mail. In front of this we’ve got load balancers that’ll send you to the least busy mail server when you use pop or smtp. We’re fully support pops, smtps and imaps. Behind the mail servers we’re using Solaris nfs servers attached directly to an EMC Clarion CX300 SAN unit providing us with the ultimate in enterprise storage and performance. Each linux web server will be a VPS. The benifit of this is ease of deployment, we can move VPS servers to less busy hardware nodes seemlessly and with minimal downtime (soon with no downtime!!). The back end provisioning system is comprised of two Parallels products, one is POA and the other is PBA, Parallels Operations Automation and Parallels Business Automation. Thats a split 40/60 for provisioning and billing. The billing end manages domain provisioning and a few other things.

Using Firefox3 in ubuntu and can’t get Sun JRE working??

Right this was a bit of a doozy.

  1. sudo apt-get install sun-java6-plugin
  2. cd .mozilla/plugins/
  3. ln -s /usr/lib/jvm/java-6-sun-
  4. restart firefox

It still doesn’t work? Ok. Type the following into your address bar about:plugins – if you see anything in relation to icedtea, remove it:

  1. apt-get remove icedtea-gcjwebplugin
  2. restart firefox

You should now have a functional JRE installed in Firefox3. This has been bugging me for a while as I’ve a few apps that use a JRE that I’ve not been able to use for a while.

icq broke in pidgin – patch available

I’ve been using ICQ for near enough 10 years. Yesterday it looks as if ICQ pushed out a network update that changed versions around. This broke ICQ for me on ubuntu hardy. I’ve been waiting since last night for a patched install but alas it hasn’t come and because time is money in business I needed ICQ working so I can liase with our Russian friends.

I’ve built a patched version which is available here:

Just download the 2 files and do:

sudo dpkg -i pidgin_2.4.1-1ubuntu3_i386.deb libpurple0_2.4.1-1ubuntu3_i386.deb

Restart pidgin and all will be well.

The long awaited product launch – also there’ll be more to come!

So we’ve been talking about stuff on the company blog, with friends, with customers etc for months. Today we’re doing a soft, low key, launch. It’s a new range of VPS plans based on Parallels Virtuozzo. It’s tacked onto our brand new billing and provisioning systems. It’s all part of the project we’ve been working on behind the scenes to automate more and more and also provide the next generation in hosting to our existing customer base and to the rest of Europe.

So here it is our new virtualisation product launch. It’s a cost effective, scalable, high performance platform that allows us to deliver services on Linux and Windows x64 Virtual Environments.

Some of the cool features include:

  • Live migration
  • CentOS 5.x and Windows 2003 x64 VPS
  • Instant activation
  • Virtuozzo Power Panel
  • Advanced blacknight PEM billing control panel
  • Ability to migrate a VPS to a dedicated server
  • Ability to migrate a dedicated server to a VPS

And I guess lots more. We’ve thrown quite a bit of hardware at this new system and I’ve personally spent 100s of hours testing, configuring, breaking and fixing it over the past few months.

Any feedback, error reports etc are welcome.

Office space in Dublin costs the earth or does it?

We’re setting out to hiring a few people that’ll have to be located beside each of the data centres we use in Dublin so we can have staff on-site primarily during business hours. Later we’ll expand on that.

One reason Blacknight never had offices in Dublin is due to cost. Now lots of business are based in Dublin, that’s fine. However for the price of an office in Dublin we can hire 2-3 competent engineering staff. This is fact.

So in investigating our options I came across the following co-working site. After having a look around I see that they have 2 locations now that you rent office space in. It’s really simple, you get a desk, chair, internet, phone and secretarial services all for a fixed relatively low monthly fee. There are other ancillary services that you can pay additionally. It really is a good idea.

This is most probably the route we’ll go down for our “Data Centre Technican” positions that we have going. Hopefully this particular company might get more office space in west Dublin that’ll be closer to our kit.

Entering the property market..

As the title suggest I’ve thought about it long and hard over the past while and I’ve decided to go and buy a house. I had been living in a rented house for 7 years. I moved out in Oct. 2007 and moved into a house not far away with Tara. Amy and Niall. While living with a few people is good and all, the house we’re in isn’t great and it’s difficult to keep clean.

I’ve gone and made an offer on the old house and this has been accepted. It’s pretty new territory for me as I’d no idea how many forms that would be required or about the amount of paper work I had to get.

So I’m getting a 92% mortgage from Permanent TSB via a broker. It’s going to work out around 250 euro a month more than it would cost to rent a house on my own. So it’s not too bad. Also I’m getting all the furniture and fixtures so it’s a pretty sweet deal.

The mortgage is a “discounted tracker mortgage” which basically means it’s at a lower rate for the first year 4.7% AER and then it’ll move to around 5.2% in year two. Lots of people don’t know what a tracker mortgage is so I’ll try and explain it in simple terms. Basically the ECB (european central bank) sets a recommended rate in Europe that banks are supposed to follow. Now in the last 30 odd years (before tracker mortgages) you could be waiting a year before you bank gives you any discount on your loan rate after a drop. Now a tracker mortgage is linked to the ECB rate and is usually 0.25% and 1% in the difference of the ECB rate. Mine is going to be 0.75%. That means that if the ECB sets a rate of say 5.125% my rate could be either 5.2% AER or 5.05% AER. It’s up to the bank to decide. The main benifit of this is that when the rate changes my bank automatically adjust it for me. They can’t decide “We’re not going to give people the benifit of the ECB dropping their rates”, they have to give me the break. It also means that they have to put it up when it goes up.