CentOS 6 Xen VM kernel panic after kernel update / 6.1 update

It looks as if the Fedora guys made a decision to update the kernel device detection in RHEL 6.1 when it was being made. I found this out the hard way when I went to build a bunch of new VMs for a customer and get them up to date before deploying them.

You have two choices here:

1) update /etc/fstab and /boot/grub/menu.lst to have the new device name which _will_ be /dev/xvde1 or /dev/xvdeX where X is the device number. This is usually only the case if the Xen 3 config file has the device name as sda1 or sdaX.
2) if you are running Xen and you are using /dev/sdaX as the device name simply change this to be /dev/xvdaX and the Guest kernel won’t have any issues booting / doing kernel upgrades.

Qmail delivery problems, symtoms and a “hacky” fix

We’re a PA partner and have clustered Qmail with 4 servers + NFS storage. We’ve seen an issue where Qmail throws the following error:

failure: Sorry,_I_could_not_find_a_mail_exchanger_or_IP_address._(#5.4.4)/

This is a permanent error and means Qmail will not retry, the sender will receive an NDR (Non delivery reply / bounce back).

I’ve seen 3 causes so far that cause this error to appear with Qmail. Other MTAs like Postfix and Exim don’t have this issue.

1) CNAMEs as MX records and or there being no A record for the $domain.tld in the absence of an MX record.
2) smtp fixup is enabled on the cisco pix/asa firewall where the MX record resides.
3) The size of the returned DNS packet from the nameservers of the domain causes Qmail issues.

Our experience isn’t limited to these scenarios but they are the most common that we’ve seen.

As we manage quite a number of Cisco ASA firewalls and we have mixed MTAs behind them (qmail, exchange, postfix, exim etc) we always disable Ciscos smtp fixup. I’ve seen too many problems caused by it and it serves no use.

Parallels should take a leaf out of Postfix’s book as it actually detects this and performs a work around on the fly:

Jul 8 09:46:00 bk1-relay relay16/smtp[8773]: 21EDC39803E: enabling PIX workarounds: disable_esmtp delay_dotcrlf for hostname.domain.tld[74.xxx.xxx.xxx]:25

How do you detect if smtp fixup is enabled?

Easy telnet to the MX record on port 25 and if the 220 banner returned looks like:

Escape character is ‘^]’.
220 ***********************

then it is enabled. I’d advise the end user in question to disable this setting on their firewall(s).

We’re using the smtproutes fix for domains that refuse to fix their end and it works fine. This is normally located in:

/usr/local/qmail/shared/control/ (typically in clustered configurations)

If the file “smtproutes” doesn’t exist simply create it and you add the route like this:

$domain.tld:relay.domain.tld where relay.domain.tld is your non qmail relay server.

Tweetdeck breaks when you dist upgrade from ubuntu 8.04 to 9.04

If like me you’ve been using LTS 8.04 for a while in work, at home or for play you may have been doing so for stability reasons.

This morning I got into the office and I was annoyed at some of the things that 8.04 was falling short on so I decided to dist-upgrade.

The main breakage post upgrade was tweetdeck, it kept saying that my ELS was corrupt and I’d need to re-enter my details. However then it would say that “A small number of people have problems with tweetdeck, we’re working on this”. I was stumped.

Easy fix was to remove Adobe Air, remove the .appdata/ folder in my users home directory and do:

# sudo dpkg -r tweetdeckfast.f9107117265db7542c1a806c8db837742ce14c21.1

This will remove tweetdeck from your system.

Now re-install Adobe Air and Tweetdeck and you should be back in business.

My version of chicken marsala

The background behind this post is important I guess. I went on holidays with Tara and her family earlier this year (for her sisters wedding) to the USA. We booked various hotels in different areas for the 15 days or so that we were going for. Tara is my girlfriend whom I’m going out with now for over a year.

While in the US we visited some of her relatives and we spent quite a bit of time in her aunt Tereasa’ house. Now Taz had given her fair warning that I’m fairly picky when it comes to food (those of you that know me well, know how bad I am!!) so she was all set for meat and chips. However while there she made this absolutely stunning dish called “Chicken Marsala” and I had seriously never tasted chicken as good before.

So when we came back I kept thinking about this great meal I had while I was away (I had many other great meals in her house, but this one sticks out the most) so I had to go about trying to make it.

First off, my only issue was really finding Marsala Wine. Thankfully O’Briens in Carlow had it stocked, a bottle cost me 12.99 which to me was a bit steep. But I got it and off I went.

I’ve tried it 4 times now and I hadn’t been able to successfully “make it the right way”, that was until today. I’ve got the correct measures down (be they a bit unaccurate for sharing) and I had the most beautiful chicken marsala again tonight. Tara concurs that it was excellent (not just said so, she ate the lot!).

The following receipe was what makes an excellent chicken Marsala (for 2 or 3, depending on what you serve with it)

  1. 3-4 chicken breasts, thinly sliced.
  2. real butter (10gramms) and olive oil (smallest amount will do)
  3. flour
  4. mushrooms
  5. chicken stock cubes, knorr seem good
  6. half a cup of Marsala Wine *
  7. 1 cup of chicken stock *
  8. A wide deep frying pan (*) or skillet

The process:

  • Put 1.5 chicken stock cubes into a cup, fill with boiling water and stir, this is now your chicken stock.
  • Cover a baking tray with flour, add salt and pepper and then cover all the pieces of chicken in flour. Heat the pan, after putting some of the butter and olive oil in, to the point just before the butter starts smoking (this is crucial) and then put the chicken on to it for approx 2-3 mins either side
  • Take the chicken off and take the pan off of the heat for a minute. Put the chicken somewhere to keep it warm and where you can gather the juices from for later.
  • Put the pan back on the heat and place the mushrooms onto it (obviously you can chop them what ever way you prefer) and cook them for 5-6 minutes or until their own moisture/juice starts to show.
  • Immediately after the mushrooms own juices starting to show add the chicken stock. You’ll now cook the mushrooms for around 5 minutes on a medium heat in the stock.
  • After 5 or so minutes have elapsed turn the heat down and now put the chicken back in, let the whole lot simmer for around 2-3 minutes and then add in the half a cup of marsala wine.
  • Let this now simmer for 10 minutes on a low heat

* the measurements for the stock and wine I’ve used are because my frying pan is wide but the sides are not very high and it can’t support a lot of sauce/juices. Ideally use a pan with high sides and double up on these measurements so that you end up with quite a bit of the juices to place over your veg, potatoes or rice.

You can now serve the dinner to 2-3 people.

Enjoy this, I’d like to welcome comments on this post if anyone tries it out :)

Blacknight launch new shared hosting system

Blacknight are going to launch their new shared hosting platform tomorrow August 7th 2008. We’ve been waiting to do this for close to 10 months now when we put development into overdrive.

What to expect?

  1. More of everything
  2. The basic package has 120 databases (30 Mysql4, 30 Mysql5, 30 Ms SQL 2005 and 30 Postgresql)
  3. You can use MS SQL 2005 on linux now, wow?
  4. The system is partially virtualised, we don’t need to install servers anymore. We simply tell our provisioning system to create a new VE (virtual environment) and apply a set of templates to it that match the shared hosting system.
  5. PHP 4 and PHP 5 are available on both linux and windows
  6. Complete DNS control, complete control over your domain and it’s name server settings
  7. Manage your own spam and antivirus settings
  8. Blacknight application vault with loads of apps you all have looked for :-)
  9. Single login gets you access to everything. i.e. once your in, you can manage your plesk vps, your shared hosting account, register domains, manage your payments, add a new credit card etc.
  10. You get access to our new site builder that comes with over 250 templates and 100s of new features.
  11. You can see the system defined mime types, you can also setup your own, awesome eh?
  12. CGI sub section of the control panel tells you the paths to perl, python and sendmail and also what permissions to set on your files.
  13. Theres a log manager where you can login and view your logs via ftp
  14. Manage multiple ftp users without issue
  15. @Mail webmail system
  16. Advanced backup system that lets the user manage their backups.

It’s a complete suite of tools that anyone who has used any other control panel will fall in love with immediately.Where is this new system, well to get a look at it you’ll need to head over to http://dev.blacknight.ie and purchase a domain or a hosting package.

Our entry level plan has 10GB of disk space and 100GB of bandwidth, thats more than our existing plans put together almost!

last but not last a bit of information on the technology. We’ve deployed clustered Qmail + LDAP for e-mail. In front of this we’ve got load balancers that’ll send you to the least busy mail server when you use pop or smtp. We’re fully support pops, smtps and imaps. Behind the mail servers we’re using Solaris nfs servers attached directly to an EMC Clarion CX300 SAN unit providing us with the ultimate in enterprise storage and performance. Each linux web server will be a VPS. The benifit of this is ease of deployment, we can move VPS servers to less busy hardware nodes seemlessly and with minimal downtime (soon with no downtime!!). The back end provisioning system is comprised of two Parallels products, one is POA and the other is PBA, Parallels Operations Automation and Parallels Business Automation. Thats a split 40/60 for provisioning and billing. The billing end manages domain provisioning and a few other things.

Using Firefox3 in ubuntu and can’t get Sun JRE working??

Right this was a bit of a doozy.

  1. sudo apt-get install sun-java6-plugin
  2. cd .mozilla/plugins/
  3. ln -s /usr/lib/jvm/java-6-sun-
  4. restart firefox

It still doesn’t work? Ok. Type the following into your address bar about:plugins – if you see anything in relation to icedtea, remove it:

  1. apt-get remove icedtea-gcjwebplugin
  2. restart firefox

You should now have a functional JRE installed in Firefox3. This has been bugging me for a while as I’ve a few apps that use a JRE that I’ve not been able to use for a while.

icq broke in pidgin – patch available

I’ve been using ICQ for near enough 10 years. Yesterday it looks as if ICQ pushed out a network update that changed versions around. This broke ICQ for me on ubuntu hardy. I’ve been waiting since last night for a patched install but alas it hasn’t come and because time is money in business I needed ICQ working so I can liase with our Russian friends.

I’ve built a patched version which is available here: http://weird.ie/files/

Just download the 2 files and do:

sudo dpkg -i pidgin_2.4.1-1ubuntu3_i386.deb libpurple0_2.4.1-1ubuntu3_i386.deb

Restart pidgin and all will be well.

The long awaited product launch – also there’ll be more to come!

So we’ve been talking about stuff on the company blog, with friends, with customers etc for months. Today we’re doing a soft, low key, launch. It’s a new range of VPS plans based on Parallels Virtuozzo. It’s tacked onto our brand new billing and provisioning systems. It’s all part of the project we’ve been working on behind the scenes to automate more and more and also provide the next generation in hosting to our existing customer base and to the rest of Europe.

So here it is http://www.blacknightvps.com our new virtualisation product launch. It’s a cost effective, scalable, high performance platform that allows us to deliver services on Linux and Windows x64 Virtual Environments.

Some of the cool features include:

  • Live migration
  • CentOS 5.x and Windows 2003 x64 VPS
  • Instant activation
  • Virtuozzo Power Panel
  • Advanced blacknight PEM billing control panel
  • Ability to migrate a VPS to a dedicated server
  • Ability to migrate a dedicated server to a VPS

And I guess lots more. We’ve thrown quite a bit of hardware at this new system and I’ve personally spent 100s of hours testing, configuring, breaking and fixing it over the past few months.

Any feedback, error reports etc are welcome.

Office space in Dublin costs the earth or does it?

We’re setting out to hiring a few people that’ll have to be located beside each of the data centres we use in Dublin so we can have staff on-site primarily during business hours. Later we’ll expand on that.

One reason Blacknight never had offices in Dublin is due to cost. Now lots of business are based in Dublin, that’s fine. However for the price of an office in Dublin we can hire 2-3 competent engineering staff. This is fact.

So in investigating our options I came across the following co-working site. After having a look around I see that they have 2 locations now that you rent office space in. It’s really simple, you get a desk, chair, internet, phone and secretarial services all for a fixed relatively low monthly fee. There are other ancillary services that you can pay additionally. It really is a good idea.

This is most probably the route we’ll go down for our “Data Centre Technican” positions that we have going. Hopefully this particular company might get more office space in west Dublin that’ll be closer to our kit.

Paul Kellys blog