Planet CentOS

If someone was to ask you why you run CentOS, what would your top 3 reasons be ?

- KB

While i’ve used (up to now) the IBM/LSI-logic solution (aka RDAC) to support multiple paths to an IBM storage solution (aka DS4xxx and DS3xxx), it was a pain because each time you wanted to install a new kernel the procedure implied to remove the old/previous rdac module, boot with the new kernel (without mpp), rebuild mpp/rdac and creating a new initrd and then another reboot (with the new initrd containing the correct module).

I’ve now switched to dm-multipath instead. The basic and provided /etc/multipath.conf normally works quite ok, but if you want to tune it to support more storage vendors/solutions you really have to read the multipath documentation. Jim already blogged about the DS4700 FC backend storage .

Here is the version for the DS3200 (SAS connections) :

devices {
device {
vendor                  “IBM”
product                 “1726-2xx  FAStT”
getuid_callout          “/sbin/scsi_id -g -u -s /block/%n”
prio_callout            “/sbin/mpath_prio_rdac /dev/%n”
features                “0″
hardware_handler        “1 rdac”
path_grouping_policy    group_by_prio
failback                immediate
rr_weight               uniform
no_path_retry           300
rr_min_io               1000
path_checker            rdac
}
}

Tru just pointed out http://i586.centos.org/ which is an archive of the fruit of the push to get the AMD K6-II / Intel i586 install ISO working.

Nice stuff, and nice to know the effort was not wasted...

The other day I had to configure a box that had to fetch some files from another machine and transfer those files from the DMZ to an external bank. While I usually use SFTP for that, in that specific case i had no choice and had to use FTP/SSL. First thing that hurted me is that to fetch the certificate/private key that the bank created for me, I had to use Internet Explorer on a Windows machine ! Ouch … (yeah, they use activex on the page you have to login to for the certificate request, you *can’t* use openssl yourself to send them the CSR …) bad, bad .. and also funny that they point you to an https website to read the documentation, in which they say how to import they Root CA (which obvsiouly you had to import yourself first to read the same manual …) .. From that time i knew i’d have troubles ..

Okay, exporting the SSL certificate/private key from Internet Exploder, using openssl to convert to PEM and i had those ready to be used on my CentOS 5.4 VM. Lftp seems good for such task and supports ssl too .. After having configured my ~/.lftprc with the correct value (like ssl:key-file and ssl:cert-file) I wasn’t able to connect : the message was : “Fatal error: gnutls_handshake: A TLS fatal alert has been received” . Hmm, strange. I decided to test with the RPMforge version (which is built against OpenSSL and not Gnutls) and that one worked correctly (without having changed the conf files). Okay it’s now working but does that mean that the lftp package from 5.x doesn’t work in ssl mode with a client certificate ? I’ve downgraded the package to the one present in the 5.x branch (before the 5.4) : lftp-3.5.1-2.fc6 instead of lftp-3.7.11-4.el5 and it worked perfectly with the same config files too. Sounds like a bug to me and not a config issue so i opened an bug upstream and on the CentOS mantis system. Let’s see how it goes. In the meantime (if you have the same issue) you can either downgrade to the lftp version you’ll find in the 5.3 tree or update to the version from RPMforge.

I've been trying to automate the CentOS-5 updates system as much as possible - however, one thing that the system cant do at this time is be smart about packages that only exist on one arch and not on the others. eg. KVM on x86_64 only.

So for the time being, there will be updates announced for these package on all arch's. Ofcourse, this will be only for packages that are compatible, so in many cases only the .src.rpm will be announced!

I shall try and fix this soon'ish. But in the mean time, be sensible and look at the announcements completely if you do get them! The fix would be that the system does the right thing : only announce packages into the arch they are built for. So KVM will not get an announcement for i386, but only on x86_64.

- KB

Feeling a bit nervous because there’s nothing new to install? Don’t fret, we just released CentOS 5.4. Release Notes can be found here, while you can find the release here. Have fun.

Looks like I might be able to work out all the last minute details and make it to the CPanel Conference 09 in Houston, Texas next week (5th, 6th and 7th of October, 2009).

Karanbir and his wife are flying over from London to make the event. We are also possibly going to have some kind of CentOS social event or meeting on the October 8th, so if you are in Houston and might like to do something like that then contact me or Karanbir on IRC (freenode, #centos-social) or e-mail.

I am really looking forward to seeing Karanbir in person for the first time and meeting his wife. I am also looking forward to meeting Garry Dale, who is helping at the event and pretty much put most of this together.

Heck, I might even get a haircut and trim up the beard for this :D

There are now posted to CentOS Extras for CentOS 4 and CentOS 5 new packages for NX (nx-3.3.0-14) and FreeNX (freenx-0.7.3-2).

NX is an enterprise-class solution for secure remote access, desktop virtualization, and hosted desktop deployment built around the self-designed and self-developed NX suite of components. Please see the NoMachine web site for more information about the NX libraries.

FreeNX is a GPL implementation of the NX Server and NX Client Components. See the FreeNX website for more information.

The NX packages in CentOS Extras are built from the OSS sources from NoMachine and provide the NX libraries, and the FreeNX packages provide the required server software to use the NX libraries.

CentOS also provides an NX client called QtNX (qtnx-0.9.0-3), but only for CentOS 5.

See the CentOS Wiki for information on using NX and FreeNX on CentOS.

The CentOS 4.8 i386 install media has an option for the older i586 processors, as well as the normal i686 installer. The only problem is that the i586 kernels are broken on the released install media for CentOS 4.8.

If you need i586 install media, you can download them from here.

Since the upstream product on which CentOS is based does not support i586, we decided to fix the issues and release an i586 installer separately so as not to impact the already released products. One of the main reasons we do not want to change the OS directory to be different than the released ISOs is that we know some people use jigdo (and probably other things) to create installable media from the mirror trees instead of downloading both the trees and the DVD, etc.

Because of the need to keep the OS directory for 4.8 the same as the original install media, if you want to use the new i586 media to do a network install, you will need to create your own tree and publish it (you can mount and publish the i586 install DVD for this), or you can install from:

http://i586.centos.org/centos/4/os/i386/

A few of us are going to be getting together for drinks on the Tuesday 29th Sep 2009, everyone is welcome to come along. I'll get there for about 18:15hrs and plan on being around till about 20:00 - Depending on how many people are around and what the feeling is - we might nip around to Ragam ( mostly authentic South Indian food ), a few doors down.

There will be a demo for CentOS-5.4 as well! If there is anything specific you might want to see, let me know a bit in advance.

The full address is :

King & Queens,
1 Foley St,
London,
W1W 6DL‎

Here is a Google Street view of the place.

If you email me, I'll get back with my mobile number - although it should be mostly easy to spot the 'CentOS Guys'.

Hope to see you there, then!

If you ever want to get to the grub menu while using pygrub - there is a really simple way of doing that. Just add :

bootargs="-i"


into the /etc/xen/{domU config file}. And the next time you start the VM, it will bring up the grub menu. Quite handy when you need to recover the root passwords or to be able to use a different init script as a one off.

Essentially, that bootargs will pass in parameters to pygrub during the domU boot phase. To get a list of all the possible options you can pass in with bootargs, try this: pygrub --help. You should get output like this :

# pygrub --help
Usage: /usr/bin/pygrub [-q|--quiet] [-i|--interactive] [--output=] [--kernel=] [--ramdisk=] [--args=] [--entry=] image

- KB

Fair is fair: After I mentioned the Definitive Guide to CentOS here, I also should mention that the CentOS Bible has been available since around August.

This one doesn’t have my name on the cover, only on the inside, as I was the technical reviewer for this book. The CentOS Bible can be used as a reference book for many things regarding CentOS, while the Definitive Guide to CentOS is more of a solution oriented book. Both are worth having, IMHO, for personal reasons (hey, I wrote some of it) I prefer the Definitive Guide, though.

By default md-raid will limit its operations to 200000k/sec - which is plenty for most desktop and 2 - 3 disk machines, but when you have more than 3 - 4 disks and there is enough cpu and i/o bandwith available, it makes sense to increase that limit.

to find out what the limit on your machine is :

$ cat /proc/sys/dev/raid/speed_limit_max
200000


Setting it to something higher :

echo 500000 >/proc/sys/dev/raid/speed_limit_max


So whats a good speed to set ? That depends on what it is that you are looking to achieve, eg: if you dont mind max'ing out your hardware platform ( cpu / io / disks ) then set it to something very high, like 2000000. On the other hand, if you want to keep some cpu and io resources back from md-raid ( like when doing a raid-1 rebuild on a production machine ) you might want to actually lower it down a bit.

The three main issues to consider when working out a raid max speed :

• Number of disks: for aggressive sync's I tend to go with 50 - 70 M/sec per disk, so on a 4 disk system the 200000 number is mostly ok, but on a 8 or 12 disk system I'd look to make that much higher. For conservative rates, or when machine resources are required elsewhere as well, 10 - 12M/sec per disk.
• Interface: What interface you use is also going to make a big difference. So consider the implications of using IDE / SATA / SCSI.
• CPU: the raid jobs,specially when run for large disks or over many disks, will be fairly cpu intensive. So workout what sort of speeds work best for the loads you have. Usually this isnt something one needs to consider unless the machine is already under load or expected to be used during the raid operation. Over the last few years, AMD's have been able to deliver slightly better throughputs than Intel's - but in the recent past, much of that has changed. So dont just go with what you hear or opinions around the place : test it yourself.

Finally, while speed_limit_max sets the rates md-raid is going to try and reach, there is the speed_limit_min - which is the rate that md-raid will try and maintain as an 'atleast' limit. I tend to be a bit more conservative about that number. Usually aiming for 25 - 30 M/sec per disk for a very aggressive run. Or 10 - 15 M/sec for a more toned down run. If you have i/o intensive ops running on the machine you might need to reduce this even further - however the default of 1M/s for the whole machine, irrespective of disk count is something I feel too low for a modern machine.

I find many people are unaware of this small detail, hopefully this post will help.

I will be at the cPanel Conference in the first week of October this year. Hope to meet lots of CentOS Users there! CentOS has a corner in the exhibitor area, and helping out over there will be Garry Dale and maybe Johnny would be able to come down as well.

On the 6th at 1:30pm I'll be doing a short 30 min talk on 'Rapid deployment & provisioning' for CentOS. Depending on how it works out for time, I'll try and get a demo / walk though as well for some of the common recommended methodologies. If you manage 2 or more machines even if they are Virtual Machines, there should be something in there for you.

If you are based in the area, but unable to make it for the conference, get in touch with me anyway - I plan on being in the city for a few days after, so we could still meet up.

- KB

I've been looking into the idea of collaborative mind mapping. Think wiki, but in a mind map. The aim being to create a knowledge pool around some very specific areas, that multiple people could contribute into. Specially areas where there might be a lot of content overlap in different zones or a workflow thats easy to define.

Early examples ( and the ones I want to start working with ) could include :

• Post-compromise content and system audit
• System lockdown for various roles, like home-server or home-nas
• setting up a uPnP server, including storage and performance considerations
• Two node, heartbeat based failover cluster for mysql

I guess its easy to see the theme here, all the tasks are almost things that could be reduced to a howto. I keep thinking there must be better ways to handle this at a small to medium sized team level than using a wiki. Say 3 to 7 active contributors with a few dozen occasional drive-by's - and general knowledge levels of each contributor being drastically different from one another.

One thing that has worked really well in the past, for me personally, is doing these based on and around an issue tracker like TicGit. Before you dismiss that idea completely, think about it. However, that does not scale to > 1 person very well. And its a bit of a pain since the only way to organise those down is into a FAQ or a list-of-things kind of way. I hate both those approaches to organisation.

Mind maps are a logical next step after the step based issue trackers and wiki - however, finding one that works well in a browser, and can have nodes outside the immediate map isnt easy. In a nutshell : I've not found any software that lets me do that. I know xmind and Free Mind both have some ways to share the maps. But neither is optimal for mass public consumption. Pimki seems to have potential, but is too much single person centric. Wikka on the other hand, seems to set itself up as the perfect candidate - integrated wiki and mind mapping. But it needs a java plugin and the content it creates seems to not be openjdk friendly.

Are there any other options out there worth considering ?

- KB

As Hp acquired Neoware several months ago, customers are searching for new thin clients .. and I received a Neoware e90 thin client (that wasn’t used anymore). What could I do with it ? … hmm, let’s try to use it at home as a small appliance to host a USB HDD that can be shared . Advantage is that it doesn’t consume a lot of electricity (in comparison with my Asus Barebone with a AMD x2 64) and doesn’t produce noise at all .. which is also a good thing. The thin client I received has a Via Nehemiah cpu @ 800mghz and 128Mb ram. It also has a small IDE-DiskOnChip disk (32mb) but that is obviously too small to setup CentOS on it. I decided to dedicate a small 1Gb USB stick gift I received from a “well-known hypervisor” company (aka Vmware) and use it for / and swap.

I disconnected the DiskOnChip module from the motherboard and configured the bios to boot in pxe as first device and local usb-hdd for the second one (if you need a password, it’s likely to be either ‘dogbites’ or ‘DOGBITES’) and i started a CentOS 5.3 setup. But that didn’t work on first try : the embedded NIC (VIA Technologies, Inc. VT6102 [Rhine-II] (rev 74) ) refused to aquire an IP address . Switching to VT3/VT4 showed me that even if via-rhine.ko kernel module was loaded, it was impossible to have a network connection. (message was related to “netdev watchdog transmit timed out” and some IRQ messages too). I then decided to add the kernel parameter ‘irqpoll’ and then the setup was able to work on the network. One problem solved … Second problem is that with 128mb ram, CentOS 5.x normally isn’t installable. Well, if you use text mode (anyway graphical mode will even refuse to start …) and use disk-druid to create the swap partition, anaconda will use it directly to simulate the missing RAM. Other thing is that I *had* to use was a NFS based setup : I tried a http based setup and it always died on me (maybe because it had to fetch stage2.img while with NFS it just loop-mounts it …). Anyway it installed succesfully on the USB stick (minimal install, so every component removed from the software selection, took 29 minutes to complete) and it rebooted normally. Don’t forget also to add the irqpoll kernel parameter in grub.conf so that you’ll have network connection after reboot … And as an image talks more than a long sentence .. :

This via Hampus : HP seems to now have a support pack, specific to CentOS for their ProLiant servers. Look at : ProLiant Support Pack for CentOS 5 (i386 and x86_64) as an example.

That's one form of endorsement!

- KB

There’s been quite a bit of interest in the XFS offerings included in the 5.4 release of RHEL, and unfortunately it hasn’t really lived up to the hype. There are a few things you’ll need to know if you want to use the included xfs support

• It’s not really included: The XFS kernel module is only in the x86_64 version of the kernel. If you’re using the x86 release, you get no XFS module.
• There aren’t any XFS tools included either: The xfsprogs package isn’t included in RHEL 5.4 Server (see RH’s own BZ #521173 about this). So basically mkfs.xfs isn’t included. That’s not very handy if you were hoping to actually USE xfs.
• Anaconda won’t let you use XFS either: If you’re doing a fresh install and you boot anaconda with the XFS option, you get all the nifty little XFS options you’d expect when you set up your partitioning scheme. The downside is that once anaconda has all the information and tries to actually format things like you told it to, it segfaults. Why? Because it can’t actually MAKE the XFS file system. The tools aren’t there, remember?

Now, many of you may be saying “But Red Hat TOLD me to use xfs in their release notes!” and yes, yes they did. Quoting the Release Notes from RHEL 5.4 we see this ->

Users of GFS2 that do not need high availability clustering are encouraged to look at migrating to other file systems like the ext3 or xfs offerings. The xfs file system is specifically targeted at very large file systems (16 TB and above).

I’d really like to see RH fix support for this, because XFS is an excellent file system, and has some excellent performance when paired with things like MySQL databases.

Red Hat: if you’re listening, please reverse the decision on https://bugzilla.redhat.com/show_bug.cgi?id=521173 and include the XFS toolkits. Your users will thank you for it.

For around a year now, I’ve been wanting to move away from IBM’s (okay, LSI’s) rdac mpp drivers used for the ds4XXX  series disk chassis on RHEL and CentOS. When RHEL 5.3 came out boasting support dm-multipath support for the DS4XXX series, I was understandably overjoyed. The only problem was I couldn’t make it work. After several rounds of cursing, muttering and poking folks smarter than me to help out, the problem became immediately clear.

The example configs which work fine for the ’supported’ platforms have a text string mismatch when using the ‘unsupported’ ds4700. Basically you have to change a bit of text slightly because the hardware identifies itself slightly differently. Who’d have thunk it, right?

Below is the multipath.conf snippet I’ve been using now for the past month with some pretty good success.


defaults {
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]*"
wwid "*"
}
blacklist_exceptions {
wwid "3600XXXX"
}
devices {
device {
vendor "IBM"
product "1814 FAStT"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_rdac /dev/%n"
features "0"
hardware_handler "1 rdac"
path_grouping_policy group_by_prio
failback immediate
rr_weight uniform
no_path_retry queue
rr_min_io 1000
path_checker rdac
}


Now clearly you’ll have to modify the wwid to suit your own environment, and you’ll also want to exclude the non-multipath device references (/dev/sdb for example) from lvm.conf if you’re using lvm.  This got things going for me, so hopefully it’ll help others out as well.

The CentOS 4 series point refresh has been released to the mirrors for a couple weeks now, and the updates it backlogged as well. But the AMD K6-II / Intel i586 install ISO was not right when we shipped, and we knew it

Akemi 'toracat' Yagi had it working in her side archive, and kept working the issue with Johnny 'hughesjr' Hughes, and candidate ISOs have been in testing in the QA back channel. I get a 'heads up' on a new testing from hughesjr yesterday afternoon, and around 5 am today, a notice that a new candidate was ready for pulling and testing

I put lftp to work, and burned the CD. Booted with the command line parameter:

: i586 text

and did a minimal install

Eureka -- it works in mainline CentOS

Coming soon to a mirror near you (for the four or five users of such old kit). The unit I am testing on was my workstation on 11 September 2001, and I long since consigned it to the boneyard

---

090902: fixed grammatical error

I had today to deploy two CentOS 5.3 xen dom0 on two blades and then some domU’s guests. Everything was fine except that when i used our traditionnal deploydomU script (which uses virt-install) it directly complained about memory issue. The exact message was ” ‘Out of memory’, “xc_dom_boot_mem_init: can’t allocate low memory for domain\n ” ” . Strange as I was sure that the dom0 had plenty of memory and the new guest was defined to use only 768Mb .. so what was the issue ? In fact, nothing related to memory : Our new machines get deployed through a pxe boot menu (with syslinux/pxelinux.0 and pxelinux.cfg) in the Labs zone, but a typo was inserted in that menu so that newer CentOS 5.3 x86_64 machines were in fact … using i386 repo ! 

It took me 5 minutes to consult the great oracle (aka google) , find the same issue and look at both new nodes to confirm with `uname -a` that I tried to deploy a x86_64 domU on a i386 dom0 …

Hehehe, strange that the message is related to memory and not arch .. but several minutes later (and a coffee cup, machines being redeployed correctly *after* the pxelinux.cfg file was modified) everything was back to normal and x86_64 domU’s running fine … hope that it can help other people having the same ‘typo’ :-p

The DRBD packages in CentOS 4 and CentOS 5 will soon be updated. Each CentOS version has two DRBD versions.

CentOS 4 DRBD

Version 0.7.25
CentOS 4 currently contains DRBD 0.7.25 (which it was released with) and the newer DRBD 8.2.6. There will be 2 upgrades to CentOS 4. The first is that a new kernel module for DRBD 0.7.25 will be released that is kABI based. This means that you will be able to use it on multiple kernels, and not need to reinstall it. The packages will be kmod-drbd-0.7.25-4.el4 and should be a direct replacement for the current kmods if you are using version 0.7.25 of DRBD.

I recommend that if you are using this version if DRBD, you upgrade it to DRBD 8.3.2 or higher instead. The upgrade from DRBD 0.7.x to DRBD 8.3.x is not trivial, however there is no longer any support for DRBD 0.7.x from Linbit, so I would recommend and upgrade after CentOS releases the 8.3.x version discussed below, that you upgrade to 8.3.x.

Here is an article from Florian Haas on upgrading from version 0.7 to 8.2 (or greater) DRBD.

Version 8.3.x
Currently CentOS 4 also contains version 8.2.6 of DRBD with newer versions of DRBD in the Testing Repo.

Linbit, the upstream provider of DRBD, has released a new 8.3.x tree and they are no longer maintaining the 8.2.x tree. The upgrade from 8.2.x to 8.3.x is not hard and can be done live, so CentOS will release a version 8.3.x DRBD update that replaces 8.2.x

If you planning new DRBD installations, please use 8.3.2 or later versions and not 8.2.x or 0.7.25 versions.

Upgrading from 8.2.x to 8.3.x
When upgrading from DRBD from 8.2.x to 8.3.x (and even when doing a 8.3.x to another newer 8.3.x) you should handle this upgrade a little differently than normal machine updates. Here is a basic outline of how to do the upgrade:

1. Login to your Secondary Node. Verify this is your secondary node with the command service drbd status.

2. Upgrade everything expect your kernel, DRBD and heartbeat. You likely (or at least should) have heartbeat, drbd, kmod-drbd and kernel excluded from your current updates. If you do, then just a the command yum upgrade. If you do not have the above files excluded, this command should work:

yum --exclude="kernel* heartbeat* drbd* kmod-drbd*" upgrade

3. Turn off heartbeat on the secondary node with the command service heartbeat stop. Check your primary node, it should still be running heartbeat and DRBD. The secondary node should only be running DRBD.

4. Turn off DRBD with the command service drbd stop. At this point, you do not have DRBD running on the secondary node.

5. Remove the current kmod that you have installed ... this command should do it: rpm -e kmod-drbd82. (substitute drbd83 for drbd82 if you are doing an inplace upgrade of drbd83)

6. Install the latest CentOS kernel on your machine. If you remove the exclusions in your CentOS-Base.repo file, you can do this with the command yum upgrade kernel\*

7. Install the upgraded drbd83 and kmod-drbd83 with the command yum install drbd83 kmod-drbd83. Note: you may need to substitute upgrade for install if this is an inplace upgrade of drbd83.

8. You will now likely need to reboot on the new kernel, but you do not want drbd and heartbeat to start up on the reboot, so for now do these commands chkconfig heartbeat off AND chkconfig drbd off. Reboot your secondary node machine and log back into it.

9. At this point, you should be running the new kernel and have the new drbd installed. Try starting the drbd with the command service drbd start. Now check the drbd status with the command service drbd status ... the secondary node should be up and running in secondary mode. Once it is also consistent, you should be able to start heartbeat with the command service heartbeat start. Check your /etc/log/ha-log to verify heartbeat is back in secondary mode. Also check heartbeat is running with service heartbeat status

10. If everything is working, turn on heartbeat and drbd to start on boot with the commands:

chkconfig drbd on
chkconfig heartbeat on

11. You should now have your secondary DRBD node back to normal. You should be able to now to login to your primary node and turn off heartbeat on it and have the secondary node pick up as primary using the command service heartbeat stop on your primary node.

12. You should now have your Normal SECONDARY NODE as your current primary and normal primary node as a DRBD secondary with heartbeat turned off. If that is the case, you should be able to upgrade the primary node using steps 2-11 above (skipping step 3) ... substituting primary node for secondary node in the instructions.

Note: When you start heartbeat on the normal primary node, all the loads will move from the secondary mode to the primary mode, so make sure DRBD is running properly before restarting heatbeat on the primary node.

CentOS 5 DRBD

Version 8.0.16
CentOS 5 currently contains DRBD 8.0.x (which it was released with) and DRBD newer 8.2.6.

DRBD 8.0.x is the stable tree, and if you want to use it, it is currently supported. Version 8.0.16 will be released as part of this update. The basic steps to upgrade will be the same as in the procedure above, substituting drbd and drbd-kmod for drbd82 or drbd83.

As long as LinBit supports the 8.0.x tree, CentOS will do the same.

Version 8.3.x
All of the information in the CentOS 4 Version 8.3.x section is applicable to CentOS 5 as well.

The upgrade outline is also the same.

The updates should be released in the next week, so be looking for them on or before Saturday, September 5th, 2009.

The CSGFS (Cluster Suite / Global File System) repository for CentOS 4 has been updated after the release of CentOS 4.8.

All the latest Source files have been built and added to the repository.

There is one known problem of Missing SRPMS for lvm2-cluster.

A bug has been filed upstream asking for the release of this SRPM. In the mean time, you need to run a lesser version of lvm2 from the main CentOS-4 repository on your Cluster servers.

Recently I had to install a new server that will act as a mail server (Zarafa but that doesn’t matter) and being member of a DRBD cluster (to replicate automagically the Zarafa MySQL DB and Attachments on disks to the other node) . Fine, except that only one physical node was at my disposal : we’ll convert the existing M$ Exchange server physical box to CentOS/DRBD after the migration. So what ?

I was thinking about that nice feature in mdadm when you want to create a Linux software Raid 1 array but with only one available disk (”mdadm –create /dev/md0 –level=1 –raid-devices=2 /dev/sda1 missing” for those of you who don’t know that nice feature) and add the second disk later .. That would be cool to do exactly the same with DRBD : one node active and then add the missing one later .. Don’t try to find a ‘missing’ parameter in the drbd.conf file .. but that’s possible (even if not documented in the online docs) . Do you remember that nice parameter you use when you initialize your first DRBD resource (drbadm — –overwrite-data-of-peer primary $resourcename) ? Why not testing it with only one available node ? Yes, it works .. In fact that remembers me the name of that parameter in the previous DRBD versions (aka  “– –do-what-I-say” ) :  that was really a way of instructing DRBD to do what you wanted it to do.

The only “issue” found so far is that it isn’t possible to use the “drbdadm resize” command online to extend its size (yes, I use the nested LVM configuration : so backend disks / LVM / LV as a DRBD device / LVM / new LV on top of the drbd device) but I can easily understand why such operation really needs a connection to the second real node (which obviously is missing here)

Oh, while i’m talking about DRBD you have to know (if you use it already) that DRBD 8.3.2 (and the corresponding kABI kmods) are available in the [testing] repo

In quite a few situations its preferred to have ssh keys dedicated for a service or a specific role. Eg. a key to use for home / fun stuff and another one to use for Work things, and another one for Version Control access etc. Creating the keys is simple, just use

ssh-keygen -t rsa -f ~/.ssh/id_rsa.work -C "Key for Word stuff"

Use different file names for each key. Lets assume that there are 2 keys, ~/.ssh/id_rsa.work and ~/.ssh/id_rsa.misc . The simple way of making sure each of the keys works all the time is to now create config file for ssh:

touch ~/.ssh/config
chmod 600 ~/.ssh/config
echo "IdentityFile ~/.ssh/id_rsa.work" >> ~/.ssh/config
echo "IdentityFile ~/.ssh/id_rsa.misc" >> ~/.ssh/config

This would make sure that both the keys are always used whenever ssh makes a connection. However, ssh config lets you get down to a much finer level of control on keys and other per-connection setups. And I recommend, if you are able to, to use a key selection based on the Hostname. My ~/.ssh/config looks like this :

Host *.home.lan
  IdentityFile ~/.ssh/id_dsa.home
  User kbsingh

Host *.vpn
  IdentityFile ~/.ssh/id_rsa.work
  User karanbir
  Port 44787

Host *.d0.karan.org
  IdentityFile ~/.ssh/id_rsa.d0
  User admin
  Port 21871

Ofcourse, if I am connecting to a remote host that does not match any of these selections, ssh will default back to checking for and using the 'usual' key, ~/.ssh/id_dsa or ~/.ssh/id_rsa

It seems this Months ( August 2009 ) issue of the Linux For You has CentOS-5.3 on the cover mounted DVD.

If you are in India, want a copy of CentOS, dont want to pay for downloading 3+ GB over the internet, that might be a good way to get the install dvd.

Know anyone else carrying CentOS in the Magazine cover mounted DVD's ? Let us know.

My name is Johnny Hughes. I am one of the lead developers at the CentOS Project, which releases the Linux operating system called CentOS. I am currently the Project Lead on the CentOS-4 version of the distribution.

The current release of CentOS 4 is version 4.8 which was released on August, 21st 2009.

CentOS 4 will be supported with security updates until February 29, 2012, although if you are planning a new deployment you should probably use the latest version of CentOS, currently CentOS 5. CentOS rebuilds sources from Red Hat Enterprise Linux, and will release updates as long as they are released upstream. For Red Hat's policies and lifetimes, please see this link.

If you are looking for help with CentOS please see Getting Help on the CentOS Wiki.

Over the last few months, I've started using my proper laptop a lot more, and at work as the exclusive desktop interface and moved the 'Desktop' into a VM host. The only issue being that while the Desktop had 8G of ram, the laptop has 'only' 1G. And my email client of choice, Thunderbird is not happy. I've been running the nightly builds for a long time ( almost 6 months ), and that really needs about 2G of ram for itself ( on a i386 machine, with my email load ). You can see how that's going to be Fail on the laptop with a Gig of ram, and I *do* intend to run other things on there along with my email client. Like *drumroll* Firefox */drumroll*. And about a dozen ssh client instances in gnome-terminal, pidgin, a desktop wiki, and the usual normal things that people do.

First thing that I considered was - it used to not be this ram hungry in the older days, so lets try and older build. Out came the tarballs for thunderbird-1.0, build and install. Works. However, its not really that much better on ram consumption. Hitting 600M of usage after a few hours. However, more irritating than that is that IMAP support in Thunderbird has improved by leaps and bounds over the last few years, and I've grown used to those. So much so that I kept getting thunderbird-1 into a state where we were fighting again. Not a good sign that.

Next thing to consider was moving to another email client. Just to see what the options really are. It wasent pretty. Evolution is unable to handle either the quantity of email I receive or be reliable enough to make it through a day without crashing on me a few times. Imap support is mostly ok. However, the UI and the way things work is still the same non-intuitive and cumbersome interface only a mother or outlook users would love.

Claws on the other hand has almost nonexistant imap support. While using it, I kept thinking the 1990's were just around the corner, peering over my shoulder and would attack me any minute! Dont get me wrong, on the Nokia n810 I *love* claws, it actually makes the nokia an extremely potent communication platform. On the desktop though, it just feels clunky and keeps getting in the way ( maybe its just me and my way of thinking, expecting it to work a lot faster than its able to ). And where is that IDLE support ? Even some of the basic things, like using a 3 pane wide-view layout ( I have a 1680x1050 display on the laptop ) are hard to get right, and after fighting it for almost 2 days and getting irritated many times, decided to give up on Claws.

Mutt and Alpine are both options, good ones. However I've been unable to get either Mutt or Alpine playing well with server-side mail filtering and even getting reasonable report-views on state of folders on a remote end. With Alpine there seems to be a way, but sitting and adding manually, dozens of folders and having it manually manage each one is quite a pain. Added to that is that while I am ok to use Mutt as a backup or a use-in-a-hurry type mail client, I'd really like to be able to just use a GUI based client for most things. Mutt can perhaps come very very close to replacing it, if I can get the mutt-sidebar working ( have completely and totally failed for me ). If nothing else works, I might revisit that and see if anything can be done to make it not segfault on load.

Kmail is another option, slightly made more attractive with the built in sieve support in addition to the local maildir stores. But having used it in anger, I didnt like the way it laysout emails or how it handles multiple ID's per mailbox, in the past. Perhaps its time to revisit Kmail again.

Balsa is another option that I considered and its not half bad a client, but I think a bit basic for my needs.

Anyway, for the time being, its back to thunderbird and nightly. And a 4G swap partition. Atleast this way I get enough time to get up and refill Tea / Water / Coffee mug everytime I start reading emails :) I just feel that thunderbird as a client is really at the best place that email can be at the moment, its matured quite nicely and with the new development features for Tbird3, its only getting better. Now they just need to see if perhaps moving from the mbox storage format is an option, specially for people who get lots of emails that can be quite a winning development. Now, if only they would stop wasting time with 'tabs' in a mail client! Wtf is that about anyway ? I've not come across a single use case where I'd want to be running tabs while 'doing' email!

Are there any other major email clients on linux that come 'recommended' ? No 'telnet' does not count as a mail client.

- KB

A few minutes back 'KingJ' came into #centos on irc.freenode.net with an odd problem running yum update. His yum process back traces out with :

# yum update
Loaded plugins: fastestmirror, priorities
Loading mirror speeds from cached hostfile
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 229, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 104, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 339, in doCommands
    self._getTs(needTsRemove)
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 101, in _getTs
    self._getTsInfo(remove_only)
  File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 112, in _getTsInfo
    pkgSack = self.pkgSack
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 591, in lambda
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 434, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 223, in populateSack
    self.doSetup()
  File "/usr/lib/python2.4/site-packages/yum/repos.py", line 71, in doSetup
    self.ayum.plugins.run('postreposetup')
  File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 176, in run
    func(conduitcls(self, self.base, conf, **kwargs))
  File "/usr/lib/yum-plugins/fastestmirror.py", line 181, in postreposetup_hook
    all_urls = FastestMirror(all_urls).get_mirrorlist()
  File "/usr/lib/yum-plugins/fastestmirror.py", line 333, in get_mirrorlist
    self._poll_mirrors()
  File "/usr/lib/yum-plugins/fastestmirror.py", line 376, in _poll_mirrors
    pollThread.start()
  File "/usr/lib/python2.4/threading.py", line 416, in start
    _start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread

At first look it does seem as if its a resource issue - and investigating further confirms that his machine / VM is running out of ram. People run and have been able to run CentOS-5 instances in very tight and small resource setups, however most people usually need atleast 128M of free usable memory for yum to do its thing.

- KB

Yesterday's post on the K6 covered getting a CentOS 4.8 beta candidate installed on ancient hardware; The careful reader may have noticed that I had an unexplained list item early on in that outline:

Add to /etc/yum.conf

exclude=kernel*

This is not something that just occurred to me unbidden, but rather came from an awareness that the upstream has had the dreaded 'Regression' from time to time in its RHEL 4 series, where a patch needed to support the K6/i586 architecture was not consistently present. In reading the bug comment notes, it seems that the 'boneyard' available to the member of the kernel testing team tasked with this is not so full of carcasses as mine, and so he cannot test his fixes as well

So, I took affirmative steps to preemptively 'partition away' the need for an updated working kernel from our 4.8 beta install candidate, and yet be able to get to a working chassis with the kernel from the 4.5 final image, which is known to work. Good thing. The regression is back in the 4.8 kernel SRPMs, and the needed patch got dropped, it seems (this from an initial workup -- detail testing will be needed to see)

---

The workaround is straightforward; Akemi 'toracat' Yagi maintains a testing 'plus' archive, containing kernels with the needed patch, and I can confirm that her candidate works fine. see: http://centos.toracat.org/kernel/centos4/centosplus-testing/i386/

Thanks, toracat