Planet CentOS

CentOS 4.x machine not rebooting and faced with a grub prompt

2008-06-25T13:44:18+00:00

One of my customer phoned me to say that one CentOS 4.x machine (acting as a apache reverse proxy) didn’t reboot after a power outage. The machine had two sata disks configured in raid 1 (through md/software raid) but instead of booting, the machine was just displaying a grub> prompt.

Of course i tried the traditional `grub-install –recheck /dev/sda` and `grub-install –recheck /dev/sdb` and also the manual procedure (already described here) to install grub on both devices .. but no luck .. still booting at the grub> prompt.

But then i looked (in rescue mode) at the (/mnt/sysimage)/etc/grub.conf and i counted 22 kernel entries in the file .. The customer had configured the nightly automatic yum update but he never cleaned the old kernels (both up and smp) … so i “cleaned up” the grub.conf file, once again installed grub with grub-install and …. machine rebooted normally ..

I’ve never thought that too many entries in the grub.conf file could block the machine from booting … Maybe that will save other people time

CentOS 5.2 has been released

2008-06-24T15:25:00+00:00

The CentOS team is pleased to announce the release of CentOS 5.2 - the latest update for the CentOS 5 series.

Major changes in CentOS 5.2 compared to CentOS 5.1 are: Firefox 3, Thunderbird 2, OpenOffice.org 2.3 and Evolution 2.12 on the Desktop side, Samba 3.0.28, xen-3.2 and an upgraded kernel with lots of driver updates on the server side of the system.

Read the Release Notes, the Release Announcement and get it while it’s hot from a mirror close to you.

5.2 Release update

2008-06-19T23:19:46+00:00

We found a very major issue with the last set of ISOS for 5.2 meaning I had to redo the distro isos today. We should start seeding the mirror network in the next 24 hrs time, so release should still be 23rd June, give or take a day or so.

UPDATE: 2008-June-23 : We found yet another issue with the x86_64 tree, so while some of the updates are now syncing out, please wait for the release announcement before pulling packages and the isos.

--
Karanbir Singh [ http://www.karan.org/ ]

RHEL backported one additional year

2008-06-19T22:38:53+00:00

At the 2008 Red Hat summit in Boston, Red Hat outlined to support Red Hat Enterprise Linux for new hardware and installation media one year longer than it did in the past.

This is a major event. In the past Red Hat offered new hardware support, bugfixes and feature enhancements (dubbed full support) for 3 years after the initial release. But now that will be for 4 years after initial release. New installation media will be release up to 5 years after initial release !

That means that RHEL5 (and CentOS 5), released on March 2007, will have full support until March 2011, new media releases until March 2012 and will have security updates until March 2014.

If you were running RHEL4 (or CentOS 4), released on Feb 2005, than the upcoming 4.7 release will not be the last, but there will be at least a 4.8 and 4.9 release until May 2009, new media releases until May 2010 and from then on the normal security updates will keep coming in until Feb 2012.

This matches much better with the normal hardware lifecycle within companies. Usually hardware is decommissioned after 4 to 5 years and since a release is often not being deployed in production when it ships, but more or less a year after initial release. That means there is a much better (and wider) overlap between supported releases.

Red Hat's Product Life Cycle page is not adapted yet, but here is an updated slide from my Enterprise Linux presentation and it explains Red Hat's Enterprise Linux offering best:

Click the picture to see it in all its glory

This is a major improvement to Enterprise Linux as a whole and I am very keen to see how Novell and Canonical are going to react to Red Hat's newest twist.

Enjoy !

Using apt in an RPM world

2008-06-17T20:00:22+00:00

Everytime I am surprised that people don't know that apt-get works on RPM-based distributions and works much better than the alternatives. Especially in a CentOS/RHEL environment where you have various distribution releases running, apt-rpm allows you to use the same apt version and the same apt features across CentOS/RHEL 2.1, 3, 4 and 5.

In an attempt to persuade you to try out apt, let me denounce some myths about the current apt-rpm:

works on RPM packages
can do multilib (coexist 32bit and 64bit)
does check GPG keys on packages
can work with repomd repositories (as used by yum)
does work a lot faster than yum
is being maintained (although could use more hands)
has python bindings
has a graphical interface (synaptics)
handles multiple repositories much better
allows to pin packages by version or repository (and manages cross-dependencies)
existed years before yum (and was the only depsolver for a few years in the Red Hat world)

Now, because I can say it works great but you may not believe me, let me make it very simple for you to try it out on CentOS. Here is a quickstart guide...

First we start off installing apt from RPMforge. Follow the guide on the CentOS wiki to configure RPMforge for your system. Then do:

yum install apt

(or alternatively install the latest apt RPM package from http://packages.sw.be/apt/ for your distribution)

and then edit the file /etc/apt/sources.list.d/os.list and add for CentOS:

repomd http://mirror.centos.org/ centos/$(VERSION)/os/$(ARCH) repomd http://mirror.centos.org/ centos/$(VERSION)/updates/$(ARCH) repomd http://mirror.centos.org/ centos/$(VERSION)/extras/$(ARCH)

This adds the official CentOS os, updates and extras repositories.

Now you can use it, for example do:

apt-get update

to update the locally cached metadata, or do:

apt-get upgrade

to upgrade your system with the latest updates, or yet, do:

apt-get install synaptic

to install a package named synaptic. Try:

synaptic

to start synaptic and use the graphical interface (also available from System > Administration > Synaptic Package Management)

There are a few commands that use the locally cached metadata, the following apt-cache commands are used most often:

apt-cache search keyword

to search the local metadata for packages related to keyword, or:

apt-cache policy package

to show the different versions of a certain package that is offered.

PS And let me also add that the project name is apt-rpm, not apt4rpm (a complete different project).

CentOS 5.2 - release update

2008-06-14T20:03:00+00:00

A short update on the release of CentOS 5.2. We are currently in the progress of doing QA testing. All packages have been build. The current plan is to be able to finish all QA test this week so we might be able to release 5.2 next weekend or in the days after it.

As always, these are just indications. It will be ready when it is ready. But we are trying our best and we know that you are waiting for it :-)

I'll post another update in a couple of days.

3ware performance in CentOS

2008-06-13T13:54:35+00:00

There is an upstream bug report here, which may be of interest to folks using 3ware and aacraid raid cards with CentOS or RHEL. The basics of the bug hinge on setting MWI, which can have a pretty hefty impact on IO performance. If you’re storing your MySQL bits on a 3ware powered array, it’s a safe bet that this fix may help improve your performance and reduce some of the IO wait seen on the system.

The downside with this is that even though the fix is known, Red Hat is sticking to their procedure, and has stated that they will not release the fix for this in the main kernel until 5.3 is released. Since 5.2 is fresh from the factory, it’s not likely that we’ll be seeing this fix pushed mainstream in the next few months.

This leaves a few choices for the RHEL and CentOS communities for how to proceed.

Weigh in with your opinion on this bug report. If enough people respond, RH will likely appease them.
Help test the patched kernels in the bug report. The more comfortable RH is with the patch, the more likely it is that they’ll tuck it in with a bug fix or security update. See #1.
Give CentOS a chance to get 5.2 out the door. Once it’s released, folks will have some time to roll up a kernel repository at http://people.centos.org similar to what was done with bz321111. Since this bug is strictly performance affecting, installs won’t be an issue, and you can update to the modified kernel, or use the stock release as you see fit.

In theory, you could also roll your own kernel with this patch if you didn’t want to wait, however if you do this, you’re accepting responsibility for building it properly and tracking all the kernel security and bug updates until the patch becomes mainstream. I wouldn’t recommend this method since it requires more time and upkeep, but for folks who roll their own it provides another alternative.

RHEL Beta-test SIG ?

2008-06-12T01:17:22+00:00

I have been playing with (and talking about) this before, so why not take it to the next level and share it with the larger CentOS and RHEL community ?

The CentOS community is pretty limited in what we can do to the core OS. Since our mantra is "aiming to be 100% compatible with Red Hat Enterprise Linux" we cannot fix bugs or improve the CentOS core without waiting for RHEL to make those modifications first. We have limited leverage and a 6-month release cycle against us.

But that is not the complete truth, Red Hat usually has an internal, a 3rd party and a public beta period and everything that is found within that period might get fixed before it is being shipped (and frozen) for the next 6 months.

So if we can improve the testing during the RHEL beta program, everyone in the CentOS community directly benefits from that as well. Therefor it makes a lot of sense to encourage the large CentOS community to take part in the RHEL beta program and help with improving the next CentOS releases. (You don't need my backpatting, start already !)

At the moment there is a RHEL 4.7 Beta release that runs until July 10 and probably could use your involvement as well. Unfortunately the beta releases require Red Hat Network access and therefor are not readily available. That leaves its mark on the RHEL4 beta mailinglist (lacking activity). Most people undoubtedly use Red Hat's bugzilla directly. But still, more involvement in the beta releases improves the next official release so we should encourage it.

So my proposal consists of a multitude of actions:

bring together interested and committed people to promote beta-testing within the CentOS community (how about another Special Interest Group ?)
set up a wiki-page to encourage beta-testing and list the relevant beta's, mailinglists and procedures for testing and reporting problems
forward the beta release announcements on the CentOS mailinglist as well (or have it send by Red Hat directly ?)
make the RHEL beta releases available to a wider audience (without having people confuse the beta for the real thing)
go over the CentOS bugs and test against the newer beta release and update ticket in Red Hat's bugzilla

Are you in charge of doing beta-testing within your company ? Or have you been doing that at home for yourself ? Are you interested in leading or helping a CentOS project in that area ? Let me know and let us start discussing on the centos-devel mailinglist.

PS If you are looking for the RHEL 4.7 Beta ISO images, go to Red Hat Network, log in, click on Downloads, expand the RHEL4 channel and go to the RHEL4 Beta channel. There you can find the RHEL 4.7 Beta ISO images.

Entering CentOS 5.2 QA mode …

2008-06-06T21:21:16+00:00

Yes, it started .. the CentOS QA-Team entered the 5.2 QA era .. meaning that we have to test a bunch of existing features and newer ones included in 5.2. For example, in the upstream announce mail i saw that the newer libvirt has support remote connections. So i decided to give it a try just after i updated my CentOS 5.1 x86_64 dom0 to 5.2QA (and my domU i386 and x86_64) … but when i tried to connect, i received a ‘connection reset by peer’ (i tested with only ssh and not tls/certs) … so i decided to read a little bit on the libvirt.org website and found which parameters should have been configured in the /etc/libvirt/libvirtd.conf (full list available here) . The only ‘problem’ so far is that the /etc/libvirt/libvirtd.conf is not provided by libvirt itself and doesn’t exist ! .. Strange because it’s referenced in the /etc/sysconfig/libvirtd (that you have to modify too) file .. So it seems you have to create it yourself , and then i was able to connect remotely (i tested only with ssh .. and important : don’t forget that you need ssh key-based auth for this …)

More informations about the QA tests later (and by other people/QA testers too … )

gconf voodoo

2008-06-06T03:01:50+00:00

The gnome desktop has tons of versatility and flexibility to suit just about any desktop type needs. Unfortunately this flexibility has a hidden cost, and a few dozen hidden options. While most options are right where you’d expect them to be in the various gnome applications like nautilus, others can be difficult to nail down. This is where gconf was supposed to come to our rescue, but instead it got all drunk and confused.

While many folks compare it to the Windows registry, this isn’t entirely accurate. GConf is a bit more user friendly than that, although some similarities can be drawn. It’s a binary set of files which requires a special utility to work, and operates in a directory/file structure type. Below, we’ll go through some of the more common changes users are likely to want.

The Basics:

There are two ways to go about playing around in your GConf registry. You can use the gconf-editor gui, which is in the gconf-editor package, or you can use gconftool-2, which is a command line driven application, and a little more cumbersome to maneuver around in. The basic command to help get you around in gconftool-2 is ‘gconftool-2 -R /’. With this command, you’ll see the directory/file structure which makes up the registry, and their associated settings. If you’re new to gconf, it’s probably best to start out with the gui.

Starting small:

A few times a month or so, users will ask how to tell gnome or nautilus to ignore blank CD input, or to at least do something useful with it, like open k3b instead of the default nautilus burn window. Setting this up with gconf-editor is relatively simple. Open it up, and browse to the ‘/desktop/gnome/volume_manager/’ directory. Inside this directory you’ll find a number of settings that you can modify for various automated media handling. You can change the default movie player, dvd player and more from this directory. Incidentally, ‘automount_drives’ and ‘automount_media’ are located here also, so if you’re having trouble with usb drives, this is one thing to check. The two options that we’re concerned with right now are ‘autoburn_data_cd_command’ and ‘autoburn_audiio_cd_command’.

By default these are both set to ‘nautilus –nodesktop burn:‘, but this isn’t the behavior we want. If you’d like to have k3b loaded up instead, simply change the string values to ‘k3b’ and you’re off and running. This is a per user setting, so you don’t have to be root to modify most of these values. If you do happen to launch the app with sudo, you’ll also have the ability to enforce this setting for all users. This can get handy, and we’ll look at it a bit later on.

System Policies

Now that you’ve had a little bit to look at the user side of gconf, if you’re planning to run lab or kiosk systems you might also want to look at enforcing some of your system policies with gconf. This should not be your only security method to lock the boxes down, simply another layer to examine for inclusion.

Inside the /desktop/gnome/lockdown directory, you’ll find several settings which can help you restrict your workstations or kiosk systems, such as ‘disable_command_line’. After launching gconf-editor with sudo, set these options the way that you want, then right click them and choose ‘Set as Mandatory‘. This will enforce these changes for all system users, and disallow the user from changing the settings individually for their accounts. This can be done for many of the settings here, including application specific options. While this means of enforcement is not perfect, it can go a long way toward helping an admin regain some control and a possibly a little sanity. It’s also one EVIL BOFH prank for other admins/users… if one were so inclined… >:-)

Searching for RHEL and CentOS information

2008-06-05T22:54:45+00:00

If you use RHEL or CentOS a lot and you often find yourself looking for good information on the web about either CentOS or RHEL, you might find the following Firefox search addons very useful.

Here's my overview, sorted by importance:

Red Hat Knowledgebase search: Search Red Hat's extensive knowledgebase, documentation and many other resources
CentOS Knowledgebase search: Search CentOS' extensive mailinglists, wiki, documentation, forums, bugtracker and website
Red Hat Bugtracker search: Search Red Hat's Bugzilla database by using keywords
Red Hat Bugtracker by Id search: Similar, but search by Red Hat Bugzilla Id. This is mostly important when you are looking at RPM changelogs
CentOS Wiki search: Search CentOS' wiki which contains many useful tricks and clues or get an account and add your own solutions
CentOS Mailinglists search: Search the CentOS mailinglists archive to find solutions from many skilled experts
Red Hat Security Advisories search: Search the Red Hat Network security advisories for packages or CVE information

I would like to have them on Firefox's addon website, but in the meantime you can install them directly from this blog-article.

Also remember that the solutions you find for CentOS are equally suited on RHEL and vice versa. So you might want to have both the RHEL and CentOS Knowledgebase search added to your Firefox.

Important: Only supported by Firefox 2 and higher ! So ironically this will not yet work for CentOS 5.1, but RHEL 5.2 should be fine.

Linuxtag 2008 in Berlin was brilliant

2008-06-02T12:51:06+00:00

The latest edition of Linuxtag was very productive. During the 4 days the CentOS crew managed to do several things, including:

professional booth with a 24" screen/laptop setup that we can now reuse for other events around Europe
proper template slides for events like this (also now a German translation thanks to Ralph and Felix)
well received "CentOS and Enterprise Linux" presentation
strengthened our ties with the Fedora project
have a much better solution for our CentOS media (both printing and burning)
met interesting individuals who (I am sure) will strengthen the CentOS project
have a new set of objectives for the CentOS project

Apart from the usual suspects, I met some very interesting people from the Gnome Foundation, Fedora project, Transifex and the VideoLan project without even having taken the time to walk around Linuxtag. From that perspective it becomes similar to FOSDEM, I never to find the time to visit the booths or attend a lot of the presentations myself in recent years.

Regardless of the busy 4 days I still feel that 4 days is a bit too long for Linuxtag. Having to sacrifies 3.5 working days as a freelancer is very painful. And having to travel 7 hours to get there (thanks Tim !) takes a lot of energy.

Next up is NLLGG, Jornadas Regionales, FroSCOn and T-DOSE. I'll catch you there !

Why CentOS will not become the next Microsoft

2008-05-30T13:11:14+00:00

The subject may sound weird to you, but all the arguments that free CentOS from becoming the next Microsoft can be used to to counter the pundits that position Red Hat as being the next Microsoft.

(You may think this statement is so nineties, but a recent opinion piece that got onto Slashdot prompted similar comments)

We can only ask ourselves why someone would want us to believe that Red Hat is the next Microsoft, but let me reiterate why neither CentOS nor Red Hat will be the next Microsoft:

The source code is freely available
There is a level playing field (read: no monopoly) for all Open Source contributors to innovate and integrate (thanks to the GPL)
Red Hat employs hundreds of Open Source developers that contribute to development for and maintains a lot of Open Source projects that form the base of every Linux distribution (more than any other company)
The Fedora project is a good example how the community and Red Hat interact to the benefit of everyone and a good entry point to become part of the community
Improvements made by Red Hat are being imported in Ubuntu, SuSE Linux and other Linux distributions (and vice versa)
Red Hat does not sell any software, it sells support, services and training
Red Hat even buys proprietary software, improves it and makes it free (GFS, FDS, MetaMatrix, ...)
Red Hat combats software patents and promotes only open standards (unlike others)
And last (but certainly not least from where I am standing) the fact that we have a project like CentOS proofs that Open Source is honest business, you only pay for value

So my believe is that Red Hat is in this position because they obviously did something right and people or companies value their offering, even though it may not appeal to everyone in every situation. Unlike Microsoft they are not in a position to lock you into something mostly because there are enough alternatives.

And if there is really a point in looking who will become the next Microsoft, the danger is more likely to come from here, here or here. Although don't expect Microsoft to give in that easy. We have seen from the recent OOXML debacle, that Microsoft is more eager to push its monopoly through everyone's throat than it cares about its public image.

I am pretty sure Microsoft will be much more careful next time though... And we will be too :-)

I am sure I did not cover all items, so if you think I missed a hot spot or I am being too favorable, feel free to add your comment.

Going to Linuxtag

2008-05-26T23:35:00+00:00

So I’m leaving for Linuxtag tomorrow - and if anyone of you wants to meet up there from Wednesday, 28th of May to Saturday 31st of May - you can find us at the booth 109 in hall 7.2b. See you there!

CentOS 5.2

2008-05-23T08:49:00+00:00

Before everyone starts asking questions let's get this cleared up here :-) Upstream released version 5.2 of their Enterprise Linux distribution on the 21st. So I understand that you are all wanting to now when CentOS 5.2 will be released.

Well, looking back at 5.1 we see that upsteam released 5.1 on November 7th 2007. CentOS 5.1 was released on December 2th 2007. So there was a delay of 3,5 weeks before the release of CentOS 5.1

So when can we expect CentOS 5.2 then, well using the same 3,5 weeks we end up with a date of around June 14th 2008. This is of course a estimate it can be later or earlier.

For some background information, why does it take 3,5 weeks ? First we need to remove all the logos and trademarks of Upstream. Secondly we need to build everything from source and this for both i386 and x86_64. Then everything that gets build goes past the QA team that verify that everything works as it should. From all the build packages install media will be created and these also need to be tested by the QA team. For each release a set of release notes are created and these are translated in different languages (12 for 5.1). Finally all the packages and media need to be uploaded in distributed to the mirror network so you can download it .

So this is why it takes a couple of weeks for a CentOS release to come out and remember that all this is done by volunteers and we could always use some more. So if you have some spare time and are willing to help you can make yourself known in the centos-devel mailinglist.

Tunneling NFS4 over SSH

2008-05-22T19:26:17+00:00

Today we had the need to mount a filesystem from a system that was almost completely isolated and instead of having to transfer a huge amount of data over a tunneled SSH connection, I thought, why not pursue mounting NFS over an SSH tunnel.

Since NFS4 by default does TCP if both client and server can do that, this would be the perfect opportunity to test the new capability. In fact, it should not be hard at all.

Consider the folowing situation:

some-server (EL4) <-> mgmt-server (Solaris) <-> nfs-server (EL4)

So we connect to our server using SSH from the mgmt-server using:

ssh -R 3049:nfs-server:2049 some-server

If "AllowTcpForwarding yes" is set in your sshd_config, this will create a tunnel back from our server to our nfs-server over the mgmt-server SSH connection. So that connecting to port 3049 on some-server will take us to port 2049 on nfs-server.

If you have a dedicated management server, you may want to hardcode this in your ~/.ssh/config as:

Host * RemoteForward 3049 nfs-server:2049

On the nfs-server side, things become a bit more complicated. Configuring NFS4 is a bit different than what I was used to do. Look at the next example config:

/srv/nfs *(ro,sync,insecure,hide,no_root_squash,fsid=0,no_subtree_check) /srv/nfs/share *(ro,sync,insecure,nohide,no_root_squash,fsid=1)

The difference is that the export with "fsid=0" is considered the root of the exported directories. No longer does NFS expect directories to be exported with the same location on the NFS server.

The downside is that you may have to bind-mount your real path to the tree that you export. In my case I would have to do:

mount -o bind /path/share /srv/nfs/share

And as a result, /srv/nfs/share will be exported as /share.

All nice and dandy.

Yes, now let's do the mount:

mkdir /path mount -t nfs4 -o port=3049,hard,intr localhost:/share /path

And this should work. At least if the permissions are set correctly. If you do have problems, the kernel messages and mountd message in /var/log/messages usually give a good indication of what the cause is. If you are unlucky, nothing is logged and it becomes guesswork.

Update: My original article indicated that doing this was not completely possible. But the problem was related to the new NFS4 configuration.

The kernel collection

2008-05-22T02:17:42+00:00

If you maintain a number of older RHEL or CentOS 3 and 4 machines, you’ve probably got a few extra kernels lying around to clutter up your /boot partition. In some instances this can cause update issues, and I ran into one such case today. An admin came to me asking why yum was attempting to install all of his packages to his /boot partition, and when I examined further, I saw this on his screen:
installing package kdegraphics-3.3.1-9.el4_6 needs 3MB on the /boot filesystem
While the error itself does look at first glance as the admin described it, this is not the case. The culprit was a kernel update further up the screen, and a 98% full /boot partition, with around 25 spare kernels. His system was simply informing him that the transaction would not occur because one of the updates was not going to succeed due to limited disk space.

While the InstallOnlyN plugin for yum will handle this quite nicely for the day to day stuff, once it’s happened it can be a little tricky to resolve. The easiest way is with a script called kernel-prune from Seth Vidal’s duke directory.

By piping the output of this script through xargs, you can remove all the tedium of manually removing packages one or two at a time. For RHEL and CentOS 3, where installonlyn isn’t really an option, this is pretty much the easiest way to periodically purge some unwanted fat from your system. Hope this helps at least 1 of you out there.

Feel free to share your own methods or comments below. I’m sure there are other methods out there so let’s hear them!

CentOS reference guide

2008-05-19T18:10:03+00:00

SElinux is a phenomenal way to protect your systems, and very few people disagree with this. The biggest complaint I hear is that it’s not user friendly. Most people seem to treat it like a binary system, and either leave it on, or turn it off. There’s very little documentation about the ins and the outs of selinux contexts and the targeted rulesets which ship with RHEL and CentOS. After some discussions with Ralph this morning on IRC, he’s graciously put together a list of the base contexts which ship in the targeted rule, and a brief explanation of what they do. If you want to take a few minutes to look through the granular protection possible through selinux, have a quick read of the new documentation at http://wiki.centos.org/TipsAndTricks/SelinuxBooleans

If you’re on IRC, feel free to stop by freenode’s #centos channel and thank Range for putting this list together.

Ubuntu's need to catch a wave

2008-05-17T22:38:58+00:00

Let me play devil's advocate here. Mark Shuttleworth's recent pledge to join a synchronised release plan for Enterprise Linux distributions is no more than a wish to benefit from a lot of work that Novell and Red Hat are already doing in the Enterprise space.

Let me explain.

Red Hat's Enterprise Linux offering is a very important proposition to businesses, not only because Red Hat guarantees (and has proven) to support each RHEL version for 7 years after general availability, but mostly because each version is updated with new hardware support every 6 months.

Why is that important ?

When a version comes out all components that comprise a distribution have been frozen half a year before and have been tested very hard. But hardware does not freeze and companies buy new hardware constantly. Red Hat needs to provide support for newer (selected) hardware from vendors to make its solution acceptable to customers. This work is not taken lightly. Red Hat has to backport fixes and backport kernel infrastructure, update drivers, perform regression tests and provide QA together with hardware vendors.

Every 6 months a new update release is being made to ensure that newer hardware is ready to be deployed with a recent RHEL version. Red Hat guarantees that new boot media is available with newer hardware support for another 3 years. That means 2 RHEL releases at every point in time are being maintained to support new hardware.

This picture explains Red Hat's Enterprise Linux offering best.

Click the picture to see it in all its glory

So what are you getting at ?

The sheer manpower to do this, together with new development and bugfixing for 4 different RHEL versions is something that Canonical/Ubuntu simply cannot take upon it. Even Novell does not support that many released versions in the way Red Hat does.

So ?

So Mark's article is wishful thinking and hoping to ride the wave that Red Hat (and Novell) are funding. If he can use that same kernel, with the same backports, fixes and regressions tests, Ubuntu LTS does not need to do anything to support the same vendor hardware. Easy, but at the expense of both Novell and Red Hat.

That explains why Mark wants 2 out of 3 Enterprise Linux distributions to enter his circle. I could imagine Novell and Mandrake joining forces to align the release cycle to try and take on Red Hat's 80% Enterprise Linux market share (sales, not necessarily install-base). There is hardly any benefit in Mark's proposal for Red Hat.

Also, Mark's Ubuntu timeline also resembles CentOS's timeline. And I guess he must have been studying Red Hat's release cycle very hard to come to his conclusion. (CentOS' timeline obviously comes from Red Hat's release dates)

Let me add that CentOS is not really involved in this discussion as CentOS does not directly compete with Red Hat (RHEL) or Canonical (Ubuntu LTS) in this market segment. Both offer paid-for services and support, while CentOS is provided as-is.

Update: I updated the CentOS timeline to include the upcoming RHEL6/CentOS-6 release.

Impact of the Debian OpenSSL vulnerability

2008-05-15T21:23:00+00:00

We have posted a warning about the impact of the Debian OpenSSL vulnerability on the CentOS-announce list, but I think it is useful to repeat it here (for readers of CentOS Planet) as well:

A severe vulnerability was found in the random number generator (RNG)
of the Debian OpenSSL package, starting with version 0.9.8c-1 (and
similar packages in derived distributions such as Ubuntu). While this
bug is not present in the OpenSSL packages provided by CentOS, it may
still affect CentOS users.

The bug barred the OpenSSL random number generator from gaining enough
entropy required for generating unpredicatable keys. In fact it
appearss that the only source for entropy was the process ID of the
process generating a key, which is chosen from a very small range and
is predictable. As such, all keys generated using the Debian OpenSSL
library should be considered compromized. Programs that use OpenSSL
include OpenSSH and OpenVPN. Note that GnuPG and GNU TLS do not use
OpenSSL, so they are not affected.

This vulnerability can affect CentOS machines through the use of keys
that were generated with the OpenSSL package from Debian. For
instance, if a user uses OpenSSH public key authentication to log on
to a CentOS server, and this user generated the key pair with a
vulnerable OpenSSL library, the server is at heavy risk because the
key can be reproduced easily.

Additionally, all (good) DSA keys that were ever used on a vulnerable
Debian machine for signing or authentication should also be considered
compromized due to a known attack on DSA keys.

As a result of this bug, everyone should audit *every* key or
cerficicate that was generated with OpenSSL, to trace its origin and
make sure that it was not generated with a vulnerable Debian OpenSSL
package. Or in the case of DSA keys care should be taken that they
were not generated or used on a system with a vulnerable OpenSSL
package. Keys that are potentially compromised should be replaced with
strong keys.

The Debian Wiki[2] has a preliminary list of affected application. A
tool to detect potentially weak keys is also provided, but it contains
an incomplete list of affected keys and can give false positives.

The Metasploit project provides a full list of weak keys in various
configurations[3].

Questions on how this may affect CentOS users should be directed to
the CentOS users list. List subscription information is available
from:

http://lists.centos.org/mailman/listinfo/centos

With kind regards,
The CentOS Team

[1] http://www.debian.org/security/2008/dsa-1571
[2] http://wiki.debian.org/SSLkeys
[3] http://metasploit.com/users/hdm/tools/debian-openssl/

Abusing MySQL

2008-05-08T17:21:19+00:00

One of the larger complaints about mysql for me has always been the hoops required to find out basic information. I want to check my GRANTS periodically to check permissions. I want backups to not take forever, and I want to be able to use find. Turns out, I can have everything I want and all I have to do is use maatkit.

I discovered this a few nights ago while reading through Jeremy Cole’s blog and noticed some of the functionality he was talking about. While I don’t fully trust their repositories (I prefer to stick with the distro mysql releases), I was very interested in the maatkit functionality.

After talking with Karanbir and waiting patiently for a few minutes, I had a nice shiny build of the maatkit MySQL toolkit in the testing repository to play with. It works exactly as advertised, and I could not be happier with it.

The majority of the maatkit tools are geared toward mysql replicated environments, so there’s a load of master/slave tracking capability including some new ways to track latency, lag, and more. This is all neat, but it’s not something I’m overly interested in just yet, because I don’t have any replication problems.

What does interesting me is this: mk-find

mk-find is essentially a perl implementation of find for mysql. I can now search for and sort tables by size, row, create/modify time, and perform actions based on what returns with –exec.

Another tool within maatkit which caught my eye was mk-parallel-dump, which while exceptionally long to type in (yes, I’m lazy) is quite a good tool for speeding up backups. Instead of simply dumping the entire db all at once into a single file and waiting for it to spool through huge amounts of information, I can pass this utility a directory, run a thread per table, and cut my backup time by a significant percentage of time. Simply run: mk-parallel-dump –tab –basedir /path/to/backups/ and you’re off, but if you want to get even more hi-tech, they include some sample scripts for scheduled backups.

If you’re looking for a way to make your mysql management tasks easier, have a look at maatkit, or just go get it from the CentOS-Testing repository and see how it can simplify your life.

Lirc capable remote

2008-05-05T13:42:12+00:00

Does anyone have a Lirc ( http://www.lirc.org/ ) compatible remote they would like to recommend ?

- KB

IBM Director 5.20.2 agent setup on CentOS/EL

2008-05-02T08:13:58+00:00

I’m used to deploy IBM Director server/agents on IBM hardware to monitor hardware/services .. and surely due to the fact that i work for an IBM business partner and that i give myself the IBM director course for IBM …

But there is something really anoying : each time you receive a IBM director cd/iso image (like the 5.20.2 that you can download from the IBM support website), it should normally contains the Linux level 2 agent for each of the supported Linux distributions (aka RHEL 3,4,5 , SLES 9,10 and Vmware esx). You can even integrate such agent in the director console to push it to remote machine (in fact it will do it through ssh … so be careful if you tuned sshd to accept only specific user/key-based auth …)

But last time i had to deploy it on CentOS machines (usually a simple change in the /etc/redhat-release file is enough ) i did it from the director console … Task was marked as successfully but nothing was installed .. (how the hell could director answer me that it was successfull if it was not the case ?) . Okay, let’s do it manually then … but then i saw that the level2 agent located on the CD (director/agent/linux/i386/FILES/dir5.20.2_agent_linux.sh -x) contained only the RHEL3 and SLES10 RPMS inside ! WTF ?

You can download the full Director Linux agent 2 package on the IBM website and that one will contain all the required RPMS …

Linuxtag 2008 update

2008-04-29T22:01:00+00:00

The meeting in #centos-social takes place on May 4th 2008 and NOT March 4th 2008. Looks like there are still some intellectual property issues with CPTM (CentOS Public Time Machine), so we had to reschedule that event.

We hope to have the machine ready for Linuxtag 1875, though.

Sorry.

Linuxtag 2008

2008-04-29T21:52:00+00:00

Linuxtag 2008 is coming closer by the minute (May 28th to May 31st 2008) - and we are going to be there. And so can you - either as a visitor or with us at the booth we have there!

To coordinate this event, there is going to be a “Meeting” in the IRC channel #centos-social on the freenode IRC network. The meeting will take place on Sunday, May 4th 2008 (not March 4th) at 22:00 CEST (that’s 20:00 UTC). Just connect to irc.freenode.net with an IRC client and /join #centos-social then.

So if you want to be part of Linuxtag 2008: Be there on sunday! Or you could subscribe to the centos-promo mailing list and discuss matters there.

See you!

Patch your kernel without rebooting

2008-04-24T23:52:00+00:00

Now this is way cool. It’s a way to apply a security patch against your kernel without having to reboot — which is one of the reasons why people normally hesitate to update the kernel.

This doesn’t work with every security update, as ksplice cannot work with patches updating data structures in the kernel code, but 42 of the last 50 kernel security patches wouldn’t have required a reboot.

I hope that the distributors take a sharp look at the mechanisms behind ksplice and incorporate that into their (enterprise) products.

CentOS 5 and aide

2008-04-10T15:41:42+00:00

In recent days, the subject of intrusion detection systems for centos has come up. To cover this and hopefully help some folks out, I’ve decided to do a brief writeup of Aide, the IDS which comes with CentOS. Please don’t confuse this with SELinux. SELinux is a Mandatory Access Control style permissioning system. SELinux stops people from getting into your system via protected applications. Aide lets you know if they actually get beyond SELinux and onto your system.

Installing Aide
yum install aide
What? You expected it to be harder? Now that we have aide installed, we need to configure it. The default config file should be okay for most folks who haven’t relocated things on the distro too much. Double check to make sure that all the directories you want to scan are listed. If you want to fine-tune the aide config, then you’ll need to edit /etc/aide.conf.

Initializing Aide’s Records

The next thing we need to do is create the initial aide database. For this, you need to run the following command:
# /usr/sbin/aide --init

This will take a little bit of time to run, and you’ll have some disk churn for minute or two while aide investigates your system and creates a baseline. Once this is done, we’re going to run an initial query of the system, just to make sure that everything’s working properly. To do this, run the command below:
# cp /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz # /usr/sbin/aide --check

This copies the initial database to the current database, then checks them against each other. In theory you should not have any differences. If you do, investigate them. As we’re still setting this up, they’re likely to be mundane .viminfo files or something similar. Keep in mind that when you update applications via ‘yum update’ that you may see aide go a bit nuts, just as tripwire or others would. You’re replacing files on your system when you update, and this is exactly what aide is designed to warn you about. In a perfect world, you should get some output like the text below:

# aide --check AIDE, version 0.13.1 ### All files match AIDE database. Looks okay!

Once we’re satisfied that aide is working as we expect, it’s time to set up a periodic check of the system. Only you can determine what’s often enough for your servers. I personally run aide as weekly cron, by creating a file in /etc/cron.weekly/ called aide.cron, with the following contents:

#!/bin/bash /usr/sbin/aide --check | /bin/mail -s "Weekly Aide Data" email@host.com
This runs my check once a week. That’s pretty much it to setting up aide. If you want to see more options for aide, please check out the documentation in /usr/share/doc/aide-*/

Update:

So it seems that by default, aide requires selinux to be enabled, or at least permissive so that it can record the selinux contexts of the files it watches. If for some reason you really, truly want to have selinux disabled, but you still want aide to watch the system, use the config file below. It is identical to the default scan, but with the selinux bits removed.

selinux-free.aide.conf

CentOS vendor support

2008-04-09T09:13:00+00:00

Official vendor support for an operating system contributes highly to the visibility of a system. Therefore it is very encouraging to see that VMWare is planning to support CentOS as a guest and host(?) system in its upcoming VMWare Workstation 6.5 product. Kudos go out to VMWare for planning to support CentOS, as well as releasing guest OS tools under a free software license.

Of course, we would love to see more vendors supporting CentOS. And given the fact that we try to be fully binary compatible with our upstream vendor, it should not require retraining of support personnel or much additional effort. It's surprising to see that some vendors do not support CentOS even when their infrastructure or developers rely on CentOS. Of course, many vendors will create their offerings based on customer demand. So, don't hesitate to speak up, and ask your software vendor to support CentOS. Maybe even drop a few lines on why you prefer CentOS over the operating systems that they do support (such as stability, long term support, etc.). Finally, let the community know if a major products starts supporting CentOS, other people may have been waiting for support as well (and as a kind "thank you" to that particular company).

ssh oneliners

2008-04-08T17:22:38+00:00

Some handy commands to remember when you really need to abuse ssh.

ssh -X remotehost # yawn. X forwarding through ssh.

ssh -Y remotehost # trusted X forwarding through ssh. Still yawn, let’s do something fun.

ssh -D2222 remotehost # This is okay. This command sets up a SOCKS proxy on port 2222 which can be used with firefox (and Internet Explorer if you really hate yourself) to avoid office internet filters…. not that I condone such anti-social behavior.

ssh -L 3306:database.example.net:3306 # okay, now we’re getting interesting. This generates an ssh tunnel between your machine and the remote box on port 3306. This works for connecting to remote mysql instances when firewalls would ordinarily interfere. Simply point your mysql client to localhost:3306 and you’re off and running. This can also be applied to other applications as well. A slight modification, and you get the string below:

ssh -L3389:remote.win2k3.server:3389 user@remote.linux.box # This command is a variation on the command above, allowing us to connect to those unsightly windows machines via rdesktop for remote administration. Best of all, we do this without opening up the remote desktop ports to the outside world. Remember folks, that windows code is expensive, you have to keep your precious little snowflakes safe after all.

With the commands above, you can alternatively add -f , if you want ssh to go to the background after the authentication portion is handled. Otherwise it’ll just leave you sitting at a remote shell prompt. If you wanted to add a built-in self-destruct, you could add ’sleep 30′ after the ssh command strings above. This tells ssh to exit after 30 seconds if nothing has made a connection via the tunnel created.

ssh -nNT -R 2222:localbox:2222 remotebox # This command lets you create a reverse ssh tunnel, so that if you connect to remotebox:2222, you’ll be connected to the local machine on port 2222 also. This is useful when you really want to go home at night, but your boss demands you keep working. This way, you each get what you want, and you can avoid the firewall your office employs to keep folks from remotely connecting to…well, if you’re using this, you don’t really care.

Crap, what if I already have an ssh session open, but I forgot to create the tunnel? Not to worry, there are escape keys to rescue you. Operating a little like screen, ~C will open an ssh command prompt so that you can start or stop tunnels as needed. It’ll look a bit like the one below:
[jperrin@server ~]$ ssh> help Commands: -Lport:host:hostport Request local forward -Rport:host:hostport Request remote forward -KRhostport Cancel remote forward

As always, we’re just scratching the surface of what ssh can do, so if you want more information, fire up ‘man ssh’ in your favorite terminal and sit down for a good read.

Customize SSH

2008-04-08T16:04:58+00:00

SSH is a staple of *nix environments, however most people don’t take the time to customize it in order to take advantage of the deeper features. If you have more than a few machines to log into or administer, it’s pretty likely that you’ve also got a few usernames to go along with them. By creating a personal ssh config file, you can shave a few seconds and a few keystrokes off your logins. You can even create profiles for several user accounts on the same machine if you have specific task based logins.

To get started with your own personal config file, create and open ~/.ssh/config in your favorite text editor. The easiest way to organize this file is to break it up into host based sections, similar to the example below.
Host SessionName # Short Name you can use to start a session HostName server.example.com # FQDN or ip of the server User myuser # Unprivileged username for this session

Host Session2
HostName server.example.com
User myadminuser #this is a 2nd user with sudo access to the same server
IdentityFile id_rsa

Host server2
HostName server2.example.com
User yetanother # yet another user on another host.
IdentifyFile second_rsa_key
Compression yes #server has limited bandwidth, so enable compression

Host backup
HostName backups.example.com
User backupuser # backup user
BindAddress 192.168.1.2 #Connect from this ip address.
Port 2222 # Use this port instead of 22.

As you can see from the example above, we change ssh keys so you don’t have to use the same one, ssh usernames to eliminate the @, and the address ssh connects from since we have multiple IP addresses on this system. This isn’t really even scratching the surface of what you can do with ssh, so if you have even more customizations in mind, have a read through the ssh_config manpage in your favorite terminal.