Planet CentOS

Dag Wieërs: Conferences and conferences

Tue, 26 Aug 2008 00:12:36 +0000

The past week was a very busy one. While the Olympic games were held in Bejing I was breaking my personal records for number of presentations at a conference in one day (3), number of presentations in a week (5) and number of conferences in a week (3).

Because JRSL in Buenos Aires was just before FrOSCon, and there was also a Debian Days in Buenos Aires. I flew to Buenos Aires on Friday afternoon, arrived on Saturday morning, I went to DebianDays on Monday, JRSL on Wednesday where I gave 3 talks (CentOS, RPM packaging and Dstat), flew back on Thursday morning to arrive on Friday at noon in Zaventem to be in Bonn on Friday evening to present a duo-presentation about Proxytunnel (Punching holes in the corporate firewall) on Saturday and a duo-presentation about giving presentation with the wiimote on Sunday.

And on Monday back to work in Antwerp...

At the Debian Days I met both Wouter Verhelst and Kurt Roeckx (Q voor de vrienden), which was a surprise (for them more than for me as you would expect ;-)), and I learned some interesting things about Debian from Wouter's talk. Probably more about that later if I find the time.

The presentations at JRSL in Buenos Aires went well, but only the Dstat presentation attracted a lot of people. (80?) I had expected more people for the two other presentations (40?), but for some reason the conference did not attract the number of people they expected to have. Wednesday seemed to be a pretty slow day at JRSL, sadly it was the only day I could be there (hence why I had all 3 presentations on one day :-/).

The presentations at FrOSCon both were very successful, the proxytunnel presentation I did together with Mark Janssen was crowded. I think we had about 200 people in a room that had 140 places with people sitting on the stairs and in front, and people standing in the back and at the side. And a lot of people had to miss the presentation because of that. For a conference it is hard to predict which presentations attract a lot of people and even we were surprised since Proxytunnel mostly is self-explanatory (at least we'd like to think the documentation is very good ;-))

The talk also got a lot of response and nice ideas at the end, so it was an interesting experience for the first Proxytunnel talk I did. I was very eager to demonstrate the proxy bouncing implementation that I wrote earlier. The whole reason for doing the talk was to promote those features of Proxytunnel that a lot of people hadn't heard about.

The "Giving presentations using the wiimote" talk together with Geerd-Dietger Hoffman was a lot of fun to do. Not only during the presentation (where we poked fun at each other), but also creating the slides and coming up with the ideas on how to use the Wiimote's technical components to the fullest in this specific domain. And even when we thought we had it all covered, the audience apparently could think of something else !

As I said at the end of the questions: once again the audience outsmarted the presenters. But to our defense, they also outnumbered us ;-)

FrOSCon uses Pentabarf, the perfect and versatile conference management system with the name that sucks. And this is the only conference I know that actually gets feedback about presentations and shares them with the presenters completely transparently. It allowed me to compare how we did with both presentations as well as the presentation of last year.

Looking at the feedback of both presentations, I am a happy man. Up next is T-DOSE and LinuxWorld Expo !

PS Thanks to both Adrian Alves (JRSL) and Andreas Kupfer (FrOSCon) for giving us the opportunity to do this.

Ralph Angenendt: Froscon 2008

Fri, 22 Aug 2008 22:46:00 +0000

If you are one of the people who are in St. Augustin for Froscon 2008 and if you are reading this blog or planet.centos.org, then you are the person who wants to come to the CentOS booth and have a talk with us!

See you all there on Saturday, 23rd of August or Sunday, the 24th of August.

Karanbir Singh: CentOS position on systems intrusion at Red Hat

Fri, 22 Aug 2008 22:40:28 +0000

Earlier in the day today Red Hat made an announcement [1] that there had been an intrusion into some of their computer systems last week. In the same announcement they mention that some of the packages for OpenSSH on RHEL-4 ( i386 and x86_64 ) as well as RHEL-5 ( x86_64 ) were signed by the intruder. In their announcement they also clarified that they were confident that none of these, potentially compromised, packages made their way into or through RHN to client and customer machines. As a security measure a script [3] was made available along with a semi-detailed description of the issue [2].

We take security issues very seriously, and as soon as we were made aware of the situation I undertook a complete audit of the entire CentOS4/5 Build and Signing infrastructure. We can now assure everyone that no compromise has taken place anywhere within the CentOS Infrastructure. Our entire setup is located behind
multiple firewalls, and only accessible from a very small number of places, by only a few people. Also included in this audit were all entry points to the build services, signing machines, primary release machines and connectivity between all these hosts.

Since OpenSSH is a critical component of any Linux machine, we considered it essential to audit the last two released package sets (openssh-4.3p2-26.el5.src.rpm, openssh-4.3p2-26.el5_2.1.src.rpm ). I have just
finished this code audit, and can assure everyone that there is no compromised code included in either of these packages. A similar check is also being done for the CentOS-4 sources.

Packages released today, by upstream, ( based on : openssh-4.3p2-26.el5_2.1.src.rpm, openssh-3.9p1-11.el4_7.src.rpm ) address two issues. Firstly they contain a fix for http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4752 . And secondly, in the remote event that someone had indeed got compromised packages via RHN, their packages would get updated to a known good state. We wanted to get these packages out right away to address the first issue, and also to cover users converting non updated RHEL installs to CentOS in the next few weeks/months. Release of these packages into the mirror.centos.org network does *not* imply that CentOS users are affected by the intrusion at Red Hat.

Finally, while we feel confident that there is no possibility of this compromise having been passed onto the CentOS userbase, we still encourage users to verify their packages independently using whatever resources they might have available.

--

[1]: https://rhn.redhat.com/errata/RHSA-2008-0855.html

[2]: http://www.redhat.com/security/data/openssh-blacklist.html

[3]: https://www.redhat.com/security/data/openssh-blacklist-1.0.sh :Its important to note that this script *only* checks for packages built within Red Hat, and will *not* be a reliable source of verification on CentOS since we rebuild from sources, using no Red Hat binary.

Russ Herrold: GnuPG -- A few minutes on using detached and clearsigned content

herrold@centos.org (herrold) — Fri, 22 Aug 2008 17:15:00 +0000

This is a re-formatted [and typo reduced ;) ] version, re-laid for the blogging software, of a post I made to the main CentOS mailing list earlier today. A test copy to verify of this which will properly verify is here, and may be retrieved with wget.

A few minutes on using detached and clearsigned content.

In light of today's CVE-2007-4752 by the CentOS project's upstream:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4752

I issue this brief piece on using GnuPG

1. View a proposed key to use, at the MIT keyserver

from: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x650D5882

2. Copy and create a local instance

[herrold@centos-5 redhat]$ vi rht-key

[herrold@centos-5 redhat]$ gpg --import rht-key
gpg: key 650D5882: duplicated user ID detected - merged
gpg: key 650D5882: public key "Red Hat, Inc. (Security Response Team)
" imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: 3 marginal(s) needed, 1 complete(s) needed, classic trust model
gpg: depth: 0 valid: 2 signed: 5 trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1 valid: 5 signed: 2 trust: 0-, 0q, 0n, 1m, 4f, 0u
gpg: next trustdb check due at 2009-03-14

3. Compute a local fingerprint of the candidate

[herrold@centos-5 redhat]$ gpg --fingerprint 650D5882
pub 1024D/650D5882 2001-11-21
Key fingerprint = 9273 2337 E5AD 3417 5265 64AB 5E54 8083 650D 5882
uid Red Hat, Inc. (Security Response Team)

sub 2048g/7EAB9AFD 2001-11-21

[herrold@centos-5 redhat]$

4. Compare and validate the fingerprint of the candidate against the RHT statement of the same fingerprint:

http://www.redhat.com/security/team/key/

5. You do NOT need to accept a key permanently to check signed content purportedly with it; consider the Red Hat notice at:
http://www.redhat.com/security/data/openssh-blacklist.html

6. We can retrieve the checking script

wget https://www.redhat.com/security/data/openssh-blacklist-1.0.sh

and the (presumptively) signed checksum of that file

wget https://www.redhat.com/security/data/openssh-blacklist-1.0.sh.asc

This is called a detached signature

7. And then we can validate ('--verify') that the signature and the file were signed by a person in possession of the private key.

Hopefully that private key is itself protected, as behind one way firewalls, and with a 'pass phrase' which matches a known public (which we retrieved and added earlier). This procedural security process is followed by me [one way firewalls, and pass phrases, and other CentOS team members], along with other measures.

[herrold@centos-5 redhat]$ gpg --verify openssh-blacklist-1.0.sh.asc openssh-blacklist-1.0.sh

gpg: Signature made Fri 22 Aug 2008 05:02:29 AM EDT using DSA key ID
650D5882
gpg: Good signature from "Red Hat, Inc. (Security Response Team)
"
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the
owner.
Primary key fingerprint: 9273 2337 E5AD 3417 5265 64AB 5E54 8083 650D 5882
[herrold@centos-5 redhat]$

8. As we have not indicated to gpg that we permanently trust this key, gpg adds the WARNING -- this is expected and correct under this outline. The validation checks out.

9. This file can be clearsigned -- the process we will follow is this:

[herrold@centos-5 .gnupg]$ gpg --clearsign import-key-howto.txt

You need a passphrase to unlock the secret key for
user: "R P Herrold "
1024-bit DSA key, ID 9B649644, created 2003-02-09

File `import-key-howto.txt.asc' exists. Overwrite? (y/N) y
[herrold@centos-5 .gnupg]$

10. That is, import-key-howto.txt is clearsigned, and a new file,
import-key-howto.txt.asc, is produced. As I did it twice, to add this text, the warning about Overwriting a file appeared.

11. This is a non-detached (clearsigned, file, and might also be tested by retrieving the indicated key contents, and doing a '--verify'

12. As I have previously certified my own key, I can do it more simply locally:

[herrold@centos-5 .gnupg]$ gpg --verify import-key-howto.txt.asc
gpg: Signature made Fri 22 Aug 2008 12:37:39 PM EDT using DSA key ID
9B649644
gpg: Good signature from "R P Herrold "
[herrold@centos-5 .gnupg]$

Note that the TIME of the signing will vary, as I have to resign the file after adding this content.

13. Previously (prior to 22 Aug 2008), I have included my PGP details in every piece of email I send. Starting today, as to email originate; I will add another line with my GPG details as well. I will send this document to the main centos mailing list.

Date: Thu, 21 Aug 2008 17:43:28 -0400 (EDT)
From: R P Herrold
To: trading-shim general mailing list
Subject: segmentation faults
In-Reply-To: 1219351509.12150.18.camel@gb07>
Message-ID:
References: 200808202117.m7KLH4rf011059@pippin.first.lan>
20080820224216.GA11712@localhost>

1219351509.12150.18.camel@gb07>
User-Agent: Alpine 1.999 (LRH 1145 2008-08-19)
X-M: Go Blue
X-OpenPGP-Key-ID: 0x7BFB98B9
MIME-Version: 1.0

In pine (alpine), one does this with Customized X-headers:

Customized Headers = X-M: Go Blue
X-GnuPG-GPG-Key-ID: ox9B649644
X-OpenPGP-Key-ID: 0x7BFB98B9

[hmmm -- a typo: o for 0 in the GnuPG line -- I'll fix that in alpine]

This piece intentionally does not address CentOS response; a preliminary statement on this has been posted in the /topic of the IRC channel #centos on irc.freenode.org, and I have done a blog posting which is up at: http://planet.centos.org/

- -- Russ herrold
herrold@owlriver.com
herrold@centos.org
security@centos.org

Russ Herrold: CVE-2007-4752 and CentOS

herrold@centos.org (herrold) — Fri, 22 Aug 2008 15:51:00 +0000

wearing my 'security@centos.org' hat, I have changed the IRC topic temporarily:

11:47 orc_orc changed the topic of #centos to: updated 22 Aug 2008 CentOS acknowledge CVE-2007-4752 and are reviewing our build and signing processes and hosts for signs of tampering subsequent to retrieval of SRPMs. // DO NOT PASTE IN HERE (unless asked; 1 line MAX), use http://pastebin.centos.org/ | See http://centos.org/irc | How to ASK a question: http://tinyurl.com/anel | CentOS mirrors:
http://centos.org/mirrors | Understanding Backporting:
http://tinyurl.com/r77l2

and had to temporarily omit:

Current Releases: CentOS 5.2, 4.6, 3.9, 2.1 | CentOS 5.2 now released

Fabian Arrotin: Extending a xvd virtual disk for a DomU machine on-the-fly ?

Thu, 21 Aug 2008 05:47:52 +0000

Recently i had to extend the space in one of my Xen DomU paravirt guest. I usually create a LV on the Dom0 that is presented to the DomU as a block device. Of course you can extend on-the-fly the LV on Dom0 but how can you tell to the DomU that the underlying block device was modified/extended ? Hmmm .. okay, people will point me to the fact that it’s possible to just create a new LV on Dom0 and give it live (aka block-attach) it to the DomU but that was not my question … Or they can tell me that shutting down the DomU and `xm create` the DomU again will work (and yes, it works of course) but that was not the goal …
On a real system (meaning non-virtualized) you can just rescan the scsi bus/adapter (or Fiber Channel if on a San storage through a LIP command) with just `echo ‘- - -’ > /sys/class/scsi_host/host0/scan ` (assuming that host0 is the adapter that has the device you modified/added/extended/whatever …) . So i expected to see the same behaviour in DomU .. but of course, block devices being not emulated because of the awareness of the DomU kernel, such command is invalid (no scsi_host even exists) .. Shocking !

Google pointed me to the answer (which didn’t satisfy me) on the Xen-users list (read the full thread) . So it seems not possible (anymore ?). Ouch … Okay, back to the alternative : instead of lvextend the LV on the Dom0, create a new LV and block-attach it to the DomU in which use LVM too to extend your VG/LV with a newly initialized PV …

Dear lazyweb, if you find me something that claims that it’s possible, let me know …

Russ Herrold: Let's get rid of disclaimers like this ...

herrold@centos.org (herrold) — Wed, 20 Aug 2008 20:47:00 +0000

... on mailing lists, as well. Or just subscribe and post from another email account. Or use more than a subject line to ask a question.

Email must be too hard for mere mortals to figure out.

Date: Wed, 20 Aug 2008 08:17:39 -0400
From: Mark T. Kennedy 
To: quickfix developers 
Subject: quickfix-d]  is there a new bug/issue tracker?

QuickFIX Documentation:
http://www.quickfixengine.org/quickfix/doc/html/index.html
QuickFIX Support: http://www.quickfixengine.org/services.html

/mark

This communication and any attachments may contain confidential/proprietary
information and is intended for information purposes only. It is not an
invitation or offer to purchase interests from Diamondback.  Any
representation to the contrary is unintentional.  This communication is
intended only for the person(s) to whom it is addressed.  If you are not the
intended recipient you are hereby notified that you have received this
document in error and that any review, dissemination, distribution, or
copying of this message or any attachments is not permitted.  If you have
received this in error, please notify the sender immediately by e-mail and
delete this message.  All e-mails sent to or received from this address will
be received by Diamondback's company e-mail system and is subject to
archival and possible review by someone other than the recipient.  This
notice is automatically appended to each e-mail message leaving Diamondback.

Where is my coffee cup, anyway?

Fabian Arrotin: NetworkManager and ipw3945 issue

Tue, 19 Aug 2008 18:10:01 +0000

I replaced recently my WiFi access-point at home and because the new AP (a Linksys WRT160n) supports WPA/WPA2 i tried to connect with WPA2 .. I had some stranges messages (in loop) from NetworkManager when trying to connect to the AP :

NetworkManager: Activation (eth1) Stage 2 of 5 (Device Configure) complete.
NetworkManager: Activation (eth1/wireless): disconnected during association, asking for new key.
NetworkManager: Activation (eth1) New wireless user key requested for network ‘$wlan-name’.
NetworkManager: Activation (eth1) New wireless user key for network ‘$wlan-name’ received.

I was sure that the PSK was correct because i was able to connect with both my eeepc and my e51 nokia mobile phone.

Querying the great oracle (translate to `using google`) told me that a *lot* of people have the same issue with the ipw3945 wireless nic. (independently of the linux distro : CentOS, Fedora, OpenSUSE, Ubuntu …..) but upgrading to a more recent wpa_supplicant (not available in the CentOS repositories !) package solved it for me.

Attention : The wpa_supplicant package available on RHEL/CentOS 5.2 is 0.4.8-10.2.el5 while Axel built version 0.5.8-16.el5 in his el5-testing repo .

As usual, read carefully instructions present on the CentOS wiki about the yum-plugin-priorities configuration or do like me : disable all third-party repositories and enable them only when wanted/needed

Fabian Arrotin: Tools to sync a RPM repository in your LAN

Tue, 19 Aug 2008 12:08:36 +0000

Due to Dag’s last blog post about his latest update to mrepo, we had several folks in the #centos and #centos-social irc channels asking how to configure it to just synchronize repositories on a server in their local network.

First of all you have to understand that mrepo isn’t only a repo synchronisation tool : it can help you to create deployment servers etc (see the mrepo features list )..

If you only need to sync repositories, you have other alternatives :

- rsync (if your remote mirror support rsync of course .. for CentOS mirrors that support rsync, see the centos.org mirrors list webpage)

- if rsync is not available, use reposync from the yum-utils package (available too on RHEL and works to mirror rhn internally because of the rhn-plugin available on RHEL yum version)

Just a reminder for people who forget that such tools (especially reposync) exist ..

Dag Wieërs: mrepo now with fuseiso and unionfs support (0.8.5 ready soon ?)

Fri, 15 Aug 2008 01:49:52 +0000

I am planning to do an mrepo 0.8.5 release very soon. For those new to mrepo, mrepo is a python tool that can download RPMs from repositories, but also from Red Hat Network and Yast Online Update (or CentOS or OpenSUSE for that matter), mount ISO images if needed, and create repositories out of it.

For the people that have heard of Red Hat Satellite, consider it a (free) light version that only covers downloading the updates and making it available.

However it does also take care of reporting newly available packages, as well as providing the necessary layout for doing kickstart installations from it. And if you know what buttons to push, also provides a nice TFTP layout for PXE booting.

In effect that means that on a single system you can download RHEL and SLES updates for all your architectures and versions on a single system and then make it available within your network.

Once set up it does everything by itself, but also allows to create your own repositories and, for instance, create staging and production repositories so you can cherrypick the things you want to distribute to groups of servers.

Now, there are many new features in this release (that is almost 2 years late). The most important ones are:

now ships with the rhnlib and up2date library code it works out of the box on non-RHEL systems (and RHEL5)
fuseiso support which allows mrepo to be used by a user (no root privileges are needed if you have fuse configured)
unionfs support which merges different discs to a single one
Yast Online Update support (youget)
rhnget can now be used to list all available packages and download some based on regexp matches
Speed ups with respect to repository updates

Even though mrepo has a lousy maintainer, it managed to get some fortune-500 companies to use it.

PS I hereby apologize to existing users for always promising the new release but never doing it. But this time the sooner you send me feedback, the sooner 0.8.5 is out of the door (and I have one less item on my TODO list for another 2 years ;-))

Russ Herrold: If a tree falls in the forest, and no one hears it, ...

herrold@centos.org (herrold) — Tue, 12 Aug 2008 15:40:00 +0000

... does it still make a sound?
-- folk equivalent of a Zen koan

hmmm. No smiley. Clearly sent by someone with a keen perception of the obvious:

Date: Tue, 12 Aug 2008 08:56:29 -0400
From: spamtools-owner @ lists.abuse.net
To: herrold @ owlriver.com
Subject: Spamtools recipient validation for herrold @ owlriver.com

This is a probe message to check the distribution of the spamtools list.
Please let me know immediately if you did not receive this message.

Regards,
John Levine, list meister

I'll hop right on getting that message out, right after another cup of coffee.

Russ Herrold: "We're going to need another Timmy!"

herrold@centos.org (herrold) — Mon, 11 Aug 2008 13:03:00 +0000

Mr. Lizard, Dinosaurs

A running gag on that show, and in IRC, the same.

04:37 msivak> umga9pej
04:37 msivak> hups
04:37 msivak> time to change another password ;)

Russ Herrold: score: pen one, orc zero

herrold@centos.org (herrold) — Sun, 10 Aug 2008 20:53:00 +0000

'Out, damned spot! out, I say.'

-- Lady Macbeth, Macbeth, Act V, Scene 1, Shakespeare

Came back from a trip out of town, and as is my usual custom, had all the dirty clothing on the top of the suitcase [for the TSA to appreciate digging through]. Now I am usually pretty careful to pull stray paper, change, and writing implements out of clothing as I disrobe. I missed a ball point pen this time, and in loading the laundry bin, missed it a second time.

We all know how this comes out, and indeed once the pen moved from the washer to the dryer, it opened up. Spots everywhere. Dr Suess would be proud, but no 'Voom' seems to be in our house. It spotted and gave its distinctive blue-black hue to the good towels, napkins, and other items which went through with a white summer weight cotton shirt in which pocket the pen was riding. There is probably nothing in the future of those towels than promotion to the 'rag box.'

But the issue remained of removing the ink from the dryer drum interior. I consulted Google, and a couple of commercial products were suggested, but it is Sunday, and I am not likely to go out again today. Household agents such as acetone (sometimes found in nail polish remover), denatured alcohol, Comet brand dry bleach powdered abrasive cleaner came to mind. Digging through the garage, I also came across an ether based starting fluid, and WD-40 brand spray lubricant.

Down to the dryer, and spot testing {ahem} began. Bottom line, alcohol on a paper towel and a bit of elbow grease triumphed.

Too late in the day for coffee, not late enough for Scotch. I'll go find a Miller Genuine Draft (bottled) in the 'fridge.

Dag Wieërs: Dag Wieers intelligent swipe at Ubuntu

Sat, 09 Aug 2008 02:15:57 +0000

Remember when I wrote an opinion piece about Ubuntu LTS titled Ubuntu's need to catch a wave ?

That night someone, nicknamed mapnjd, submitted the article to Slashdot with the above title (Dag Wieers intelligent swipe at Ubuntu) but I guess the Slashdot editors thought it would be a better headline if they phrased it Dag Wieers Scoffs at Coordinated Linux Release Proposal ... and overnight I became an Ubuntu-hater ...

Nothing is farther from the truth though. Fact is that I am part of the CentOS team and I do believe that in a lot of cases CentOS (or its cousin RHEL) is a very good fit, but that obviously doesn't mean that I am against Fedora or Ubuntu, or Ubuntu LTS.

What's even more, in my Enterprise Linux presentations I actually promote Ubuntu LTS in many ways. But the Slashdot article somehow symbolizes the one-track mind of its readers. Apparently if you write a critical piece about Ubuntu LTS's difficulties to catch onto an existing Enterprise market, you must have something against Ubuntu (and deserve dishonest Slashdot comments).

Well, let me clear that up a bit, I think Ubuntu is great because:

Ubuntu introduced Linux to a lot of people thanks to the free media they ship
Ubuntu got a lot of media attention around the world, something I doubt would have happened otherwise
Ubuntu's competition is indirectly a driving force within the Fedora community (and possibly other distributions) and positively affected desktop Linux progress
Ubuntu's upgrade path from (normal) Ubuntu to Ubuntu LTS makes it attractive for laptops and desktops to migrate (in time) to a stable environment (something CentOS/RHEL and SLES are lacking)

However, I still remain sceptic about the role Ubuntu LTS can play in the Enterprise market and whether Cannonical is able to create a viable business around Ubuntu LTS and indirectly sustain the funding of the Ubuntu project.

My biggest concerns for using Ubuntu LTS in the enterprise are:

Who pays for support if Ubuntu LTS is free ? Why not just pay the moment you actually need the support ?
Ubuntu LTS' support prices are much more expensive for comparable offerings, why would companies not use RHEL or SLES instead ?
How to turn a large community of free users in the consumer market into a happy paying customer base in the enterprise market ?
With only limited staffing, how to give comparable support as Red Hat ?
How to support a feature-full kernel, while Red Hat cherry-picks the feature they only support ?
How to support much more applications and features in Ubuntu LTS with considerably less staff than Red Hat or Novell ?
How to backport kernel patches against a 4 year old kernel with a limited staff of kernel developers ?
The confusion between (normal) Ubuntu and Ubuntu LTS in the marketplace diffuses search results and the difference is lost on many people, something I experience at conferences and tradeshows.

A lot of these concern question the long-term viability of Canonical/Ubuntu LTS given the extended support of 4 years and the funds needed to keep this up. Whether Canonical can get away with introducing Ubuntu LTS bottom up in large enterprises is questionable, although it might work in the SME market. I hope Canonical can find a sustainable business model because competition is important for Linux' growth.

Even red hatters and green geckos need to be kept on their toes.

Dag Wieërs: How I stumbled into Linux

Tue, 05 Aug 2008 23:33:11 +0000

Today I had an interesting conversation with a colleague about the Linux ~~provisioning~~ (how I dislike that word) deployment system we are developing at a customer. And in the midst of things he brought up how he started with Linux.

Apparently we share the same story, and I wondered how many other people were driven to Linux by frustration over some unexplained Windows bug at the time.

My story goes back to 1995, involved Windows 95 and an expensive CD burner I bought. I was already using Linux on a 80386, but that one was slower and did not have an internet connection.

In those days I specifically told people not to do dual-boot to learn Linux since rebooting from one to the other OS to find clues or get information clearly does get tiresome quickly. In those days Internet access on Linux was far from obvious and virtualization was unheard of (at least on Intel).

Anyway, using Windows 95 on the newer hardware with the CD burner caused Blue Screens of Death for 80% of the CD-Rs. As a student this was costing me a lot of money (and time). I fiddled with drivers, reinstalled Windows, used Soft-ICE to discover anything that could help, all to no avail. Windows was not giving much away and I suspected hardware related problems.

But dual-booting Linux on the same hardware made the CD burner and system stable, and I never trashed any of my CD-Rs again. Since I had spent so much time on Windows, which did not teach me anything about the inners of Windows and I was able to solve a few issues myself getting the CD burner to work on Linux, clearly taught me that Linux was much more efficient and educational for me than Windows was ever going to be.

I must say that in those days I had 2 very invaluable tutors available on IRC, both to which I am still grateful: Peter De Schrijver and Ulrik De Bie assisted me often with RTFM and answering questions with questions :-)

Although I stopped using Windows after the experience for myself, I still needed some Windows skills to keep my family's computers going, and those skills have proven to be useful from time to time, even if only to show I do know something about computers or to get free Internet access (by volunteering to administer the local libraries computers or to fiddle with Hotel paid-for Internet access clocks).

Life can take strange turns and often is unpredictable, however if it wasn't for the CD burner or the Windows 95 BSOD, something else would definitely have flipped me to Linux and Open Source. I certainly was lucky with the people surrounding me and the accessibility to technology.

Let me use this opportunity to thank everyone who was part of this, my parents, my family (and uncle Marcus), my friends, teachers and colleagues over the years. If you read this and we are out of touch, contact me or link me !

Fabian Arrotin: CentOS 5.2 on the Apple iMac

Tue, 05 Aug 2008 07:48:56 +0000

I’ve always heard that a picture tells more than long sentences ..

For various reasons (including the fact that i like the iMac design and that as a musician i have recording hardware that is only recognized/usable with Mac OS X), i decided to buy me a shining new Apple iMac 24″. But of course Linux remains my OS of choice ..

So i dediced to use it in dual-boot mode and i installed of course CentOS (don’t need to explain why i think … ) . I decided to use rEfit as the efi boot menu (better than the included bootcamp because to boot an alternative OS at boot you have to press a key, while rEfit always displays a boot menu and boots a (configurable) default OS)

I’ll write of course a page on the CentOS wiki explaining in details what has been tested, what works and what doesn’t … One little note about the setup : i *always* setup linux through the network (with or without kickstart) so i tested the netinstall boot.iso on the mac. I had to play with some options : for example anaconda is always trying to mount the cdrom before asking you which method you want to use (you can of course use the ‘method= ‘ to override this behaviour though.

But i noticed that it was really slow to ‘inspect’ the cd .. using the option hda=ide-scsi helped me for the setup (i installed from my local nfs repo )

So the full line i used (you specifiy more paramaters of course) was : “linux vnc hda=ide-scsi”

More informations to come on the CentOS wiki …

Dag Wieërs: Undeleting an open file by inode

Mon, 04 Aug 2008 16:49:40 +0000

At a customer today I was confronted with a situation where VMware ESX processes had log-files open that were already deleted. This can happen when logrotate was incorrectly configured, or when operational staff removed big files to clean up diskspace quickly.

However when the file is still open, you can remove the file-entry (link) to the inode, but the diskspace will not become available until all kernel references to the inode are gone (and a process having the file open counts as a reference too).

Killing the VMware processes that had the file open was not an option since we just wanted to truncate the file without impacting the guests.

There are 2 possibilities to truncate the file. The first one is very easy. First we have to find the entry in the /proc filesystem:

[root@system root]# ls -l /proc/2334/fd/ | grep delete lrwx------ 1 root root 64 Aug 4 18:34 3 -> /path/to/vmware.log (deleted)

Now we know the file-descriptor, we can do and truncate the (unreal) symlink to the inode:

[root@system root]# echo -n >/proc/2334/fd/3

Now this procedure is safe (given that the process did not do anything with the file-descriptors in between), but it will not help with recovering the file itself (in case you wanted that).

So the longer procedure involves debugfs. By using lsof one can find the inode of the open file:

[root@system root]# lsof -p 2334 | grep delete vmware-vm 2334 root 3u REG 8,5 43003904 46087 /path/to/vmware.log (deleted)

This gives us the original location (/path/to/vmware.log) and the inode (46087) of the deleted file. Now we have to find out on what filesystem/device this inode is from. The easiest is:

[root@system root]# df -h /path/to/ Filesystem Size Used Avail Use% Mounted on /dev/sda5 1.7G 1.7G 0 100% /path

Beware: the below commands are not for the faint of heart. If you do something wrong you might trash your filesystem, crash your system or inadvertently kill a dozen puppies.

So we now know the device is /dev/sda5 and we can start creating a file-entry to the inode on the correct filesystem by doing:

[root@system root]# debugfs -w /dev/sda5 -R 'link <46087> /path/to/vmware.log'

Now the file entry is back, albeit it has no links referenced to it, so it is still considered to be deleted by the filesystem if the file is eventually closed.

[root@system root]# ls -l /path/to/vmware.log -rw-r--r-- 0 vmware vmware 43003904 Apr 18 14:48 /path/to/vmware.log

Now you can access the original content, or you can truncate the file to free up diskspace.

[root@system root]# echo -n >/path/to/vmware.log

If you happen to know how one can also restore the number of links of that inode so that the file is in fact restored, please let me know. I tried using modify_inode <46087> but that did not update the inode information when leaving debugfs.

Dag Wieërs: CentOS 4.7 close to release

Wed, 30 Jul 2008 01:54:56 +0000

On the heels of the CentOS 5.2 release last month, the first spin of CentOS 4.7 went out to the QA team yesterday and if there are not too many hurdles in the QA process CentOS 4.7 might be out soon.

As a teaser, here is the upstream announcement as well as the release notes.

PS No need to ask when it will be out. If it is *that* important to you, you might as well do us all a favor and consider moving to RHEL !

Dag Wieërs: CentOS very much a-live

Sat, 26 Jul 2008 01:16:40 +0000

It did not get the media attention that it deserved, even though Linux Weekly News did pick it up: the new CentOS 5.2 Live CD has been released.

The CentOS Live CD is one of the important sub-projects of the CentOS project as it gives people the opportunity to test out CentOS' hardware support without the need to install it.

Admittedly, the reasons why CentOS is great for servers (API stability, 7 years of support, proven environment) sometimes interfere with its success on the desktop. And if you are not blessed by picking the hardware in function of CentOS, than the Live CD is a great tool to learn and appreciate CentOS on the desktop.

Again great work by Patrice Guay and the rest of the CentOS QA team !

Dag Wieërs: Stop software "piracy", support Open Source !

Tue, 22 Jul 2008 21:43:09 +0000

De Standaard is once again helping the BSA and Microsoft with their scare tactics, so let me give some counter-weight...

Is commercial software too expensive or "piracy" against your ethics ? Afraid of the BSA raiding your home or business ? Start using Open Source software ! For everything you do, a free ('no strings attached') Open Source alternative exists. And you can still pay someone, if you want to :-)

Karanbir Singh: Emergency trip to India

Sun, 13 Jul 2008 17:26:12 +0000

Just had news this afternoon that my Grandma, who was already in hospital for a few days, has taken a turn for the worse and the doctors are worried that she isnt improving as fast as she should. So, I am heading off to India to be with my family for a bit. I will still be on email and mobile phone ( prefer SMS rather than Voice ).

All the various projects I am involved with will still keep on moving along, maybe a bit slower over the next few weeks.

- KB

Johnny Hughes: Attacks on Package Managers - ummm...

Sat, 12 Jul 2008 16:25:26 +0000

In a recent article entitled Attacks on Package Managers, there are many things discussed by a group of Computer Science students (and maybe some instructors) at the University of Arizona. While I can not address how Debian protects their APT repositories or how Fedora (or anyone else) protects their YUM repositories, I can discuss how CentOS protects its update system used by default to deliver updates to users.

CentOS Mirrors

First, let me explain the CentOS mirror system. CentOS directly controls about 30 mirror servers from which we serve updates via yum and rsync to other public mirrors and to users directly. These mirrors are members of the CentOS.org domain and are totally controlled by the CentOS project. These mirrors can be totally trusted because only CentOS Project personel have login or update access to these machines.

A second set of mirrors are called the CentOS "Public Mirrors". These mirrors are monitored by a system called mirmon, the results are listed here . These public mirrors are also listed here in another format for ease of finding a close mirror. The first major fault in the study linked above is that these mirmon monitored mirrors are only the first step in being assinged to provide updates directly to CentOS users. Just being listed as a mirror DOES NOT MEAN that yum (as configured by CentOS by default) is going to use that mirror. Please see the mirrorlists sectionbelow for that selection process.

Mirmon uses a couple files within the mirror to verify that a mirror is doing updates and it is a "Coarse" test that we use to decide which mirrors will be subjected to the mirrorlist tests below. If you are rsyncing a local mirror of your own from one of the public mirrors, I recommend that you use more than one, then you can be sure one person is not in any way modifying anything.

Mirrorlists

The aspect of CentOS security which this study totally ignores is called the CentOS mirrorlists. This system is the one that is actually used (in a default setup as published by CentOS ) to deliver updates to users. CentOS uses a script to download a file called repomd.xml from every repo on every server listed as active in the CentOS Public Mirrors. Once we have that file, we check it against the same file from the master CentOS server. If the file from the public mirror is different than the file on our CentOS master mirror, then that server is not published on our mirrorlists.

The mirrorlist generation process runs non-stop in a loop testing each and every CentOS "Public Mirror" on every run. With the current number of public mirrors it takes a maximum of 2-3 hours for a mirror that does not have the same repomd.xml file in a repo to be removed from the mirrorlist.

CentOS does not just check one repo on a given public mirror, we check each and every repomd.xml file from each and every published repository on each and every public mirror.

The CentOS mirrorlist is only 10 servers long, even though we have about 200 mirrors listed in our list. There is a different list for each country where the 10 listed servers is geographically picked and if 10 public servers are not found, CentOS.org mirrors back fill the list to 10 servers. This means that every mirror listed is NOT even used in our mirrorlists.

If you are using the default CentOS update method, you can rest assured that you are being provided a geographically accurate (by country) and updated (tested and regenerated every 2-3 hours) mirrorlist.

The Study

Now I will discuss the issues brought up by the above article and discuss if these issues apply to CentOS.

First, lets discuss the Metadata Replay attack that they list. This attack is not at all a concern with CentOS if the default method (the CentOS mirrorlists) is used to do updates because every 2-3 hours if an individual repository on a mirror is not updated, it is not on the mirrorlist. If you are running your own mirror that you update from a public mirror, you can write your own script to download the repomd.xml file from the proper place at http://mirror.centos.org/ and you can check yours file against this file, if they are the same then you have a good set of metadata ... with that you will get the correct updates.

The next attack they discuss is the Mirror Control attack. This one is also NOT a problem for people using the default CentOS update system, since each and every repomd.xml file (the same one you get if you use yum and the default mirrorlist) is verifed on every mirror every 2-3 hours. Is it possible for someone to provide a fake file to the centos testing machine, and a different one to other people. Yes, if they know the IP address of each and every machine we might possibly use to test the mirror then they might be able to give us a redirected file and give everyone else a different metadata file. Even IF they did that, they do not have packages signed by a centos.org key. Because of our repomd.xml file checking, the likelyhood of this attack (as with the first one) is almost 0.

The other major problem that they discuss is a man-in-the-middle attack. Without using HTTPS, a person MIGHT be able to use a man in the middle attack. It is not a simple thing to do, and it does not get any "malicious" software (since CentOS requires signed packages) ... though it might be possible to list old files so that updates are not done. This would be a very hard thingto undertake just to prevent an update, though possible in theory.

The Bottom Line

Being listed as a "Public Mirror" DOES NOT MEAN that a system is being listed in the CentOS yum mirrorlist. CentOS does other checks and our mirrorlists are safe. You can easily make sure that your system is updated by running yum at consistent intervals and requiring all packages are signed.

If you are creating your own mirror, you can check your repomd.xml file against those at http://mirror.centos.org/

If you use packages signed by a centos.org key, you can be sure we released it. If you monitor the CentOS Announce mailing list you can see when Security Patches are released.

Should you be concerned about security updates and install them when they are released ... YES.

Is the sky falling ... NO.

Dag Wieërs: My rationale for the Nokia E71

Sat, 12 Jul 2008 16:20:20 +0000

Last week I bought a Nokia E71, a few days before the iPhone 3G was available in stores. You may think I must be crazy for not giving into Apple, but I have my reasons.

I had the following list of requirements:

Full keyboard (and not an on-screen keyboard)
OS that I could develop for (Symbian ?)
Not based on Windows
Needed Wifi, GPRS, UMTS
Wanted an SSH client (preferably putty)
USB connection and bluetooth
Small enough to fit well in my pocket

Looking at the Nokia E61i and E90, the E61i was lacking some features and the E90 is simply too big. The Nokia E71 actually combines a lot of both phones and is smaller, thinner and lighter. So when I read about the E71, I was sold.

The fact that Nokia is Open Sourcing Symbian is a welcome surprise as well, even when currently I am a bit disappointed about the availability of Symbian Open Source software. I hope a surge of Symbian developers can address that a bit, although I am happy with the Symbian putty, OGG player and Google apps.

I am still looking or hoping for:

OGG support included in Symbian
iCal integration in Symbian calendar
~~Task manager for Symbian~~ (Keep Home key pressed)
Good alarm clock application that can fade in and play OGG files (oggplay not sufficient)
Open Source VOIP application or Skype for Symbian
Pidgin for symbian
Open Office document support

I was also pleasantly surprised about how well the Nokia E71 keyboard worked. The raised keys on the keyboard makes it very reliable for keyboard input even on this small format. That was one of the more important reasons for not going for an on-screen keyboard and ignoring the iPhone.

The Nokia E71 is also 100 Euro less expensive than the iPhone, but at these prices I bet that does not make the difference.

I have bluetooth working with CentOS to access the micro-SDHC card or use it to have Internet access over UMTS from my laptop. A future blog article will detail how to do this.

Dag Wieërs: Package manager vulnerability study flawed ?

Sat, 12 Jul 2008 14:43:34 +0000

A study from the University of Arizona (recently posted on slashdot) looked at weaknesses in package managers (and mirror setup). By becoming an official mirror and delaying or stalling a mirror's updates they tried to lower the security of servers using that mirror and increasing the window of opportunity for a successful attack.

In itself it is very useful to make people aware of weaknesses in technology or abuse of trust, but in this case (and certainly for CentOS) I think they overstated the impact or at least ignored mechanisms used to prevent possible security risks.

By default CentOS uses yum with mirrorlist enabled. This means that instead of using a single mirror all the time, you are not using one, but different mirrors. This reduces the risk of a single mirror being out-of-date somewhat. But next to that CentOS has several tiers of mirrors depending on the update-frequency of each mirror (and the form of control the CentOS project has of those mirrors).

And the mirrorlist that CentOS users actually use is being created based on the correctness of the individual mirrors, we are continuously verifying mirror content, metadata, filesizes and signatures on checksums. This means that CentOS users are only working with an up-to-date mirrorlist and mirrors that stall or delay packages are left out of this mirrorlist. You can see how it works on the CentOS mirror status page.

Unfortunately this mechanism was not mentioned in the study.

The conclusion seems to be that any theoretical risk is very minimal and indirect, however some of the recommendations for improving the package manager's robustness should definitely be taken seriously by their developers.

Update: CentOS developer Johnny Hughes blogged about the same topic.

Karanbir Singh: CentOS and reissue of updated packages for CVE-2008-1447

Sat, 12 Jul 2008 13:00:08 +0000

Some people will notice that a second set ( i386 x86_64 ) of announcements were just made to address the issue raised in CVE-2008-1447 after the initial announcement ( i386 x86_64 ).

These are indeed newer packages based on bind-9.3.4-6.0.2.P1.el5_2 ( the original update was based on bind-9.3.4-6.0.1.P1.el5_2 ). Reason for this reissue from upstream is explained at : https://bugzilla.redhat.com/show_bug.cgi?id=454852 and I highly recommend you look at it. Specially if you run ipv6 on the wire.

Of-course it would have been nicer if upstream had issued another RHSA rather than just update the existing one with newer packages. I wonder if there were operational issues or release process issues to blame for this.

- KB

Karanbir Singh: puppet fact for CentOS Version

Fri, 11 Jul 2008 16:45:30 +0000

I looked and found nothing for this, so wrote this quick fact for puppet.

# centos_version.rb

Facter.add("centos_version") do
  setcode do
    %x{/bin/rpm --qf "%{version}\n" -q centos-release}.chomp
  end
end

Once its in your facts/ you can do things like this in your puppet manifests :
case $centos_version { 4: { ... } 5: { ... } default: { .... } }

- KB

Jim Perrin: Spacewalk’s first steps

Thu, 10 Jul 2008 15:12:50 +0000

A few days ago RedHat announced that they had open-sourced their satellite product under the moniker of Spacewalk, and I’ve taken a few days to play around with it and get some first impressions of what’s been put out. I do not by any means claim to be an expert on the RHN satellite from whence this came, or the current spacewalk incarnation. This is simply meant to be a linux enthusiast’s first look at the product they’ve put out.

The directions to install spacewalk are very clear and relatively simple to follow. It only took about 10 minutes to get it up and running. From there, it’s a whole different story.

Spacewalk requires its own machine

The directions don’t say this, and it’s not really 100% true, but for nearly all real-world cases, it’s much simpler just to give it a box. There are two basic reasons for this. First up is that spacewalk drops a number of packages on top of other things you may be running, like apache. The spacewalk server setup drops in a ’satellite-httpd’ process instead of using the distro provided httpd package. Since RHN satellite was/is a boxed solution, this fact can be overlooked as I figure it’s probably something that will change as the project matures and gains popularity. The second issue with spacewalk is storage, which is primarily an organization based issue. Sure it’s going to take a few gigs of disk space to mirror your favorite distribution, updates and any associated 3rd party repositories that you might want. However:

Channels cannot cross organisations

This one kind of surprised me considering that RHN seems to do this just fine, though it’s probably due to a different back end. To illustrate this point a little, lets assume that we’re running the Spacewalk server for a university. The IT department has their own organization for the university infrastructure, with a CentOS5 channel for base, a child channel for updates, and another child for Extras. A fairly boring example to be sure, but a good foundation to work from. The CS department runs CentOS for this systems as well, using it for both instruction, and the servers related to instruction. They have require the exact same channels the IT department uses, but Spacewalk currently requires them to duplicate the entire tree; Base, Updates, Extras, all of it. If you expand this out for a few more organizations, and figure 20G or so per channel for the life of the distribution, you’re very easily looking at a few hundred gigs of storage. And while you’re busy pushing these packages to the Spacewalk server, you’ll be doing so manually.

Syncing Repositories

Part of the RHN satellite feature was that it would sync with redhat’s RHN proper, and then you could move out with your updates locally. The old RHN satellite would pull from RedHat’s RHN proper, and then you could manage your machines locally. With Spacewalk, the RHN sync capability was removed, and no base for syncing to other repositories (via yum, rsync or otherwise) currently exists. If you want to keep spacewalk updated with the latest and greatest for your distribution, you’ll have to script something up yourself. The methods for doing so are not difficult, and anyone with a basic grasp of shell scripting should be able to pull this off.

The good stuff

I really don’t want this to seem like I’m simply complaining about Spacewalk as it is a very good product, and RedHat did a good thing by releasing it. It simply has a bit of growing to do as it begins life as an open source project. There’s already a rather vibrant community springing up around it, both as a mailing list and in irc on freenode. Additionally, Spacewalk provides functional centralized management of multiple boxes across different distributions which is indeed quite useful.

If you’re in the market for centralized system management and you have a box with storage to spare, then I would highly recommend folks take a look at Spacewalk. It is definitely a project to keep an eye on as it matures.

Fabian Arrotin: CentOS 4.x machine not rebooting and faced with a grub prompt

Wed, 25 Jun 2008 13:44:18 +0000

One of my customer phoned me to say that one CentOS 4.x machine (acting as a apache reverse proxy) didn’t reboot after a power outage. The machine had two sata disks configured in raid 1 (through md/software raid) but instead of booting, the machine was just displaying a grub> prompt.

Of course i tried the traditional `grub-install –recheck /dev/sda` and `grub-install –recheck /dev/sdb` and also the manual procedure (already described here) to install grub on both devices .. but no luck .. still booting at the grub> prompt.

But then i looked (in rescue mode) at the (/mnt/sysimage)/etc/grub.conf and i counted 22 kernel entries in the file .. The customer had configured the nightly automatic yum update but he never cleaned the old kernels (both up and smp) … so i “cleaned up” the grub.conf file, once again installed grub with grub-install and …. machine rebooted normally ..

I’ve never thought that too many entries in the grub.conf file could block the machine from booting … Maybe that will save other people time

Ralph Angenendt: CentOS 5.2 has been released

Tue, 24 Jun 2008 15:25:00 +0000

The CentOS team is pleased to announce the release of CentOS 5.2 - the latest update for the CentOS 5 series.

Major changes in CentOS 5.2 compared to CentOS 5.1 are: Firefox 3, Thunderbird 2, OpenOffice.org 2.3 and Evolution 2.12 on the Desktop side, Samba 3.0.28, xen-3.2 and an upgraded kernel with lots of driver updates on the server side of the system.

Read the Release Notes, the Release Announcement and get it while it’s hot from a mirror close to you.

Karanbir Singh: 5.2 Release update

Thu, 19 Jun 2008 23:19:46 +0000

We found a very major issue with the last set of ISOS for 5.2 meaning I had to redo the distro isos today. We should start seeding the mirror network in the next 24 hrs time, so release should still be 23rd June, give or take a day or so.

UPDATE: 2008-June-23 : We found yet another issue with the x86_64 tree, so while some of the updates are now syncing out, please wait for the release announcement before pulling packages and the isos.

--
Karanbir Singh [ http://www.karan.org/ ]