Planet CentOS

If you are a web hosting company and deploying CentOS 5 or 6 images in a Cloud, I want to speak with you. Please get in touch via one of the mechanisms listed at : http://www.karan.org/contact.html

Alternatively, if you use services from a hosting company that uses CentOS images in a cloud : drop me a line and let me know about them, I will try and reach out directly.

Thanks in advance and look forward to getting in touch with some of you guys.

- KB

We implemented Puppet as a configuration management system at $work ,  and Puppet is a great tool. Then I heard about some dashboards that could be used on top of it. I've heard about different dashboards ($management_people *like* dashboards) like Puppet-dashboard and Foreman.

I was advised by several people to give Foreman a try and it's really simple to install. Their wiki covers basic installation and there is even a yum repo that can be used (Epel has to be enabled too). As i have a small network to manage, I decided to setup Foreman on the same host as puppetmaster. Configuring /etc/foreman/* is easy and missing parts can be configured just by looking at the Foreman website wiki/FAQ. But troubles came when I enabled reports : puppetmasterd config was changed to include :

[master]
reports = store, foreman

and the foreman.rb script (copied and modified from /usr/share/foreman/extras/puppet/foreman/templates/foreman-report.rb.erb) integrated in the correct /usr/lib/ruby/site_ruby/1.8/puppet/reports dir. (Note : don't forget to update $foreman_url).

But no reports were coming in Foreman. hmmm .... error message was :

Report foreman failed: Could not send report to Foreman at http://puppetmaster.mybeautifuldomain.com:3000/reports/create?format=yml: Permission denied - connect(2)

That was not an iptables issue, but selinux one :

type=AVC msg=audit(1329830711.788:28372): avc:  denied  { name_connect } for  pid=13144 comm="puppetmasterd" dest=3000 scontext=unconfined_u:system_r:puppetmaster_t:s0 tcontext=system_u:object_r:ntop_port_t:s0 tclass=tcp_socket

Here is my locally generated selinux for Foreman :

module foreman 1.0;

require {
type puppetmaster_t;
type http_port_t;
type ntop_port_t;
class tcp_socket name_connect;
}

#============= puppetmaster_t ==============
allow puppetmaster_t http_port_t:tcp_socket name_connect;
allow puppetmaster_t ntop_port_t:tcp_socket name_connect;

Things work really better after I added my foreman.pp selinux module on that host. If you don't know how to compile selinux custom policies, please read the nice Selinux page on the CentOS wiki, and especially the "Manually customizing selinux policies" section. Tools like sealert (from setroubleshoot-server package) and audit2allow are really helpful when there is no pre-defined selinux boolean that can be used.

Hope this helps .. and now going back enjoying reports, including error reports by mail (nice feature)

Looks like EL5.8 just got released upstream. Lots of sources just showed up on the ftp sites. Time to start up the CentOS-5 buildsystem.

The ELRepo Project has DRBD packages for CentOS-5 and CentOS-6, named drbd83-utils or drbd84-utils.  The CentOS Project does not want to maintain extra packages that exist in other places unless we need to change them ... so we are not going to create DRBD packages for CentOS-6.

Since CentOS-4 is being EOL'ed in less than a month, we are also not going to publish updates for the DRBD in CentOS-4.

This leaves the DRBD for CentOS-5 that are part of CentOS Extras.  Since these have been released for CentOS-5, we will continue to maintain the DRBD version 8.3.x  tree (drbd83) in CentOS Extras.

A new version of DRBD 8.3 (drbd83-8.3.12) has been released to the testing repository for CentOS-5.  You can see the details here:

DRBD 8.3.12 for CentOS-5

If you want to use DRBD 8.4.x for CentOS-5, rather than releasing it separately, the CentOS Project recommends that you use drbd84-utils from ELRepo (linked above).

For users who want to use the drbd83-8.3.12 version ... please test the version that is currently in CentOS Testing and provide feedback.  With enough feedback I will move the packages from testing to CentOS Extras.

I am often surprised by the sort of questions asked in the forums or on irc around open source projects - it just feels as if people are going out of their way to inflict pain and suffering upon themselves by trying to find the most awkward and most complicated way to do things. So how can we better help these people ? We dont need to save them or anything as drastic like that, its just a case of being able to show or explain that there might be a better way.

The first thing that I've started now doing, when asked a strange question is ask the person 'What are you really trying to achieve?'. You might be amazed how many times the answer has nothing to do with the question being asked. Try to establish what the end goal is, and in many cases its clear that the person has been lead astray by random posts on the internet, some of which are perfectly fine in their own context, but can be quite a kludge outside that context.

Establishing, clearly what the goal is before advice or opinion is thrown at people will always result in a better overall experience. And to the people spending their time in the irc channels, web forums and mailing lists helping others out : must respect. You guys are the ones making the idea of Communities and Open Source work.

- KB

While Johnny was explaining to the rest of the world how CentOS 6.1 and 6.2 were released, I received quite some questions about the QA tests and how they were performed. Well, let me explain in some words how it's now organized. Previously, there was only a Tests Matrix that was shared between the QA team members : each member of that group had access to the QA bits, could download/rsync the complete tree (with ISO images too) and do his tests, and then reported the results in one way or the other (irc, mailing-list). Of course it didn't scale out very well. Too much manual intervention, and when someone was busy with personal (or work related) issues, no feedback was coming back to the CentOS devteam.

So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

• The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
• There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
• Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
• Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
• A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !

Solid State drives provide a pretty substantial performance boost over traditional hard drive technology, but they have some limitations that require some additional planning. There are basically two big things to do, enable discard/trim support in the filesystem, and limit write operations to the SSD. You want to enable discard to deal with underlying drive specific performance degradation

Oracle seems to be dragging its feet in certifying the enterprise linux 6 line (RHEL, CentOS, and similar) for use with its flagship database products, but that doesn't appear to be stopping the smart folks over at dell from putting together directions for getting it working. Adam M has put together a nice piece on making Oracle and RAC work with RHEL 6, and done all the hard work for you. Have

The first thing I want to do is congratulate Karanbir and Tasha on the birth of their new baby girl Millie. She is the quite cute ... hello Millie :)

The CentOS Project has spent much time and effort into getting a new build system in place for CentOS 6 that can generate good and timely builds, as well as inform us of newly released upstream SRPMS and keep the CentOS QA team informed when we build any new packages.

The release of CentOS-6.2 on 12/20/2011, in less than 2 weeks and at the same time as Oracle's OEL as noted on Distrowatch, is where we would like to have all our future releases be. I think that we should see the standard 2-4 week time frame for point releases and within 24 hours for updates now that we have this new build system in place.

We have also put a Continuous Release (CR) repository in place for both CentOS 5 and CentOS 6. This repository can be installed via the simple command:

yum install centos-release-cr

The purpose of the CR repository is to allow the CentOS Project to push some of the security updates if we are having issues with a point release build (like we did with both CentOS-6.0 and CentOS-6.1). If we are not going to meet the 2-4 week goal for our point release, we will push out the packages we have gotten to build properly while continuing to work on the problem packages. This repository is totally optional and was not needed with CentOS-6.2, but we recommend it be installed if you want to get your security updates as fast as possible.

Karanbir gets the credit for the new build system, called reimzul. It uses beanstalkd work queues and allows adding new builders to process the work as required.

The build system has the flexibility to allow us to import SRPMS into a git repo for packages we want to change, generate a new SRPM after edits for those packages, and submit those modified SRPMS into the work queues. It also allows for the submission of non-modified SRPMS directly without the need to import them into git. It automates several things that we have done in the past by hand (automatically knowing which packages are not built by CentOS (for example the RHN packages that deal with upstream subscriptions) and automatically copies multilib 32bit packages into the 64 bit tree. The system also reliably produces the Yum-Presto DeltaRPMS and metadata for minimizing download times for updates.

We do need to announce that CentOS-4 will be reaching the End Of Life at the end of February 2012. That means that there will be no more CentOS-4 updates after March 1st, 2012. If you are still using CentOS-4, you need to upgrade to CentOS-5 or CentOS-6 or switch to Red Hat's paid Extended Update Support for EL4 to continue to get updates. Please see the CentOS-4 EOL announcement for more details.

So, news on the CentOS front for 2012 is very promising and we are looking forward to great things in the new year.

I do not claim to be a security expert by any stretch of the imagination. The extent of my malicious network behavior ends at clicking 'start' on a nessus scan. And yet despite this, I find that I am constantly astounded at the inability of corporations to learn from each other when it comes to network security. Sony made security considerations front page news for over a month. Websites were

We use DRBD at work on several CentOS 5.x nodes to replicate data between our two computer rooms (in different buildings but linked with Gigabit fiber). It's true that you can know if something wrong happens at the DRBD level if you have configured the correct 'handlers' and the appropriate notifications scripts (Have a look for example at the Split Brain notification script). Those scripts are 'cool' but what if you could 'plumb' the DRBD status in your actual monitoring solution ? We use Zabbix at $work and I was asked to centralize events from differents sources and Zabbix doesn't support directly monitoring DRBD devices. But one of the cool thing with Zabbix is that it's like a Lego system : you can extend what it does if you know what to query and how to do it. If you want to monitor DRBD devices, the best that Zabbix can do (on the agent side, when using the zabbix agent running as a simple zabbix user with /sbin/nologin as shell) is to query and parse /proc/drbd . So here we go : we need to modify the Zabbix agent to use Flexible User Parameters, like this (in /etc/zabbix/zabbix_agentd.conf) :

UserParameter=drbd.cstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep cs|cut -f 2 -d ':'|grep Connected |wc -l
UserParameter=drbd.dstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep ds|cut -f 2 -d ':'|cut -f 1 -d '/'|grep UpToDate|wc -l

We just need to inform the Zabbix server of the actual Connection State (cs) and Disk State (ds) . For that we just need to create Application/Items and Triggers .. but what if we could just create a Zabbix Template so that we can just link that template to a DRBD host ? I attach to this post the DRBD Zabbix template (xml file that you can import in your zabbix setup) and you can just link it to your drbd hosts. Here is the link . That XML file contains both two Items (cstate and dstate) and the associated triggers. Of course you can extend it, especially if you use multiple resources , drbd disks. Because we used the Flexible parameters, you can for example in the Zabbix item, create a new one (based on the template) and monitor the /dev/drbd1 device just by using the drbd.dstate[1] key in that zabbix item.

Happy Monitoring and DRBD'ing ...

Computer security is once again becoming a hot topic for administrators.  There are dozens of new sites springing up around the web, and each is slinging their own ‘Perfect’ setup instructions.  They have the usual bell curve of good advice, okay advice, and advice that will effectively leave you with a smoldering pile of rubble where your data used to be. What disturbs me is the growing

ssh -X remotehost # yawn. X forwarding through ssh. ssh -Y remotehost # trusted X forwarding through ssh. Still yawn, let’s do something fun. ssh -D2222 remotehost # This is okay. This command sets up a SOCKS proxy on port 2222 which can be used with firefox (and Internet Explorer if you really hate yourself) to avoid office internet filters…. not that I condone such anti-social behavior. ssh

It seems that in decommissioning my old blog setup (I got bored keeping up with wordpress security), I removed some useful bits for those of you (thanks to all 3 of you!) who actually read the stuff I write. So here's the rewrite functionality  for removing the ~ character from user pages within apache. RewriteEngine On #RewriteLog logs/rewrite.log # Uncomment for rewrite logging #

I've now had the SSD in my laptop for about 10 days. Its made a massive difference to the way I work.

Its striking as to how much of a difference having this extra performance in the laptop would make. In march I upgraded the memory on this laptop from 2GB to 8GB - which also made a massive difference, specially since I almost never reboot the device and the filesystem cache get very good at handling just the right kind of stuff - but what kills them is my email ( ~ 30 gb ) and VMs ( upto 5 running at any given time ). Having the SSD now means that I no longer need to drop back to 10 seconds for jedit startup after I've been running a couple of VMs.

One thing that hasn't gone quite to expectation is the battery life. The HP 2540p had ~ 4 hrs or so, doing what I do, when I got it new. That had dropped to just over 3 hrs with the 250gb sata disk in. With the SSD its now gone to 2 hrs ~30 min or so. Initially that felt quite strange, I was expecting it to go in the other direction. And while I havent been able to put a finger on exactly what this is, it seems like there are 2 interesting side effects from the SSD upgrade.

1) The four cores on this i5 laptop now run at full speed ( 2.53 Ghz ) a lot more often than they did in the past, trending this over the last 48 hrs and its averaged 1.87Ghz; Not sure what it was earlier but the cpu governors used to stay blue a lot more than they do now.

2) Heat. The cooling fan is on a lot more, and the heat vent seems a lot warmer than it ever did in the past. This might be due to the cpus running a lot faster, a lot more. The disk itself does not 'seem' to be any warmer. The bottom left side of the laptop which houses the disk feels cool.

The big win of-course, is performance of everything. Almost every app just starts in place ( even eclipse! ). Doing a search in large code projects is instantaneous. Git operations are visibly quicker. Even using svn isn't nearly as boring as it used to be, if I can stop adding -a to all my svn commits it would not get in my way.

The only thing that isn't quite as quick as it needs to be on this machine now is the graphics interface ( intel HD ).

Also worth keeping in mind is that use CentOS-6 for the SSD hosted content, and make sure you have 'discard' enabled as a mount option.

- KB

Turns out there's at least one fan of CentOS inside dell. A dell community blog showed up today touting a CentOS based LiveCD with all the built-in goodies needed to update the firmware for dell servers. There's a complete walk through for how to use it as well as screen shots of the disk in action. If you're using dell kit in your shop, this is definitely something worth checking out, and it's

For those of you upgrading to RHEL6 or CentOS 6 who are using nfs automappings, some of you are probably noticing some interesting behavior. When we stumbled across this issue, it was because our home directories were all mounted nobody:nobody. Turns out in RHEL6, nfsv4 is now the default, and this can catch people off guard. There are a few ways to fix this however.  Least effort: uncomment the

I find that vim's syntax highlighting is incredibly useful in making documents more readable. So much so that I find myself wishing that I had this feature when simply paging through files quickly with 'less'. Turns out the vim folks thought of this too and include a script that allows you to use your finely honed vim environment as a replacement for less. Buried within your vim install is a

The number of questions I received from different people regarding the LiveCD/LiveDVD tools and the kickstart files used to produce the ISO images was quite "high". People looking at the normal place will be disappointed because we haven't used the original livecd subversion repo to produce the actual Live medias.  So in the meantime, if people want to use the livecd-creator tool, they can fetch the SRPM here : http://people.centos.org/arrfab/CentOS6/SRPMS/livecd-tools-0.3.6-1.el6.src.rpm . I've just copied also the two kickstart files used for both LiveCD and LiveDVD here : http://people.centos.org/arrfab/CentOS6/LiveCD-DVD/

Hope that people will be satisfied .. faster to push those files there than to change the whole 'used behind the scene' infra

As you've probably seen if you're subscribed to the CentOS announce list (or if you just rsync/mirror the whole CentOS tree) , the CentOS 6.0 LiveCD was released last monday. This is the first of our CentOS custom spins ! While I'm writing that blog post, the CentOS 6.0 LiveDVD is on its way to the external mirrors too and will normally be announced shortly (when enough mirrors will have it) ! It will be the second CentOS respin and we have more in the pipe for you ! As Karanbir announced it in the 6.0 release mail , we planned also to provide two other spins : the minimal one and the lws one. Good news is that the minimal one is almost finished and being intensively tested. If things don't change (or bugs appear during QA), the iso image will be only ~250Mb for the i386 arch and ~300Mb for the x86_64 one. It's meant to be used as a real basic CentOS system (even less packages that the @core group on a normal install if used with the proper kickstart invocation !) : 186 packages only on your disk. You'll have a very basic CentOS system with only openssh-server and yum. We are even testing the luks/lvm/md devices combination to be sure to meet your needs.

The next custom respin (LWS code name - for LightWeigth Server edition) will still be a CD iso image (but pushed to the limit) that will include basic server packages, more or less in the idea of the ServerCD that existed during the CentOS 4.x days ... That one still needs to be finished while work has already being done.

Stay tuned for more informations when it will be pushed to mirrors and announced .. all that at the same time as 6.1 and 5.7 (in parallel) builds ..Interesting times !

I decided to put CentOS 6 on my iMac. It was running in dual-boot mode with OSX and CentOS 5. Installing through the network (from a NFS share) was really easy and no bug encountered but at the end of the install, when it asked me to reboot, nothing : after having selected the Linux partition in the rEfit boot manager screen, nothing. hmm ....

I restarted the install process to see if at least anaconda tried to install grub on the first sector of the /boot partition and not in the MBR but that was correctly seen and chosen by anaconda . So the issue was somewhere else. I had a /boot ext3 partition (on /dev/sda3) while /dev/sda4 is the VolumeGroup in which I had defined my Logical Volumes. There was a big rewrite in Anaconda for the storage part and el6/CentOS 6 suffers from one bug found on the upstream bugzilla when having to deal with Apple computers *and* using rEfit at the same time : https://bugzilla.redhat.com/show_bug.cgi?id=505817

Long story short : to have CentOS 6 running on your iMac (if using refit as the EFI boot manager) :

• install CentOS 6 as usual (check that grub will be installed on the first sector of /boot and not in the MBR , normally correctly seen/proposed by Anaconda)
• on the first reboot, enter the rEFIt shell and launch 'gptsync' (it will say that it has to 'sync' the gpt, accept the sync)
• select now the Linux partition : it will fail with a black screen
• power down the iMac and start it up : select Linux in the refit boot manager and enjoy your CentOS 6 installation on the iMac

We are pleased to announce the immediate availability of CentOS-6.0 for 
i386 and x86_64 Architectures.

CentOS-6.0 is based on the upstream release EL 6.0 and includes packages 
from all variants. All upstream repositories have been combined into 
one, to make it easier for end users to work with.

There are some important changes to this release compared with the 
previous versions of CentOS and we highly recommend reading this 
announcement along with the Release Notes at 
http://wiki.centos.org/Manuals/ReleaseNotes/CentOS6.0

There are no CD images being released with CentOS-6, however we have 
some CD variants in the pipeline. Details for these are mentioned below.

Since upstream has a 6.1 version already released, we will be using a 
Continous Release repository for 6.0 to bring all 6.1 and post 6.1 
security updates to all 6.0 users, till such time as CentOS-6.1 is 
released itself. There will be more details about this posted within the 
next 48 hours.

+++++++++++++++++++++++
Upgrading from CentOS-4 or CentOS-5:

We recommend everyone run through a reinstall rather than attempt an 
inplace upgrade from CentOS-4 or CentOS-5

+++++++++++++++++++++++
LiveCD and LiveDVD

LiveCDs and LiveDVDs for i386 and x86_64 will be released within the 
next few days. These will bring in the ability to directly install from 
the livemedia.

+++++++++++++++++++++++
Minimal Install CD

We have also created a minimal install CD, that would bring up a base 
machine with just enough content to have a usable platform. This CD 
image will be released in the next few days.

+++++++++++++++++++++++
The LightWeightServer (LWS) CD

In order to bring back the CentOS-4 Server CD style single iso image, we 
are creating a LWS varient of the main distro. Details for this will be 
posted in the next few days with release happening after the live media 
and the minimal cd editions.

+++++++++++++++++++++++
Downloading CentOS-6.0 for new installs:

When possible, consider using torrents to run the downloads. In most 
cases you will find its also the fastest means to download the distro. 
There are currently over a thousand  people seeding CentOS-6 and it's 
possible to get upto 100mbps downloads via these torrents.

Torrent files for the DVD's are avilable at :
http://mirror.centos.org/centos/6.0/isos/i386/CentOS-6.0-i386-bin-DVD.torrent
http://mirror.centos.org/centos/6.0/isos/x86_64/CentOS-6.0-x86_64-bin-DVD.torrent

You can also use a mirror close to you :
http://www.centos.org/modules/tinycontent/index.php?id=30

Most mirrors will allow direct DVD downloads over http, ftp and rsync.

Please keep in mind that not all mirrors are currently updated, some 
might take upto another 24 hours before they have all the content.

+++++++++++++++++++++++
sha1sum for the CentOS-6.0 ISOS:

i386:
fcf49e875cd4494f2af68cf257ab9e93523c9427  CentOS-6.0-i386-bin-DVD.iso
862815623d2e7990207dd78a281837c7eb719e83  CentOS-6.0-i386-netinstall.iso

x86_64:
9de87b0c696ebd72b952edb4cc06c24cbdc37d81  CentOS-6.0-x86_64-bin-DVD1.iso
5e3834621f11fbcca78cf7d70625c647045f45f5  CentOS-6.0-x86_64-bin-DVD2.iso
23f9e606cbcbd52d2e5df3716a85cdde336f7bfe  CentOS-6.0-x86_64-netinstall.iso

+++++++++++++++++++++++
Sources and Debuginfo packages:

SRPMS and debuginfo packages are still making their way to the CentOS 
mirrors and should be available within the next 24 to 48 hours. We are 
prioritising the centos modified packages.

+++++++++++++++++++++++
Getting Help:

The best place to start when looking for help with CentOS is at the wiki 
( http://wiki.centos.org/GettingHelp ) which lists various options and 
communities who might be able to help. If you think there is a bug in 
the system, do report it at http://bugs.centos.org/ - but keep in mind 
that the bugs system is *not* a support mechanism.

+++++++++++++++++++++++
Contributing and joining the project:

We are always looking for people to join and help with various things in 
the project. If you are keen to help out a good place to start is the 
wiki page at http://wiki.centos.org/Contribute . If you have questions 
or a specific area you would like to contribute towards that is not 
covered on that page, feel free to drop in on 
#centos-devel at irc.freenode.net for a chat or email the centos-devel list 
(http://lists.centos.org).

+++++++++++++++++++++++
Thanks to everyone who contributed towards making 6.0


Enjoy!

One thing that I had to have a look at (during CentOS 6 QA), is the way anaconda (the Red Hat/Fedora/CentOS installer) pre-defines some 'tasks' . People used to those kind of install know what I'm talking about : the "Mininal", "Desktop", "Basic Server" and other choices you have during setup. From that first selection, you can decide (or not) to customize the software selection which then leads you to a screen containing categories / groups / packages defined in the comps.xml file present under /repodata on the tree/install media.

If you don't 'see' which screen i'm talking about, a small screenshot of the upcoming CentOS 6 will explain better than words :

Those pre-defined tasks aren't defined in the comps.xml file but rather at build time within anaconda. Fine but how can you 'modify' anaconda behaviour and test it without having to patch anaconda SRPM, rebuild it and launch a new build to generate the tree and install medias ? Easy , thanks to a simple file on the tree !

People wanting to modify anaconda behaviour at install time without having to regenerate the whole tree can just create a small file (updates.img) , put it in the /images directory in the tree. Anaconda (when installing over the network, http/ftp/nfs) always try to see if an updates.img file exists, and if so, use it. Fine, so I could easily try to "patch" it without having to modify the whole tree.

Creating that updates.img (it's just a ext2 filesystem on top) is really easy :

dd if=/dev/zero of=/tmp/updates.img bs=1k count=1440
losetup `losetup -f` /tmp/updates.img

losetup -a|grep updates.img
mkfs.ext2 /dev/loop3           # was loop3 in my case
mkdir /mnt/loop ; mount -o loop /tmp/updates.img /mnt/loop/ ; ll /mnt/loop
drwx------. 2 root root 12288 Jun 11 15:43 lost+found

From now, it's just a matter of putting the new files that you want to test and that will "overwrite" at run-time the defaults anaconda ones.

(in our current example, it was the installclasses/rhel.py that needed to be modified, so I just had to create a installclasses dir and drop my version of rhel.py in there on the loop device)

When you're done, umount the updates.img, copy it to /path/to/your/install/tree/images , restart a http install (verify that permissions and selinux contexts are of course correct !) and enjoy !

Easier and faster. Thanks to the Anaconda team which decided to permit modifying the anaconda behaviour at run-time with a simple file

In my first post in this small series, I closed without addressing matters of securing an IPv6 connection, and in matters of debugging where a connection failure is occurring. Thinking about it, the diagnostic post needs to come first, because tightening down a conneciton can cause hard to diagnose symptoms. So, on to diagnosis ...

We examined the interface results last time. Looking at just the routing related to ipv6 is straightfowrard as well:

Some familiar tools:

/sbin/ifconfig eth0
/sbin/ifconfig sit1
/sbin/route -A inet6

So using those tools:

[herrold@hostname ~]$ /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::4cf2:1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:1691 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1693 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:177195 (173.0 KiB)  TX bytes:210468 (205.5 KiB)

[herrold@hostname ~]$ /sbin/route -n -A inet6 | grep 2604
2604:aa:bb:cc::/64      ::          U     256    1658       0 sit1
2604:aa:bb:cc::/128     ::          U       0       0       2 lo
2604:aa:bb:cc::2/128    ::          U       0    1691       1 lo
[herrold@mailhub ~]$   

That is a pretty ordinary routing table for a non-gateway endpoint. Off-box traffic (to the '/64' netmask) is handed to the sit interface, and local traffic (to the '/128') retained on the local lo interface

We use a unfamiliar tool: ping6 -- The common '127.0.0.1' localhost has a new form under ipv6:

ping6 ::1 -c 2

and testing

[root@hostname ~]# ping6 ::1 -c 2
PING ::1(::1) 56 data bytes
64 bytes from ::1: icmp_seq=0 ttl=64 time=0.157 ms
64 bytes from ::1: icmp_seq=1 ttl=64 time=0.132 ms

--- ::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.132/0.144/0.157/0.017 ms, pipe 2
[root@ostname ~]# 

And we can ping by IP on the 'Global' link, both on the local end, and remotely with differing transit times for the packets:

Looking at the network connections, we examine the tunelling interface:

[root@hostname ~]# /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::4cf2:1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:1714 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1731 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:179587 (175.3 KiB)  TX bytes:215180 (210.1 KiB)

[root@hostname ~]# ping6 2604:aa:bb:cc::2 -c 2
PING 2604:aa:bb:cc::2(2604:aa:bb:cc::2) 56 data bytes
64 bytes from 2604:aa:bb:cc::2: icmp_seq=0 ttl=64 time=0.135 ms
64 bytes from 2604:aa:bb:cc::2: icmp_seq=1 ttl=64 time=0.137 ms

--- 2604:aa:bb:cc::2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.135/0.136/0.137/0.001 ms, pipe 2
[root@hostname ~]# ping6 2604:aa:bb:cc::1 -c 2
PING 2604:aa:bb:cc::1(2604:aa:bb:cc::1) 56 data bytes
64 bytes from 2604:aa:bb:cc::1: icmp_seq=0 ttl=64 time=55.1 ms
64 bytes from 2604:aa:bb:cc::1: icmp_seq=1 ttl=64 time=53.7 ms

--- 2604:8800:100:bb::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 53.781/54.473/55.165/0.692 ms, pipe 2
[root@hostname ~]# 

Turning to DNS and name resoluton, it is quite familiar. One does not need an IPv6 link to query nameservers and receive back results, as they will answer questions _about_ ipv6 hostnames ('AAAA' records) to any authorized inquirant. Try these:

dig +trace www.ipv6.sixxs.net
dig www.kame.net aaaa 

I get answers like this:

[herrold@centos-5 ~]$ dig www.kame.net aaaa

; 
> DiG 9.3.6-P1-RedHat-9.3.6-16.P1.el5 
> www.kame.net aaaa
;; global options:  printcmd
;; Got answer:
;; ->>HEADER- opcode: QUERY, status: NOERROR, id: 45595
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 2, ADDITIONAL: 3

;; QUESTION SECTION:
;www.kame.net.                  IN      AAAA

;; ANSWER SECTION:
www.kame.net.           85164   IN      CNAME   orange.kame.net.
orange.kame.net.        85164   IN      AAAA    2001:200:dff:fff1:216:3eff:feb1:44d7

;; AUTHORITY SECTION:
kame.net.               85164   IN      NS      mango.itojun.org.
kame.net.               85164   IN      NS      orange.kame.net.

;; ADDITIONAL SECTION:
mango.itojun.org.       2364    IN      A       210.155.141.200
mango.itojun.org.       2364    IN      AAAA    2001:2f0:0:8800:206:5bff:fe8d:940
mango.itojun.org.       2364    IN      AAAA    2001:2f0:0:8800::1:1

;; Query time: 1 msec
;; SERVER: 10.16.1.112#53(10.16.1.112)
;; WHEN: Thu Jun  9 17:17:20 2011
;; MSG SIZE  rcvd: 195

[herrold@centos-5 ~]$

which is certainbly a mess to read -- let's trim out the interesting parts:

[herrold@centos-5 ~]$ dig www.kame.net aaaa
;; ANSWER SECTION:
www.kame.net.           85164   IN      CNAME   orange.kame.net.
orange.kame.net.        85164   IN      AAAA    2001:200:dff:fff1:216:3eff:feb1:44d7 

Which is the familiar information: a CNAME record is pointed in fact at a AAAA record at a ipv6 -type IP. We can ping (ping6) it by IP:

[root@hostname ~]# ping6 2001:200:dff:fff1:216:3eff:feb1:44d7 -c 2
PING 2001:200:dff:fff1:216:3eff:feb1:44d7(2001:200:dff:fff1:216:3eff:feb1:44d7) 56 data bytes
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=0 ttl=52 time=246 ms
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=1 ttl=52 time=256 ms

--- 2001:200:dff:fff1:216:3eff:feb1:44d7 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 246.833/251.705/256.577/4.872 ms, pipe 2
[root@hostname ~]#

or ping it by name, as DNS is working:

[root@hostname ~]# ping6 www.kame.net -c 2
PING www.kame.net(2001:200:dff:fff1:216:3eff:feb1:44d7) 56 data bytes
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=0 ttl=52 time=227 ms
64 bytes from 2001:200:dff:fff1:216:3eff:feb1:44d7: icmp_seq=1 ttl=52 time=244 ms

--- www.kame.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1008ms
rtt min/avg/max/mdev = 227.291/235.678/244.066/8.401 ms, pipe 2
[root@hostname ~]# 

Some other examples to try as 'beacons' might include:

ping6 2604:8800:100:9a::1 -c 2
ping6 2001:200:0:8002:203:47ff:fea5:3085 -c 2
ping6 ftp.ipv6.uni-muenster.de -c 2
ping6 -I eth0 ipv6.google.com -c 2 

So the familiar diagnostic methods of examining interfaces, checking routing, testing connectivity by IP, and connectvity after resolution by name are all present

It seems quite a lot of people blogged about IPV6 day . It's true that it's always a good idea to speak about IPV6. I'm using IPV6 natively on my server hosted at Hetzner (they offer a /64 IPV6 subnet, which is more than enough for a CentOS server hosting several xen domU Virtual Machines). At home, that's another story. I use a HE.net free tunnel to be able to reach ipv6 hosts. Yes, even in 2011, you still have to use tunnels to use IPV6 ! Why ? that's indeed a good question. Even if my CentOS ipv6 tunnel end-point/router/radvd at home is working correctly, I decided to ask my belgian provider if they had plans on implementing native IPV6. Well, not for my home connection, as I already know that Belgacom (the biggest provider in belgium) doesn't support IPV6 on their BBOX2 modems that they give to customers when ordering a DSL connection at home (while i'm talking about Belgacom, please stop sending me direct advertisement to my mailbox - the real one and not the electronic one - with your invoices about a service - VDSL2/BelgacomTV - that you *can't* offer to all your customers ... thanks) . So I decided to ask their 'professional services' because we have two 'professional and business' lines that we used at $work. Long story short (to avoid explaining how much emails/cases I had to send/open to have an answer) : "no, even on the business lines we can't support IPV6 and we have no plans (*sic*, I hope that guy was just kidding or probably doesn't know the real answer ..) nor dates about future implementation of the IPV6 services/connectivity " ..

Nice .. now /me goes back to CentOS QA mode ...

There is a 'ipv6 readiness testing day' today, April June 8, 2011, and so it seems appropriate to post my personal checklist for putting a CentOS box up on that network fabric

• Apply for an account with SixXs. Their reply takes a couple of days, as it is a volunteer run organization
• Have a deployed, updated, and hardened unit at a routable static IPv4 address
• Amend /etc/sysconfig/iptables to include a line passing the tunnelling protocol. I place the entry after the IPSEC protocol entries in a stock setup. Restart iptables
  

  ... 
  -A RH-Firewall-1-INPUT -p 50 -j ACCEPT
  -A RH-Firewall-1-INPUT -p 51 -j ACCEPT
  # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
  -A RH-Firewall-1-INPUT -p ipv6 -j ACCEPT
  # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  ... 

• Strip out any previous efforts at disabling the ipv6 / net-pf-10 kernel modules from loading in /etc/modules.conf, and in the files sourced in /etc/modprobe.d/ . Then rebuild the modules dependency table: /sbin/depmod -a
• Amend /etc/sysconfig/network to carry the following lines:
  

  #
  NETWORKING_IPV6=yes
  IPV6INIT=yes
  IPV6FORWARDING=yes
  IPV6_DEFAULTDEV=sit1
  #

  Which anticipates that the configuration details for the ipv6 tunnel will live in a file: /etc/sysconfig/network-scripts/ifcfg-sit1
• and add that mentioned file: /etc/sysconfig/network-scripts/ifcfg-sit1 -- I have elided site-specific details as to IP addresses with: aa.bb.cc and aa:bb:cc placeholders
  

  #
  DEVICE=sit1
  BOOTPROTO=none
  ONBOOT=yes
  IPV6INIT=yes
  IPV6_TUNNELNAME="SixXS"
  #
  IPV6_AUTOTUNNEL=yes
  PHYSDEV=eth0
  IPV6_ROUTER=yes
  #
  IPV6TUNNELIPV4="38.229.76.3"
  #    38.229.76.3 is the remote end of the tunnel at the tunnel broker
  IPV6TUNNELIPV4LOCAL="198.aa.bb.cc"
  #    198.aa.bb.cc is the local ipv4 static IP
  IPV6ADDR="2604:aa:bb:cc::2/64"
  #    2604:aa:bb:cc::2/64 shows both the local gateway IP, and netmask
  #    the remote end gateway IP is by convention, the :1 
  IPV6_MTU="1280"
  TYPE=sit
  # 

At this point, simply restarting networking should bring up the ipv6 link, and properly route it -- so: /sbin/service network restart

The interfaces will look something like this:

[herrold@nostname ~]$ /sbin/ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:01:02:aa:bb:cc
          inet addr:76.aa.bb.cc  Bcast:76.aa.bb.dd  Mask:255.255.255.248
          inet6 addr: fe80::201:aaff:bb05:cc16/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11088057 errors:0 dropped:0 overruns:1 frame:0
          TX packets:10668738 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1726307345 (1.6 GiB)  TX bytes:3178496052 (2.9 GiB)
          Interrupt:3 Base address:0x6f80

[herrold@hostname ~]$ /sbin/ifconfig sit1
sit1      Link encap:IPv6-in-IPv4
          inet6 addr: 2604:aa:bb:cc::2/64 Scope:Global
          inet6 addr: fe80::bbf2:cc1c/128 Scope:Link
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:500 errors:0 dropped:0 overruns:0 frame:0
          TX packets:502 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:53331 (52.0 KiB)  TX bytes:62784 (61.3 KiB)

[herrold@hostname ~]$ 

To wrap this up, ipv6 hardening, and connection debugging are worthy topics, and it may well be that a cautious sysadmin wants to lock down /etc/sysconfig/ip6tables and examine how one has hardened /etc/hosts.deny ... But rather than rush out content (I have a couple of mailing list posts I need to re-work), I'll leave these for later posts, while you, gentle reader, go apply for an account at a tunnel broker