April 06, 2021

CentOS Community Newsletter, April 2021 (#2104)

April 06, 2021 04:07 AM

Dear CentOS Community,

Thanks for joining us for another edition of our monthly newsletter. Here's what's happening in the CentOS community.

Upcoming CentOS Dojo

Yesterday we closed the Call For Presentations (CFP) for the upcoming CentOS Dojo in May, and we hope to publish the schedule of selected presentations this week. Meanwhile, you can register today for the event. Registration is free, but you will need to register to attend. The event will be online, and feature presentations about many aspects of the CentOS project, including several presentations about CentOS Stream. We look forward to seeing you there.

New AAA Infra

As you are hopefully already aware, we are in the midst of rolling out a new AAA (authentication) infrastructure, which we will share with Fedora. If you have an account on either the CentOS or Fedora account system, you should read Fabian's email (and the responses to it) on the centos-devel mailing list for details of what you need to do.

You will be hearing more about this changeover in the coming days, particularly if you are one of the people who has accounts on both projects with conflicting details (ie, same email address but different usernames, or vice versa).

CPE Update

The most recent CPE Update, which you may have seen on the centos-devel mailing list, covers most of the month of March, and contains a lot of exciting new developments, including the new AAA infrastructure (see above) and the list of priorities for the next quarter. Read the whole thing on the blog.

SIG Reports

SIGs - Special Interest Groups - are communities who are building something on top of CentOS. SIGs are expected to report quarterly on their progress, health, and opportunities for community participation.

This month we have a report from the Hyperscale SIG.

Meanwhile, the Software Collections SIG, scheduled to report this month, appears to be having a rather slow quarter.

If you're interested in running a SIG around your particular area of interest or expertise, we'd love to hear from you. Bring your proposal to the centos-devel mailing list

Updates/Releases

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during March:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during March:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during March:

Other releases

The following releases also happened during March:

Get Involved

There's a number of places where you can get involved in the CentOS Community. These are documented on the "Contribute" page in the wiki.

Ongoing efforts include the wiki itself, which has accumulated a lot of content over the past decade, much of which could stand to be freshened. If you're interested in assisting with the wiki refresh project, the best place to volunteer is on the centos-docs mailing list. The process for proposing changes to the wiki is covered in part 4 of the above-mentioned document.

This newsletter is another place where we always need help - finding stories, writing the content, translating into non-English languages, and working with community members to provide blog posts, technical howtos, or other content for publication to the blog. Please join us on the centos-promo mailing list to step up to do some of that work.

And the every-day work of answering user questions on the centos@centos.org mailing list, the forums, or IRC (#centos on Freenode) is open to anybody with knowledge, patience, and time. Just drop in and introduce yourself.

Get Gear

If you want to proclaim your love for the CentOS Project, we have two main options for obtaining CentOS Gear. Head over to the Red Hat Cool Stuff Store for CentOS shirts and hats. And HelloTux has CentOS tshirts, polo shirts, and sweatshirts!

April 05, 2021

CentOS Hyperscale SIG Quarterly Report for 2021Q1

April 05, 2021 02:35 PM

This report covers work that happened between January 13th and April 2nd 2021.

Purpose

The Hyperscale SIG focuses on enabling CentOS Stream deployment on large-scale infrastructures and facilitating collaboration on packages and tooling.

Membership update

The SIG was established in January with six founding members (Davide Cavalca, Filipe Brandenburger, Matthew Almond, Justin Vreeland,Thomas Mackey, David Johansen). Since then, four more members have joined (Igor Raits, Neal Gompa, Anita Zhang, Michel Salim).

We welcome anybody that’s interested and willing to do work within the scope of the SIG to join and contribute. See the membership section on the wiki for the current members list and how to join.

Releases and Packages

The SIG releases packages in a main repository. Sources for these packages are maintained in c8s-sig-hyperscale branches in dist-git (example).

Packages released in main are designed to be drop-in replacements for the corresponding packages on a stock CentOS Stream 8 system. This repository can be enabled by installing the centos-release-hyperscale package.

systemd

We ship a backport of systemd 247 based on the Fedora packaging. This includes a variety of bug fixes in existing features such as timers and cgroups, as well as new properties that take advantage of the latest kernel features. You can also look forward to new knobs in the various tools and daemons to make debugging and configuration easier.

By default, this backport will boot the system using the unified cgroup hierarchy (i.e. cgroup2), in line with systemd upstream policy. This can be changed with the appropriate kernel cmdline knobs.

This systemd backport also includes a SELinux overlay module, which allows running systemd 247 on a system in enforcing mode. Nonetheless, the SELinux integration has only seen limited testing and should be considered experimental at this point.

Grep

We ship a backport of Grep 3.6 based on the Fedora packaging. Compared to the stock 3.1 version shipped with the distribution, it includes major performance improvements and several bugfixes.

iptables

The iptables package on CentOS Stream 8 ships with only the nftables backend enabled. As part of the work to enable the legacy backend as well, we have packaged legacy-enabled versions of arptables and ebtables.

MTR

We ship a backport of MTR 0.94 based on the Fedora packaging. This includes several bug fixes, notably improving reliability when running in TCP mode.

dwarves

We ship a backport of dwarves 1.20 based on the Fedora packaging. This includes several improvements to the pahole tool, notably including much better BTF support.

Meson

We ship the latest version (0.55.3 ⇒ 0.57.1) of meson and the latest version (1.8.2 ⇒ 1.10.2) of ninja-build based on the Fedora packaging. This includes many different bug fixes and improvements. We will keep updating them as new versions will get released.

Health and Activity

The SIG was approved by the CentOS board on January 13. So far we’ve been able to maintain a healthy development pace, and hope to continue doing so in the future.

Meetings

The SIG holds regular bi-weekly meetings on Wednesdays at 16:00 UTC. Meetings are logged and the minutes for past meetings are available.

The SIG also uses the #centos-hyperscale IRC channel for ad-hoc communication and work coordination, and the centos-devel mailing list for async discussions and announcements.

Conference talks

The SIG was first introduced at CentOS Dojo, FOSDEM, 2021 (recording). SIG activities were also covered as part of DevConf.cz 2021 (recording) and FOSDEM 2021 (recording).

Future SIG-related talks are planned for CentOS Dojo, May 2021 and for LISA21.

Planned work

The SIG tracks pending work as issues on our Pagure repository. Notable projects currently in flight include:

  • Setting up the experimental repository and publishing a Copy-on-Write enabled build of the packaging stack and optional support for Btrfs-based atomic updates using Micro DNF
  • Shipping up-to-date versions of libvirt backported from Fedora
  • Completing the work to publish an iptables with the legacy backend enabled
  • Engaging with the Cloud SIG to build a set of Cloud images with the Hyperscale repo and packages already baked in

Issues for the Board

We have no issues to bring to the board’s attention at this time.

April 01, 2021

CPE Weekly: 2021-03-31

April 01, 2021 07:50 PM

Hi Everyone,

Sorry for the two week gap since my last report, we had a busy time in
the CPE team with the new fedora accounts deployment, our quarterly
planning cycle started for Q2 and Ireland had a bank holiday mid week
which *seemed* like a great idea at the time. Until no-one knew what
day it was for about a week!

So here I am, right at the end of Q1 with the CPE teams final weekly
report for January, February and March... two days early

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

CPE had our quarterly planning call last Thursday 26th March to
prioritize our project work going into Q2 (quarter 2, which is April,
May & June).
Our initiative repo quarterly boards have been updated
https://pagure.io/cpe/initiatives-proposal/boards/2021Q2
and our repo can be accessed here: https://pagure.io/cpe/initiatives-proposal
Our 2021 Quarterly Planning timetable can also be viewed here if you
are curious on when our next quarterly planning session is:
https://docs.fedoraproject.org/en-US/cpe/time_tables/
And finally, details on initiative requesting/how to work with us on
new projects here:
https://docs.fedoraproject.org/en-US/cpe/initiatives/

Going into Q2, the CPE team will work on rpmautospec
https://pagure.io/cpe/initiatives-proposal/issue/11 and aim to deliver
this project within the months of April, May & June. We are starting
this project on Monday 12th April and will keep you posted on where
the team will track work and what IRC channel they will use for comms.

You can also expect a Q1 blog post from us in the next week or two
highlighting the work that the team delivered over the last quarter
too.

Misc

* CentOS Dojo for May 13th & 14th CFP closes on Monday 5th April so
please submit your talks asap!
https://wiki.centos.org/Events/Dojo/May2021

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Account Migration is scheduled for next Tuesday 6th April
* Please read this important email from Fabian Arrotin on
verifying/updating your CentOS and Fedora email address
https://lists.centos.org/pipermail/centos-devel/2021-March/076690.html
* CentOS CI is also updating ocp.stg to 4.7.3 & will roll out to
production by the end of the week if all goes well

CentOS Stream

* Centpkg is build and available in Fedora and EPEL!
* MBS is being deployed
* ODCS is deployed
* Scripts for mass rebuild are ready
* CVE Dashboard for CentOS 8 Stream is up
* In short, lots of good things coming!

Fedora

* F34 beta is out!

* Mass reboot is scheduled for tomorrow, April 1st so please expect
some issues due to this required outage
* Final Freeze is due to start on Tuesday April 6th @ 1400 UTC - F34
schedule can be viewed here
https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html

Noggin/AAA

* Fedora Accounts is out!!!!!!!!!!!!!!!!
* There are still some corner case issues being worked through but
users should be able to access fedora services as normal. **NOTE** you
will need to reset your password if you have not already done so if
you receive an Unable to call ID or some note like that. Please
request a password reset and wait for the mail to land. Then follow
the link and reset your password.
* For any issues, please open a ticket on
https://pagure.io/fedora-infrastructure/issues
* The team can be found on #fedora-aaa for discussions on IRC
* And please report any issues you find relating to the Noggin
application in the repo https://github.com/fedora-infra/noggin
**ANOTHER NOTE** Thank you so so so much to all of the members of the
fedora community and wider open source communities who assisted our
team last week when we were deploying the new system. Your help did
not go unnoticed and unappreciated and we could not have done this
work without any of you. You know who you are, and you have my and the
wider teams sincerest thanks and gratitude

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

March 15, 2021

CPE Weekly: 2021-03-05

March 15, 2021 07:15 PM

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

* CentOS Newsletter for March is out!
https://blog.centos.org/2021/03/centos-community-newsletter-march-2021-2103/
* CentOS Dojo scheduled for May 13th & 14th CFP is open, details on
event and CFP link can be found here
https://wiki.centos.org/Events/Dojo/May2021
* New community podcast is out from Red Hat Community
https://twitter.com/redhatopen/status/1367113857936809984
* Lightning Talks & some others from DevConf.cz are now uploaded!
https://www.youtube.com/c/DevConf_INFO/featured
* Check out the most recent blog post on the Fedora Code of Conduct
here https://communityblog.fedoraproject.org/fedora-code-of-conduct-report-2020/

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Legacy CentOS CI Infra Openshift 3.6 has been retired
* CentOS CI OCP clusters updated to 4.6.18

CentOS Stream

* Testing centpkg against the new buildsystem and CBS
* Developing a style guide for CentOS Stream - first draft will be in
a repo on git.centos.org to view/comment by mid-March
* Building CentOS Stream only packages, eg logos, etc for Stream 9

Fedora

* Still in Beta Freeze
* Working on progressing flatpak-indexer, its currently in staging
* Processed 100+ fedscm requests!
* In staging got pagure on dist-git working with the git user instead
of each packager having their own shell account
* Working with debuginfo-d folks to get them set up with resources to
enable it in Fedora infra

Noggin/AAA

* Reviewing dates for a planned outage in March - early estimated
dates are 18th & 19th for production migration. Formal email to follow
to all fedora lists once outage period is confirmed early next week.
* Community blog coming middle of next week on the new account system work
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6
* The team use #fedora-aaa for discussions on IRC
* And please report any issues you find in the repo
https://github.com/fedora-infra/noggin

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

 

March 03, 2021

CentOS Online Dojo, May 13th, 14th, CfP now open

March 03, 2021 06:02 PM

TL;DR:

As promised in my email a few weeks ago, we're going to try to do online Dojos quarterly for the coming few quarters, and we're pleased to announce the first of these.

We will be holding the event May 13th and 14th, once again using the Hopin platform that we used last time. (Those of you who sent feedback about the platform may like to know that this was all passed on to Hopin, and at least some of it has been addressed since then.)

Details of the event are here: https://wiki.centos.org/Events/Dojo/May2021

More details will of course come soon, once we have the online event created.

Registration will be free.

The Call for Presentations (CfP) is now live, at forms.gle/wRa8r5ZRHrqnZ6VF6

We are looking for presentations about CentOS. This can be CentOS Linux, CentOS Stream, CentOS SIG work, any work in the CentOS community, or any project you're running on top of CentOS distributions.

Attendees of the February event specifically asked for more content about:

  • CentOS Stream
  • Koji (and similar ways of managing rpm builds)
  • Creating your own module/package in a personal repo
  • Gitlab
  • CentOS Stream contribution flow
  • FreeIPA
  • Keycloak
  • SIG updates

Please let me know if you have any further questions, and I hope to see your presentation proposals soon!

--Rich

March 02, 2021

CentOS Community Newsletter, March 2021 (#2103)

March 02, 2021 01:25 AM

Dear CentOS Community,

Here's a glimpse into what's happening in the CentOS community, and what's coming in the next few months.

This month in CentOS Stream

We've had a lot going on in CentOS Stream this month, here are some major things we've been working on:

- CentOS Linux Extras and CentOS Stream Extras are composed separately, that way if a SIG only ships on Stream, you can do so

- The CentOS Linux -> CentOS Stream migration instructions were shortened from  3 steps to 2 steps, check them out here:
https://www.centos.org/centos-stream/

[root@centos ~]# dnf swap centos-linux-repos centos-stream-repos
[root@centos ~]# dnf distro-sync

Carl George has provided a short screencast to demonstrate the process.

- The Stream 9 buildsystem is coming along, one architecture (s390x) is having hardware wired in very shortly

- Generated and pushed CentOS Stream 8 containers to https://quay.io/repository/centos/centos

- Kept our business as usual builds and pushes going
- Here are some interesting modules that got released recently:
https://lists.centos.org/pipermail/centos-devel/2021-February/076528.html

Landed in CentOS Stream

The following community contributions have, so far, landed in CentOS Stream.

We appreciate your patience as we improve the contribution process, as well as the reporting around it.

DevConf.cz

The annual developers conference that is usually held in Brno, Czechia, was held online in February. The CentOS Project was featured in two presentations there.

Davide Cavalca presented on the use of CentOS Stream at Facebook.

And Brian Stinson, Tomas Tomecek, and Carl George presented on Consuming CentOS Stream.

We expect video from both of those presentations to be available within the next couple of weeks.

Special Interest Groups

Special interest groups (SIGs) are the place where communities build things on top of CentOS - whether this is software, such as the Cloud SIG or the Storage SIG, or a non-software effort, such as the Promo SIG or Artwork SIG.

SIGs are asked to report quarterly on their progress, so that you know what they're up to, and where you can get involved. We also encourage you to attend the SIG's meetings, where you can find out about the day-to-day work.

CentOS OpsTools SIG Quarterly Report

Purpose

Provide tools and documentation, recommendation and best practices for operators of large infrastructure.

Membership update

Sadly, we did not attract new volunteers to contribute to the SIGs purposes, but at the same time, we didn't lose any.

Activity

After initial euphoria setting up the SIG and getting to work, we can now observe a kind of settlement. For the next quarter, we are planning to upgrade collectd to version 5.12.

There is also an interest in investigating in loki as tool to allow multi-tenant log aggregation.

Issues for the board

Nothing to report.

Infrastructure SIG report

DONE

* Updated SIG wiki
* Updated meeting calendar - Infra SIG will meet every 2nd Monday @ 1500 UTC on #centos-meeting. Next meeting is scheduled on 1st March
* Added x2 new members giving limited access to core infrastructure machines for CentOS Stream that CentOS Infrastructure is assisting in bootstrapping.
* * Leonardo Rossetti & Mohan Boddu. Both are senior engineers at Red Hat and are heavily involved in the development of release engineering and compose building for the upcoming C9S release.
* Completed a full infrastructure-wide upgrade of the monitoring stack to latest version of Zabbix

IN PROGRESS

* Review ticket for mbox koji access
* Review ticket for registering domain name in AU
* Account merge to Fedora account auth system Noggin

TODO

* Develop criteria for infrastructure access
* Admin, there can always be more tidying and refinement on admin 🙂

Governance Documentation

Over the years, the CentOS Governance has been documented in a handful of web pages and blog posts, which has served the purpose. With greater scrutiny on our governance, we are discovering that there are assumptions and "common knowledge" which is not documented anywhere.

As a consumer community (ie, we make stuff, you use it), the governance model was perceived to be less important to clarify before now. This has resulted in some unforeseen problems and misunderstandings over the past few months. Over the coming months we are trying to address this in two ways.

We plan to take the existing governance documents, and consolidate them into more formal project bylaws.

But, also, in this process, we are asking for community input on the governance model, to identify areas where we can improve. We are already implementing more openness around the Board of Directors process, and this is just the start. By telling us how the CentOS governance could better serve you, you can help us make this step towards being a more inclusive, collaborative project.

As we move towards a more contributor model, it becomes more important to clearly document how the project governs itself, and how you are part of that model. We hope that you'll be hearing a lot more about this in the coming months. We also hope that you will engage in this process and help us continue to make the project more transparent and collaborative.

Rebuild News

As various RHEL rebuild projects start up, we're very aware that we're all part of the same family. We look forward to collaborating in any ways that make sense. And, regardless of how (or if) we end up working together, we wish these projects the best of luck and success.

The Rocky Linux project is sending out monthly community updates. The latest edition can be seen here.

And the Alma Linux project has just announced the release candidate (R.C.) for their 8.3 release.

Note: We've highlighted two rebuild projects here, but we're not trying to pick favorites. We're just mentioning the ones that have been brought to our attention. If you're running a RHEL rebuild project, and want us to link to your news in future newsletters, please get in touch with Rich, at rbowen@centosproject.org, or join us on the centos-promo mailing list.

Other Community News

If you have any community news, big or small, please do let us know, on the centos-promo list.

The Foreman Project tweeted about a presentation around managing CentOS Stream hosts:

Phil wrote this excellent article about the merits of CentOS Stream. We think it's worth your time to read. https://medium.com/swlh/centos-stream-why-its-awesome-5c45d944fb22

Get Involved

In addition to the usual avenues for getting involved in The CentOS Project, I would like to draw attention to two specific opportunities.

First, this newsletter: If you would like to be involved in the process of putting this together each month, come join us on the centos-promo mailing list. We start drafting the next month's newsletter as soon as the last one "goes to press", and the more help we can get, the broader vision of the community we can present.

Second, after the recent CentOS Dojo, we committed to doing a quarterly online Dojo until such time as we can travel again (and possibly after that!). We need help in finding and scheduling the content, and promoting the event. Again, fo rthis, join us on the centos-promo mailing list, where that work will be happening.

Get Gear

If you want to proclaim your love for the CentOS Project, we have two main options for obtaining CentOS Gear. Head over to the Red Hat Cool Stuff Store for CentOS shirts and hats. And HelloTux has CentOS tshirts, polo shirts, and sweatshirts!

Until next time ...

Thanks for reading this far, and thanks for being part of the CentOS community!

February 26, 2021

CPE Weekly Report: 2021-02-26

February 26, 2021 07:08 PM

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

I hope you all enjoyed DevConf.cz last week! There were some great
talks and I am looking forward to catching up on the ones I
unfortunately missed when they are posted in a few weeks!
Also if you missed the CentOS Dojo at FOSDEM, you can watch all talks
on the CentOS YouTube channel here
https://www.youtube.com/playlist?list=PLuRtbOXpVDjC7RkMYSy-gk47s5vZyKPbt

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS CI updated OCP to 4.6.17
* Rolled out security fixes to ci.centos.org Jenkins and cert update

CentOS Stream

* Documentation updated on the shortened CentOS Linux -> CentOS Stream
conversion, see the demo here https://asciinema.org/a/393875
* CentOS Extras is now separately delivered for Stream and Linux
* CentOS Stream 8 container images are published now to quay.io

Fedora

* We are now in F34 freeze! All changes to frozen hosts take 2 +1s
* Bodhi updates-testing activated for F34
* Fedscm-admin work started on default branches
* Openh264 repos are hosted on Cisco CDN

Noggin/AAA

* User migration script has been successfully re-run
* Lots of docs updates - check out the docs section for more
information https://noggin-aaa.readthedocs.io/en/latest/
* PR opened for changes to docs to add pkinit to docs to allow
applicable certs be shipped for packages but it seems fedora-packager
Fedora package has to be built with the change applied
https://pagure.io/fedora-packager/pull-request/166
* If you are experiencing any issues with your application
authenticating with Noggin, please reach out to the team on IRC
channel #fedora-aaa
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6
* And please report any issues you find in the repo
https://github.com/fedora-infra/noggin

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

February 15, 2021

CPE Weekly: 2021-02-14

February 15, 2021 04:36 PM

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

Conferences!

* DevConf.cz is on 18th - 20th Feb! Get your ticket here if you
haven't already https://hopin.com/events/devconf-cz-2021
* CentOS Dojo @ FOSDEM was really great last week, and if you missed
it be sure to check out the CentOS youtube channel where all of the
talks are now uploaded and available to view
https://www.youtube.com/thecentosproject

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Our CI infra has been updated from Ocp.ci / ocp.stg.ci to 4.6.15
* Monitoring stack updated to zabbix 5.0.8
* Kojihub now supports x86_64,ppc64le & aarch64

CentOS Stream

* CentOS Stream container images are now readily available!Check out
the mail from Brian Stinson to the CentOS-devel & announce list here
for more details on tags and where to pull
https://lists.centos.org/pipermail/centos-devel/2021-February/076503.html

Fedora

* Mass branching of packages was completed last week
* Mass branching of modules is underway
* There is already have a branched compose
* The main branch changes are also almost complete with just docs left
* tests namespace in dist-git has migrated to “main” with “master” as
symlink for now with it being removed after F34 release, so mark your
calendar!

Noggin/AAA

* Security fixes on Content Security Policy
* Re-installed FreeIPA schema to test a faster way to import user data
as part of tuning & performance testing while still in staging
* If you are experiencing any issues logging in, please reach out to
the team on IRC channel #fedora-aaa
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6
* And please report any issues you find in the repo
https://github.com/fedora-infra/noggin

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

February 09, 2021

CentOS Community Newsletter, February 2021 (#2102)

February 09, 2021 02:07 AM

Dear CentOS Community,

This month's newsletter is running a little late, because I wanted to include the report from our annual FOSDEM CentOS Dojo, which was held last Thursday and Friday.

CentOS Dojo at FOSDEM

We had 216 registrations, with 164 (75.9%) of registrants actually showing up. The average attendee spent 475 minutes at the event.

Over the two days of the event, we had 8 presentations, all of which are now available on YouTube, if you missed any of them.

We started the day with a round-table discussion with the board of directors. This started slowly, but developed into a useful Q&A with the community, covering everything from CentOS Stream (of course) to the new SIGs, to deep-dives into specific technical issues. We then had presentations about various SIGs (Cloud, Hyperscale) and various use cases around the community.

Overall, we were pleased with the turnout and the interactions, especially in the "hallway" track. We are considering doing more of these events - at least quarterly during the remainder of the pandemic, and then hopefully continue them in the future, for those who remain unable or unwilling to travel to in-person events.

We would love to hear from you about what content you'd like to see at future events, or, better yet, if you want to present about what you're working on.

Upcoming events

In just under 2 weeks, DevConf.cz will be happening (February 18th - 20th). This event is usually held right before, or right after, FOSDEM, in nearby Brno. This year, it's online, with content scheduled so as to be convenient for attendees in Europe time zones.

As every year, there's a  lot of deep technical content covering a wide range of topics. We want to specifically draw attention to two presentations:

On Friday at 14:45 (CET), Davide Cavalca will be talking about the use of CentOS Stream at Facebook. And then at 17:30, Tomas Tomecek, Brian Stinson, and Carl George will be talking about Consuming CentOS Stream.

Details and (Free!) registration are available at https://devconf.cz/.

SIG Reports

Special Interest Groups (SIGs) are one important place where the community can get involved in making CentOS more useful. This month we hear from several of our SIGs about what they've been doing for the past quarter.

Active SIGs hold regular meetings, where you can find out what's happening, and where you can get involved.

Hyperscale SIG

Although not scheduled to report this month, the Hyperscale SIG presented at last week's Dojo about what they have planned, and what they have done so far. You can watch the full presentation on YouTube, and read more about the SIG here.

* Alt Arch

Cloud SIG

Purpose

Packaging and maintaining different FOSS based Private cloud infrastructure applications that one can install and run natively on CentOS.

https://wiki.centos.org/SpecialInterestGroup/Cloud

Membership Update

We have reached out to all current and pending members of the SiG to confirm their continued interest as we revitalize the SiG. Once the membership lists are updated we will be holding nominations and elections for chair and co-chair.

We are always looking for new members, especially representation from other cloud technologies and we’ve reached out to Shaken Fist to see if they would like to join though they are currently Ubuntu only.

Releases and Packages

RDO

Nov 16 Victoria release: https://blogs.rdoproject.org/2020/11/rdo-victoria-released. Interesting features in the Victoria release include:
Source tarballs are being validated using the upstream GPG signature, to ensure the integrity of the packaged source code..

Openvswitch/OVN are not shipped as part of RDO. Instead RDO relies on builds from the CentOS NFV SIG.
The full release notes are at https://releases.openstack.org/victoria/highlights.html

Health and Activity

The Cloud SIG has been very active in regards to creating and publishing builds though it has not held a meeting over the past months. Efforts are being made to revitalize the SiG by re-establishing meetings and grow both the membership and projects involved. At this time, the SiG is only OpenStack.
The OpenStack group is focusing on the Wallaby release, which will be available for CentOS Stream 8 once it is finished. For additional details about the CloudSiG’s plans for CentOS Stream adoption in Wallaby, and previous releases, see the following blog post: https://blogs.rdoproject.org/2021/01/rdo-plans-to-move-to-centos-stream/

Alan Pevec held a RDO and CentOS Stream AMA which is now available on YouTube: https://www.youtube.com/watch?v=MlhAhClVaEI&feature=youtu.be

Alfredo Moralejo and Javier Peña presented 'How OpenStack became boring (and successful)' at the CentOS Dojo on February 5th. https://youtu.be/H0JDgsafFD0

Issues for the Board

We have no issues to bring to the board’s attention at this time.

Storage SIG

Repository Status and Updates

  • Ceph Nautilus updates: 14.2.16 (c7 and c8)
  • Ceph Octopus updates: 15.2.8 (c8 only) too
  • luarocks packages were added for upcoming Ceph Pacific
  • GlusterFS updates:  9.0, 8.3, and 7.9 for c7 and c8.  (Note: glusterfs-7 is now EOL)
  • NFS-Ganesha updates: 3.5 and 2.8.4 for c7 and c8; including associated libntirpc.
  • Samba updates: 4.11.17(c7 and c8) & 4.12.11 & 4.13.4(c8 only). v4.11.x is now EOL

Group Status and Actions from meeting

  • SIG needs to update the wiki and the calendar page moving to #centos-meeting2
  • SIG will work on automating cephadm builds

Links and other general informations

Meetings agenda https://hackmd.io/Epc35JIESaeotoGzwu5R5w

Messaging SIG

During the past quarter, there has not been much change in or with the messaging SIG, and there is nothing to report. Its artifacts are consumed by both Cloud SIG and Opstools SIG.

Release and Updates

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during January:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during January:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during January:

 

February 08, 2021

CPE Report: 2021-02-05

February 08, 2021 07:51 PM

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

Conferences!

* CentOS Dojo @ FOSDEM is on right now! Links to talks from Thursday
are on the CentOS youtube channel and Rich is playing a blinder
getting all the content uploaded in record time
https://www.youtube.com/TheCentOSProject
* NOTE: 'playing a blinder' means doing an excellent job for
anyone unfamiliar with the term
* Fedora has a booth as well @ FOSDEM this weekend! Make sure you stop
by and say hi to all those great Fedorans who will be manning it this
weekend https://chat.fosdem.org/#/room/#fedora-stand:fosdem.org

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CI team members are migrating Fedora-Infra and Fedora-apps namespace
whcih is one of the last few before we shut down legacy cluster
* There is also an investigation spike on Zabbix upgrade to current
LTS version which will then be rolled-out on the CentOS Infra once
complete

CentOS Stream

* Python39 built and ready to compose
* Dist-git repos are regularly up to date
* Repos are populated in the CentOS Stream GitLab instance and will be
publically viewable in the coming weeks
* Very detailed talks on CentOS Stream given by Brian Stinson & Brian
'Bex' Exelbierd are watchable now on the CentOS YouTube channel -
check them out!

Fedora

* Infra team are assisting with the testing of ipa/noggin for
otp/other cases in stg
* Their also doing a cleanup of a bunch of broken links on koji volume
* Mass rebuild of rpms is done, modules are underway
* FTBFS for the mass rebuild are filled

CPE ARC TEAM

(Community Platform Engineering Advanced Reconnaissance Team....Team)
We have a new sub team in our team, led by Pingou, who are running
advance investigations on some of the tech debt and bigger initiatives
that the CPE team have in our backlog and they have been tackling
Datanomer/Datagrepper tech debt first.
The team have been partitioning the ‘messages’ table of datagrepper's
DB, & hope to be able to test this setup next week
* prod like in openshift
https://datagrepper-monitor-dashboard.app.os.fedoraproject.org
* prod like with a default delta of 3 days
http://datagrepper.arc.fedorainfracloud.org/datagrepper/
* partitioned table + default delta of 3 days
http://datagrepper-test.arc.fedorainfracloud.org/datagrepper/
* using the timescale postgresql plugin [not implemented yet]
http://datagrepper-timescale.arc.fedorainfracloud.org

Noggin/AAA

* We faced some issues with IPA limits and tuning, and 2FA & still
trying to figure out the best way to enforce 2FA with sudo.
* We are getting closer to migrating from stg to prod and once the
Fedora migration is complete, the CentOS accounts will be then
imported.
* NOTE: If you have an account in both CentOS & Fedora and have
different email addresses associated with each, please update your
preferred email address in your profile and look out for an email next
week on your options.
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* Elections pr reviewed https://pagure.io/elections/pull-request/90
* Next is Greenwave & waiverdb
* Board the issues are tracked on are here
https://github.com/orgs/fedora-infra/projects/7

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

CentOS Dojo @ FOSDEM, 2021

February 08, 2021 07:20 PM

Last week we held our traditional annual CentOS Dojo at FOSDEM. We had 216 people registered, of whom 164 (75.9%) actually showed up to attend some part of it. A big thank you to those that turned up and made it a successful event.

In case you missed it, or some part of it, all of the content is now on YouTube.

On Thursday we had four presentations:

  • The Board of Directors had an "ask me anything" session, where questions were fielded from attendees. [Video]
  • Brian Exelbierd and Brian Stinson talked about CentOS Stream. [Video, Slides]
  • Tomas Tomecek talked about the contribution workflow of CentOS Stream, and how that is the process to land changes in RHEL. [Video, Slides]
  • David Duncan talked about building elastic configurations with EC2-Hibernate [Video, Slides]

And on Friday, we had four more:

  • Javier Peña and Alfredo Moralejo Alonso talked about how OpenStack became boring (and successful) [Video, Slides]
  • Davide Cavalca gave an introduction to the new Hyperscale SIG [Video, Slides]
  • Matthew Almond talked about speeding up DNF/RPM using copy on write [Video, Slides]
  • David Duncan talked about building an image pipeline with CentOS Stream and Image Builder [Video]

It was great to get together with the community, even though it was online. We had some great impromptu discussions in the "hallway track", and it was good to see some faces.

We want to do these at least quarterly for the remainder of this year - watch Twitter and the mailing lists for announcements of dates for the next event! We would also like to hear from you what content you would like to see at upcoming events, especially if you'd like to give a presentation about what you're working on.

January 15, 2021

CPE Weekly Report: 2021-01-15

January 15, 2021 06:54 PM

Hi Everyone,

New Year, same CPE weekly(ish)

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

General Project Updates

We are kicking off Q1 this year with some familiar project faces,
namely Noggin, the replacement of the current FAS system and
continuing our development of CentOS Stream.

Most of our initiatives live here
https://pagure.io/cpe/initiatives-proposal and you can use the new
issue button to submit your own proposal.

Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.

We also have updated our docs section on the initiative process we
follow as we cannot accept everything so please do check it out if you
want to understand our process more
https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

GitLab

Being very honest, I've found myself a little bit strapped for time to
give this project its due diligence over the last few months, but
please bear with us/me and expect a more concentrated effort on this
coming into Q2 (April, May, June) of this year. I apologise for the
time a resolution is taking and I really do appreciate all of your
patience.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Community newsletter can be read here
https://blog.centos.org/2021/01/centos-community-newsletter-january-2020-2101/

CentOS Stream

* Continuing to work on Stream 8 pushes and builds
* Investigating how to automate some module pushes
* Reviewing documentation that is available on Stream currently to
identify gaps and where needs improvement

Fedora

* OSBS is building for aarm64 & x86_64 in production since December!
* All of the projects under the fedora-infra and releng namespaces on
pagure have had their default branch migrated from “master” to “main”.
* F34 mass rebuild due to start next week

Noggin/AAA

* New sprint started focusing on testing correct access has been given
per user/account
* Last remaining apps being configured & tested with fasjson API
* Work will be tracked here https://github.com/fedora-infra/aaa-tracker/issues/4
* Our open issues board can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* We are working through supybot and greenwave applications currently
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

## Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

December updates

January 15, 2021 03:55 PM

I usually include the below report in the monthly newsletter, and overlooked it this month. So, without further ado, here are the CentOS 7 updates that were pushed out in December:

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during December:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during December:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during December:

Other releases

The following releases also happened during December:

January 12, 2021

CentOS Community Newsletter, January 2021 (#2101)

January 12, 2021 12:12 AM

Dear CentOS Community,

As we enter the new year, I'm sure there's really only one thing on your mind, and so we'll start there.

As you are no doubt aware, the CentOS project has shifted focus from CentOS Linux - the RHEL rebuild - to CentOS Stream - the continuously delivered distribution that reflects what will be delivered in the next release of Red Hat Enterprise Linux (RHEL).

Many, many articles have been written about this, and I want to take an opportunity to call out some of the better ones, to help you understand what's happening, and where we go from here.

To those who claim that CentOS Stream will be somehow unstable, I would encourage you to read Brendan's article about how RHEL is made. Things that go into RHEL are not bleeding edge or continually shifting sands. They are small incremental changes which have been baked for a long time.

To those objecting to the term "rolling release", see Stef's article about continuous delivery, and how CentOS Linux and CentOS Stream related to RHEL.

And to those who are pre-judging CentOS Stream without the benefit of even trying it, you should read Jack's article about not knocking it until you try it. (Jack's an Ubuntu fan, but makes a lot of good points.)

Karsten has written an article about the various things that are kept in balance around the CentOS project, and some of the history that led to where we are.

Finally, Scott's article about ... well, all of it ... is definitely worth your time if you want to have a deeper understanding about why people are angry, and why they are right, and wrong, to be angry.

For those of you who are planning to move to Rocky, CloudLinux, or one of the other projects that has sprung up to take the place of CentOS Linux, we wish you - and these projects - all the best. But we caution you to understand that building an OS is a big project, and it's going to take a while for them to get where they're going. Please plan your migration accordingly.

There are other things happening in the CentOS community, but we understand that this one is pretty overshadowing right now.

Hyperscale SIG proposed

A group of developers has proposed a Hyperscale SIG, which will be voted on in Wednesday's board meeting. They propose to focus on solutions around large-scale infrastructures, such as those at organizations such as Facebook and Twitter.

If you are interested in this kind of SIG, and particularly if you are running a hyperscale infrastructure, we welcome your comments and participation.

CentOS Linux 8 (20-11) released

The fourth release of CentOS 8 is now available, as of December 7th. This release is labelled 8.2011 (ie, November 2020) and is based on the 8.3 release of RHEL.

Q1 CPE Priorities

In Q1, CPE will be working on the following priorities:

  • CentOS Stream
  • Noggin/AAA replacement
  • Fedora-Messaging Schemas 
  • Flatpak indexer code merge
  • Debuginfo-d
  • Datanomer & Datagrepper V.2

We'll be updating the centos-devel list as progress is made on these projects.

Happy New Year

We wish you a 2021 that is happy and productive, and hope to see you in person before the year is out. Thanks, as always, for being part of our community.

 

December 19, 2020

Balancing the needs around the CentOS platform

December 19, 2020 06:40 AM

These past few weeks I’ve read through and listened to a lot people’s reactions and responses to our news about the future of the CentOS Project. I see a lot of surprise and disappointment, and I also see people worried about the future and how this is going to affect them, their livelihoods, and the ecosystem as a whole. I feel a strong sense of betrayal from people, I hear that.

I don’t know if my story here is going to help you or not, but I appreciate you reading it through and listening to what I have to say. The history I cover I think is necessary to understand where we are today. From here I’m going to be available on the CentOS devel list and Twitter if you want to talk further about why I think it’s going to turn out okay.

I’ve been on the CentOS Project Governing Board since its creation. I also was part of the consensus decision that we recently announced about shifting the project’s focus.  I’ve cared about this space for a long time, for my 19 years at Red Hat and prior to that. I was involved in the Fedora Project since the earliest days, leading the documentation project and sitting on the then-Fedora Board, among other roles. I led the team at Red Hat that brought the CentOS Project in closer to Red Hat in 2013/2014, and as a result of that work I earned a seat on the CentOS Governing Board, where I was the Red Hat Liaison and Board Secretary until Spring 2020.

Let’s go back to 2003 where Red Hat saw the opportunity to make a fundamental change to become an enterprise software company with an open source development methodology.

To do so Red Hat made a hard decision and in 2003 split Red Hat Linux into Red Hat Enterprise Linux (RHEL) and Fedora Linux. RHEL was the occasional snapshot of Fedora Linux that was a product—slowed, stabilized, and paced for production. Fedora Linux and the Project around it were the open source community for innovating—speedier, prone to change, and paced for exploration. This solved the problem of trying to hold to two, incompatible core values (fast/slow) in a single project. After that, each distribution flourished within its intended audiences.

But that split left two important gaps. On the project/community side, people still wanted an OS that strived to be slower-moving, stable-enough, and free of cost—an availability gap. On the product/customer side, there was an openness gap—RHEL users (and consequently all rebuild users) couldn’t contribute easily to RHEL. The rebuilds arose and addressed the availability gap, but they were closed to contributions to the core Linux distro itself.

In 2012, Red Hat’s move toward offering products beyond the operating system resulted in a need for an easy-to-access platform for open source development of the upstream projects—such as Gluster, oVirt, and RDO—that these products are derived from. At that time, the pace of innovation in Fedora made it not an easy platform to work with; for example, the pace of kernel updates in Fedora led to breakage in these layered projects.

We formed a team I led at Red Hat to go about solving this problem, and, after approaching and discussing it with the CentOS Project core team, Red Hat and the CentOS Project agreed to “join forces.” We said joining forces because there was no company to acquire, so we hired members of the core team and began expanding CentOS beyond being just a rebuild project. That included investing in the infrastructure and protecting the brand. The goal was to evolve into a project that also enabled things to be built on top of it, and a project that would be exponentially more open to contribution than ever before—a partial solution to the openness gap.

Bringing home the CentOS Linux users, folks who were stuck in that availability gap, closer into the Red Hat family was a wonderful side effect of this plan. My experience going from participant to active open source contributor began in 2003, after the birth of the Fedora Project. At that time, as a highly empathetic person I found it challenging to handle the ongoing emotional waves from the Red Hat Linux split. Many of my long time community friends themselves were affected. As a company, we didn’t know if RHEL or Fedora Linux were going to work out. We had made a hard decision and were navigating the waters from the aftershock. Since then we’ve all learned a lot, including the more difficult dynamics of an open source development methodology. So to me, bringing the CentOS and other rebuild communities into an actual relationship with Red Hat again was wonderful to see, experience, and help bring about.

Over the past six years since finally joining forces, we made good progress on those goals. We started Special Interest Groups (SIGs) to manage the layered project experience, such as the Storage SIG, Virt Sig, and Cloud SIG. We created a governance structure where there hadn’t been one before. We brought RHEL source code to be housed at git.centos.org. We designed and built out a significant public build infrastructure and CI/CD system in a project that had previously been sealed-boxes all the way down.

However, the development of RHEL itself still remained closed behind the Red Hat firewall.  This had been true for almost twenty years. For the open source development ecosystem this has been an important and often painful gap—it’s the still same openness gap as 2003.

This brings us to today and the current chapter we are living in right now. The move to shift focus of the project to CentOS Stream is about filling that openness gap in some key ways. Essentially, Red Hat is filling the development and contribution gap that exists between Fedora and RHEL by shifting the place of CentOS from just downstream of RHEL to just upstream of RHEL.

Just as when we joined forces, Red Hat approached the CentOS Project with its plan, and the CentOS Board signed on to it. That plan centered around not just closing the feedback-loop part of the openness gap, but in finding a way to help evolve RHEL development from happening inside of Red Hat to outside of it.

The Board was fully aware that in filling one gap we risked reopening the availability gap on the end-user side of the equation. While CentOS Stream would be open to contribution in a way that it never had been before, it would stand the risk of being somewhat different than CentOS Linux has been.

But we also knew as a project trying to do two antithetical things at once would mean doing both poorly. Providing our community with a solid, reliable distro that is good-enough for your workloads is a strong part of the CentOS brand. We’re confident that CentOS Stream can do this.

And while I’m certain now that CentOS Linux cannot do what CentOS Stream can to solve the openness gap, I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the end.

Beginning now is the time to genuinely help the CentOS Project understand what you need in a CentOS Linux replacement, in some detail. Even your angriest of posts are being read, and your passionate viewpoints are being seen and understood. I’m not the only Linux old-timer working on this.

This is your chance to be recognized for where you land in the availability or the openness gap, and how it is being there, so that the people crafting RHEL solutions are doing it with your use case(s) in mind. This input is happening right now. The new email address centos-questions@redhat.com goes directly to the people in the business unit (who are not in Sales) trying to solve your problems using this open source development method.

It is hard to balance the needs and processes of making business decisions with the needs and processes of making open community decisions. Arguably, Red Hat has been among the best organizations at straddling this hard, thin line. If you trust our code enough to run it for this long, I ask you to trust us to make good decisions here. I ask you to trust Red Hat and the CentOS Board to work with you to find a way to bring the community along into the next chapter.

If you want to talk with me further, the best place is the centos-devel list or Twitter.

December 11, 2020

How RHEL is Made

December 11, 2020 05:49 PM

This week Red Hat announced its plan to put all its energy into CentOS Stream 8, resulting in the discontinuation of CentOS Linux 8 in one year’s time.  CentOS Stream, originally announced in September of 2019, is a continuous release of RHEL which provides updates as soon as they are developed and verified.  Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable?  If you want to know what to expect from CentOS Stream, the best starting point is knowing how Red Hat Enterprise Linux is built.  Let’s get into it!

Red Hat has been making Linux releases for such a long time, its original development methodology predates the agile manifesto.  Historically, RHEL has been built behind closed doors, its plans held close enough that even the announcement of predictable 6-month minor / 3-year major releases seemed a monumental reveal during the RHEL 8 launch.  Fortunately, how Red Hat makes Linux distributions has evolved, not just since calendar years started with “19”, and there have been multiple process generations since RHEL 8 launched just 18 months ago.  While fundamentals like upstream first, copious quality engineering, ecosystem partnership, and customer care remain the same, we work continuously to improve how those fundamentals are implemented.  

Let’s start with grounding: every RHEL minor release is based on the previous release, plus targeted backports of upstream development.  Often, Red Hatters are the original authors of those patches, but there are no shortcuts: upstream acceptance is the first test every patch must go through before we start it through the journey that eventually leads to a patch’s integration in the release.  Even then, this is about an upstream patch existing, but that alone will not guarantee a patch’s inclusion.

Any decision to introduce an upstream change into RHEL is a team decision and the team is large: developers, quality engineers, support personnel, product owners, and various partners all work together on priority and feasibility.  Once a decision is reached and commitments are made, only then do developers and quality engineers begin writing code.  As you probably know, in the most congenial of rivalries, developers try to write code that nobody can break and quality engineers create batteries of ways to break the code developers write.  This brings us to the second key place where Red Hat invests: tests.

We write tests for everything: unit tests, systemic tests, kernel and userspace ABI conformance tests, performance tests, dependency tests, architecture tests, driver tests, load tests, and many more.  Having tests is foundational, but it is their application that brings meaning.  This brings us to the third key area where Red Hat invests: process infrastructure.

For the last several years, Red Hat has worked on a series of “Always Ready” operating system initiatives.  The goal is as simple as the name suggests: at any moment in time the OS is ready for release.  It’s easier to describe than it is to implement. In complex systems, so many things can have unintended consequences.  To handle this we use layers of automation, incrementally building confidence in changes, before they are integrated and released into the distribution.  Here is a high-level sketch of the process every single change in RHEL must go through to be included:

When a change is targeted at RHEL, multiple incremental steps occur before it is actually included.  Changes are built, but the only certain outcome is that a CI system will run a suite of tests on the builds (the build is not yet made available for general use).  If those tests pass, a second round of preverification specific to the code change occurs (not yet good enough).  And if those tests pass, the change is tentatively included in the errata system and subject to further verification (it’s still not ready to publish).  Systemic test suites run on the combined whole to verify the gestalt functionality.  And if those tests pass, the build will finally make it into the space where CentOS Stream systems recognize it as an available update.  It’s a long pipeline and many changes move through it every single day.  For those interested in more of the vision and architecture of this system, you can read more in CentOS Stream is Continuous Delivery!

While the description of this system may seem elegant and reassuring, watching it in action can feel quite the opposite: The more testing is done the more bugs are found- and Red Hat does a whole lot of testing.  Historically, RHEL development has been done behind closed doors, isolating people from the routine bug identification and remediation process, only allowing the world to see the end result.  In the future, as RHEL development becomes more transparent, as we approach RHEL 9, this process will become uncomfortably visible.  While the testing systems are built to prevent such failures from reaching end users, anybody who wants to look deeper may be surprised at how messy operating system development can be!

Finally, for those who wonder how soon all this will map to CentOS Stream, we have good news: it is already happening today with RHEL 8.4 and CentOS Stream 8!  At the same time these RHEL builds are verified, they are also delivered to CentOS Stream.  Of course we aren’t done yet: CentOS Stream has not yet realized its mission of adding a developer community around RHEL, that is where we are headed, into a place where there are more options to engage with Red Hat and shape future RHEL.  There is always room for improvement, from better tests to more facets in future collaboration, we are excited to share building RHEL with you so that we build a better OS with and for you.

Minutes for CentOS Board of Directors for 2020-11-11

December 11, 2020 05:49 PM

On 2020-11-11 the CentOS Board of Directors met to discuss ongoing business.

First, the board would like to thanks everybody involved in CentOS Linux 7.9 release.

The Board was in an Executive session, where Red Hat CTO, Chris Wright joined to present Red Hat plan around CentOS Linux and CentOS Stream. A Board discussion followed.

Following up the discussion around the different users' communities impacted by proposed changes, Chris Wright, mentioned to the Board that Red Hat is also reshaping and expanding the RHEL Developer program. The details will be communicated through standard Red Hat channels.

The following resolutions were approved by the majority of the Board :

  • CentOS Stream 8 will continue with contributions for the full-support phase of RHEL 8. APPROVED
  • CentOS Stream 9 will start on schedule with the RHEL 9 Beta. APPROVED
  • CentOS Linux 9 will not start. APPROVED
  • CentOS Linux 8 ends in December 2021. APPROVED

An announcement and detailed FAQ will be prepared in next weeks.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

Minutes for CentOS Board of Directors for 2020-09-09

December 11, 2020 05:41 PM

On 2020-09-09 the CentOS Board of Directors met to discuss ongoing business. The Board discussed the creation of the infrastructure SIG to streamline and foster contributions in this challenging area. 

The creation of the Infrastructure SIG was approved and Aoife Moloney will be the new chair in charge to gather requirements from all actors and define the SIG baseline contribution model. Board members insisted that administrative accesses to the CentOS infrastructure will need to be carefully granted and on the principle of least privilege. Rich Bowen kindly drafted a blog post that goes in deeper details for the next steps and the challenges ahead.

A lengthy exchange happened around the feedback loop from SIG to the CentOS Board. The board would like to invite SIG chairs (or their representative) for discussion when needed, but also hear from them on a regular basis. A communication will follow-up when the board agrees on the best format and frequency for these sessions.

Ralph Angenendt announced he decided to step down from the CentOS Board of Directors. The Board would like to thank Ralph for all his service and hard work over the years.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

CentOS Stream is Continuous Delivery

December 11, 2020 04:14 PM

Continuous Delivery 101: Do the hard things continuously,
so that they become easy.

From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.

I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.

Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.

Obvious, no? … No.

The Linux distribution is the grand challenge of
continuous integration and delivery.

What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.

Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.

The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.

If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.

With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.

The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.

Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.

The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.

Are we done? … No.

To the untrained eye,  CentOS Stream is
already 
as stable as RHEL.

But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..

Fedora, CentOS Stream and RHEL delivery

Diagram licensed CC-SA: https://creativecommons.org/licenses/by-sa/4.0/

Here’s how the flow of delivery looks for 8 and 9:You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.

Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.

The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.

CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.

But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.

As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.

While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.

Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...

And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.

CentOS Stream is the stable and reliable
continuous delivery of RHEL

December 08, 2020

CentOS Project shifts focus to CentOS Stream

December 08, 2020 01:57 PM

The future of the CentOS Project is CentOS Stream, and over the next year we’ll be shifting focus from CentOS Linux, the rebuild of Red Hat Enterprise Linux (RHEL), to CentOS Stream, which tracks just ahead of a current RHEL release. CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021. CentOS Stream continues after that date, serving as the upstream (development) branch of Red Hat Enterprise Linux.

Meanwhile, we understand many of you are deeply invested in CentOS Linux 7, and we’ll continue to produce that version through the remainder of the RHEL 7 life cycle.

CentOS Stream will also be the centerpiece of a major shift in collaboration among the CentOS Special Interest Groups (SIGs). This ensures SIGs are developing and testing against what becomes the next version of RHEL. This also provides SIGs a clear single goal, rather than having to build and test for two releases. It gives the CentOS contributor community a great deal of influence in the future of RHEL. And it removes confusion around what “CentOS” means in the Linux distribution ecosystem.

When CentOS Linux 8 (the rebuild of RHEL8) ends, your best option will be to migrate to CentOS Stream 8, which is a small delta from CentOS Linux 8, and has regular updates like traditional CentOS Linux releases. If you are using CentOS Linux 8 in a production environment, and are concerned that CentOS Stream will not meet your needs, we encourage you to contact Red Hat about options.

We have an FAQ to help with your information and planning needs, as you figure out how this shift of project focus might affect you.

[See also: Red Hat's perspective on this.]

Thoughts on CentOS Stream

December 08, 2020 08:00 AM

I’m excited to see the CentOS project and Red Hat work together and collaborate around CentOS Stream, and I’d like to explain why I think this is a good move. I’ve been a member of the CentOS project for the last 16 years, and in that time I’ve had countless conversations with developers who were targeting enterprise deployments, but who wanted to push things just a little beyond what was currently available. In my early days with the project it was “I just need this new feature from PHP” or “I need this one option enabled in postfix”. This was so common it spawned an entire cottage industry of 3rd party repositories like Elrepo, IUS, nux-desktop, and even our own CentOSPlus. Often these were features Red Hat would include in a future version of RHEL, but the timing and communication around these features was a mystery. Red Hat never announced release dates or upcoming features and for many developers, even for those internal to Red Hat this was a PROBLEM.

When I joined Red Hat to work on CentOS full time, they outlined the goal pretty clearly: “We want to showcase our upstream community work we intend to put in our layered products”. Red Hat’s developers were using CentOS to do their upstream development work, and our role was to help them. This quickly became a problem, because the way to get new work into RHEL was Fedora, but that’s often not practical for a variety of reasons (Software Collections, modularity structure, release cadence, etc). Nothing here is new. Red Hat’s Josh Boyer and Brendan Conoboy spoke at length about this challenge in their Penrose Triangle talk at Flock in 2018

CentOS Stream represents several positive steps for Red Hat here.

  1. It makes RHEL development more transparent and reliable.
  2. It provides a way for ISVs and developers to contribute fixes and features.
  3. It provides a way for the community to provide feedback.

Did you download the RHEL8 public beta? Did you notice that the python setup looked VERY different between the GA and the beta? If you spent the 6 months between the RHEL beta and GA developing your app to work, it’s possible you had the rug yanked out from under you. CentOS Stream solves that by providing constant updates so you can see what changes are coming, and adjust accordingly. Because RHEL’s development is now transparent to the public, devs shouldn’t be surprised by changes, they’ll finally be able to see them coming.

Seeing what Red Hat is doing is one thing, but for those users I mentioned earlier who just needed that one new feature - they now have a way to collaborate with Red Hat to make it a reality. CentOS Stream provides a way for users to submit pull requests and to make their case for why it should be included. This obviously doesn’t mean everyone will get their way, but it’s a stark improvement from the past.

I’m excited to see the CentOS project and Red Hat work together and collaborate around CentOS Stream, and I’d like to explain why I think this is a good move. I’ve been a member of the CentOS project for the last 16 years, and in that time I’ve had countless conversations with developers who were targeting enterprise deployments, but who wanted to push things just a little beyond what was currently available. In my early days with the project it was “I just need this new feature from PHP” or “I need this one option enabled in postfix”. This was so common it spawned an entire cottage industry of 3rd party repositories like Elrepo, IUS, nux-desktop, and even our own CentOSPlus. Often these were features Red Hat would include in a future version of RHEL, but the timing and communication around these features was a mystery. Red Hat never announced release dates or upcoming features and for many developers, even for those internal to Red Hat this was a PROBLEM.

December 01, 2020

CPE Weekly: November 22 2020

December 01, 2020 08:14 PM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-11-22 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

Final project submission date for consideration in Q1 is Friday 27th
November. If you have an initiative that may take weeks/months and
multiple people to work on and want to request it to CPE, please
follow the steps outlined in our initiatives repo and create your
issue before 27th November https://pagure.io/cpe/initiatives-proposal
Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/

Below are the projects the CPE team are currently working on for the
months of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

New GitLab topic sent to devel-announce@lists.fedoraproject.org &
centos-devel@centos.org on Message Bus is out. See email in hackmd
here
https://hackmd.io/oZrDwbSeSWO-l_X65A1ndg?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 6 is EOL 30th November
* CFP for FOSDEM Dojo - https://wiki.centos.org/Events/Dojo/FOSDEM2021
* Updated CentOS CI Openshift staging cluster to latest 4.6.4, Waiting
for stable release in the 4.6 branch before rolling out to production.
* CentOS 7.9.2009 is released! (for x86_64, i386, ppc64, ppc64le,
armhfp and aarch64 architectures)
* Lot of work done for Noggin/AAA

CentOS Stream

* Use centos-stream-release package to convert from CentOS 6 to CentOS
Stream before 30th November
* Working on integrating ODCS in Stream
* Curating out t_functional suite
https://github.com/centos/sig-core-t_functional
* Refining our testing for finding issues at distro-level

Fedora

Staging Environment

* Completed - any issues you find please report them in fedora infra
https://pagure.io/fedora-infrastructure/issues

Noggin/AAA

* Testing team owned apps in staging with Noggin
* We will be requesting community member testing before December so
keep an eye out for an announcement!
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

OSBS for aarch64

* Basic OKD 3.11 working on aarm64 with F31
* Working on repeating that install with F33
* Next step will be to

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

 

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-12-17 @ 1300 UTC on #fedora-meeting-1

#centos-meeting

* Next Meeting: 2020-12-15 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
ReplyForward

_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel

 

Minutes for CentOS Board of Directors for 2020-10-14

December 01, 2020 10:49 AM

On 2020-10-14 the CentOS Board of Directors met to discuss ongoing business. The Board was in an Executive session, where Red Hat CTO, Chris Wright, requested feedback from the participants on the progress of the CentOS Stream project since its creation, last year. Also, a discussion around resources for both CentOS Linux and CentOS Stream took place. A plan for improving and fostering development around CentOS Stream, as a RHEL upstream platform, will be presented and discussed when details are sorted out, in a future board meeting.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

CentOS Community Newsletter, December 2020 (#2012)

December 01, 2020 12:36 AM

Dear CentOS Enthusiast,

With many of you celebrating one holiday or another this time of year, we want to extend to you the warmest wishes for your Thanksgiving, Diwali, Christmas, Hanukkah, Kwanzaa, New Years, and holiday season. We hope for each of you that 2021 brings new opportunities, and much happiness.

We have a few news items to share with you in this newsletter.

CentOS Linux 6 EOL

As has been announced everywhere for the past year (and more!) CentOS 6 has been moved to End Of Life (EOL) status as of November 30th, 2020. During the first week in December 2020, the 6.10 directory will move to vault.centos.org

Packages will still be available at: http://vault.centos.org/centos/6.10/. However, once moved, there will be no more updates pushed to vault.centos.org. Therefore, security issues will no longer be fixed.

Should you require continued support for this version, we encourage you to contact Red Hat about Extended el6 support for RHEL.

CentOS Linux 7.2009

We are pleased to announce the general availability of CentOS Linux 7 (2009). Effectively immediately, this is the current release for CentOS Linux 7 and is tagged as 2009, derived from Red Hat Enterprise Linux 7.9 Source Code.

As always, read through the Release Notes at : http://wiki.centos.org/Manuals/ReleaseNotes/CentOS7 - these notes contain important information about the release and details about some of the content inside the release from the CentOS QA team. These notes are updated constantly to include issues and incorporate feedback from the users.

See the mailing list announcements for the x86_64 and altarch releases.

CentOS Linux 8 (Release to come)

Stay tuned! We expect to have a release of CentOS Linux 8, based on RHEL 8.3, any day now. (Indeed it may already be released when you are reading this.) Watch the centos-announce mailing list for the announcement!

FOSDEM Dojo: CFP now open

As has been our tradition for a decade now, we'll be hosting the CentOS Dojo on the day before FOSDEM 2021. FOSDEM has gone virtual this year, and our Dojo will also be online. The Call for Presentations (CFP) is now open. Details are on the Dojo wiki page.

The event will be held on Friday, February 5th, 2021. We will expand to include February 4th if we receive enough talk submissions.

We are looking for talks about:

  • CentOS Linux
  • CentOS Stream and the RHEL contribution workflow
  • Anything that you are doing in the CentOS ecosystem (including, but not limited to: SIG activity, CentOS infra, governance, community, etc)
  • Anything you're doing *on* CentOS - stuff you're running on CentOS, interesting research projects, useful CentOS tools/utilities/techniques.
  • Larger ecosystem topics - Linux, Cloud, Open Source, etc.

I encourage you to look at previous event schedules for further inspiration: 

CentOS is also usually well represented in the Distributions devroom.

CPE Q1 Priorities

In the coming days, you'll see a thread on the centos-devel mailing list regarding CPE's (Community Platform Engineering) priorities in Q1. We have a vote, as a community, to influence what they'll be working on. So we encourage you to watch for that thread, and express your opinions, so that we can ensure that CPE is using their time in a way that benefits us the most.

SIGs building against CentOS Stream

In case you missed it: Johnny mentioned a few days ago that SIGs can now build against CentOS Stream. If your SIG isn't yet, and wants to, please check out that thread, and get in touch with the list to get started!

SIG reports

SIGs - Special Interest Groups - are communities who build various things on top of CentOS. This month we have reports from two of our SIGs:

Virtualization SIG

oVirt: Upstream released 4.4.3 which introduces cluster compatibility level 4.5 with additional features enablement but requires RHEL AV 8.3 in order to work. So we are waiting for CentOS 8.3 to be released so the Advanced Virtualization team will be able to rebuild it from RHEL-AV packages.

CentOS OpsTools SIG

Only two notable changes happened during this quarter: a rebuild of collectd-sensubility fixing major issues and the addition of collectd-libpodstats, which is a plugin to monitor pods and to report their usage via collectd.

No new members were attracted, but also no member voiced they are not interested anymore. We've had a request from outside to include a package, but the requester himself was not interested in any contribution.

Happy New Year!

Once again, we hope that you have a safe, happy, and prosperous new year in 2021, and that we see all of you again soon.

November 24, 2020

What is CPE up to: CentOS Stream

November 24, 2020 05:23 PM

Last week I had a short chat with Brian Stinson and Carl George, from Red Hat's CPE (Community Platform Engineering) team about the work they are doing to enable CentOS Stream. (9 minutes)

November 18, 2020

An update on our Gitforge

November 18, 2020 02:21 PM

Hey everyone,

Back in March we published this blog where the CPE team came to a decision about a future Gitforge. The decision was made to opt for Gitlab as the Forge of choice. We can now announce that the service has been stood up successfully, with Gitlab running this as a SaaS offering on behalf of the community. We are still in the process of making configuration changes and starting to seed content. In the background the CPE team are working through their tooling and configurations to ensure when we launch that the process and experience is seamless for the community.

We expect CentOS Stream distribution repositories to be the first content that will show up in Gitlab. We plan on giving plenty of notice whenever we intend to relocate other things like SIG repos or Downstream source drops from Red Hat. Until then git.centos.org will remain up and running to support your work.

November 03, 2020

CentOS Community Newsletter, November 2020 (#2011)

November 03, 2020 01:48 AM

Dear CentOS Enthusiast,

It's been another fairly quiet month in CentOS, but I have a few things to share with you.

News

CPE

Last week I spoke with Aoife Moloney and Stefan Mattejiet of Red Hat's Community Platform Engineering (CPE) group. CPE do a bunch of infrastructure work for Fedora and CentOS, and we've been trying to bring more attention to what they're working on, on our behalf, over the last few months. My conversation with them is on YouTube, and I hope to have more of these interviews over the coming weeks to tell you about specific initiatives.

I would encourage you also to read CPE's weekly updates (October 17th, October 25th), and their Q3 summary for more detail.

If you have suggestions or requests for what CPE should work on in coming sprints, please get in touch with your ideas. You can email the centos-devel mailing list, or contact Rich Bowen directly off-list at rbowen@centosproject.org

CentOS 6 End Of Life, November 30th.

This is your final warning that CentOS 6 will be designated "End Of Life" on November 30th. After that time, it will receive no more updates.

Fedora 33

Our friends over at Fedora released Fedora 33 last week. Celebrate with them by attending their release party this weekend! Register here: https://hopin.to/events/fedora-33-release-party

Updates

A number of people have noticed that there has been no activity on the centos-announce mailing list in October. That's because all of the activity is over on the centos-cr-announce list, as we prepare for a 7.9 point release, which we expect to happen in the coming couple of weeks.

Meanwhile, we encourage you to keep an eye on the Building 7 page for daily updates of the status of that release.

Events

FOSDEM Dojo, February 4th and 5th

Although FOSDEM looks different this year, we plan to continue our tradition of holding a CentOS Dojo on the day (or days) before FOSDEM begins.

This event will take place online. Doing this online gives us access to a wider pool of speakers and attendees - this means YOU! The call for presentations is now open.  The actual schedule will depend on submissions that we receive, but we are tentatively hoping to run the event on February 4th and 5th, ahead of FOSDEM starting on the 6th.

We are looking for presentations about anything that you are doing in, or on, CentOS Linux and CentOS Stream. This includes, but is certainly not limited to, SIG activity, research using CentOS, useful tools/applications that can be deployed on CentOS, or topics around community engagement.

More details about the event will be available on the event wiki page as soon as we know more.

DevConf.cz, DevConf.in

The Call for Presentations for both DevConf.in and DevConf.cz are now open. The .cz CFP closes on November 6th, and the .in CFP closes on November 15th. You can access both of them at cfp.devconf.info.

October 27, 2020

Using connection delegation with mitogen for Ansible

October 27, 2020 11:00 PM

This should be a very short blog post, but long enough to justify a blog post instead of a 'tweet' : I had myself a small issue with mitogen plugin in our Ansible infra.

To cut a long story short, everybody knows that ansible relies on ssh as transport. So one can use traditional ~/.ssh/config tuning to declare ProxyJump for some hosts, etc

But when you use mitogen (we do), in the official doc there is a mention of specific parameter for connection delegation : mitogen_via

The simple example on the webpage seems trivial and if you have multiple hosts that need to be configured from remote ansible+mitogen combo, using mitogen would speed things up as it would know about the host topology.

That's what I thought when having a look at the simple inventory on that web page:

[dc2]
web1.dc2
web2.dc2
web3.dc2

[dc2:vars]
mitogen_via = bastion.dc2

Sounds easy but when I tried quickly to use mitogen_via , something that I thought would be obvious in fact wasn't. My understanding was that mitogen would automatically force agent forwarding when going through the bastion host. A simple ansible -m ping (let's assume web1.dc2 in their example) returned me :

web1.dc2 | UNREACHABLE! => {
    "changed": false,
    "msg": "error occurred on host bastion.dc2: SSH authentication is incorrect",
    "unreachable": true
}

Well, we can see from the returned json that it was trying to pass through bastion.dc2 and that's confirmed on web1.dc2 :

Oct 28 15:52:36 web1.dc2 sshd[12913]: Connection closed by <ip_from_bastion.dc2> port 56728 [preauth]

Then I thought about something that was obvious to me but that mitogen (just reusing underlying ssh) doesn't do automatically : Forwarding the ssh agent to the nodes behind.

We can easily solve that with one simple ansible parameter : ansible has the ansible_ssh_common_args and ansible_ssh_extra_args parameters, specific to the SSH connection

So what about we force Agent Forward just on that bastion host and see how that works ? That means that in our inventory (but can go to host_vars/bastion.dc2 too) we just have to add parameter:

bastion.dc2 ansible_ssh_extra_args='-o ForwardAgent=yes'

Let's try again :

web1.dc2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

Good, so we can push that for our bastion hosts (used in inventory for mitogen_via) in host_vars or group_vars and call it a day. The reason why I prefer using ansible_ssh_extra_args is that it will merge and add settings, in case you have already something like this in your ansible.cfg :

[ssh_connection]
ssh_args =

I like the logic that we don't need to modify ~/.ssh/config with all exceptions to reflect the infra layout but we can just reflect it in ansible inventory

October 25, 2020

CPE Weekly: 2020-10-25

October 25, 2020 12:02 AM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-25 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

Updated initative timetable can be viewed here
https://docs.fedoraproject.org/en-US/cpe/time_tables/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

Sent a mail to the devel lists for both Fedora & CentOS with questions
that had answers relating to the topic Accouns & Permissions. It has
been sent to devel-announce at fedoraproject.org &
centos-devel at centos.org. Here is the link to the hackmd doc I used to
write the email before copying it across to my email to send
https://hackmd.io/1pjX1cVnTjekOLVowj5UiQ?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 7.9.2009 pkgs pushed to CR repo, next step is to install tree
and artifacts such as iso & cloud images.
* Working a lot with the AAA/Noggin team on CentOS account integration
and making good progress

CentOS Stream

* There are now cloud image updates available for CentOS Stream
* There are some documentation updates here on unshipped packages &
Stream feel free to read up!
https://wiki.centos.org/FAQ/CentOS8/UnshippedPackages

Fedora

General

* F33 will be released next week!!
* The team got an F33 nightly compose finished too!
* There is a dashboard available to view the performance of Anitya
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/8Zi9LU5Mz/anitya?orgId=1
* And there is capability to do the same for the packager workflow
pipeline which we are still working on

Staging Environment

* Build system nearly done - waiting on a firewall change

Noggin/AAA

* New estimated deployment date for Noggin is 29th January 2021.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 (On Freenode IRC)

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting (On Freenode IRC)

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

October 17, 2020

CPE Q3 Achievements 2020

October 17, 2020 03:17 PM

Hi there,

I'm Aoife Moloney. You may remember me from such communications as the CPE office hours, Data Centre - what it means for you, and The Future of Communishift.

Over the last three months, the Community Platform Engineering team (or CPE for short as it's long to keep typing) have been working on a few projects, and generally surviving 2020 like everyone else. But we made it, and so did our projects! Mostly… 🙂

 

Over the last three months we worked on:

  • The Great Fedora Data Centre Move of 2020
  • Noggin
  • CentOS Stream
  • Packager Workflow Healthcare (Always check with your maintainer before taking this workflow. Side effects may be, but not limited to, frustration, tears, elation at successful builds)
  • Fedora-Messaging Schemas

 

We also had our long standing (and long suffering) ‘sustaining team’ on the front lines who are daily maintaining and running both the Fedora and CentOS infrastructures and responding to issues, bugs, etc. And doing a damn fine job too.

 

And we attended and participated in a few conferences too, namely Nest with Fedora & DevConf US.

 

So, what did we as a team overall achieve in these last few months?

 

CPE Infra & Releng Team 

This team was led by Pingou, and its members in Q3 were Mark O’Brien, Michal Konecky, Fabian Arrotin, David Kirwan, Kevin Fenzi, Vipul Siddharth, Stephen John Smoogen & Tomas Hckra.

This team is a sub team of CPE and focuses on lights on work in both the Fedora and CentOS infrastructures. We will always have some of our team members working in this way each quarter as it is good to have a break from scheduled project workloads and take a foray into the (sometimes) chaotic world of infrastructure maintenance, aka FIRE!!! 🙂

What they did: 

  • Changed their name. May we present: CPE Infra & Releng Team - oooooh, aaaaah!
  • Vipul & David worked with Fabian in the CentOS infra and did something with openshift clusters & migrated the kojihub for https://cbs.centos.org to a new infra
  • Kevin and Smooge moved all of the Fedora infrastructure. 117 servers.  Let that number sink in.
  • Pingou & Michal did a ton of babysitting toddlers 🙂 They moved a lot of scripts over and things are working well
  • Tomas helped bootstrap F33 - oh yeah!
  • And Mark had (not biologically,  but in sentiment) a baby! And became an admin for some of the Fedora Infrastructure.

Why its good:

  • The name change represents what this team works on and is easy to understand instantly. Plus, naming is hard so we wanted to keep it simple 🙂
  • Helped release F33 beta
  • We doubled down on toddlers allowing to build more automation around the infrastructure
  • Assisted with the fedora datacentre move for minimal disruption to the fedorans day to day lives
  • Helped keep CentOS CI operational - and then helped put out the flames when it caught fire 🙂
  • Over 500 tickets across both Fedora & CentOS infra + releng resolved by this team - that is some seriously good firefighting!

 

Fedora Data Centre Move

This dynamic duo was Kevin Fenzi and Stephen Smoogen, with supporting cast members from both CPE and the community along the way. The goal of this project was to successfully move a (large) number of the Fedora infrastructure hardware from one datacentre to the other without too much chaos. And considering the world wide pandemic that happened right at the beginning, they did a pretty fine job succeeding. Some additional services are still being added to the infrastructure in its new home in IAD, so if you notice a few still missing, we are getting to them slowly but surely and thank you again for your patience and understanding during these last few months!

What they did: 

  • Moved a ton of servers across the country of the United States
  • Kept critical services in Fedora Infrastructure alive during the move
  • Worked an uncountable amount of hours!

Why its good:

  • We got some new hardware!
  • The team carried out some resilience testing in the new data centre which means more reliability for the infrastructure should bad things happen 
  • Updated records and warranties were a passive benefit of this move too

 

Noggin

This team was led by Aurelien Bompard, and its members in Q3 were Ryan Lerch, Nils Philippsen & James Richardson. The goal of this project is to replace the current FAS system with a newer one and migrate the CentOS accounts to the one FAS instance (Noggin), which will mean our team has one authentication system to maintain for two infrastructures long term. This team has been working to a November 2020 deadline, but unfortunately during Q3 the team faced a number of challenges such as a delayed staging environment to test in due to the data centre move, then when we got it, realized their plugin they spent time developing was not going to work long term and now have to redo a bit of work in Q4. There were also a lot of holidays and personal events for the team in Q3 because everyone is human and entitled to a life 🙂 They have re-scoped their work for Q4 to make sure what's delivered is sustainable and reliable long term, more people have joined the team including some sys-admin for support along the way, and are now looking at delivering Noggin in full by the end of January 2021. 

What they did: 

  • A lot of ipsilon investigation
  • Added a spam curtailer service to Noggin
  • Added an agreements section for users to select their user preferences 
  • Deployed Noggin to staging but found out the way they did it wont be good for the project long term
  • Had a little cry about developing a plugin unnecessarily, hugged it out and then re-planned dates and the work we need to do in Quarter 4 to be able to deliver a better, more reliable and robust service in January 2021. Queue Noggin’  - Rise of the Phoenix Project

Why its good:

  • We knew where we went wrong, learned a lot both technically and as a project team for it, and were able to call the mistakes out and get the support we need to get the project back on track. Just a little bit later than we wanted.
  • We still created a solution that will meet the needs of both the CentOS and Fedora community users, and once we have the correct configurations in place and are ready to be tested we look forward to your feedback!

CentOS Stream:

This team was led by Brian Stinson, and its members in Q3 were Johnny Hughes, Carl George, Mohan Boddu, Leonardo Rosetti, James Antill & Siteshwar Vashisht

What they did: 

  • A lot of darn package & module building
  • Light hearted threats to their PO to teach her how to convert a CentOS Linux distro to CentOS Stream using the new release package - which she did! 🙂
  • Kept CentOS Stream compose up to date with RHEL nightlies
  • Launched the centos-stream-release package - Big deal. Like, huge.

Why its good:

  • CentOS Stream is continuing to stand on its own and becoming a more robust distro
  • There's lots more content in Stream for its users
  • Users can now swap from CentOS Linux to Stream easily

Packager Workflow Healthcare:

This team was led by Will Woods and its team members were Adam Saleh and Stephen Coady with Pingou in a part-time consulting/reviewing role. The team took a look into the Fedora packager workflow and tried to identify weaker points in the chain, and spot times that are more prone to downtimes.  They are finalizing a report of their findings to send to the community lists with hopefully a ‘next steps’ section that they feel will help reduce the issues packagers face sometimes in Fedora. Its a work in progress, but to have some data to read and understand is a great launching point.

What they did:

  • Refined the monitor-gating script that monitors the packager pipeline to enhance its performance
  • Picked a certain date range and got a database dump to pull metrics from into graphana to chart uptimes of applications within the pipeline
  • Created a diagram of the pipeline to help understand how packages flow through the fedora infra

Why its good:

  • The diagram of the packager workflow process is a great resource for both packagers and new contributors of the fedora community to refer to and help understand how things work
  • The team also have some recommendations they are working through with management and the wider CPE team to identify possible next steps and how we can improve the packager experience long term by adopting better monitoring.

 

Fedora-Messaging Schemas:

This project was also being worked on by the Noggin team part-time, so Aurelien Bompard, Nils Philippsen & Ryan Lerch. We needed to pause this work around the start of September and we hope to be able to return to it over the next quarter - October, November & December.

The guys have a github board here with a cookie-cutter schema available and a list of apps they were working on, so if you want to help out on this one, please feel free to visit the board and grab a card! 🙂 

What they did: 

  • Created a board to track the work being done and whats left to do https://github.com/orgs/fedora-infra/projects/7
  • Created a template schema 
  • Created a list of applications that require a schema update
  • Added some schemas to applications that need them

Why its good:

  • This will help us progress the retirement of fed-msg in 2021
  • It will also give applications, and application maintainers, access to new fedora messaging schemas for more faster & reliable notifications.

 

And that, my dear friends, is Quarter 3 for CPE.

Take care all, and see you around IRC! 🙂

 

Aoife


Powered by Planet!
Last updated: April 16, 2021 05:45 PM