February 15, 2021

CPE Weekly: 2021-02-14

February 15, 2021 04:36 PM

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

Conferences!

* DevConf.cz is on 18th - 20th Feb! Get your ticket here if you
haven't already https://hopin.com/events/devconf-cz-2021
* CentOS Dojo @ FOSDEM was really great last week, and if you missed
it be sure to check out the CentOS youtube channel where all of the
talks are now uploaded and available to view
https://www.youtube.com/thecentosproject

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Our CI infra has been updated from Ocp.ci / ocp.stg.ci to 4.6.15
* Monitoring stack updated to zabbix 5.0.8
* Kojihub now supports x86_64,ppc64le & aarch64

CentOS Stream

* CentOS Stream container images are now readily available!Check out
the mail from Brian Stinson to the CentOS-devel & announce list here
for more details on tags and where to pull
https://lists.centos.org/pipermail/centos-devel/2021-February/076503.html

Fedora

* Mass branching of packages was completed last week
* Mass branching of modules is underway
* There is already have a branched compose
* The main branch changes are also almost complete with just docs left
* tests namespace in dist-git has migrated to “main” with “master” as
symlink for now with it being removed after F34 release, so mark your
calendar!

Noggin/AAA

* Security fixes on Content Security Policy
* Re-installed FreeIPA schema to test a faster way to import user data
as part of tuning & performance testing while still in staging
* If you are experiencing any issues logging in, please reach out to
the team on IRC channel #fedora-aaa
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6
* And please report any issues you find in the repo
https://github.com/fedora-infra/noggin

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

February 09, 2021

CentOS Community Newsletter, February 2021 (#2102)

February 09, 2021 02:07 AM

Dear CentOS Community,

This month's newsletter is running a little late, because I wanted to include the report from our annual FOSDEM CentOS Dojo, which was held last Thursday and Friday.

CentOS Dojo at FOSDEM

We had 216 registrations, with 164 (75.9%) of registrants actually showing up. The average attendee spent 475 minutes at the event.

Over the two days of the event, we had 8 presentations, all of which are now available on YouTube, if you missed any of them.

We started the day with a round-table discussion with the board of directors. This started slowly, but developed into a useful Q&A with the community, covering everything from CentOS Stream (of course) to the new SIGs, to deep-dives into specific technical issues. We then had presentations about various SIGs (Cloud, Hyperscale) and various use cases around the community.

Overall, we were pleased with the turnout and the interactions, especially in the "hallway" track. We are considering doing more of these events - at least quarterly during the remainder of the pandemic, and then hopefully continue them in the future, for those who remain unable or unwilling to travel to in-person events.

We would love to hear from you about what content you'd like to see at future events, or, better yet, if you want to present about what you're working on.

Upcoming events

In just under 2 weeks, DevConf.cz will be happening (February 18th - 20th). This event is usually held right before, or right after, FOSDEM, in nearby Brno. This year, it's online, with content scheduled so as to be convenient for attendees in Europe time zones.

As every year, there's a  lot of deep technical content covering a wide range of topics. We want to specifically draw attention to two presentations:

On Friday at 14:45 (CET), Davide Cavalca will be talking about the use of CentOS Stream at Facebook. And then at 17:30, Tomas Tomecek, Brian Stinson, and Carl George will be talking about Consuming CentOS Stream.

Details and (Free!) registration are available at https://devconf.cz/.

SIG Reports

Special Interest Groups (SIGs) are one important place where the community can get involved in making CentOS more useful. This month we hear from several of our SIGs about what they've been doing for the past quarter.

Active SIGs hold regular meetings, where you can find out what's happening, and where you can get involved.

Hyperscale SIG

Although not scheduled to report this month, the Hyperscale SIG presented at last week's Dojo about what they have planned, and what they have done so far. You can watch the full presentation on YouTube, and read more about the SIG here.

* Alt Arch

Cloud SIG

Purpose

Packaging and maintaining different FOSS based Private cloud infrastructure applications that one can install and run natively on CentOS.

https://wiki.centos.org/SpecialInterestGroup/Cloud

Membership Update

We have reached out to all current and pending members of the SiG to confirm their continued interest as we revitalize the SiG. Once the membership lists are updated we will be holding nominations and elections for chair and co-chair.

We are always looking for new members, especially representation from other cloud technologies and we’ve reached out to Shaken Fist to see if they would like to join though they are currently Ubuntu only.

Releases and Packages

RDO

Nov 16 Victoria release: https://blogs.rdoproject.org/2020/11/rdo-victoria-released. Interesting features in the Victoria release include:
Source tarballs are being validated using the upstream GPG signature, to ensure the integrity of the packaged source code..

Openvswitch/OVN are not shipped as part of RDO. Instead RDO relies on builds from the CentOS NFV SIG.
The full release notes are at https://releases.openstack.org/victoria/highlights.html

Health and Activity

The Cloud SIG has been very active in regards to creating and publishing builds though it has not held a meeting over the past months. Efforts are being made to revitalize the SiG by re-establishing meetings and grow both the membership and projects involved. At this time, the SiG is only OpenStack.
The OpenStack group is focusing on the Wallaby release, which will be available for CentOS Stream 8 once it is finished. For additional details about the CloudSiG’s plans for CentOS Stream adoption in Wallaby, and previous releases, see the following blog post: https://blogs.rdoproject.org/2021/01/rdo-plans-to-move-to-centos-stream/

Alan Pevec held a RDO and CentOS Stream AMA which is now available on YouTube: https://www.youtube.com/watch?v=MlhAhClVaEI&feature=youtu.be

Alfredo Moralejo and Javier Peña presented 'How OpenStack became boring (and successful)' at the CentOS Dojo on February 5th. https://youtu.be/H0JDgsafFD0

Issues for the Board

We have no issues to bring to the board’s attention at this time.

Storage SIG

Repository Status and Updates

  • Ceph Nautilus updates: 14.2.16 (c7 and c8)
  • Ceph Octopus updates: 15.2.8 (c8 only) too
  • luarocks packages were added for upcoming Ceph Pacific
  • GlusterFS updates:  9.0, 8.3, and 7.9 for c7 and c8.  (Note: glusterfs-7 is now EOL)
  • NFS-Ganesha updates: 3.5 and 2.8.4 for c7 and c8; including associated libntirpc.
  • Samba updates: 4.11.17(c7 and c8) & 4.12.11 & 4.13.4(c8 only). v4.11.x is now EOL

Group Status and Actions from meeting

  • SIG needs to update the wiki and the calendar page moving to #centos-meeting2
  • SIG will work on automating cephadm builds

Links and other general informations

Meetings agenda https://hackmd.io/Epc35JIESaeotoGzwu5R5w

Messaging SIG

During the past quarter, there has not been much change in or with the messaging SIG, and there is nothing to report. Its artifacts are consumed by both Cloud SIG and Opstools SIG.

Release and Updates

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during January:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during January:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during January:

 

February 08, 2021

CPE Report: 2021-02-05

February 08, 2021 07:51 PM

Hi Everyone,

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

Initiative FYI Links

Initiatives repo here: https://pagure.io/cpe/initiatives-proposal
2021 Quarterly Planning timetable here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.
Details on initiative requesting/how to work with us on new projects
here: https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

Conferences!

* CentOS Dojo @ FOSDEM is on right now! Links to talks from Thursday
are on the CentOS youtube channel and Rich is playing a blinder
getting all the content uploaded in record time
https://www.youtube.com/TheCentOSProject
* NOTE: 'playing a blinder' means doing an excellent job for
anyone unfamiliar with the term
* Fedora has a booth as well @ FOSDEM this weekend! Make sure you stop
by and say hi to all those great Fedorans who will be manning it this
weekend https://chat.fosdem.org/#/room/#fedora-stand:fosdem.org

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CI team members are migrating Fedora-Infra and Fedora-apps namespace
whcih is one of the last few before we shut down legacy cluster
* There is also an investigation spike on Zabbix upgrade to current
LTS version which will then be rolled-out on the CentOS Infra once
complete

CentOS Stream

* Python39 built and ready to compose
* Dist-git repos are regularly up to date
* Repos are populated in the CentOS Stream GitLab instance and will be
publically viewable in the coming weeks
* Very detailed talks on CentOS Stream given by Brian Stinson & Brian
'Bex' Exelbierd are watchable now on the CentOS YouTube channel -
check them out!

Fedora

* Infra team are assisting with the testing of ipa/noggin for
otp/other cases in stg
* Their also doing a cleanup of a bunch of broken links on koji volume
* Mass rebuild of rpms is done, modules are underway
* FTBFS for the mass rebuild are filled

CPE ARC TEAM

(Community Platform Engineering Advanced Reconnaissance Team....Team)
We have a new sub team in our team, led by Pingou, who are running
advance investigations on some of the tech debt and bigger initiatives
that the CPE team have in our backlog and they have been tackling
Datanomer/Datagrepper tech debt first.
The team have been partitioning the ‘messages’ table of datagrepper's
DB, & hope to be able to test this setup next week
* prod like in openshift
https://datagrepper-monitor-dashboard.app.os.fedoraproject.org
* prod like with a default delta of 3 days
http://datagrepper.arc.fedorainfracloud.org/datagrepper/
* partitioned table + default delta of 3 days
http://datagrepper-test.arc.fedorainfracloud.org/datagrepper/
* using the timescale postgresql plugin [not implemented yet]
http://datagrepper-timescale.arc.fedorainfracloud.org

Noggin/AAA

* We faced some issues with IPA limits and tuning, and 2FA & still
trying to figure out the best way to enforce 2FA with sudo.
* We are getting closer to migrating from stg to prod and once the
Fedora migration is complete, the CentOS accounts will be then
imported.
* NOTE: If you have an account in both CentOS & Fedora and have
different email addresses associated with each, please update your
preferred email address in your profile and look out for an email next
week on your options.
* The work tracker for this project can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* Elections pr reviewed https://pagure.io/elections/pull-request/90
* Next is Greenwave & waiverdb
* Board the issues are tracked on are here
https://github.com/orgs/fedora-infra/projects/7

Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

CentOS Dojo @ FOSDEM, 2021

February 08, 2021 07:20 PM

Last week we held our traditional annual CentOS Dojo at FOSDEM. We had 216 people registered, of whom 164 (75.9%) actually showed up to attend some part of it. A big thank you to those that turned up and made it a successful event.

In case you missed it, or some part of it, all of the content is now on YouTube.

On Thursday we had four presentations:

  • The Board of Directors had an "ask me anything" session, where questions were fielded from attendees. [Video]
  • Brian Exelbierd and Brian Stinson talked about CentOS Stream. [Video, Slides]
  • Tomas Tomecek talked about the contribution workflow of CentOS Stream, and how that is the process to land changes in RHEL. [Video, Slides]
  • David Duncan talked about building elastic configurations with EC2-Hibernate [Video, Slides]

And on Friday, we had four more:

  • Javier Peña and Alfredo Moralejo Alonso talked about how OpenStack became boring (and successful) [Video, Slides]
  • Davide Cavalca gave an introduction to the new Hyperscale SIG [Video, Slides]
  • Matthew Almond talked about speeding up DNF/RPM using copy on write [Video, Slides]
  • David Duncan talked about building an image pipeline with CentOS Stream and Image Builder [Video]

It was great to get together with the community, even though it was online. We had some great impromptu discussions in the "hallway track", and it was good to see some faces.

We want to do these at least quarterly for the remainder of this year - watch Twitter and the mailing lists for announcements of dates for the next event! We would also like to hear from you what content you would like to see at upcoming events, especially if you'd like to give a presentation about what you're working on.

January 15, 2021

CPE Weekly Report: 2021-01-15

January 15, 2021 06:54 PM

Hi Everyone,

New Year, same CPE weekly(ish)

If you would like to see this report and toggle to the section you are
most interested in, I would suggest visiting this link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar
on your left to skip to where you want to go!

General Project Updates

We are kicking off Q1 this year with some familiar project faces,
namely Noggin, the replacement of the current FAS system and
continuing our development of CentOS Stream.

Most of our initiatives live here
https://pagure.io/cpe/initiatives-proposal and you can use the new
issue button to submit your own proposal.

Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/ so you know when
I need it in by to review it.

We also have updated our docs section on the initiative process we
follow as we cannot accept everything so please do check it out if you
want to understand our process more
https://docs.fedoraproject.org/en-US/cpe/initiatives/

Misc

GitLab

Being very honest, I've found myself a little bit strapped for time to
give this project its due diligence over the last few months, but
please bear with us/me and expect a more concentrated effort on this
coming into Q2 (April, May, June) of this year. I apologise for the
time a resolution is taking and I really do appreciate all of your
patience.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Community newsletter can be read here
https://blog.centos.org/2021/01/centos-community-newsletter-january-2020-2101/

CentOS Stream

* Continuing to work on Stream 8 pushes and builds
* Investigating how to automate some module pushes
* Reviewing documentation that is available on Stream currently to
identify gaps and where needs improvement

Fedora

* OSBS is building for aarm64 & x86_64 in production since December!
* All of the projects under the fedora-infra and releng namespaces on
pagure have had their default branch migrated from “master” to “main”.
* F34 mass rebuild due to start next week

Noggin/AAA

* New sprint started focusing on testing correct access has been given
per user/account
* Last remaining apps being configured & tested with fasjson API
* Work will be tracked here https://github.com/fedora-infra/aaa-tracker/issues/4
* Our open issues board can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* We are working through supybot and greenwave applications currently
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

## Team Info

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great weekend!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

December updates

January 15, 2021 03:55 PM

I usually include the below report in the monthly newsletter, and overlooked it this month. So, without further ado, here are the CentOS 7 updates that were pushed out in December:

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during December:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during December:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during December:

Other releases

The following releases also happened during December:

January 12, 2021

CentOS Community Newsletter, January 2021 (#2101)

January 12, 2021 12:12 AM

Dear CentOS Community,

As we enter the new year, I'm sure there's really only one thing on your mind, and so we'll start there.

As you are no doubt aware, the CentOS project has shifted focus from CentOS Linux - the RHEL rebuild - to CentOS Stream - the continuously delivered distribution that reflects what will be delivered in the next release of Red Hat Enterprise Linux (RHEL).

Many, many articles have been written about this, and I want to take an opportunity to call out some of the better ones, to help you understand what's happening, and where we go from here.

To those who claim that CentOS Stream will be somehow unstable, I would encourage you to read Brendan's article about how RHEL is made. Things that go into RHEL are not bleeding edge or continually shifting sands. They are small incremental changes which have been baked for a long time.

To those objecting to the term "rolling release", see Stef's article about continuous delivery, and how CentOS Linux and CentOS Stream related to RHEL.

And to those who are pre-judging CentOS Stream without the benefit of even trying it, you should read Jack's article about not knocking it until you try it. (Jack's an Ubuntu fan, but makes a lot of good points.)

Karsten has written an article about the various things that are kept in balance around the CentOS project, and some of the history that led to where we are.

Finally, Scott's article about ... well, all of it ... is definitely worth your time if you want to have a deeper understanding about why people are angry, and why they are right, and wrong, to be angry.

For those of you who are planning to move to Rocky, CloudLinux, or one of the other projects that has sprung up to take the place of CentOS Linux, we wish you - and these projects - all the best. But we caution you to understand that building an OS is a big project, and it's going to take a while for them to get where they're going. Please plan your migration accordingly.

There are other things happening in the CentOS community, but we understand that this one is pretty overshadowing right now.

Hyperscale SIG proposed

A group of developers has proposed a Hyperscale SIG, which will be voted on in Wednesday's board meeting. They propose to focus on solutions around large-scale infrastructures, such as those at organizations such as Facebook and Twitter.

If you are interested in this kind of SIG, and particularly if you are running a hyperscale infrastructure, we welcome your comments and participation.

CentOS Linux 8 (20-11) released

The fourth release of CentOS 8 is now available, as of December 7th. This release is labelled 8.2011 (ie, November 2020) and is based on the 8.3 release of RHEL.

Q1 CPE Priorities

In Q1, CPE will be working on the following priorities:

  • CentOS Stream
  • Noggin/AAA replacement
  • Fedora-Messaging Schemas 
  • Flatpak indexer code merge
  • Debuginfo-d
  • Datanomer & Datagrepper V.2

We'll be updating the centos-devel list as progress is made on these projects.

Happy New Year

We wish you a 2021 that is happy and productive, and hope to see you in person before the year is out. Thanks, as always, for being part of our community.

 

December 19, 2020

Balancing the needs around the CentOS platform

December 19, 2020 06:40 AM

These past few weeks I’ve read through and listened to a lot people’s reactions and responses to our news about the future of the CentOS Project. I see a lot of surprise and disappointment, and I also see people worried about the future and how this is going to affect them, their livelihoods, and the ecosystem as a whole. I feel a strong sense of betrayal from people, I hear that.

I don’t know if my story here is going to help you or not, but I appreciate you reading it through and listening to what I have to say. The history I cover I think is necessary to understand where we are today. From here I’m going to be available on the CentOS devel list and Twitter if you want to talk further about why I think it’s going to turn out okay.

I’ve been on the CentOS Project Governing Board since its creation. I also was part of the consensus decision that we recently announced about shifting the project’s focus.  I’ve cared about this space for a long time, for my 19 years at Red Hat and prior to that. I was involved in the Fedora Project since the earliest days, leading the documentation project and sitting on the then-Fedora Board, among other roles. I led the team at Red Hat that brought the CentOS Project in closer to Red Hat in 2013/2014, and as a result of that work I earned a seat on the CentOS Governing Board, where I was the Red Hat Liaison and Board Secretary until Spring 2020.

Let’s go back to 2003 where Red Hat saw the opportunity to make a fundamental change to become an enterprise software company with an open source development methodology.

To do so Red Hat made a hard decision and in 2003 split Red Hat Linux into Red Hat Enterprise Linux (RHEL) and Fedora Linux. RHEL was the occasional snapshot of Fedora Linux that was a product—slowed, stabilized, and paced for production. Fedora Linux and the Project around it were the open source community for innovating—speedier, prone to change, and paced for exploration. This solved the problem of trying to hold to two, incompatible core values (fast/slow) in a single project. After that, each distribution flourished within its intended audiences.

But that split left two important gaps. On the project/community side, people still wanted an OS that strived to be slower-moving, stable-enough, and free of cost—an availability gap. On the product/customer side, there was an openness gap—RHEL users (and consequently all rebuild users) couldn’t contribute easily to RHEL. The rebuilds arose and addressed the availability gap, but they were closed to contributions to the core Linux distro itself.

In 2012, Red Hat’s move toward offering products beyond the operating system resulted in a need for an easy-to-access platform for open source development of the upstream projects—such as Gluster, oVirt, and RDO—that these products are derived from. At that time, the pace of innovation in Fedora made it not an easy platform to work with; for example, the pace of kernel updates in Fedora led to breakage in these layered projects.

We formed a team I led at Red Hat to go about solving this problem, and, after approaching and discussing it with the CentOS Project core team, Red Hat and the CentOS Project agreed to “join forces.” We said joining forces because there was no company to acquire, so we hired members of the core team and began expanding CentOS beyond being just a rebuild project. That included investing in the infrastructure and protecting the brand. The goal was to evolve into a project that also enabled things to be built on top of it, and a project that would be exponentially more open to contribution than ever before—a partial solution to the openness gap.

Bringing home the CentOS Linux users, folks who were stuck in that availability gap, closer into the Red Hat family was a wonderful side effect of this plan. My experience going from participant to active open source contributor began in 2003, after the birth of the Fedora Project. At that time, as a highly empathetic person I found it challenging to handle the ongoing emotional waves from the Red Hat Linux split. Many of my long time community friends themselves were affected. As a company, we didn’t know if RHEL or Fedora Linux were going to work out. We had made a hard decision and were navigating the waters from the aftershock. Since then we’ve all learned a lot, including the more difficult dynamics of an open source development methodology. So to me, bringing the CentOS and other rebuild communities into an actual relationship with Red Hat again was wonderful to see, experience, and help bring about.

Over the past six years since finally joining forces, we made good progress on those goals. We started Special Interest Groups (SIGs) to manage the layered project experience, such as the Storage SIG, Virt Sig, and Cloud SIG. We created a governance structure where there hadn’t been one before. We brought RHEL source code to be housed at git.centos.org. We designed and built out a significant public build infrastructure and CI/CD system in a project that had previously been sealed-boxes all the way down.

However, the development of RHEL itself still remained closed behind the Red Hat firewall.  This had been true for almost twenty years. For the open source development ecosystem this has been an important and often painful gap—it’s the still same openness gap as 2003.

This brings us to today and the current chapter we are living in right now. The move to shift focus of the project to CentOS Stream is about filling that openness gap in some key ways. Essentially, Red Hat is filling the development and contribution gap that exists between Fedora and RHEL by shifting the place of CentOS from just downstream of RHEL to just upstream of RHEL.

Just as when we joined forces, Red Hat approached the CentOS Project with its plan, and the CentOS Board signed on to it. That plan centered around not just closing the feedback-loop part of the openness gap, but in finding a way to help evolve RHEL development from happening inside of Red Hat to outside of it.

The Board was fully aware that in filling one gap we risked reopening the availability gap on the end-user side of the equation. While CentOS Stream would be open to contribution in a way that it never had been before, it would stand the risk of being somewhat different than CentOS Linux has been.

But we also knew as a project trying to do two antithetical things at once would mean doing both poorly. Providing our community with a solid, reliable distro that is good-enough for your workloads is a strong part of the CentOS brand. We’re confident that CentOS Stream can do this.

And while I’m certain now that CentOS Linux cannot do what CentOS Stream can to solve the openness gap, I am confident that CentOS Stream can cover 95% (or so) of current user workloads stuck on the various sides of the availability gap. I believe that Red Hat will make solutions available as well that can cover other sides of the gap without too much user heartburn in the end.

Beginning now is the time to genuinely help the CentOS Project understand what you need in a CentOS Linux replacement, in some detail. Even your angriest of posts are being read, and your passionate viewpoints are being seen and understood. I’m not the only Linux old-timer working on this.

This is your chance to be recognized for where you land in the availability or the openness gap, and how it is being there, so that the people crafting RHEL solutions are doing it with your use case(s) in mind. This input is happening right now. The new email address centos-questions@redhat.com goes directly to the people in the business unit (who are not in Sales) trying to solve your problems using this open source development method.

It is hard to balance the needs and processes of making business decisions with the needs and processes of making open community decisions. Arguably, Red Hat has been among the best organizations at straddling this hard, thin line. If you trust our code enough to run it for this long, I ask you to trust us to make good decisions here. I ask you to trust Red Hat and the CentOS Board to work with you to find a way to bring the community along into the next chapter.

If you want to talk with me further, the best place is the centos-devel list or Twitter.

December 11, 2020

How RHEL is Made

December 11, 2020 05:49 PM

This week Red Hat announced its plan to put all its energy into CentOS Stream 8, resulting in the discontinuation of CentOS Linux 8 in one year’s time.  CentOS Stream, originally announced in September of 2019, is a continuous release of RHEL which provides updates as soon as they are developed and verified.  Many people who use CentOS Linux today now wonder if CentOS Stream 8 will be a suitable distribution for their use: is it tested, will it be stable?  If you want to know what to expect from CentOS Stream, the best starting point is knowing how Red Hat Enterprise Linux is built.  Let’s get into it!

Red Hat has been making Linux releases for such a long time, its original development methodology predates the agile manifesto.  Historically, RHEL has been built behind closed doors, its plans held close enough that even the announcement of predictable 6-month minor / 3-year major releases seemed a monumental reveal during the RHEL 8 launch.  Fortunately, how Red Hat makes Linux distributions has evolved, not just since calendar years started with “19”, and there have been multiple process generations since RHEL 8 launched just 18 months ago.  While fundamentals like upstream first, copious quality engineering, ecosystem partnership, and customer care remain the same, we work continuously to improve how those fundamentals are implemented.  

Let’s start with grounding: every RHEL minor release is based on the previous release, plus targeted backports of upstream development.  Often, Red Hatters are the original authors of those patches, but there are no shortcuts: upstream acceptance is the first test every patch must go through before we start it through the journey that eventually leads to a patch’s integration in the release.  Even then, this is about an upstream patch existing, but that alone will not guarantee a patch’s inclusion.

Any decision to introduce an upstream change into RHEL is a team decision and the team is large: developers, quality engineers, support personnel, product owners, and various partners all work together on priority and feasibility.  Once a decision is reached and commitments are made, only then do developers and quality engineers begin writing code.  As you probably know, in the most congenial of rivalries, developers try to write code that nobody can break and quality engineers create batteries of ways to break the code developers write.  This brings us to the second key place where Red Hat invests: tests.

We write tests for everything: unit tests, systemic tests, kernel and userspace ABI conformance tests, performance tests, dependency tests, architecture tests, driver tests, load tests, and many more.  Having tests is foundational, but it is their application that brings meaning.  This brings us to the third key area where Red Hat invests: process infrastructure.

For the last several years, Red Hat has worked on a series of “Always Ready” operating system initiatives.  The goal is as simple as the name suggests: at any moment in time the OS is ready for release.  It’s easier to describe than it is to implement. In complex systems, so many things can have unintended consequences.  To handle this we use layers of automation, incrementally building confidence in changes, before they are integrated and released into the distribution.  Here is a high-level sketch of the process every single change in RHEL must go through to be included:

When a change is targeted at RHEL, multiple incremental steps occur before it is actually included.  Changes are built, but the only certain outcome is that a CI system will run a suite of tests on the builds (the build is not yet made available for general use).  If those tests pass, a second round of preverification specific to the code change occurs (not yet good enough).  And if those tests pass, the change is tentatively included in the errata system and subject to further verification (it’s still not ready to publish).  Systemic test suites run on the combined whole to verify the gestalt functionality.  And if those tests pass, the build will finally make it into the space where CentOS Stream systems recognize it as an available update.  It’s a long pipeline and many changes move through it every single day.  For those interested in more of the vision and architecture of this system, you can read more in CentOS Stream is Continuous Delivery!

While the description of this system may seem elegant and reassuring, watching it in action can feel quite the opposite: The more testing is done the more bugs are found- and Red Hat does a whole lot of testing.  Historically, RHEL development has been done behind closed doors, isolating people from the routine bug identification and remediation process, only allowing the world to see the end result.  In the future, as RHEL development becomes more transparent, as we approach RHEL 9, this process will become uncomfortably visible.  While the testing systems are built to prevent such failures from reaching end users, anybody who wants to look deeper may be surprised at how messy operating system development can be!

Finally, for those who wonder how soon all this will map to CentOS Stream, we have good news: it is already happening today with RHEL 8.4 and CentOS Stream 8!  At the same time these RHEL builds are verified, they are also delivered to CentOS Stream.  Of course we aren’t done yet: CentOS Stream has not yet realized its mission of adding a developer community around RHEL, that is where we are headed, into a place where there are more options to engage with Red Hat and shape future RHEL.  There is always room for improvement, from better tests to more facets in future collaboration, we are excited to share building RHEL with you so that we build a better OS with and for you.

Minutes for CentOS Board of Directors for 2020-11-11

December 11, 2020 05:49 PM

On 2020-11-11 the CentOS Board of Directors met to discuss ongoing business.

First, the board would like to thanks everybody involved in CentOS Linux 7.9 release.

The Board was in an Executive session, where Red Hat CTO, Chris Wright joined to present Red Hat plan around CentOS Linux and CentOS Stream. A Board discussion followed.

Following up the discussion around the different users' communities impacted by proposed changes, Chris Wright, mentioned to the Board that Red Hat is also reshaping and expanding the RHEL Developer program. The details will be communicated through standard Red Hat channels.

The following resolutions were approved by the majority of the Board :

  • CentOS Stream 8 will continue with contributions for the full-support phase of RHEL 8. APPROVED
  • CentOS Stream 9 will start on schedule with the RHEL 9 Beta. APPROVED
  • CentOS Linux 9 will not start. APPROVED
  • CentOS Linux 8 ends in December 2021. APPROVED

An announcement and detailed FAQ will be prepared in next weeks.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

Minutes for CentOS Board of Directors for 2020-09-09

December 11, 2020 05:41 PM

On 2020-09-09 the CentOS Board of Directors met to discuss ongoing business. The Board discussed the creation of the infrastructure SIG to streamline and foster contributions in this challenging area. 

The creation of the Infrastructure SIG was approved and Aoife Moloney will be the new chair in charge to gather requirements from all actors and define the SIG baseline contribution model. Board members insisted that administrative accesses to the CentOS infrastructure will need to be carefully granted and on the principle of least privilege. Rich Bowen kindly drafted a blog post that goes in deeper details for the next steps and the challenges ahead.

A lengthy exchange happened around the feedback loop from SIG to the CentOS Board. The board would like to invite SIG chairs (or their representative) for discussion when needed, but also hear from them on a regular basis. A communication will follow-up when the board agrees on the best format and frequency for these sessions.

Ralph Angenendt announced he decided to step down from the CentOS Board of Directors. The Board would like to thank Ralph for all his service and hard work over the years.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

CentOS Stream is Continuous Delivery

December 11, 2020 04:14 PM

Continuous Delivery 101: Do the hard things continuously,
so that they become easy.

From the outside, it may appear that the way we build RHEL (and thus the CentOS Linux content) hasn’t changed in a decade. But beneath the covers, we’re pulling off a monumental transformation of how we develop RHEL without impacting our customers.

I've told this story at various conferences, but the announcements about CentOS Linux 8 and CentOS Stream have provided the impetus to tell the story here.

Three years ago, several of us working in RHEL Engineering had an idea: what if we applied modern development practices to RHEL such as continuous integration, continuous delivery, predictable release cadence … paired with open source development practices like release early release often, pull requests, forking, and code review.

Obvious, no? … No.

The Linux distribution is the grand challenge of
continuous integration and delivery.

What drew me into open source has always been this integration challenge. There is an infinite sea of uncoordinated projects. It really is an amazing example of evolution. If you squint your eyes like so, you can just about see the strange organisms, the mutations, the microcosms, and the natural selection all happening before you.

Over the last 20 years, I’ve contributed to over a hundred different projects. My contributions focused on making projects function seamlessly together so the user would have a coherent experience.

The Cockpit project is the most visible example of this. We connected about 95 Linux APIs and components, each developed separately, and released on different schedules in over 10 different distros, into a coherent user experience, delivering stable releases every other week for six years and counting.

If Linux is the grand challenge of continuous integration and delivery, then I saw RHEL as the unparalleled absolute: take ten thousand uncoordinated projects, thousands of contributors, add additional structure (like kABI) and additional guarantees (like 10 + 3 years of hardware enablement), integrate them constantly, and deliver a stable release every single day.

With dreamy (well, watery) eyes, we called such an effort “Always Ready RHEL”.

The effort started painstakingly onboarding the thousands of packages into continuous integration. It shocked many that we didn’t already have CI for all RHEL components back in 2017. But if it was easy, it would have happened much earlier.

Today, any update that goes into RHEL has to pass continuous integration gating before landing in our nightly compose, which runs automated tests for that component. Then, each change needs to be explicitly verified to a RHEL quality (mostly by Quality Engineering) before it can land in the RHEL nightly builds.

The “Always Ready RHEL” effort now continues with continuous delivery, which you now know as CentOS Stream: the RHEL nightly composes are already delivered in CentOS Stream. The whole point of continuous delivery is to make each release as stable as the one before. We’re delivering daily.

Are we done? … No.

To the untrained eye,  CentOS Stream is
already 
as stable as RHEL.

But the challenge here is unparalleled, and RHEL engineers carry awareness of that. The way the different teams do their work integrating RHEL is as diverse as the upstream communities themselves. Yet because so many people are iterating together toward different aspects of this goal, I’m convinced we can make Continuous Delivery a reality..

Fedora, CentOS Stream and RHEL delivery

Diagram licensed CC-SA: https://creativecommons.org/licenses/by-sa/4.0/

Here’s how the flow of delivery looks for 8 and 9:You can see the Fedora releases on the left. And the chart illustrates how CentOS Stream is synonymous with the work on RHEL X.Y releases. Technically speaking, CentOS Stream and RHEL updates are two binary packages built from the same source. An update will be published to CentOS Stream if and only if it is published to the RHEL nightly builds. Thus the RHEL nightly builds are the CentOS Stream updates you get. Once we branch from Fedora, our development gets into a stride where each change is integrated cleanly on top of everything that went before. An update is pushed to CentOS Stream if and only if it is published to the unreleased minor update of RHEL. RHEL customers later see each of these as a RHEL Errata update.

Each of these changes, whether bug fixes or features, is tested via automated tests and verified by Quality Engineering processes before landing in CentOS Stream.

The only work not directly and immediately visible in Stream is the work we do on the already-released RHEL minor versions themselves (indicated as “errata” in the diagram). Often this work is done under NDA, are embargoed, or are backports of changes already in CentOS Stream.

CentOS Stream intends to be as stable as RHEL,
It’s fundamental to continuous delivery.

But hey, even the RHEL-released product is not completely stable. Back in July, a RHEL (andCentOS) fix for the “boot hole” vulnerabilities ended up being far worse than the CVE itself: it caused many systems not to boot. Oh, man.

As a result, we’re not only investing time in reworking upstream components, but also adapting our process to ensure that this cannot happen again. Rinse, repeat.

While I wasn’t part of the decision to EOL CentOS Linux 8, I’m committed to putting my effort toward pulling off CentOS Stream. Doubly so, because it makes RHEL be Open Source: Where we can work together with an entire ecosystem on this exciting continuous integration and delivery challenge.

Open sourcing a product is hard, yet we’ve made amazing progress. So far we’ve aligned the RHEL development process with Fedora, placed all the actual sources of RHEL in one readable place, enabled contributors to open a pull request against any part of RHEL, released early and often ...

And this is just the start. There are hundreds of people working toward this CentOS Stream change, all while not missing a beat delivering the RHEL releases you’ve come to expect.

CentOS Stream is the stable and reliable
continuous delivery of RHEL

December 08, 2020

CentOS Project shifts focus to CentOS Stream

December 08, 2020 01:57 PM

The future of the CentOS Project is CentOS Stream, and over the next year we’ll be shifting focus from CentOS Linux, the rebuild of Red Hat Enterprise Linux (RHEL), to CentOS Stream, which tracks just ahead of a current RHEL release. CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021. CentOS Stream continues after that date, serving as the upstream (development) branch of Red Hat Enterprise Linux.

Meanwhile, we understand many of you are deeply invested in CentOS Linux 7, and we’ll continue to produce that version through the remainder of the RHEL 7 life cycle.

CentOS Stream will also be the centerpiece of a major shift in collaboration among the CentOS Special Interest Groups (SIGs). This ensures SIGs are developing and testing against what becomes the next version of RHEL. This also provides SIGs a clear single goal, rather than having to build and test for two releases. It gives the CentOS contributor community a great deal of influence in the future of RHEL. And it removes confusion around what “CentOS” means in the Linux distribution ecosystem.

When CentOS Linux 8 (the rebuild of RHEL8) ends, your best option will be to migrate to CentOS Stream 8, which is a small delta from CentOS Linux 8, and has regular updates like traditional CentOS Linux releases. If you are using CentOS Linux 8 in a production environment, and are concerned that CentOS Stream will not meet your needs, we encourage you to contact Red Hat about options.

We have an FAQ to help with your information and planning needs, as you figure out how this shift of project focus might affect you.

[See also: Red Hat's perspective on this.]

Thoughts on CentOS Stream

December 08, 2020 08:00 AM

I’m excited to see the CentOS project and Red Hat work together and collaborate around CentOS Stream, and I’d like to explain why I think this is a good move. I’ve been a member of the CentOS project for the last 16 years, and in that time I’ve had countless conversations with developers who were targeting enterprise deployments, but who wanted to push things just a little beyond what was currently available. In my early days with the project it was “I just need this new feature from PHP” or “I need this one option enabled in postfix”. This was so common it spawned an entire cottage industry of 3rd party repositories like Elrepo, IUS, nux-desktop, and even our own CentOSPlus. Often these were features Red Hat would include in a future version of RHEL, but the timing and communication around these features was a mystery. Red Hat never announced release dates or upcoming features and for many developers, even for those internal to Red Hat this was a PROBLEM.

When I joined Red Hat to work on CentOS full time, they outlined the goal pretty clearly: “We want to showcase our upstream community work we intend to put in our layered products”. Red Hat’s developers were using CentOS to do their upstream development work, and our role was to help them. This quickly became a problem, because the way to get new work into RHEL was Fedora, but that’s often not practical for a variety of reasons (Software Collections, modularity structure, release cadence, etc). Nothing here is new. Red Hat’s Josh Boyer and Brendan Conoboy spoke at length about this challenge in their Penrose Triangle talk at Flock in 2018

CentOS Stream represents several positive steps for Red Hat here.

  1. It makes RHEL development more transparent and reliable.
  2. It provides a way for ISVs and developers to contribute fixes and features.
  3. It provides a way for the community to provide feedback.

Did you download the RHEL8 public beta? Did you notice that the python setup looked VERY different between the GA and the beta? If you spent the 6 months between the RHEL beta and GA developing your app to work, it’s possible you had the rug yanked out from under you. CentOS Stream solves that by providing constant updates so you can see what changes are coming, and adjust accordingly. Because RHEL’s development is now transparent to the public, devs shouldn’t be surprised by changes, they’ll finally be able to see them coming.

Seeing what Red Hat is doing is one thing, but for those users I mentioned earlier who just needed that one new feature - they now have a way to collaborate with Red Hat to make it a reality. CentOS Stream provides a way for users to submit pull requests and to make their case for why it should be included. This obviously doesn’t mean everyone will get their way, but it’s a stark improvement from the past.

I’m excited to see the CentOS project and Red Hat work together and collaborate around CentOS Stream, and I’d like to explain why I think this is a good move. I’ve been a member of the CentOS project for the last 16 years, and in that time I’ve had countless conversations with developers who were targeting enterprise deployments, but who wanted to push things just a little beyond what was currently available. In my early days with the project it was “I just need this new feature from PHP” or “I need this one option enabled in postfix”. This was so common it spawned an entire cottage industry of 3rd party repositories like Elrepo, IUS, nux-desktop, and even our own CentOSPlus. Often these were features Red Hat would include in a future version of RHEL, but the timing and communication around these features was a mystery. Red Hat never announced release dates or upcoming features and for many developers, even for those internal to Red Hat this was a PROBLEM.

December 01, 2020

CPE Weekly: November 22 2020

December 01, 2020 08:14 PM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-11-22 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

Final project submission date for consideration in Q1 is Friday 27th
November. If you have an initiative that may take weeks/months and
multiple people to work on and want to request it to CPE, please
follow the steps outlined in our initiatives repo and create your
issue before 27th November https://pagure.io/cpe/initiatives-proposal
Our updated initative timetable can be viewed here for 2021
https://docs.fedoraproject.org/en-US/cpe/time_tables/

Below are the projects the CPE team are currently working on for the
months of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

New GitLab topic sent to devel-announce@lists.fedoraproject.org &
centos-devel@centos.org on Message Bus is out. See email in hackmd
here
https://hackmd.io/oZrDwbSeSWO-l_X65A1ndg?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 6 is EOL 30th November
* CFP for FOSDEM Dojo - https://wiki.centos.org/Events/Dojo/FOSDEM2021
* Updated CentOS CI Openshift staging cluster to latest 4.6.4, Waiting
for stable release in the 4.6 branch before rolling out to production.
* CentOS 7.9.2009 is released! (for x86_64, i386, ppc64, ppc64le,
armhfp and aarch64 architectures)
* Lot of work done for Noggin/AAA

CentOS Stream

* Use centos-stream-release package to convert from CentOS 6 to CentOS
Stream before 30th November
* Working on integrating ODCS in Stream
* Curating out t_functional suite
https://github.com/centos/sig-core-t_functional
* Refining our testing for finding issues at distro-level

Fedora

Staging Environment

* Completed - any issues you find please report them in fedora infra
https://pagure.io/fedora-infrastructure/issues

Noggin/AAA

* Testing team owned apps in staging with Noggin
* We will be requesting community member testing before December so
keep an eye out for an announcement!
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

OSBS for aarch64

* Basic OKD 3.11 working on aarm64 with F31
* Working on repeating that install with F33
* Next step will be to

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

 

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-12-17 @ 1300 UTC on #fedora-meeting-1

#centos-meeting

* Next Meeting: 2020-12-15 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
ReplyForward

_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel

 

Minutes for CentOS Board of Directors for 2020-10-14

December 01, 2020 10:49 AM

On 2020-10-14 the CentOS Board of Directors met to discuss ongoing business. The Board was in an Executive session, where Red Hat CTO, Chris Wright, requested feedback from the participants on the progress of the CentOS Stream project since its creation, last year. Also, a discussion around resources for both CentOS Linux and CentOS Stream took place. A plan for improving and fostering development around CentOS Stream, as a RHEL upstream platform, will be presented and discussed when details are sorted out, in a future board meeting.

No other issue has been discussed this month, and updates will be amended to tickets if necessary.

CentOS Community Newsletter, December 2020 (#2012)

December 01, 2020 12:36 AM

Dear CentOS Enthusiast,

With many of you celebrating one holiday or another this time of year, we want to extend to you the warmest wishes for your Thanksgiving, Diwali, Christmas, Hanukkah, Kwanzaa, New Years, and holiday season. We hope for each of you that 2021 brings new opportunities, and much happiness.

We have a few news items to share with you in this newsletter.

CentOS Linux 6 EOL

As has been announced everywhere for the past year (and more!) CentOS 6 has been moved to End Of Life (EOL) status as of November 30th, 2020. During the first week in December 2020, the 6.10 directory will move to vault.centos.org

Packages will still be available at: http://vault.centos.org/centos/6.10/. However, once moved, there will be no more updates pushed to vault.centos.org. Therefore, security issues will no longer be fixed.

Should you require continued support for this version, we encourage you to contact Red Hat about Extended el6 support for RHEL.

CentOS Linux 7.2009

We are pleased to announce the general availability of CentOS Linux 7 (2009). Effectively immediately, this is the current release for CentOS Linux 7 and is tagged as 2009, derived from Red Hat Enterprise Linux 7.9 Source Code.

As always, read through the Release Notes at : http://wiki.centos.org/Manuals/ReleaseNotes/CentOS7 - these notes contain important information about the release and details about some of the content inside the release from the CentOS QA team. These notes are updated constantly to include issues and incorporate feedback from the users.

See the mailing list announcements for the x86_64 and altarch releases.

CentOS Linux 8 (Release to come)

Stay tuned! We expect to have a release of CentOS Linux 8, based on RHEL 8.3, any day now. (Indeed it may already be released when you are reading this.) Watch the centos-announce mailing list for the announcement!

FOSDEM Dojo: CFP now open

As has been our tradition for a decade now, we'll be hosting the CentOS Dojo on the day before FOSDEM 2021. FOSDEM has gone virtual this year, and our Dojo will also be online. The Call for Presentations (CFP) is now open. Details are on the Dojo wiki page.

The event will be held on Friday, February 5th, 2021. We will expand to include February 4th if we receive enough talk submissions.

We are looking for talks about:

  • CentOS Linux
  • CentOS Stream and the RHEL contribution workflow
  • Anything that you are doing in the CentOS ecosystem (including, but not limited to: SIG activity, CentOS infra, governance, community, etc)
  • Anything you're doing *on* CentOS - stuff you're running on CentOS, interesting research projects, useful CentOS tools/utilities/techniques.
  • Larger ecosystem topics - Linux, Cloud, Open Source, etc.

I encourage you to look at previous event schedules for further inspiration: 

CentOS is also usually well represented in the Distributions devroom.

CPE Q1 Priorities

In the coming days, you'll see a thread on the centos-devel mailing list regarding CPE's (Community Platform Engineering) priorities in Q1. We have a vote, as a community, to influence what they'll be working on. So we encourage you to watch for that thread, and express your opinions, so that we can ensure that CPE is using their time in a way that benefits us the most.

SIGs building against CentOS Stream

In case you missed it: Johnny mentioned a few days ago that SIGs can now build against CentOS Stream. If your SIG isn't yet, and wants to, please check out that thread, and get in touch with the list to get started!

SIG reports

SIGs - Special Interest Groups - are communities who build various things on top of CentOS. This month we have reports from two of our SIGs:

Virtualization SIG

oVirt: Upstream released 4.4.3 which introduces cluster compatibility level 4.5 with additional features enablement but requires RHEL AV 8.3 in order to work. So we are waiting for CentOS 8.3 to be released so the Advanced Virtualization team will be able to rebuild it from RHEL-AV packages.

CentOS OpsTools SIG

Only two notable changes happened during this quarter: a rebuild of collectd-sensubility fixing major issues and the addition of collectd-libpodstats, which is a plugin to monitor pods and to report their usage via collectd.

No new members were attracted, but also no member voiced they are not interested anymore. We've had a request from outside to include a package, but the requester himself was not interested in any contribution.

Happy New Year!

Once again, we hope that you have a safe, happy, and prosperous new year in 2021, and that we see all of you again soon.

November 24, 2020

What is CPE up to: CentOS Stream

November 24, 2020 05:23 PM

Last week I had a short chat with Brian Stinson and Carl George, from Red Hat's CPE (Community Platform Engineering) team about the work they are doing to enable CentOS Stream. (9 minutes)

November 18, 2020

An update on our Gitforge

November 18, 2020 02:21 PM

Hey everyone,

Back in March we published this blog where the CPE team came to a decision about a future Gitforge. The decision was made to opt for Gitlab as the Forge of choice. We can now announce that the service has been stood up successfully, with Gitlab running this as a SaaS offering on behalf of the community. We are still in the process of making configuration changes and starting to seed content. In the background the CPE team are working through their tooling and configurations to ensure when we launch that the process and experience is seamless for the community.

We expect CentOS Stream distribution repositories to be the first content that will show up in Gitlab. We plan on giving plenty of notice whenever we intend to relocate other things like SIG repos or Downstream source drops from Red Hat. Until then git.centos.org will remain up and running to support your work.

November 03, 2020

CentOS Community Newsletter, November 2020 (#2011)

November 03, 2020 01:48 AM

Dear CentOS Enthusiast,

It's been another fairly quiet month in CentOS, but I have a few things to share with you.

News

CPE

Last week I spoke with Aoife Moloney and Stefan Mattejiet of Red Hat's Community Platform Engineering (CPE) group. CPE do a bunch of infrastructure work for Fedora and CentOS, and we've been trying to bring more attention to what they're working on, on our behalf, over the last few months. My conversation with them is on YouTube, and I hope to have more of these interviews over the coming weeks to tell you about specific initiatives.

I would encourage you also to read CPE's weekly updates (October 17th, October 25th), and their Q3 summary for more detail.

If you have suggestions or requests for what CPE should work on in coming sprints, please get in touch with your ideas. You can email the centos-devel mailing list, or contact Rich Bowen directly off-list at rbowen@centosproject.org

CentOS 6 End Of Life, November 30th.

This is your final warning that CentOS 6 will be designated "End Of Life" on November 30th. After that time, it will receive no more updates.

Fedora 33

Our friends over at Fedora released Fedora 33 last week. Celebrate with them by attending their release party this weekend! Register here: https://hopin.to/events/fedora-33-release-party

Updates

A number of people have noticed that there has been no activity on the centos-announce mailing list in October. That's because all of the activity is over on the centos-cr-announce list, as we prepare for a 7.9 point release, which we expect to happen in the coming couple of weeks.

Meanwhile, we encourage you to keep an eye on the Building 7 page for daily updates of the status of that release.

Events

FOSDEM Dojo, February 4th and 5th

Although FOSDEM looks different this year, we plan to continue our tradition of holding a CentOS Dojo on the day (or days) before FOSDEM begins.

This event will take place online. Doing this online gives us access to a wider pool of speakers and attendees - this means YOU! The call for presentations is now open.  The actual schedule will depend on submissions that we receive, but we are tentatively hoping to run the event on February 4th and 5th, ahead of FOSDEM starting on the 6th.

We are looking for presentations about anything that you are doing in, or on, CentOS Linux and CentOS Stream. This includes, but is certainly not limited to, SIG activity, research using CentOS, useful tools/applications that can be deployed on CentOS, or topics around community engagement.

More details about the event will be available on the event wiki page as soon as we know more.

DevConf.cz, DevConf.in

The Call for Presentations for both DevConf.in and DevConf.cz are now open. The .cz CFP closes on November 6th, and the .in CFP closes on November 15th. You can access both of them at cfp.devconf.info.

October 27, 2020

Using connection delegation with mitogen for Ansible

October 27, 2020 11:00 PM

This should be a very short blog post, but long enough to justify a blog post instead of a 'tweet' : I had myself a small issue with mitogen plugin in our Ansible infra.

To cut a long story short, everybody knows that ansible relies on ssh as transport. So one can use traditional ~/.ssh/config tuning to declare ProxyJump for some hosts, etc

But when you use mitogen (we do), in the official doc there is a mention of specific parameter for connection delegation : mitogen_via

The simple example on the webpage seems trivial and if you have multiple hosts that need to be configured from remote ansible+mitogen combo, using mitogen would speed things up as it would know about the host topology.

That's what I thought when having a look at the simple inventory on that web page:

[dc2]
web1.dc2
web2.dc2
web3.dc2

[dc2:vars]
mitogen_via = bastion.dc2

Sounds easy but when I tried quickly to use mitogen_via , something that I thought would be obvious in fact wasn't. My understanding was that mitogen would automatically force agent forwarding when going through the bastion host. A simple ansible -m ping (let's assume web1.dc2 in their example) returned me :

web1.dc2 | UNREACHABLE! => {
    "changed": false,
    "msg": "error occurred on host bastion.dc2: SSH authentication is incorrect",
    "unreachable": true
}

Well, we can see from the returned json that it was trying to pass through bastion.dc2 and that's confirmed on web1.dc2 :

Oct 28 15:52:36 web1.dc2 sshd[12913]: Connection closed by <ip_from_bastion.dc2> port 56728 [preauth]

Then I thought about something that was obvious to me but that mitogen (just reusing underlying ssh) doesn't do automatically : Forwarding the ssh agent to the nodes behind.

We can easily solve that with one simple ansible parameter : ansible has the ansible_ssh_common_args and ansible_ssh_extra_args parameters, specific to the SSH connection

So what about we force Agent Forward just on that bastion host and see how that works ? That means that in our inventory (but can go to host_vars/bastion.dc2 too) we just have to add parameter:

bastion.dc2 ansible_ssh_extra_args='-o ForwardAgent=yes'

Let's try again :

web1.dc2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

Good, so we can push that for our bastion hosts (used in inventory for mitogen_via) in host_vars or group_vars and call it a day. The reason why I prefer using ansible_ssh_extra_args is that it will merge and add settings, in case you have already something like this in your ansible.cfg :

[ssh_connection]
ssh_args =

I like the logic that we don't need to modify ~/.ssh/config with all exceptions to reflect the infra layout but we can just reflect it in ansible inventory

October 25, 2020

CPE Weekly: 2020-10-25

October 25, 2020 12:02 AM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-25 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

Updated initative timetable can be viewed here
https://docs.fedoraproject.org/en-US/cpe/time_tables/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

Sent a mail to the devel lists for both Fedora & CentOS with questions
that had answers relating to the topic Accouns & Permissions. It has
been sent to devel-announce at fedoraproject.org &
centos-devel at centos.org. Here is the link to the hackmd doc I used to
write the email before copying it across to my email to send
https://hackmd.io/1pjX1cVnTjekOLVowj5UiQ?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 7.9.2009 pkgs pushed to CR repo, next step is to install tree
and artifacts such as iso & cloud images.
* Working a lot with the AAA/Noggin team on CentOS account integration
and making good progress

CentOS Stream

* There are now cloud image updates available for CentOS Stream
* There are some documentation updates here on unshipped packages &
Stream feel free to read up!
https://wiki.centos.org/FAQ/CentOS8/UnshippedPackages

Fedora

General

* F33 will be released next week!!
* The team got an F33 nightly compose finished too!
* There is a dashboard available to view the performance of Anitya
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/8Zi9LU5Mz/anitya?orgId=1
* And there is capability to do the same for the packager workflow
pipeline which we are still working on

Staging Environment

* Build system nearly done - waiting on a firewall change

Noggin/AAA

* New estimated deployment date for Noggin is 29th January 2021.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 (On Freenode IRC)

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting (On Freenode IRC)

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

October 17, 2020

CPE Q3 Achievements 2020

October 17, 2020 03:17 PM

Hi there,

I'm Aoife Moloney. You may remember me from such communications as the CPE office hours, Data Centre - what it means for you, and The Future of Communishift.

Over the last three months, the Community Platform Engineering team (or CPE for short as it's long to keep typing) have been working on a few projects, and generally surviving 2020 like everyone else. But we made it, and so did our projects! Mostly… 🙂

 

Over the last three months we worked on:

  • The Great Fedora Data Centre Move of 2020
  • Noggin
  • CentOS Stream
  • Packager Workflow Healthcare (Always check with your maintainer before taking this workflow. Side effects may be, but not limited to, frustration, tears, elation at successful builds)
  • Fedora-Messaging Schemas

 

We also had our long standing (and long suffering) ‘sustaining team’ on the front lines who are daily maintaining and running both the Fedora and CentOS infrastructures and responding to issues, bugs, etc. And doing a damn fine job too.

 

And we attended and participated in a few conferences too, namely Nest with Fedora & DevConf US.

 

So, what did we as a team overall achieve in these last few months?

 

CPE Infra & Releng Team 

This team was led by Pingou, and its members in Q3 were Mark O’Brien, Michal Konecky, Fabian Arrotin, David Kirwan, Kevin Fenzi, Vipul Siddharth, Stephen John Smoogen & Tomas Hckra.

This team is a sub team of CPE and focuses on lights on work in both the Fedora and CentOS infrastructures. We will always have some of our team members working in this way each quarter as it is good to have a break from scheduled project workloads and take a foray into the (sometimes) chaotic world of infrastructure maintenance, aka FIRE!!! 🙂

What they did: 

  • Changed their name. May we present: CPE Infra & Releng Team - oooooh, aaaaah!
  • Vipul & David worked with Fabian in the CentOS infra and did something with openshift clusters & migrated the kojihub for https://cbs.centos.org to a new infra
  • Kevin and Smooge moved all of the Fedora infrastructure. 117 servers.  Let that number sink in.
  • Pingou & Michal did a ton of babysitting toddlers 🙂 They moved a lot of scripts over and things are working well
  • Tomas helped bootstrap F33 - oh yeah!
  • And Mark had (not biologically,  but in sentiment) a baby! And became an admin for some of the Fedora Infrastructure.

Why its good:

  • The name change represents what this team works on and is easy to understand instantly. Plus, naming is hard so we wanted to keep it simple 🙂
  • Helped release F33 beta
  • We doubled down on toddlers allowing to build more automation around the infrastructure
  • Assisted with the fedora datacentre move for minimal disruption to the fedorans day to day lives
  • Helped keep CentOS CI operational - and then helped put out the flames when it caught fire 🙂
  • Over 500 tickets across both Fedora & CentOS infra + releng resolved by this team - that is some seriously good firefighting!

 

Fedora Data Centre Move

This dynamic duo was Kevin Fenzi and Stephen Smoogen, with supporting cast members from both CPE and the community along the way. The goal of this project was to successfully move a (large) number of the Fedora infrastructure hardware from one datacentre to the other without too much chaos. And considering the world wide pandemic that happened right at the beginning, they did a pretty fine job succeeding. Some additional services are still being added to the infrastructure in its new home in IAD, so if you notice a few still missing, we are getting to them slowly but surely and thank you again for your patience and understanding during these last few months!

What they did: 

  • Moved a ton of servers across the country of the United States
  • Kept critical services in Fedora Infrastructure alive during the move
  • Worked an uncountable amount of hours!

Why its good:

  • We got some new hardware!
  • The team carried out some resilience testing in the new data centre which means more reliability for the infrastructure should bad things happen 
  • Updated records and warranties were a passive benefit of this move too

 

Noggin

This team was led by Aurelien Bompard, and its members in Q3 were Ryan Lerch, Nils Philippsen & James Richardson. The goal of this project is to replace the current FAS system with a newer one and migrate the CentOS accounts to the one FAS instance (Noggin), which will mean our team has one authentication system to maintain for two infrastructures long term. This team has been working to a November 2020 deadline, but unfortunately during Q3 the team faced a number of challenges such as a delayed staging environment to test in due to the data centre move, then when we got it, realized their plugin they spent time developing was not going to work long term and now have to redo a bit of work in Q4. There were also a lot of holidays and personal events for the team in Q3 because everyone is human and entitled to a life 🙂 They have re-scoped their work for Q4 to make sure what's delivered is sustainable and reliable long term, more people have joined the team including some sys-admin for support along the way, and are now looking at delivering Noggin in full by the end of January 2021. 

What they did: 

  • A lot of ipsilon investigation
  • Added a spam curtailer service to Noggin
  • Added an agreements section for users to select their user preferences 
  • Deployed Noggin to staging but found out the way they did it wont be good for the project long term
  • Had a little cry about developing a plugin unnecessarily, hugged it out and then re-planned dates and the work we need to do in Quarter 4 to be able to deliver a better, more reliable and robust service in January 2021. Queue Noggin’  - Rise of the Phoenix Project

Why its good:

  • We knew where we went wrong, learned a lot both technically and as a project team for it, and were able to call the mistakes out and get the support we need to get the project back on track. Just a little bit later than we wanted.
  • We still created a solution that will meet the needs of both the CentOS and Fedora community users, and once we have the correct configurations in place and are ready to be tested we look forward to your feedback!

CentOS Stream:

This team was led by Brian Stinson, and its members in Q3 were Johnny Hughes, Carl George, Mohan Boddu, Leonardo Rosetti, James Antill & Siteshwar Vashisht

What they did: 

  • A lot of darn package & module building
  • Light hearted threats to their PO to teach her how to convert a CentOS Linux distro to CentOS Stream using the new release package - which she did! 🙂
  • Kept CentOS Stream compose up to date with RHEL nightlies
  • Launched the centos-stream-release package - Big deal. Like, huge.

Why its good:

  • CentOS Stream is continuing to stand on its own and becoming a more robust distro
  • There's lots more content in Stream for its users
  • Users can now swap from CentOS Linux to Stream easily

Packager Workflow Healthcare:

This team was led by Will Woods and its team members were Adam Saleh and Stephen Coady with Pingou in a part-time consulting/reviewing role. The team took a look into the Fedora packager workflow and tried to identify weaker points in the chain, and spot times that are more prone to downtimes.  They are finalizing a report of their findings to send to the community lists with hopefully a ‘next steps’ section that they feel will help reduce the issues packagers face sometimes in Fedora. Its a work in progress, but to have some data to read and understand is a great launching point.

What they did:

  • Refined the monitor-gating script that monitors the packager pipeline to enhance its performance
  • Picked a certain date range and got a database dump to pull metrics from into graphana to chart uptimes of applications within the pipeline
  • Created a diagram of the pipeline to help understand how packages flow through the fedora infra

Why its good:

  • The diagram of the packager workflow process is a great resource for both packagers and new contributors of the fedora community to refer to and help understand how things work
  • The team also have some recommendations they are working through with management and the wider CPE team to identify possible next steps and how we can improve the packager experience long term by adopting better monitoring.

 

Fedora-Messaging Schemas:

This project was also being worked on by the Noggin team part-time, so Aurelien Bompard, Nils Philippsen & Ryan Lerch. We needed to pause this work around the start of September and we hope to be able to return to it over the next quarter - October, November & December.

The guys have a github board here with a cookie-cutter schema available and a list of apps they were working on, so if you want to help out on this one, please feel free to visit the board and grab a card! 🙂 

What they did: 

  • Created a board to track the work being done and whats left to do https://github.com/orgs/fedora-infra/projects/7
  • Created a template schema 
  • Created a list of applications that require a schema update
  • Added some schemas to applications that need them

Why its good:

  • This will help us progress the retirement of fed-msg in 2021
  • It will also give applications, and application maintainers, access to new fedora messaging schemas for more faster & reliable notifications.

 

And that, my dear friends, is Quarter 3 for CPE.

Take care all, and see you around IRC! 🙂

 

Aoife

CPE Weekly: 2020-10-17

October 17, 2020 12:49 AM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-17.
Im gone a little bi-weekly lately with this report. This has been more
circumstantial with our quarter 3 projects ending and launching
quarter 4 work, and will get back to a weekly report now that Q4 is
underway.
So, the updates for both Fedora & CentOS are below, and if you want to
visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
you can then use the header bar on your left to skip to Fedora or
CentOS updates that interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

There is now a blog post out on the Fedora community blog on the AMA,
you can read it here
https://communityblog.fedoraproject.org/gitlab-ama-follow-up/
Im going to be sending one mail per week to the CentOS and Fedora
devel lists on Fridays, dedicated to one topic.
The first topic will be Permission and Access in GitLab. I will pull
the questions and answers from the hackmd doc into an email body to
try to facilitate dedicated discussion around this topic. I would like
to just set expectations now that I don't consider myself to be
technical enough to weigh in on the discussion, but I am looking
forward to reading and learning from the conversation.

Project Updates

*The below updates are pulled directly from our CPE team call we have every week.*

Fedora

General

* Go NoGo meeting was on 15th Oct. F33 was No GO and new date is moved
to 2020-10-27

Staging Environment

* Working on getting OpenQA moved over
* Adding boxes to CentOS to test IPA

Noggin

* After the team reviewed the work still left to do to be able to
deploy a robust and secure solution, we discovered that there is more
development work to do and we are now working towards a release date
of January 29th 2021.
* Accounts & data will remain secure in the current solution until we
can cut over, we have established a monitoring plan to derisk this
extended time the team needs to complete their work.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

CentOS Updates

CentOS

* New CI admin added - mobrien, welcome!
* Plumbing for duffy being worked on currently and will be deploying
to staging in the coming weeks

CentOS Stream

* Looking at ODCS services in the build system
* Also working on deploying mbbox operator to the system too
* centos-stream-release package is also out, go check it out!

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

October 13, 2020

CentOS Community Newsletter, October 2020 (#2010)

October 13, 2020 05:51 PM

Dear CentOS enthusiasts,

The past month has been quieter than recent months, in terms of news, and I was very occupied with some other things last week, so I hope you can forgive the delay in getting this month's newsletter out.

If you want to receive notifications of new newsletters in future months, please subscribe to the centos-newsletter mailing list to be the first to hear of new editions.

News:

Embroidered shirts and sweatshirts at HelloTux

You can now purchase embroidered CentOS tshirts, polo shirts, and sweatshirts, in the store at HelloTux. HelloTux also produces garments featuring our friends Debian, Ubuntu, openSUSE, and various other Linux and open source projects and brands.

An article with a little more detail about the process may be found on the CentOS blog.

Board welcomes Brian Exelbierd as director

On 2020-07-08 the CentOS Board of Directors welcomed Brian "bex" Exelbierd as a new member Director and Red Hat Liaison. This, and other board news may be found in the minutes from that board meeting.

Creation of Infrastructure SIG

In the September board meeting, the Board approved the creation of the Infrastructure SIG, which will provide a place for coordination of all planning and decisions which effect the infrastructure that enables the CentOS project. Aoife Moloney was named as the initial chair of this group. More details about the function of this SIG may be found in the article announcing the creation.

Updates:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during September:

 

September 23, 2020

CPE Weekly, September 20th 2020

September 23, 2020 05:42 PM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-09-20.

I found that if you want to skip to the hackmd, you can use the view
link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and then use the
header bar on your left to skip to either the Fedora or CentOS
updates, whichever interest you.

I'll also be adjusting these updates in the coming weeks to make them
a bit more direct to consume. Thanks for giving me this feedback in
the CPE survey, I want to deliver value to you all, so it's great to
KNOW what you find valuable first hand

General Project Updates

As a reminder, below are the projects the CPE team are working on for
the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

We have recently held our Q4 planning session and the CPE review team,
Fedora, CentOS and RHEL BU have voted the following projects for
action in Q4, which is the months of October November & December:

* OSBS for aarch64
* Fedora-messaging schemas

We are continuing to work on CentOS Stream and Noggin and took these
projects as confirmed when looking at what other work our team could
realistically complete in the Q4 period, given that there's both
Thanksgiving and Christmas time off to consider, plus any time off our
team wishes to take.

The taiga cards of Noggin, CentOS Stream, OSBS for aarch64 and
fedora-messaging schemas will be updated next week with what our team
hopes to deliver in the next quarter on each of the projects.
Our project board is here (it's just not updated properly - yet)
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

Misc

GitLab

Thank you so much to everyone for adding your questions to the doc for
the GitLab AMA session on Thursday 10th September, and for your
attendance on the day during the call.
Here is the full AMA transcript
https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_with_gitlab.2020-09-10-13.31.log.html
however it is a bit confusing to read so we got a few great
suggestions to have dedicated topics like Message Bus and Branching,
etc go out to the devel lists to discuss. I'm happy to start this next
week, but I will collect the questions related to each topic and
propose a cadence to send them out first to discuss, so people dont
miss mails and know the week ending 2nd October will be (for example)
the topic of Group Permissions - What do you think?

GitLab have also agreed to answer the questions, we have asked them to
do so within 2 weeks of the AMA so as soon as this is complete I will
let you know so you can read through them on the hackmd link.
The link is here where we asked you to contribute your questions and I
will be posting answers once we have them underneath
https://hackmd.io/RW8HahOeR7OJPON1dwuo3w

I really appreciate your involvement with this as we begin to dig
deeper into how this might play out next year and what way it should
for everyone's benefit.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Deployed new 4.5.9 openshift cluster for Stream
* The team provisioned EC2 infra for team responsible for
registry.centos.org (we don’t maintain it, so just providing infra,
like Fedora does for Copr)
* They also migrated a bunch of nodes to the new Ansible CI inventory

CentOS Stream

* Using Openshift cluster for engineering work and will be using it to
deploy & test mbbox in our infra
* Scoping and refining work for October November & December

Fedora

General

* 6 of 8 Beta-blockers have fixes for F33 beta
* New release of fedscm_admin
* FMW mac and windows binaries are signed

Staging Environment

* About 70% done installing vm’s (27 left out of 88)
* Still need to bring up aarch64/armv7/ppc64le builders
* Databases need syncing

AAA Replacement

* The team are working on testing Ipsilon in Staging and adding OpenID
Connect Capability
* they are also testing fas2ipa migration script in tiny-stage and improve it
* Add Noggin to tiny-stage environment and test
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is on hold until Noggin completes.
* It will be resumed around December timeframe and is part of our Q4
workload to complete
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been working on more improvements and fixes to the
monitor-gating
* These improvements have led to
* Finding a bug in our testing script
* Improved log messages
* We actually caught a problem!
* The data the team have been reviewing have been from April - July
and have already discovered that so far it looks like Pagure, koji and
bodhi work well
* We see some intermittent problems, but nothing too big, mostly
only spikes in runtime
* Fedora CI still looks like a POC, but functional
* Our test-script hitting timeouts/failing 10% of the time
* Gating (greenwave/resultsdb/waiverdb) looks functional, but
relies on CI and doesn't have as much packages going through the
workflow
* A more formal report will be published soon as part of the project
deliverable so keep an eye on their work!
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Team Info

Changes to CPE Product Owner Office Hours

Following the feedback received in the CPE survey, I will be reducing
my IRC office hours to once per month.

#fedora-meeting-1

* Next Meeting: 2020-10-15 @ 1300 UTC on #fedora-meeting-1

#centos-meeting

* Next Meeting: 2020-10-13 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

September 11, 2020

CPE Weekly, September 1st

September 11, 2020 06:44 PM

Hi everyone,

Welcome to September! Below are the most recent Community Platform
Engineering project updates, and if you want to know more about our
team, see our wiki page here for more information on who our team is:
https://docs.fedoraproject.org/en-US/cpe/

Here are some upcoming IRC meetings:

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-09-03

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-09-01

GitLab AMA Session

* September 10th @ 1330 UTC on #fedora-meeting-1

Below are the project & community updates this week:

GitLab

There will be an IRC based AMA session with GitLab on Thursday 10th
September @ 1330 UTC in place of the CPE PO office hours.
We are still talking to GitLab but we are deliberately taking our time
to make sure all of the technical blockers can be met and the move
will be worth it in the end.
There is very little to no updates in the tracker, but I will include
it nonetheless https://gitlab.com/gitlab-org/gitlab/-/issues/217350
I will also be sending a separate email on details of the AMA session
later this week, such as how to submit questions in advance so there
is content ready on the day.

Note: Since I'm posting this to the blog after the meeting, you may wish to read the

CentOS Updates

CentOS

* Updated ocp.stg to OCP v4.5.6.
* Added a number of users to the jump.ci host.
* Adding monitoring/alerting for NFS slowness to the ocp cluster.

CentOS Stream

* Module push tweaks.
* Exploring how to enable fedora messaging in Stream
* Reviewing documentation on contributor policies before publishing
them later this quarter.

Fedora Updates

Staging Environment

* Services will begin to be deployed this week
* Please be patient as some services will inevitably not work due to
networking errors that the team don't know until they deploy
* Thank you again for your patience and understanding during these
last few months!

AAA Replacement

* Deployment to staging for testing is delayed due to missing firewalls in IAD2
* This has just recently been unblocked so the team will begin some
deployment and testing of Noggin this week
* Wider community testing will be available, estimated next week
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* List of applications that require messaging schemas can be found
here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been reviewing data on how packages are built in the
fedora infrastructure for the last 8 weeks and have gathered enough
information to create a report on their findings.
* This report is currently in draft format, and is going to be
reviewed by the team first, and then sent to the devel and infra lists
in the next 2 weeks est.
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Here is a reminder of what our team has committed to work on in this
quarter of the year:

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

September 10, 2020

Deploying OpenShift in KVM/libvirt guests

September 10, 2020 10:00 PM

This week I had to work on a PoC to deploy OpenShift in Virtual Machines instead of bare-metal, like we did recently for the CentOS CI infra

Why in Virtual Machines (KVM guests) and not on bare-metal ? Well, there are cases where you have powerful/beefy machines, but not enough to meet the minimum number of nodes (at least 3 etcd nodes, and not even counting the real workers, at least 2 so 5 in total for bare minimum), while these nodes would perfectly (both at cpu/memory and storage) support the whole infra (assuming that you don't deploy all etcd/control planes nodes on the same physical node of course, and same for workers)

If you have a look at the official openshift documentation, you'll see that while all major cloud providers (AWS, Azure, GCP) are listed, there are also ways to deploy on bare-metal (what we did for CI infra), but also on RHEV, vSphere and Openstack too .. but nothing for plain KVM hypervisors (managed by libvirt in our cases).

But a VM is more or less like a bare-metal install, so what about we treat the VMs as bare-metal ? problem solved, right ? For our bare-metal deployment, we just used Ansible and with a simple ad-hoc playbook, so nothing fancy : just creating pxe boot entries, using ipmi to remotely power on the nodes and ensure they'd boot on network, RHCOS is installed and has all the kernel parameters for network settings and where to find RHCOS image to install, and where to find ignition files

So reusing that was my first idea, as we can easily create a VM with a fixed mac-address, and boot from the network. But then I thought about what we already use for our traditional KVM deploy : a simple ad-hoc playbook just templating a virt-install command that is kicked on the hypervisor.

If you have used virt-install yourself, you know that there is the --location parameter (that we used already). Extracted from man virt-install :

      -l, --location OPTIONS
           Distribution tree installation source. virt-install can recognize certain distribution trees and fetches a
           bootable kernel/initrd pair to launch the install.

How does that work ? Well, virt-install grabs kernel and initrd from that location, but to know where to find it (name/path), it uses a .treeinfo file. Example of http://mirror.centos.org/centos/7/os/x86_64/.treeinfo :

[general]
name = CentOS-7
family = CentOS
timestamp = 1587405659.3
variant = 
version = 7
packagedir = 
arch = x86_64

[stage2]
mainimage = LiveOS/squashfs.img

[images-x86_64]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img
boot.iso = images/boot.iso

[images-xen]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img

So let's combine this option with the Red Hat CoreOS tree that we'll generate on our httpd deployment server : such .treeinfo doesn't exist, but let's just template it . From that point, it's easy, let's just use a variant for ad-hoc playbook that will :

  • Download kernel/initrd.img and deployer image for openshift to our local httpd server
  • Ensure we'll have correct .treeinfo file in place
  • Create a virt-install wrapper that will just point to correct path with --location, and deploy VMs with RHCOS and automatically calling ignition

While I admit that I'm surely not the most experienced openshift admin (just started to play with it), I still like the fact that RHCOS is still more or less linux that we use to know, and so combinining tools we already use allow us to deploy it , but surely not the way it's officially documented :)

Board approves creation of Infrastructure SIG

September 10, 2020 01:40 AM

As any open source project grows and matures, the people who have always done all the things can’t do everything any more. CentOS is at that point (really, we have been for a long time) and we’ve been struggling with those kind of growing pains for some years.

Recently, we made a small change in process, so that SIG leads can manage their own SIG membership and permissions associated with SIG membership. This removed some of the load on the people who have traditionally handled this on behalf of the SIGs. This change was made possible by updates in the authentication tooling, and has the Board looking at other areas where we can move more operational things out of the Board, which can then focus on governance and executive tasks.

Another area where this need for maturity has been clear for a while is our infrastructure, where a small number of people handle all of the infrastructure tasks. As our infrastructure grows (in both size and complexity) this has put us in situations where work was blocked because those people didn’t have time to get tasks done. And, sometimes, people like to take a holiday.

This week, the Board of Directors approved a plan to move infrastructure oversight into a SIG structure, so that different parts of the infra can be delegated to a larger group of people. We have already been in partnership with CPE (Red Hat’s Community Platform Engineering team) to do some of these tasks, and this will give greater liberty to spread the load around even further, but also greater transparency around what it takes to be granted that access.

We are still in the process of figuring out what is in scope, and collecting a community of interested volunteers willing to do the work. The CentOS infrastructure is very broad, including build machines, the mirror network, systems with donated hosting, and so on, and we want to be sure that we do this in a way that doesn’t interfere with the work that’s already being done by so many people, while at the same time working to identify more places where volunteers can make a difference.

To be clear, giving administrative access to any systems still requires a great deal of trust and assurance of competence. We’re not giving the keys away to anyone who asks. But people who have demonstrated that competence and trustworthiness will be able to do some of the tasks that have, thus far, been handled by 2 or 3 people. We’ll be working to develop and publish some clear guidelines around who can be trusted with this access, and how to demonstrate that competence.

The exact details of how this SIG will operate are still being worked out. But the Board has approved Aoife Moloney as the first Infrastructure SIG chair, since she has already been sending infrastructure updates to the centos-devel mailing list roughly weekly for some months now, and is a logical person to take on this organizational responsibility. So, thank you, and welcome, to Aoife, for her willingness to tackle this job.
We welcome participation and comment from anyone with the skills or interest to be helpful, particularly if you are already administering any of the infrastructure that the CentOS project relies on.

September 09, 2020

Minutes for CentOS Board of Directors for 2020-07-08

September 09, 2020 07:45 PM

On 2020-07-08 the CentOS Board of Directors met to discuss ongoing business. The Board welcomed Brian "bex" Exelbierd as a new member Director as Red Hat Liaison. Additional topics included clarifying the Secretary documents/minutes handling, reviewing the new SIG process document proposed by Rich Bowen.

A reboot of the NFV SIG was approved and the new SIG chair is ​Alfredo Moralejo. Join us to welcome Alfredo, the NFV SIG meetings minutes can be checked for further details !

Rich Bowen and the board, congratulate Alain Reguera Delgado and Fabian Arrotin for the new CentOS website design and contribution process.

The Board came to the following decisions, resolutions, and agreements discussing the issues listed at https://git.centos.org/centos/board/issues :

  • Action #5 #7 and #14 can be closed.
  • #17 Board members' links need updating Board member ACTION to review their Biography over the next weeks.
  • #12 Adding SIG leaders to Board meetings AGREED to invite SIG chairs when SIG input is needed or having a dedicated meeting to have a shorter agenda. More info will come when we redefine the SIG process following Rich Bowen proposal.
  • #01 Shift Board to be more transparent in support of becoming a contributor-focused open source project ACTION To provide a list of concrete actions over the next weeks. Now we have public issues and a way to contact the board, another change will be to improve the SIG process for contribution.
  • #10 Add Secretary role to the governance ; ACTION this task is ongoing and should be closed in next weeks.
  • #13 Get an official CentOS image in to WSL ; ACTION To pursue this item the board needs to draft a list of requirements. and review needed actions.
  • #18 Rebuilding OVS/OVN in NFV Sig ; AGREED Unanimous consensus to name Alfredo Moralejo as chair.

Note: my apologies for the slow publication of these minutes; transitioning into this new Secretary role has been a little overwhelming, but I am committed to publish next meeting minutes in a timely manner.


Powered by Planet!
Last updated: March 01, 2021 09:30 PM