October 27, 2020

Using connection delegation with mitogen for Ansible

October 27, 2020 11:00 PM

This should be a very short blog post, but long enough to justify a blog post instead of a 'tweet' : I had myself a small issue with mitogen plugin in our Ansible infra.

To cut a long story short, everybody knows that ansible relies on ssh as transport. So one can use traditional ~/.ssh/config tuning to declare ProxyJump for some hosts, etc

But when you use mitogen (we do), in the official doc there is a mention of specific parameter for connection delegation : mitogen_via

The simple example on the webpage seems trivial and if you have multiple hosts that need to be configured from remote ansible+mitogen combo, using mitogen would speed things up as it would know about the host topology.

That's what I thought when having a look at the simple inventory on that web page:

[dc2]
web1.dc2
web2.dc2
web3.dc2

[dc2:vars]
mitogen_via = bastion.dc2

Sounds easy but when I tried quickly to use mitogen_via , something that I thought would be obvious in fact wasn't. My understanding was that mitogen would automatically force agent forwarding when going through the bastion host. A simple ansible -m ping (let's assume web1.dc2 in their example) returned me :

web1.dc2 | UNREACHABLE! => {
    "changed": false,
    "msg": "error occurred on host bastion.dc2: SSH authentication is incorrect",
    "unreachable": true
}

Well, we can see from the returned json that it was trying to pass through bastion.dc2 and that's confirmed on web1.dc2 :

Oct 28 15:52:36 web1.dc2 sshd[12913]: Connection closed by <ip_from_bastion.dc2> port 56728 [preauth]

Then I thought about something that was obvious to me but that mitogen (just reusing underlying ssh) doesn't do automatically : Forwarding the ssh agent to the nodes behind.

We can easily solve that with one simple ansible parameter : ansible has the ansible_ssh_common_args and ansible_ssh_extra_args parameters, specific to the SSH connection

So what about we force Agent Forward just on that bastion host and see how that works ? That means that in our inventory (but can go to host_vars/bastion.dc2 too) we just have to add parameter:

bastion.dc2 ansible_ssh_extra_args='-o ForwardAgent=yes'

Let's try again :

web1.dc2 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "ping": "pong"
}

Good, so we can push that for our bastion hosts (used in inventory for mitogen_via) in host_vars or group_vars and call it a day. The reason why I prefer using ansible_ssh_extra_args is that it will merge and add settings, in case you have already something like this in your ansible.cfg :

[ssh_connection]
ssh_args =

I like the logic that we don't need to modify ~/.ssh/config with all exceptions to reflect the infra layout but we can just reflect it in ansible inventory

October 25, 2020

CPE Weekly: 2020-10-25

October 25, 2020 12:02 AM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-25 for both
Fedora & CentOS, and if you want to visit the hackmd link
https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view you can then use the
header bar on your left to skip to Fedora or CentOS updates that
interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

Updated initative timetable can be viewed here
https://docs.fedoraproject.org/en-US/cpe/time_tables/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

Sent a mail to the devel lists for both Fedora & CentOS with questions
that had answers relating to the topic Accouns & Permissions. It has
been sent to devel-announce at fedoraproject.org &
centos-devel at centos.org. Here is the link to the hackmd doc I used to
write the email before copying it across to my email to send
https://hackmd.io/1pjX1cVnTjekOLVowj5UiQ?view

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* CentOS 7.9.2009 pkgs pushed to CR repo, next step is to install tree
and artifacts such as iso & cloud images.
* Working a lot with the AAA/Noggin team on CentOS account integration
and making good progress

CentOS Stream

* There are now cloud image updates available for CentOS Stream
* There are some documentation updates here on unshipped packages &
Stream feel free to read up!
https://wiki.centos.org/FAQ/CentOS8/UnshippedPackages

Fedora

General

* F33 will be released next week!!
* The team got an F33 nightly compose finished too!
* There is a dashboard available to view the performance of Anitya
https://monitor-dashboard-web-monitor-dashboard.app.os.fedoraproject.org/d/8Zi9LU5Mz/anitya?orgId=1
* And there is capability to do the same for the packager workflow
pipeline which we are still working on

Staging Environment

* Build system nearly done - waiting on a firewall change

Noggin/AAA

* New estimated deployment date for Noggin is 29th January 2021.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6
* And we have a project tracker available to be viewed here
https://github.com/fedora-infra/aaa-tracker

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.Below are the logs from the
most recent meetings and dates for the next ones.

#fedora-meeting-1

* Next Meeting: 2020-11-12 @ 1300 UTC on #fedora-meeting-1 (On Freenode IRC)

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting (On Freenode IRC)

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

October 17, 2020

CPE Q3 Achievements 2020

October 17, 2020 03:17 PM

Hi there,

I'm Aoife Moloney. You may remember me from such communications as the CPE office hours, Data Centre - what it means for you, and The Future of Communishift.

Over the last three months, the Community Platform Engineering team (or CPE for short as it's long to keep typing) have been working on a few projects, and generally surviving 2020 like everyone else. But we made it, and so did our projects! Mostly… 🙂

 

Over the last three months we worked on:

  • The Great Fedora Data Centre Move of 2020
  • Noggin
  • CentOS Stream
  • Packager Workflow Healthcare (Always check with your maintainer before taking this workflow. Side effects may be, but not limited to, frustration, tears, elation at successful builds)
  • Fedora-Messaging Schemas

 

We also had our long standing (and long suffering) ‘sustaining team’ on the front lines who are daily maintaining and running both the Fedora and CentOS infrastructures and responding to issues, bugs, etc. And doing a damn fine job too.

 

And we attended and participated in a few conferences too, namely Nest with Fedora & DevConf US.

 

So, what did we as a team overall achieve in these last few months?

 

CPE Infra & Releng Team 

This team was led by Pingou, and its members in Q3 were Mark O’Brien, Michal Konecky, Fabian Arrotin, David Kirwan, Kevin Fenzi, Vipul Siddharth, Stephen John Smoogen & Tomas Hckra.

This team is a sub team of CPE and focuses on lights on work in both the Fedora and CentOS infrastructures. We will always have some of our team members working in this way each quarter as it is good to have a break from scheduled project workloads and take a foray into the (sometimes) chaotic world of infrastructure maintenance, aka FIRE!!! 🙂

What they did: 

  • Changed their name. May we present: CPE Infra & Releng Team - oooooh, aaaaah!
  • Vipul & David worked with Fabian in the CentOS infra and did something with openshift clusters & migrated the kojihub for https://cbs.centos.org to a new infra
  • Kevin and Smooge moved all of the Fedora infrastructure. 117 servers.  Let that number sink in.
  • Pingou & Michal did a ton of babysitting toddlers 🙂 They moved a lot of scripts over and things are working well
  • Tomas helped bootstrap F33 - oh yeah!
  • And Mark had (not biologically,  but in sentiment) a baby! And became an admin for some of the Fedora Infrastructure.

Why its good:

  • The name change represents what this team works on and is easy to understand instantly. Plus, naming is hard so we wanted to keep it simple 🙂
  • Helped release F33 beta
  • We doubled down on toddlers allowing to build more automation around the infrastructure
  • Assisted with the fedora datacentre move for minimal disruption to the fedorans day to day lives
  • Helped keep CentOS CI operational - and then helped put out the flames when it caught fire 🙂
  • Over 500 tickets across both Fedora & CentOS infra + releng resolved by this team - that is some seriously good firefighting!

 

Fedora Data Centre Move

This dynamic duo was Kevin Fenzi and Stephen Smoogen, with supporting cast members from both CPE and the community along the way. The goal of this project was to successfully move a (large) number of the Fedora infrastructure hardware from one datacentre to the other without too much chaos. And considering the world wide pandemic that happened right at the beginning, they did a pretty fine job succeeding. Some additional services are still being added to the infrastructure in its new home in IAD, so if you notice a few still missing, we are getting to them slowly but surely and thank you again for your patience and understanding during these last few months!

What they did: 

  • Moved a ton of servers across the country of the United States
  • Kept critical services in Fedora Infrastructure alive during the move
  • Worked an uncountable amount of hours!

Why its good:

  • We got some new hardware!
  • The team carried out some resilience testing in the new data centre which means more reliability for the infrastructure should bad things happen 
  • Updated records and warranties were a passive benefit of this move too

 

Noggin

This team was led by Aurelien Bompard, and its members in Q3 were Ryan Lerch, Nils Philippsen & James Richardson. The goal of this project is to replace the current FAS system with a newer one and migrate the CentOS accounts to the one FAS instance (Noggin), which will mean our team has one authentication system to maintain for two infrastructures long term. This team has been working to a November 2020 deadline, but unfortunately during Q3 the team faced a number of challenges such as a delayed staging environment to test in due to the data centre move, then when we got it, realized their plugin they spent time developing was not going to work long term and now have to redo a bit of work in Q4. There were also a lot of holidays and personal events for the team in Q3 because everyone is human and entitled to a life 🙂 They have re-scoped their work for Q4 to make sure what's delivered is sustainable and reliable long term, more people have joined the team including some sys-admin for support along the way, and are now looking at delivering Noggin in full by the end of January 2021. 

What they did: 

  • A lot of ipsilon investigation
  • Added a spam curtailer service to Noggin
  • Added an agreements section for users to select their user preferences 
  • Deployed Noggin to staging but found out the way they did it wont be good for the project long term
  • Had a little cry about developing a plugin unnecessarily, hugged it out and then re-planned dates and the work we need to do in Quarter 4 to be able to deliver a better, more reliable and robust service in January 2021. Queue Noggin’  - Rise of the Phoenix Project

Why its good:

  • We knew where we went wrong, learned a lot both technically and as a project team for it, and were able to call the mistakes out and get the support we need to get the project back on track. Just a little bit later than we wanted.
  • We still created a solution that will meet the needs of both the CentOS and Fedora community users, and once we have the correct configurations in place and are ready to be tested we look forward to your feedback!

CentOS Stream:

This team was led by Brian Stinson, and its members in Q3 were Johnny Hughes, Carl George, Mohan Boddu, Leonardo Rosetti, James Antill & Siteshwar Vashisht

What they did: 

  • A lot of darn package & module building
  • Light hearted threats to their PO to teach her how to convert a CentOS Linux distro to CentOS Stream using the new release package - which she did! 🙂
  • Kept CentOS Stream compose up to date with RHEL nightlies
  • Launched the centos-stream-release package - Big deal. Like, huge.

Why its good:

  • CentOS Stream is continuing to stand on its own and becoming a more robust distro
  • There's lots more content in Stream for its users
  • Users can now swap from CentOS Linux to Stream easily

Packager Workflow Healthcare:

This team was led by Will Woods and its team members were Adam Saleh and Stephen Coady with Pingou in a part-time consulting/reviewing role. The team took a look into the Fedora packager workflow and tried to identify weaker points in the chain, and spot times that are more prone to downtimes.  They are finalizing a report of their findings to send to the community lists with hopefully a ‘next steps’ section that they feel will help reduce the issues packagers face sometimes in Fedora. Its a work in progress, but to have some data to read and understand is a great launching point.

What they did:

  • Refined the monitor-gating script that monitors the packager pipeline to enhance its performance
  • Picked a certain date range and got a database dump to pull metrics from into graphana to chart uptimes of applications within the pipeline
  • Created a diagram of the pipeline to help understand how packages flow through the fedora infra

Why its good:

  • The diagram of the packager workflow process is a great resource for both packagers and new contributors of the fedora community to refer to and help understand how things work
  • The team also have some recommendations they are working through with management and the wider CPE team to identify possible next steps and how we can improve the packager experience long term by adopting better monitoring.

 

Fedora-Messaging Schemas:

This project was also being worked on by the Noggin team part-time, so Aurelien Bompard, Nils Philippsen & Ryan Lerch. We needed to pause this work around the start of September and we hope to be able to return to it over the next quarter - October, November & December.

The guys have a github board here with a cookie-cutter schema available and a list of apps they were working on, so if you want to help out on this one, please feel free to visit the board and grab a card! 🙂 

What they did: 

  • Created a board to track the work being done and whats left to do https://github.com/orgs/fedora-infra/projects/7
  • Created a template schema 
  • Created a list of applications that require a schema update
  • Added some schemas to applications that need them

Why its good:

  • This will help us progress the retirement of fed-msg in 2021
  • It will also give applications, and application maintainers, access to new fedora messaging schemas for more faster & reliable notifications.

 

And that, my dear friends, is Quarter 3 for CPE.

Take care all, and see you around IRC! 🙂

 

Aoife

CPE Weekly: 2020-10-17

October 17, 2020 12:49 AM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-10-17.
Im gone a little bi-weekly lately with this report. This has been more
circumstantial with our quarter 3 projects ending and launching
quarter 4 work, and will get back to a weekly report now that Q4 is
underway.
So, the updates for both Fedora & CentOS are below, and if you want to
visit the hackmd link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view
you can then use the header bar on your left to skip to Fedora or
CentOS updates that interest you.

General Project Updates

We have a CPE Q3 Achievements blog out on the Fedora and CentOS websites
https://blog.centos.org/2020/10/cpe-q3-achievements-2020/
https://communityblog.fedoraproject.org/cpe-achievements-during-q3-2020/

And below are the projects the CPE team are working on for the months
of October, November & December:
* CentOS Stream Phase 4 - Build system services
* Noggin Phase 4 - Data Migration of Fedora & CentOS Accounts, Community testing
* OSBS for aarch64 - this will begin in November
* Fedora Messaging Schemas - this work is continuing from Q3 and is
being worked on part-time

Misc

GitLab

There is now a blog post out on the Fedora community blog on the AMA,
you can read it here
https://communityblog.fedoraproject.org/gitlab-ama-follow-up/
Im going to be sending one mail per week to the CentOS and Fedora
devel lists on Fridays, dedicated to one topic.
The first topic will be Permission and Access in GitLab. I will pull
the questions and answers from the hackmd doc into an email body to
try to facilitate dedicated discussion around this topic. I would like
to just set expectations now that I don't consider myself to be
technical enough to weigh in on the discussion, but I am looking
forward to reading and learning from the conversation.

Project Updates

*The below updates are pulled directly from our CPE team call we have every week.*

Fedora

General

* Go NoGo meeting was on 15th Oct. F33 was No GO and new date is moved
to 2020-10-27

Staging Environment

* Working on getting OpenQA moved over
* Adding boxes to CentOS to test IPA

Noggin

* After the team reviewed the work still left to do to be able to
deploy a robust and secure solution, we discovered that there is more
development work to do and we are now working towards a release date
of January 29th 2021.
* Accounts & data will remain secure in the current solution until we
can cut over, we have established a monitoring plan to derisk this
extended time the team needs to complete their work.
* The team are working on completing a full staging environment to
deploy Noggin in right now and will hope to have this in place in the
next few weeks
* We also have some members of CentOS working in this team now to help
with the work required for the migration of the CentOS accounts and
data to Noggin
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is worked on on a part time basis as we are
prioritizing completing Noggin first before fully committing to its
completion
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

CentOS Updates

CentOS

* New CI admin added - mobrien, welcome!
* Plumbing for duffy being worked on currently and will be deploying
to staging in the coming weeks

CentOS Stream

* Looking at ODCS services in the build system
* Also working on deploying mbbox operator to the system too
* centos-stream-release package is also out, go check it out!

Team Info

CPE Product Owner Office Hours

IRC office hours are now once per month.

#centos-meeting

* Next Meeting: 2020-11-10 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

October 13, 2020

CentOS Community Newsletter, October 2020 (#2010)

October 13, 2020 05:51 PM

Dear CentOS enthusiasts,

The past month has been quieter than recent months, in terms of news, and I was very occupied with some other things last week, so I hope you can forgive the delay in getting this month's newsletter out.

If you want to receive notifications of new newsletters in future months, please subscribe to the centos-newsletter mailing list to be the first to hear of new editions.

News:

Embroidered shirts and sweatshirts at HelloTux

You can now purchase embroidered CentOS tshirts, polo shirts, and sweatshirts, in the store at HelloTux. HelloTux also produces garments featuring our friends Debian, Ubuntu, openSUSE, and various other Linux and open source projects and brands.

An article with a little more detail about the process may be found on the CentOS blog.

Board welcomes Brian Exelbierd as director

On 2020-07-08 the CentOS Board of Directors welcomed Brian "bex" Exelbierd as a new member Director and Red Hat Liaison. This, and other board news may be found in the minutes from that board meeting.

Creation of Infrastructure SIG

In the September board meeting, the Board approved the creation of the Infrastructure SIG, which will provide a place for coordination of all planning and decisions which effect the infrastructure that enables the CentOS project. Aoife Moloney was named as the initial chair of this group. More details about the function of this SIG may be found in the article announcing the creation.

Updates:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during September:

 

September 23, 2020

CPE Weekly, September 20th 2020

September 23, 2020 05:42 PM

Hi Everyone,

Below is this week's CPE weekly for week ending 2020-09-20.

I found that if you want to skip to the hackmd, you can use the view
link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and then use the
header bar on your left to skip to either the Fedora or CentOS
updates, whichever interest you.

I'll also be adjusting these updates in the coming weeks to make them
a bit more direct to consume. Thanks for giving me this feedback in
the CPE survey, I want to deliver value to you all, so it's great to
KNOW what you find valuable first hand

General Project Updates

As a reminder, below are the projects the CPE team are working on for
the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

We have recently held our Q4 planning session and the CPE review team,
Fedora, CentOS and RHEL BU have voted the following projects for
action in Q4, which is the months of October November & December:

* OSBS for aarch64
* Fedora-messaging schemas

We are continuing to work on CentOS Stream and Noggin and took these
projects as confirmed when looking at what other work our team could
realistically complete in the Q4 period, given that there's both
Thanksgiving and Christmas time off to consider, plus any time off our
team wishes to take.

The taiga cards of Noggin, CentOS Stream, OSBS for aarch64 and
fedora-messaging schemas will be updated next week with what our team
hopes to deliver in the next quarter on each of the projects.
Our project board is here (it's just not updated properly - yet)
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

Misc

GitLab

Thank you so much to everyone for adding your questions to the doc for
the GitLab AMA session on Thursday 10th September, and for your
attendance on the day during the call.
Here is the full AMA transcript
https://meetbot.fedoraproject.org/fedora-meeting-1/2020-09-10/ama_session_with_gitlab.2020-09-10-13.31.log.html
however it is a bit confusing to read so we got a few great
suggestions to have dedicated topics like Message Bus and Branching,
etc go out to the devel lists to discuss. I'm happy to start this next
week, but I will collect the questions related to each topic and
propose a cadence to send them out first to discuss, so people dont
miss mails and know the week ending 2nd October will be (for example)
the topic of Group Permissions - What do you think?

GitLab have also agreed to answer the questions, we have asked them to
do so within 2 weeks of the AMA so as soon as this is complete I will
let you know so you can read through them on the hackmd link.
The link is here where we asked you to contribute your questions and I
will be posting answers once we have them underneath
https://hackmd.io/RW8HahOeR7OJPON1dwuo3w

I really appreciate your involvement with this as we begin to dig
deeper into how this might play out next year and what way it should
for everyone's benefit.

Project Updates

*The below updates are pulled directly from our CPE team call we have
every week.*

CentOS Updates

CentOS

* Deployed new 4.5.9 openshift cluster for Stream
* The team provisioned EC2 infra for team responsible for
registry.centos.org (we don’t maintain it, so just providing infra,
like Fedora does for Copr)
* They also migrated a bunch of nodes to the new Ansible CI inventory

CentOS Stream

* Using Openshift cluster for engineering work and will be using it to
deploy & test mbbox in our infra
* Scoping and refining work for October November & December

Fedora

General

* 6 of 8 Beta-blockers have fixes for F33 beta
* New release of fedscm_admin
* FMW mac and windows binaries are signed

Staging Environment

* About 70% done installing vm’s (27 left out of 88)
* Still need to bring up aarch64/armv7/ppc64le builders
* Databases need syncing

AAA Replacement

* The team are working on testing Ipsilon in Staging and adding OpenID
Connect Capability
* they are also testing fas2ipa migration script in tiny-stage and improve it
* Add Noggin to tiny-stage environment and test
* The teams kanban board where they track their work can be found here
https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is on hold until Noggin completes.
* It will be resumed around December timeframe and is part of our Q4
workload to complete
* There is a list of applications that require messaging schemas can
be found here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been working on more improvements and fixes to the
monitor-gating
* These improvements have led to
* Finding a bug in our testing script
* Improved log messages
* We actually caught a problem!
* The data the team have been reviewing have been from April - July
and have already discovered that so far it looks like Pagure, koji and
bodhi work well
* We see some intermittent problems, but nothing too big, mostly
only spikes in runtime
* Fedora CI still looks like a POC, but functional
* Our test-script hitting timeouts/failing 10% of the time
* Gating (greenwave/resultsdb/waiverdb) looks functional, but
relies on CI and doesn't have as much packages going through the
workflow
* A more formal report will be published soon as part of the project
deliverable so keep an eye on their work!
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Team Info

Changes to CPE Product Owner Office Hours

Following the feedback received in the CPE survey, I will be reducing
my IRC office hours to once per month.

#fedora-meeting-1

* Next Meeting: 2020-10-15 @ 1300 UTC on #fedora-meeting-1

#centos-meeting

* Next Meeting: 2020-10-13 @ 1500 UTC on #centos-meeting

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

September 11, 2020

CPE Weekly, September 1st

September 11, 2020 06:44 PM

Hi everyone,

Welcome to September! Below are the most recent Community Platform
Engineering project updates, and if you want to know more about our
team, see our wiki page here for more information on who our team is:
https://docs.fedoraproject.org/en-US/cpe/

Here are some upcoming IRC meetings:

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-09-03

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-09-01

GitLab AMA Session

* September 10th @ 1330 UTC on #fedora-meeting-1

Below are the project & community updates this week:

GitLab

There will be an IRC based AMA session with GitLab on Thursday 10th
September @ 1330 UTC in place of the CPE PO office hours.
We are still talking to GitLab but we are deliberately taking our time
to make sure all of the technical blockers can be met and the move
will be worth it in the end.
There is very little to no updates in the tracker, but I will include
it nonetheless https://gitlab.com/gitlab-org/gitlab/-/issues/217350
I will also be sending a separate email on details of the AMA session
later this week, such as how to submit questions in advance so there
is content ready on the day.

Note: Since I'm posting this to the blog after the meeting, you may wish to read the

CentOS Updates

CentOS

* Updated ocp.stg to OCP v4.5.6.
* Added a number of users to the jump.ci host.
* Adding monitoring/alerting for NFS slowness to the ocp cluster.

CentOS Stream

* Module push tweaks.
* Exploring how to enable fedora messaging in Stream
* Reviewing documentation on contributor policies before publishing
them later this quarter.

Fedora Updates

Staging Environment

* Services will begin to be deployed this week
* Please be patient as some services will inevitably not work due to
networking errors that the team don't know until they deploy
* Thank you again for your patience and understanding during these
last few months!

AAA Replacement

* Deployment to staging for testing is delayed due to missing firewalls in IAD2
* This has just recently been unblocked so the team will begin some
deployment and testing of Noggin this week
* Wider community testing will be available, estimated next week
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* List of applications that require messaging schemas can be found
here https://hackmd.io/@nilsph/H1i8CAbkP/edit
* There is a readme which contains documentation on messaging schemas,
a cookie-cutter template to create the schema and a definition of Done
for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* The team have been reviewing data on how packages are built in the
fedora infrastructure for the last 8 weeks and have gathered enough
information to create a report on their findings.
* This report is currently in draft format, and is going to be
reviewed by the team first, and then sent to the devel and infra lists
in the next 2 weeks est.
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

Here is a reminder of what our team has committed to work on in this
quarter of the year:

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

September 10, 2020

Deploying OpenShift in KVM/libvirt guests

September 10, 2020 10:00 PM

This week I had to work on a PoC to deploy OpenShift in Virtual Machines instead of bare-metal, like we did recently for the CentOS CI infra

Why in Virtual Machines (KVM guests) and not on bare-metal ? Well, there are cases where you have powerful/beefy machines, but not enough to meet the minimum number of nodes (at least 3 etcd nodes, and not even counting the real workers, at least 2 so 5 in total for bare minimum), while these nodes would perfectly (both at cpu/memory and storage) support the whole infra (assuming that you don't deploy all etcd/control planes nodes on the same physical node of course, and same for workers)

If you have a look at the official openshift documentation, you'll see that while all major cloud providers (AWS, Azure, GCP) are listed, there are also ways to deploy on bare-metal (what we did for CI infra), but also on RHEV, vSphere and Openstack too .. but nothing for plain KVM hypervisors (managed by libvirt in our cases).

But a VM is more or less like a bare-metal install, so what about we treat the VMs as bare-metal ? problem solved, right ? For our bare-metal deployment, we just used Ansible and with a simple ad-hoc playbook, so nothing fancy : just creating pxe boot entries, using ipmi to remotely power on the nodes and ensure they'd boot on network, RHCOS is installed and has all the kernel parameters for network settings and where to find RHCOS image to install, and where to find ignition files

So reusing that was my first idea, as we can easily create a VM with a fixed mac-address, and boot from the network. But then I thought about what we already use for our traditional KVM deploy : a simple ad-hoc playbook just templating a virt-install command that is kicked on the hypervisor.

If you have used virt-install yourself, you know that there is the --location parameter (that we used already). Extracted from man virt-install :

      -l, --location OPTIONS
           Distribution tree installation source. virt-install can recognize certain distribution trees and fetches a
           bootable kernel/initrd pair to launch the install.

How does that work ? Well, virt-install grabs kernel and initrd from that location, but to know where to find it (name/path), it uses a .treeinfo file. Example of http://mirror.centos.org/centos/7/os/x86_64/.treeinfo :

[general]
name = CentOS-7
family = CentOS
timestamp = 1587405659.3
variant = 
version = 7
packagedir = 
arch = x86_64

[stage2]
mainimage = LiveOS/squashfs.img

[images-x86_64]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img
boot.iso = images/boot.iso

[images-xen]
kernel = images/pxeboot/vmlinuz
initrd = images/pxeboot/initrd.img

So let's combine this option with the Red Hat CoreOS tree that we'll generate on our httpd deployment server : such .treeinfo doesn't exist, but let's just template it . From that point, it's easy, let's just use a variant for ad-hoc playbook that will :

  • Download kernel/initrd.img and deployer image for openshift to our local httpd server
  • Ensure we'll have correct .treeinfo file in place
  • Create a virt-install wrapper that will just point to correct path with --location, and deploy VMs with RHCOS and automatically calling ignition

While I admit that I'm surely not the most experienced openshift admin (just started to play with it), I still like the fact that RHCOS is still more or less linux that we use to know, and so combinining tools we already use allow us to deploy it , but surely not the way it's officially documented :)

Board approves creation of Infrastructure SIG

September 10, 2020 01:40 AM

As any open source project grows and matures, the people who have always done all the things can’t do everything any more. CentOS is at that point (really, we have been for a long time) and we’ve been struggling with those kind of growing pains for some years.

Recently, we made a small change in process, so that SIG leads can manage their own SIG membership and permissions associated with SIG membership. This removed some of the load on the people who have traditionally handled this on behalf of the SIGs. This change was made possible by updates in the authentication tooling, and has the Board looking at other areas where we can move more operational things out of the Board, which can then focus on governance and executive tasks.

Another area where this need for maturity has been clear for a while is our infrastructure, where a small number of people handle all of the infrastructure tasks. As our infrastructure grows (in both size and complexity) this has put us in situations where work was blocked because those people didn’t have time to get tasks done. And, sometimes, people like to take a holiday.

This week, the Board of Directors approved a plan to move infrastructure oversight into a SIG structure, so that different parts of the infra can be delegated to a larger group of people. We have already been in partnership with CPE (Red Hat’s Community Platform Engineering team) to do some of these tasks, and this will give greater liberty to spread the load around even further, but also greater transparency around what it takes to be granted that access.

We are still in the process of figuring out what is in scope, and collecting a community of interested volunteers willing to do the work. The CentOS infrastructure is very broad, including build machines, the mirror network, systems with donated hosting, and so on, and we want to be sure that we do this in a way that doesn’t interfere with the work that’s already being done by so many people, while at the same time working to identify more places where volunteers can make a difference.

To be clear, giving administrative access to any systems still requires a great deal of trust and assurance of competence. We’re not giving the keys away to anyone who asks. But people who have demonstrated that competence and trustworthiness will be able to do some of the tasks that have, thus far, been handled by 2 or 3 people. We’ll be working to develop and publish some clear guidelines around who can be trusted with this access, and how to demonstrate that competence.

The exact details of how this SIG will operate are still being worked out. But the Board has approved Aoife Moloney as the first Infrastructure SIG chair, since she has already been sending infrastructure updates to the centos-devel mailing list roughly weekly for some months now, and is a logical person to take on this organizational responsibility. So, thank you, and welcome, to Aoife, for her willingness to tackle this job.
We welcome participation and comment from anyone with the skills or interest to be helpful, particularly if you are already administering any of the infrastructure that the CentOS project relies on.

September 09, 2020

Minutes for CentOS Board of Directors for 2020-07-08

September 09, 2020 07:45 PM

On 2020-07-08 the CentOS Board of Directors met to discuss ongoing business. The Board welcomed Brian "bex" Exelbierd as a new member Director as Red Hat Liaison. Additional topics included clarifying the Secretary documents/minutes handling, reviewing the new SIG process document proposed by Rich Bowen.

A reboot of the NFV SIG was approved and the new SIG chair is ​Alfredo Moralejo. Join us to welcome Alfredo, the NFV SIG meetings minutes can be checked for further details !

Rich Bowen and the board, congratulate Alain Reguera Delgado and Fabian Arrotin for the new CentOS website design and contribution process.

The Board came to the following decisions, resolutions, and agreements discussing the issues listed at https://git.centos.org/centos/board/issues :

  • Action #5 #7 and #14 can be closed.
  • #17 Board members' links need updating Board member ACTION to review their Biography over the next weeks.
  • #12 Adding SIG leaders to Board meetings AGREED to invite SIG chairs when SIG input is needed or having a dedicated meeting to have a shorter agenda. More info will come when we redefine the SIG process following Rich Bowen proposal.
  • #01 Shift Board to be more transparent in support of becoming a contributor-focused open source project ACTION To provide a list of concrete actions over the next weeks. Now we have public issues and a way to contact the board, another change will be to improve the SIG process for contribution.
  • #10 Add Secretary role to the governance ; ACTION this task is ongoing and should be closed in next weeks.
  • #13 Get an official CentOS image in to WSL ; ACTION To pursue this item the board needs to draft a list of requirements. and review needed actions.
  • #18 Rebuilding OVS/OVN in NFV Sig ; AGREED Unanimous consensus to name Alfredo Moralejo as chair.

Note: my apologies for the slow publication of these minutes; transitioning into this new Secretary role has been a little overwhelming, but I am committed to publish next meeting minutes in a timely manner.

September 04, 2020

Remotely reinstalling a node on CentOS 8 with DuD (Driver Disk Update / kernel module for nic/hba)

September 04, 2020 10:00 PM

Recently in the CentOS Infra, we got a new sponsor giving us access to a server that has a HBA needing a kernel module that was deprecated in the RHEL8 (and thus CentOS 8) kernel by default.

What can you do in such situation ? Answer is easy : Elrepo ! They provide (for years now) kernel modules ready to go for network cards, raid/hba controllers, wifi nics, etc, and for various versions of RHEL/CentOS and other rebuilds using same kernel.

I wanted to give it a try on a node I have at least remote KVM/ipmi access, to reset the node in case of problem. Let's use the following ~8y old IBM blade for this example , that has the following network interface card and also hba :

# lspci |egrep -i 'ethernet|Serial'
0c:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2004 PCI-Express Fusion-MPT SAS-2 [Spitfire] (rev 03)
16:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 03)
# lspci -n |egrep '0c:00.0|16:00.0'
0c:00.0 0107: 1000:0070 (rev 03)
16:00.0 0200: 19a2:0710 (rev 03)
# ethtool -i eth0|grep driver
driver: be2net
# modinfo be2net|grep 0710
alias:          pci:v000019A2d00000710sv*sd*bc*sc*i*
# lsmod|grep sas
mpt2sas               249763  2
# modinfo mpt2sas|grep 0070
alias:          pci:v00001000d00000070sv*sd*bc*sc*i*

As you can see above, we were searching for the kernel module in use, and compare in kernel module the pci id. We know which kmod to search for and which pci id the kmod is supposed to support. Let's verify this in CentOS 8 : Let's try with network module :

# modinfo be2net|grep 0710 || echo "Sorry, doesn't seem supported"
Sorry, doesn't seem supported

Ouch, be2net is present but doesn't support our pci id so it was deprecated ... we need a different one. Let's try now with the hba :

# modinfo mpt2sas|grep 0070 || echo "Sorry, doesn't seem supported"  
Sorry, doesn't seem supported

empty so also not supported. By chance, Elrepo has this packaged as rpm :

  • http://elrepo.reloumirrors.net/elrepo/el8/x86_64/RPMS/kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.x86_64.rpm
  • http://elrepo.reloumirrors.net/elrepo/el8/x86_64/RPMS/kmod-be2net-12.0.0.0-5.el8_2.elrepo.x86_64.rpm

Of course we can also use a DuD as Elrepo already provides such .iso images.

In our case though, we have to do things differently, as we need two kmods and also no way to fetch it through network ( Obviously, as we need first nic kernel module/driver ) ...

So here was my idea :

  • build a DuD .iso that has both kmods/kernel modules
  • inject that .iso inside the initrd.img (as we need kernel module loaded before we can reach network for stage2 and no way to grab network driver through network obviously)

Let's go back on the CentOS 7 node that needs to be reinstalled with CentOS 8 :

# yum install -y genisoimage createrepo_c
# cd $(mktemp -d)
# mkdir -p {./dd/rpms/x86_64/,./dd/src}
# echo -e "Driver Update Disk version 3\c" > ./dd/rhdd3
# pushd dd/rpms/x86_64/
# wget http://elrepo.reloumirrors.net/elrepo/el8/x86_64/RPMS/{kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.x86_64.rpm,kmod-be2net-12.0.0.0-5.el8_2.elrepo.x86_64.rpm}
# createrepo_c ./
# popd
# pushd dd/src
# wget http://elrepo.reloumirrors.net/elrepo/el8/SRPMS/{kmod-be2net-12.0.0.0-5.el8_2.elrepo.src.rpm,kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.src.rpm}
# popd
# mkisofs -quiet -lR -V OEMDRV -input-charset utf8 -o mpt3sas-be2net-kmod.iso ./dd

Now that we have mpt3sas-be2net-kmod.iso we can use it with inst.dd= .. but as we have no network, anaconda needs to find it early in the process. So let's inject it into initrd.img ( you can do the same with kickstart )

Let's retrieve vmlinuz and initrd.img to remotely kick a CentOS 8 reinstall on itself (node is actually running CentOS 7)

# pushd /boot
# mirror_url="http://mirror.centos.org/centos/8/"
# curl --location --fail ${mirror_url}/BaseOS/x86_64/os/images/pxeboot/initrd.img > initrd.img.install
# curl --location --fail ${mirror_url}/BaseOS/x86_64/os/images/pxeboot/vmlinuz > vmlinuz.install
# popd
# echo mpt3sas-be2net-kmod.iso |cpio -c -o >> /boot/initrd.img.install 
2005 blocks

Now that we have injected .iso into initrd.img, we can reference it for anaconda/install process as /<name>.iso. Let's then just use kexec (usual) to remotely launch the install and use also VNC to see if everything is working : network should respond and be configured, and then we'll be able to see storage too.

# pushd /boot
# yum install -y kexec-tools
# kexec -l vmlinuz.install --append="net.ifnames=0 biosdevname=0 ksdevice=eth2 inst.repo=http://mirror.centos.org/centos/8/BaseOS/x86_64/os/ inst.lang=en_GB inst.keymap=be-latin1 inst.dd=/mpt3sas-be2net-kmod.iso inst.vnc inst.vncpassword=DuDTest ip=172.22.0.16 netmask=255.255.254.0 gateway=172.22.1.254 nameserver=172.22.0.1 hostname=test.ci.centos.org pcie_aspm=off" --initrd=initrd.img.install && kexec -e

From that point it is like described in the previous link about kexec and kick a reinstall : kernel boots, loads initrd.img (but this time we see the DuD iso image being loaded and then it starts anaconda as usual. We can from there connect over vnc to finish the install (we have network and hba kernel module loaded and able to configure hardware)

Once machine is installed and rebooted, we can just ssh into it and clearly we can see that both rpm/kmods were installed ok (otherwise, no network nor storage and of course no install :) )

# modinfo mpt3sas|egrep 'filename|signer'
filename:       /lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/mpt3sas/mpt3sas.ko
signer:         ELRepo.org Secure Boot Key
# modinfo be2net|egrep 'filename|signer'
filename:       /lib/modules/4.18.0-193.14.2.el8_2.x86_64/weak-updates/be2net/be2net.ko
signer:         ELRepo.org Secure Boot Key
# rpm -qa|grep elrepo
kmod-be2net-12.0.0.0-5.el8_2.elrepo.x86_64
kmod-mpt3sas-28.100.00.00-3.el8_2.elrepo.x86_64
# yum installl -y elrepo-release

Of course, as shown above, don't forget to also install elrepo-release pkg, to then access newer kmods when needed, in case of a rebase between major.minor releases.

Hope you found that useful in case you need to upgrade working hardware but with deprecated drivers in the CentOS 8 kernel.

September 02, 2020

Embroidered CentOS Shirts and Sweatshirt

September 02, 2020 02:59 AM

Linux clothes specialist HELLOTUX from Europe, who makes the official Fedora shirts, signed an agreement with Red Hat to make embroidered CentOS t-shirts, polo shirts and sweatshirts. They have been making Debian, Ubuntu, openSUSE, and other Linux shirts for 18 years, and now the collection is extended to CentOS.

Instead of printing, they use programmable embroidery machines to make the CentOS embroidery. All of the design work is made exclusively with Linux, using Inkscape, GIMP and a machine independent embroidery tool (Embird) under Wine.

Wearing these shirts you can proudly show your commitment to CentOS and free software in the upcoming years.

You can get the CentOS polos and t-shirts in blue or black and the sweatshirt in gray from HELLOTUX.

September 01, 2020

CentOS Community newsletter, September 2020 (#2009)

September 01, 2020 01:51 AM

Hi, CentOS enthusiasts, and thanks for coming back for another edition of the CentOS monthly community newsletter.

If you want to receive notifications of new newsletters in the future, subscribe to the centos-newsletter mailing list!

News:

Boothole:

In last month's newsletter, we told you about Boothole, and the fix for it. And if you were following that issue, you probably know that for some users, the fix was broken, causing a failure to boot. Since then (actually, the next day) Red Hat addressed the issue in RHEL, and CentOS released that same fix shortly thereafter. Red Hat has written up a detailed analysis of the flaw, which also applies to CentOS users, since CentOS Linux is a rebuild of RHEL.

The short version is that it is fixed, and you should update to the latest kernel, grub, and shim, to mitigate this flaw.

Events:

Nest With Fedora

The Fedora community's annual conference is usually named Flock. As in, flock together with all of your Fedora friends! But, this year, we cannot flock, and so we had to stay home in our nests. This year's Nest with Fedora conference was held the first week of August, and is now over. But if you missed it, you can still see the presentations on the Fedora YouTube channel.

Gitlab AMA

If you're interested in what's coming with the new Gitlab infrastructure for CentOS, you should read the CPE Weekly report each week. But if you have questions, your opportunity to ask them is on September 10th at 13:30 UTC, on the #fedora-meeting-1 channel on Freenode. CPE staff will be on-hand to discuss what's planned and how it will affect you, and answer all of your questions.

SIG reports

This month we have reports from two of our Special Interest Groups (SIGs): OpsTools and Virtualization.

OpsTools SIG Quarterly report

Purpose

Provide tools for operators and build up operational knowledge for
large infrastructures

Membership update

Over the past quarter, we did not attract new contributors nor we lost
one.

Activity update

Over the past quarter, we mostly worked on migrating the CI
hosted on RDO project infrastructure from CentOS 7 to CentOS
8. Recently, we also started building collectd plugins written in Go,
which also requires a bit of dependencies provided.

VIRT SIG REPORT

oVirt

- oVirt 2020 online conference will be held on September 7th, registration open. More info at https://blogs.ovirt.org/ovirt-2020-online-conference/
- oVirt 4.4.2 expected in the first half of September.
- Sandro Bonazzola is now co-chair of Virtualization SIG
- Dominik Holler also joined NFV SIG, starting to move openvswitch ovn there.

Advanced Virtualization

- We have new maintainer: Eduardo Lima (etrunko)
- Advanced Virtualization from 8.2.1.z expected in the first half of September.

Kata containers

- recently added kata containers repository shipping same version shipped by Fedora: http://mirror.centos.org/centos/8/virt/x86_64/kata-containers/

Updates

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during August:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during August:

Get Involved!

In our recent CentOS community survey, many of you said you wanted to be move involved in the work that the community does. Here are some places where you can do that.

Contribute to the website and wiki. The CentOS website is (and always will be!) a work in progress. You can submit pull requests to improve it. The website is primarily for information about the CentOS project, and its governance, while the wiki is for information that changes more frequently, such as SIGs, events, and releases. If you want to contribute to the wiki (which is much larger than the website), join the centos-docs mailing list, and request edit access for the portion of the wiki you'd like to help improve.

Would you like to help us produce a virtual Dojo? (CentOS Dojos are one-day events focusing on technical topics.) We would like to do several over the next 6 months, and are looking for people to take ownership of an event. Pick a theme or topic emphasis. Find speakers. Promote the event. If you'd like to help, get in touch with us on the centos-promo mailing list to get started.

CentOS has a number of online discussion forums where people gather to ask questions and dispense solutions. The official CentOS Forum is the one that we run ourselves, but there's also the very active (and very helpful!) r/CentOS on Reddit, and the CentOS Facebook group, where thousands of people come every day with their questions and answers. Your expertise and patient, compassionate help, is always welcome, as we try to help beginner become experts. And, of course, the centos-devel mailing list, as well as the centos mailing list, are our primary communication channels with the community, and a great place to help out.

Finally, there's this newsletter. At the moment, it's a one-person show, and we would really like to expand that to more contributors. We need news from around the CentOS ecosystem, articles about using CentOS, and stories about what interesting things people are doing with CentOS. Come join us on the centos-promo mailing list with your suggestions.

August 14, 2020

CPE Weekly: 2020-08-14

August 14, 2020 02:44 AM

Background:

The Community Platform Engineering group, or CPE for short, is the Red
Hat team combining IT and release engineering from Fedora and CentOS.
Our goal is to keep core servers and services running and maintained,
build releases, and other strategic tasks that need more dedicated
time than volunteers can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

General Project Updates

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-08-20

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-08-18

 

Misc

Nest With Fedora Note

Thank you to everyone who attended Nest With Fedora over the weekend
and engaged with us during our talks and social sessions. It was a
fantastic event, very well put together and run by the Fedora team and
it was both my pleasure and honour to have been a part of what was a
fantastic schedule! I cant wait to catch up on talks I missed when
they are uploaded, and the CPE team, with thanks to Michal Konecny who
put the structure of our piece together, will be submitting a
collaborative blog post to the Community blog space in the coming
weeks, recapping our experience at Nest this year. Well done Marie
Nordin, Matthew Miller and the wider team, and our very own Vipul
Siddarth, on what was a very successful and enjoyable event!

Engagement Email Feedback

At the beginning of July I sent an email to devel-announce requesting
feedback from the community on changes I and the wider CPE team have
made when scheduling projects to work on, in an effort to find the
balance between work and life. We are still searching
However, I got some very good tips that I will definitely be
incorporating which I shared at Nest during the CPE AMA Session and in
reply to the original mail.

The link to the mails are here for full reading
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/UNWTLOI2U4VKAU4IKJZGZLZ2DTMPEFBW/

But the suggestions and actions are here for quicker reference:
Continue to communicate regularly on projects & updates
- Will do, weekly emails have been a bit more sporadic lately as I have had
some time off

Less acronyms & abbreviations in comms
- Sure, that's an easy fix on my end and makes sense

Publish team members timezone on docs.fpo/cpe to help define our ‘working
hours’
- Will do, I hope to get to this by end of August and they will be
reflected on docs.fpo/cpe

Publish the workflow diagram to docs.fpo/cpe and add filtered versions that
are user specific
- Same as above, publishing it on docs.fpo/cpe-initiatives

Office Hours on IRC are a useful way to contact team Product Owner
- Great to hear, please feel free to stop by when you can/want to. They are
on Thursdays @ 1300 UTC on #fedora-meeting-1 and every second Tuesday @
1500 UTC on #centos-meeting

Public tracker for bugs
- Our team meets twice a day, every day on IRC to review tickets and issues
to work on. They meet @ 0830 UTC on #centos-meeting and again @ 1800 UTC on
#fedora-admin. These are public meetings so please feel free to attend.

If you would like to offer any further suggestions or feedback on the
Community Platform Engineering team, please feel free to complete our
survey which is open until August 30th
https://fedoraproject.limequery.com/696793?lang=en

Fedora Updates

Data Centre Move

* Nearly done!!!!!!
* Firewalls for staging are going up this week
* Last of the hardware has been set up for networking changes
* The bringup of Communishift is being made into a dedicated project
for work as soon as the team have capacity to do so - this may be in
late September as the members of the colo team will (hopefully) take
some very well deserved time off work before tackling this one

AAA Replacement

* Some of the Noggin team have been enjoying some very well deserved
time off work over the last week or two so work has, naturally slowed.
* The code is currently being security audited by Patrick Uiterwijk,
thank you Patrick!
* Next steps will be to successfully deploy Noggin to staging when it
has been brought back up - we are estimating this to be late next week
* Once Noggin has been deployed to staging, we would love some
community feedback on the application and its performance. We will be
emailing the infra/devel lists on when to test and how to give us
feedback when we are fully deployed in stg.
* In the meantime. Please feel free to check out the team kanban board
for more information on the features the team are working on and have
already completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* This project is being worked on by the same team as Noggin, so there
has been some progress made but not a lot as team members enjoy some
holidays
* The team have already built a list of applications that require
messaging schemas, list can be found here
https://hackmd.io/@nilsph/H1i8CAbkP/edit
* They also have completed a readme which contains documentation on
messaging schemas, a cookie-cutter template to create the schema and a
definition of Done for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* Project information: This is an investigative project that aims to
look at the entire packager workflow as a single piece of tooling to
identify where failures happen, and try to identify when and why
packages fail at different points within the workflow. We hope to have
two possible outcomes from this project at the end of the quarter
(September):
* The workflow breaks at X point and we will work on a solution to fix
* OR
* The workflow works fine, but we will need better monitoring on
the pipeline so we will work on a solution for this
* The team are using this Monitor-Script
https://pagure.io/fedora-ci/monitor-gating/ and are making
improvements to it on resiliency/reliability.
* They are finishing the investigation phase of the project and are
going to document the packager workflow (with graphs I have been
promised!) showcasing how the different systems interact with one
another
* And are working on an outline of the workflow steps (from packager
PoV) and systems involved (CPE team PoV), identifying metrics to be
measured
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

CentOS Updates

CentOS

* CentOS-infra team have created a ticket board, check it out here
https://pagure.io/centos-infra/boards/CentOS%20CI%20Infra
* CBS Koji x86 builders moved to new(er) infrastructure
(https://lists.centos.org/pipermail/centos-devel/2020-August/055988.html)
* The team also caught up C6, C7 and C8 Linux .. 2 outstanding
Bootstrap Modules for C8 Linux (eclipse and the latest rust-toolset).

CentOS Stream

* Not too much to report this week - The team are mostly working on
developing utility scripts that will ease the CentOS 8 and CentOS
Stream packaging workflow and business as usual updates to CentOS
Stream.

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

August 12, 2020

CPE Feedback Survey

August 12, 2020 01:44 PM

Hey folks,

CPE need your help 🙂

Over the last several months we've been trying to improve how we interact and share information with you all. From the blog posts, to mails and how we work on the tickets you send us.

Here is a link to a very short survey we've put together to learn how we can give you the best experience possible going forward by understanding the experiences you've had recently.

If you could take the time (5mins max) to complete it for us it would be hugely valuable as we work on this continuous improvement - https://fedoraproject.limequery.com/696793

Cheers,

Ant.

 

August 04, 2020

CentOS Community newsletter, August 2020 (#2008)

August 04, 2020 04:42 AM

Dear CentOS enthusiasts,

Here's what's been happening over the past month.

News:

Boothole

Last week we were made aware of a security hole in grub2, and released a fix for that. You can see the details about this on the centos-announce mailing list. Unfortunately, the fix itself had problems that caused a small percentage of systems to be unbootable. We have since released fixes for that situation, too, which you can read about in the resolution of this bug - https://bugs.centos.org/view.php?id=17631 - and additional information may be found in the RHEL ticket, here: https://access.redhat.com/solutions/5272311

Further discussion of the issue may be found in recent threads on the centos@centos.org mailing list.

User Survey

Over the past 3 or 4 months, we have been running a survey about how you use CentOS. Many thanks to those who participated in this, to help us better understand how we can give you what you need.

I've written up the results of the survey here - https://blog.centos.org/2020/07/centos-community-user-survey/

While some of the results were expected - most of you use CentOS in small to medium shops, running services either for work or personal use - there were some eye-opening things in this. To me personally, the volunteerism question shows that a lot of you are looking for places to get involved, and that we haven't done a great job telling you where and how. We'll be working to fix that.

For those of you who missed the survey, we intend to do this again, perhaps as often as twice a year, and also provide opportunities to give more feedback beyond the simple 5-minute questionnaire.

Thank you all for participating, and for being part of our community.

New website: Contribution information

By now you've noticed the fresh look on the centos.org website. But it's not just a new look. It's also a new back-end, which, among other things, makes it easier for you to contribute changes and updates to the content. Please see this blog post for information about how you can help us improve the CentOS website.

CPE Updates

The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give.

Events:

Nest

This weekend is Nest - the Fedora conference. This conference is usually known as Flock, or "Flock To Fedora", but this year, since we cannot flock, we're going to nest at home.

Register at the event website (it's free!) and join us this weekend for 3 days of great Fedora (and CentOS!) content.

DevConf.US

DevConf is next month, and is another free, online event. DevConf targets developers - especially those developing on Linux. Event details and registration are available on the event website.

SIG reports:

Storage SIG:

NFS-Ganesha and libntirpc were updated to 2.8.4/1.8.1 and 3.3/3.3 respectively.
To help gfidente out and get Ceph back on track, I migrated the Ceph dist-git repos to git.centos.org from the old repos on github and converted them to the new format. I went ahead and built ceph-14.2.9 (el7, el8) and 15.2.3 (el8 only because python3) — which were the current versions at the time. But since then 14.2.10 and 15.2.4 have been released, and the release of 14.2.11 is imminent.
It's a bit beyond the quarter time frame, but glusterfs-coreutils was updated finally after many years to version 0.3.1 and built for el7 and el8; gluster-block was updated to version 0.4 and built for el8. The previous version (0.3) remains in place for el7.

Updates

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during July:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during July:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during July:

Other releases

The following releases/updates also happened during July:

July 30, 2020

centos.org contribution process

July 30, 2020 03:12 PM

I hope that by now you've seen the fresh new look on centos.org. If not, you should go look now.

Community member Alain Reguera Delgado, Fabian Arrotin, and a number of others proposed some style updates to the site, as well as a change to the way the site is generated and served. In addition to looking great, this also greatly simplifies the process of updating the content of the site.

If there are changes that you want to suggest for the CentOS website, the process for doing so is documented HERE, and the site content is there, too. You can send a pull request there, and we'll review them and publish once approved.

The CentOS website is *your* website, and we welcome your contributions to it.

July 29, 2020

CentOS Community User Survey Results

July 29, 2020 01:43 PM

Over the past several months we have been running a user survey to learn more about how CentOS is used. This was a very short survey - 4 questions - designed to give us a very high-level view of how the community breaks down in terms of usage. We hope that next year we can expand on this and get more detail.

There were 426 responses to the survey, with 96 of those individuals indicating that they would be open to us contacting them for more detail about their answers.

Without further introduction, here are the answers. (Note: we did an early report on results 2 months ago, and results have not changed significantly since then.)

Question 1: What we use it for.

Note: Percentages do not add up to 100% because multiple responses were allowed.

  • Professional software development: 30.4% (129 responses)
  • Personal/Hobby software development: 31.1% (132)
  • Running services for a job/company: 62.6% (266)
  • Running services for personal/home use: 62.8% (267)
  • Desktop computer 27.5% (117)

Takeway: CentOS is still used primarily as the workhorse on which you run your network services, both at work and at home, but using it as a software development platform is a strong second. Lots of alternate answers entered under "Other" were almost all just rephrasings of the above categories.

Question 2: Volunteerism

This question asked where participants would be interested in volunteering. Approximately half of respondents skipped this question.

  • Testing: 58.4% (132)
  • Packaging: 37.6% (85)
  • Design: 15.5% (35)
  • Promotion: 22.1% (50)
  • Event planning: 5.3% (12)

Takeaway: A lot of you want to help with testing and packaging. Why aren't you? This probably indicates that our onboarding documentation/process around the testing/qa effort needs a lot of improvement, and that we need to do a better job of pointing people to packaging opportunities like SIGs and EPEL. Those of you who want to help with design and promotion, please join us on the centos-promo mailing list!

Question 3: Size of installation

In this question we offered some very broad categories of installation sizes to get an idea of whether you're running this on a single machine or a cluster of thousands.

  • 1-5 systems: 42.1% (179)
  • 6-50 systems: 34.6% (147)
  • 51-100 systems: 8.5% (36)
  • 101-1000 systems: 10.4% (44)
  • 1000+ systems: 4.5% (19)

Takeaway: While most of our fascinating stories tend to focus on the massive installations - the supercomputing clusters and massive social networks of the world - the overwhelming majority of users are smaller installations in the 1-50 systems range.

Question 4: Update frequency

With this question we were trying to get a clearer idea of how important frequent updates are - how often do you consume new packages with a dnf update

  • Daily - 19.4% (82)
  • Weekly - 36.3% (153)
  • Monthly - 26.8% (113)
  • New releases - 5.2% (22)
  • Less frequently - 12.3% (52)

Takeaways: Almost all of you update at least monthly, with more than half of you updating at least once a week. We think this means that regular updates are pretty important to you. We're fascinated with the 12% of you who install a system and then leave it running with no updates.

What did we learn?

The biggest thing that I learned from this is that there's a lot of you out there that want to help and don't know what opportunities are available. We obviously need to do a much better job of telling you about those opportunities.

There's so much more that we wanted to ask, but we know that surveys are annoying, and you get a million of them every week. We do intend to do more of these in the future, but will try to do them in such a way that they are as uninavasive as possible, while giving the opportunity for those of you who want to tell us more, to do so.

Those of you who indicated that we can contact you to ask more - we're still trying to figure out what we want to ask you. Thanks for your willingness and your patience. You'll hear from us in the coming months.

July 25, 2020

CPE Weekly: 2020-07-25

July 25, 2020 02:10 PM

Background:

The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

General Project Updates

The CPE team are working on the following projects for Quarter 3,
which is the months of July, August & September:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-07-30

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-08-04

Misc

* CFP for Nest with Fedora submissions are closing on Monday 27th July
- dont forget to submit your ideas/talks here https://pagure.io/flock
* CPE Engagement email was sent a few weeks ago, please read if you
have not already and feel free to send me your feedback on or off list
before 31st July. A follow up email will be sent with feedback
received and actions taken where it makes sense then afterwards.
Thanks for your
engagement! https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/UNWTLOI2U4VKAU4IKJZGZLZ2DTMPEFBW/

Fedora Updates

Data Centre Move

* The team have been working incredibly hard over the last few weeks
and are now working on reinstalling the staging environment.
* We estimate this work to be completed by July 31st, however things
happen so we thank you in advance for your patience
* The project card on taiga for builder - staging - remaining app
bringup dates here
https://tree.taiga.io/project/amoloney1-cpe-team-projects/us/42?kanban-status=2139945
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* And finally a reminder to please be patient when opening tickets for
service issues in general as we are in the critical point in this move
and all of our sys-admins and wider teams are assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.

AAA Replacement

* The team are focusing work on this ticket and collaborating with the
Sustaining team for the next sprint, with the goal to have noggin
installed & deployed to staging
https://pagure.io/fedora-infrastructure/issue/9152
* The have also been addressing how to curtail spam email and more
information on that effort can be found here
https://github.com/fedora-infra/noggin/issues/27
* And have added the feature to allow groups with a prefix for groups
to import to avoid overlaps between CentOS and Fedora
https://github.com/fedora-infra/fas2ipa/issues/5
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Fedora Messaging Schemas

* The team are using a kanban approach for this project as Noggin
takes priority until fully complete
* The team have already built a list of applications that require
messaging schemas, list can be found here
https://hackmd.io/@nilsph/H1i8CAbkP/edit
* They also have completed a readme which contains documentation on
messaging schemas, a cookie-cutter template to create the schema and a
definition of Done for writing a schemas
https://github.com/fedora-infra/fedora-messaging-schemas-issues
* The board they are working from can be viewed here
https://github.com/orgs/fedora-infra/projects/7

Packager Workflow Healthcare

* Project information: This is an investigative project that aims to
look at the entire packager workflow as a single piece of tooling to
identify where failures happen, and try to identify when and why
packages fail at different points within the workflow. We hope to have
two possible outcomes from this project at the end of the quarter
(September):
* The workflow breaks at X point and we will work on a solution to fix
* OR
* The workflow works fine, but we will need better monitoring on
the pipeline so we will work on a solution for this
* The team are using this Monitor-Script
https://pagure.io/fedora-ci/monitor-gating/
* Current status: failing consistently because of CI test timeouts
* The jenkins instance being used is under-resourced, jobs waiting
up to 40 minutes for a node
* They are using a workable dump of historical datanommer/datagrepper
data for queries to pull metrics on the historical/aggregate Packager
Workflow from as a baseline
* And are working on an outline of the workflow steps (from packager
PoV) and systems involved (CPE team PoV), identifying metrics to be
measured
* The teams work is being tracked here
https://teams.fedoraproject.org/project/cpe-cicd/kanban

CentOS Updates

CentOS

* CentOS-infra project created https://pagure.io/centos-infra/
* Day to Day working with CentOS-infra being documented at:
https://pagure.io/cpe/docs/pull-request/10
* New website is live https://www.centos.org

CentOS Stream

* Business as usual, the team are churning through building packages in Stream
* They are also working on tooling to bring back some rpkg features

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

July 07, 2020

CentOS Community newsletter, July 2020 (#2007)

July 07, 2020 04:42 AM

Dear CentOS enthusiasts,

Thanks for coming back for another edition of the CentOS community newsletter.

News

8.2.2004 release

We are pleased to announce the general availability of CentOS Linux 8.2.2004. Effectively immediately, this is the current release for CentOS Linux 8 and is tagged as 2004, derived from Red Hat Enterprise Linux 8.2 Source Code.

As always, read through the Release Notes at : http://wiki.centos.org/Manuals/ReleaseNotes/CentOS8.2004  - these notes contain important information about the release and details about some of the content inside the release from the CentOS QA team. These notes are updated constantly to include issues and incorporate feedback from the users.

More information at https://lists.centos.org/pipermail/centos-announce/2020-June/035756.html

You can track the status of upcoming 7.x and 8.x releases at  https://wiki.centos.org/About/Building_7 and https://wiki.centos.org/About/Building_8 respectively

centos.org refresh

With a great deal of help from community members Fabian Arrotin and Alain Reguera Delgado, we are proud to announce our new centos.org website, which you can see today at https://centos.org/   The new site is built using Jekyll - https://jekyllrb.com/ - which makes it easier to contribute to, and easier to build and deploy.

If you want to contribute patches or changes to the website, that is now done via https://git.centos.org/centos/centos.org  The new workflow makes it easy to test your changes locally, and send pull requests which can be approved and rolled out automatically to the live site.

Additionally, of course the site design has been refreshed and redesigned, updating the site that was last refreshed in 2013. So a huge thank you in particular to Alain for that work.

Board meeting minutes

The minutes of the June CentOS Board of Directors meeting may be found on the CentOS Blog at https://blog.centos.org/2020/06/minutes-for-centos-board-of-directors-for-2020-06-10/

Highlights include:

Board Liaison role has passed from Karsten Wade to Brian “bex” Exelbierd, due to his strategic placement in Red Hat’s RHEL business unit. Please welcome bex to the board.

The Community Architect (currently Rich Bowen) has been added as a permanent invite to board meetings. This is to give you, the community, another way to have your voice heard.

The Secretary role has passed from Karsten Wade to Thomas Oulevey as part of the regular rotation of that position.

The July board meeting will be held on July 8th. (it’s always on the second Wednesday of the month.) If you have issues that you need to raise to the board, you are encouraged to open a ticket at https://git.centos.org/centos/board/issues

CPE Updates

CPE (Community Platform Engineering) is the group in Red Hat which provides infrastructure support to the CentOS and Fedora projects. Their weekly updates, posted both to the centos-devel mailing list and to the CentOS blog, provide insight into what they’re working on, and what’s coming next.

To learn more about what CPE is doing, come to the CPE office hours, every other Tuesday at 15:00 UTC on the #centos-meeting IRC channel, on Freenode.

Events

Please plan to join us September 24th and 25th for DevConf.US, an event for open source developers. The call for presentations is still open for one more day!

Updates

Errata and Enhancements Advisories

We issued the following CEEA (CentOS Errata and Enhancements Advisories) during June:

Errata and Security Advisories

We issued the following CESA (CentOS Errata and Security Advisories) during June:

Errata and Bugfix Advisories

We issued the following CEBA (CentOS Errata and Bugfix Advisories) during June:

Other releases

The following releases also happened during June:

CentOS Stream

We've done a lot of work over the past few weeks to keep up to date with Red Hat Enterprise Linux Development. Currently Red Hat is in the middle of the development cycle for the upcoming RHEL 8.3, and you should be seeing some content from 8.3 reflected in CentOS Stream. Expect to see more content coming through as we add more modules. There is an updated installer and refreshed install media on the mirrors for you to try right now!

RealTime in CentOS Stream

One major addition to CentOS Stream is the RealTime (RT) repository. This is a set of packages that is developed alongside Red Hat Enterprise Linux, focused on latency-sensitive workloads. Because these packages are developed so closely with a given RHEL release, it makes perfect sense to include these packages in and gather feedback from CentOS Stream. The packages in the RealTime repository represent what's coming in the Red Hat Enterprise Linux for Real Time addon for RHEL 8.3.

You can see the documentation for Red Hat Enterprise Linux for Real Time here:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/8/html/installation_guide/index

To enable this repository on your system, make sure you have a fresh install of CentOS Stream or `dnf update` to be sure you have the latest `centos-release-stream package` and enable the `Stream-RT` repository

Yum repo files are located in: `/etc/yum.repos.d/CentOS-Stream-RT.repo`

As always, bugs can be reported against the CentOS Stream version in bugzilla: https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%208&version=CentOS%20Stream

SIG Reports

Software Collections SIG Report

 Membership update

We are always looking for new members. However, no changes in membership happened in the last six months.

Process changes and releases

  1. The SIG started to reap benefits of the new, streamlined process of releasing packages from the CentOS build system. As a consequence, packages successfully build in CBS should be now coming to you with less bureaucracy.

    However, this move required changes in the release CI. Although most of the associated issues are now resolved, there are still some kinks to iron out as we go along.

  1. The Red Hat Software Collections 3.5 were sucessfully rebuilt and should be presently available in the testing (buildlogs) repository. Due to the changes mentioned in the previous point, not all of the collections reached the release repositories (mirrors) yet, but they should be appearing gradually in a few weeks.

 New collections included in this release are:

    -   Perl 5.30 (rh-perl530)
    -   Python 3.8 (rh-python38)
    -   Ruby 2.7 (rh-ruby27)

    Several other existing collections also received updates.

    For details, please see the official release notes.

  1. The https://www.softwarecollections.org/ website was migrated to run in OpenShift environment. The benefits should include increased availability and stability. As the backend changes necessary were not insignificant, there may be unforeseen new issues still present despite the testing done beforehand. Please do not hesitate to report them to the sclorg mailing list.

Health and activity

The SIG remains active and tries to respond to any issues raised while keeping the official Red Hat collections available and up-to-date on CentOS.

However, most of the actual work is done by a single person (jstanek). More active members would be more than welcome, at least in order to increase the bus factor above 1 🙂

 

[Video] KDE on CentOS 8

July 07, 2020 02:32 AM

Enabling and configuring KDE on CentOS 8 (A presentation from Red Hat Summit)

July 06, 2020

CPE Weekly: 2020-07-05

July 06, 2020 01:55 PM

Background:

The Community Platform Engineering group is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give.

See our wiki page here for more information: https://docs.fedoraproject.org/en-US/cpe/

General Project Updates

The CPE team have finished our Quarterly Planning for Q3, July -
September, and will begin work on the following projects starting from
Monday:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
* Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Look out for our Q2 project achievements blog post coming soon!

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Next Meeting: 2020-07-09

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Next Meeting: 2020-07-07

Fedora Updates

Data Centre Move

* We are now officially operating under reduced Fedora services until
est 28th July to facilitate the final shipment of hardware to the new
data centre.
* Please View project card on taiga for builder - staging - remaining
app bringup dates here
https://tree.taiga.io/project/amoloney1-cpe-team-projects/us/42?kanban-status=2139945
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.
* Most recent update to devel-announce is here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5DNRZ4OUUNGSUJONQLEXXP3CKME43SCE/

AAA Replacement

* User agreements for both CentOS and Fedora accounts are nearly
complete. We will be able to test this in more detail once staging has
been reinstalled (est July 25th due to colo move)
* Fasjson now supports search feature
* App migreations work still ongoing
* Spam detection service is being investigated and scoped for Q3 work
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Mbbox

* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBBox handover to CentOS Stream
* The only thing that remains is putting the operator in the
public operator registry which is pending Stream team feedback
* MBox shared CRD is done
* Deployment guide - https://mbbox-operator.readthedocs.io/en/latest/
* Blog post published - check it out here
https://communityblog.fedoraproject.org/mbbox-module-building-in-a-box/

Gitforge

We are still discussing technical aspects of the project and these are
tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
Planning a date in late August for an AMA session with GitLab that
will be run through IRC for any questions the Fedora and CentOS
communities may have to ask direct.
We will keep you up to date with the developments as and when we have
information to share and thank you again for your patience.

CentOS Updates

CentOS

* New website went live (https://www.centos.org)
* A new AWS CDN setup for Stream composes
(https://composes.centos.org) is being worked on with a goal to have
the latest stream artifacts to be publicly available and at faster
speed than through normal mirror.centos.org cdn (and/or external
mirrors)

Centos Other

* CentOS CI began onboarding Fedora CI, CoreOS CI and Fedora CoreOS as
tenants this week
* Work going well and stable!

CentOS Stream

* RealTime for Stream is released!
* The team are also working on more module builds for Stream

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week ahead!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

June 30, 2020

[Video] CentOS Buildsystems and infrastructure

June 30, 2020 08:32 AM

(A presentation from Red Hat Summit)

There's a lot of work behind the scenes to support the CentOS community. In this presentation, we'll learn a little about CentOS community buildsystems and infrastructure.

June 29, 2020

CPE Weekly: 2020-06-28

June 29, 2020 08:38 PM

Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

## General Project Updates

The CPE team have finished our Quarterly Planning for Q3, July -
September, and will begin work on the following projects starting from
Monday 5th July:
* Data Centre Move - Final Works
* CentOS Stream Phase 3
* Noggin Phase 3
* Packager Workflow Healthcare
* Fedora Messaging Schemas

Details of the above projects, and of projects currently in progress,
done and what projects are in our backlog, can be found on our taiga
board per project card:
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

We also have an updated initiative timetable for briefing in new
projects to our team & key dates
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

CPE Product Owner Office Hours

#fedora-meeting-1

* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Meeting log from 2020-06-25:
https://meetbot.fedoraproject.org/teams/cpe_project_owner_office_hours/cpe_project_owner_office_hours.2020-06-25-13.01.log.html
* Next Meeting: 2020-07-02

#centos-meeting

* Every second Tuesday @ 1500 UTC on #centos-meeting
* Meeting log from 2020-06-23:
https://www.centos.org/minutes/2020/June/centos-meeting.2020-06-23-15.00.log.html
* Next Meeting: 2020-07-07

Fedora Updates

Data Centre Move

* We are now officially operating under reduced Fedora services until
est 28th July to facilitate the final shipment of hardware to the new
data centre.
* Please View project card on taiga for builder - staging - remaining
app bringup dates here
https://tree.taiga.io/project/amoloney1-cpe-team-projects/us/42?kanban-status=2139945
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.
* Most recent update to devel-announce is here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5DNRZ4OUUNGSUJONQLEXXP3CKME43SCE/

AAA Replacement

* The team are finishing some work on applications including
* Codebase changes to applications
* Application maintainers can use the API to look up group and
account information
* And addressing how to add a spam blocking feature
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Mbbox

* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBBox handover to CentOS Stream
* Mbbox has been deployed to the staging environment
* Cleaning old code from the project was done
* Deployment guide - https://mbbox-operator.readthedocs.io/en/latest/
* Blog post published - check it out here
https://communityblog.fedoraproject.org/mbbox-module-building-in-a-box/

* Tasks expected to be complete by 3rd July
* MBox shared CRD - tests and polishing

Gitforge

We are still discussing technical aspects of the project and these are
tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
And are working on a date in August/September for an AMA session with
GitLab that will be run through IRC for any questions the Fedora and
CentOS communities may have to ask direct.
We will keep you up to date with the developments as and when we have
information to share and thank you for your patience.

CentOS Updates

 

CentOS

* CentOS Linux 8.2.2004 has been released
https://lists.centos.org/pipermail/centos-announce/2020-June/035756.html
* CentOS 7.x AMI images to land soon
* Infra Nodes are also moving
* The CentOS CPE team members are also working on a new website design
with the community, so this should be pushed soon too.

Centos Other

* Fedora CI and Noobaa tenants have been onboarded to CentOS CI infra
* The team are also looking at automation work within CI

CentOS Stream

* RealTime for Stream is built!
* The team have also finished laying out repos and are doing final
tests with the installer
* This will be pushed to the mirrors very soon so keep an eye out!

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week ahead!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

June 23, 2020

[Video] CentOS PaaS (Platform as a Service) SIG

June 23, 2020 08:29 AM

(A presentation from Red Hat Summit)

The CentOS PaaS SIG builds, tests, and delivers CentOS packages for platform as a service software. In this talk, we learn specifically about the work on OKD (OpenShift Origin Kubernetes Distribution).

June 22, 2020

Minutes for CentOS Board of Directors for 2020-06-10

June 22, 2020 11:57 PM

On 2020-06-10 the CentOS Board of Directors met with Chris Wright and Deb Bryant to settle open discussions on the future of the Red Hat Liaison role on the CentOS Board. The Board also welcomed Rich Bowen as an ex-officio member, Thomas Oulevey as the new Board Secretary, and approved Brian Exelbierd as a new member Director and Red Hat Liaison. Additional topics included clarifying that voting does not occur in the new issue tracker, and asking Rich Bowen to lead an open vision and strategy process to refresh the project’s goals.

In support of these efforts, the Board came to the following decisions, resolutions, and agreements:

  1. Shall the CentOS Community Architect (Rich Bowen) be permanently added as an ex-officio attendee to all future CentOS Board meetings as a non-voting voice for the wider community?
    1. AGREED Yes, this should be a permanent seat in all Board discussions, public and private.
    2. ACTION Karsten and Rich need to coordinate and make changes to add to governance.
  2. Shall Brian Exelbierd be appointed to the CentOS Board as a member Director based on his experience and understanding of the CentOS Project and its many relationships, on his working years as a community architect, and on his performance over the years in the broader CentOS and Fedora Linux communities?
    1. AGREED Yes, Brian Exelbierd (bex) is hereby welcomed to the Board as a full voting Director.
    2. ACTION Rich to change the website to reflect this addition.
  3. Does the Board accept Red Hat’s change of the Red Hat Liaison role from Karsten Wade to Brian Exelbierd, who can fulfill this role directly out of his work at Red Hat?
    1. AGREED Yes, effective immediately.
    2. ACTION Rich Bowen to change the website to reflect this change.
  4. Does the Board accept Red Hat’s request to change the Red Hat Liaison role to be always representing Red Hat on the Board? I.e., remove the “hat on/hat off” aspect of the Liaison role.
    1. AGREED Yes, the Board welcomes this clarification that helps the Liaison be an effective two-way voice between Red Hat and the CentOS Project.
    2. ACTION Karsten and Rich need to coordinate and make changes to add to governance.
  5. Shall the role of Board Secretary transfer from Karsten Wade to Thomas Oulevey, to last for a period no shorter than six months and no longer than eighteen months?
    1. AGREED Yes, Thomas shall take over the Secretary role beginning 01 July, transition to begin immediately.
    2. ACTION Thomas and Karsten need to finalize the Secretary duties and details to be rolled into governance; Karsten needs to update the governance to reflect the changes; Rich to update the website to reflect this change.
  6. Should the new issue tracker be used for conducting voting, or only for announcing the results of discussions occurring elsewhere?
    1. AGREED No, the issue tracker should not be used for discussions or debate, public or private, on the item in the issue. Discussions should happen in the appropriate forum, with relevant links, notes, and decisions reflected into the individual ticket.
    2. ACTION Voting portion of governance should include this clarification, if it’s unclear. Karsten and Rich to review and propose changes to governance, if needed.
  7. Does the Board request Rich Bowen to take lead on an update to the project’s vision and strategy, to bring all the stakeholders to the discussion in working with the planned co-authors?
    1. AGREED Yes, the Board thinks this is a better approach in having Rich work from his position in the middle of the many voices and stakeholders.
    2. ACTION Rich to announce this effort and next steps.
  8. No Community Architect report nor triage of the issue tracker occurred in this meeting due to time constraints.

Present at the meeting:

  • Jim Perrin
  • Johnny Hughes
  • Pat Riehecky
  • Thomas Oulevey
  • Tru Hyunh
  • Ralph Angenendt
  • Karanbir Singh (Chair)
  • Chris Wright (Guest)
  • Deb Bryant (Guest)
  • Mike McLean
  • Rich Bowen
  • Karsten Wade (Secretary)

June 21, 2020

CPE Weekly: 2020-06-21

June 21, 2020 08:22 PM

CPE Weekly: 2020-06-21

Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

General Project Update

Please check out our updated initiative timetable for briefing in new
projects to our team
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Dont forget to view our taiga board to see the projects we are
currently working on, what we have scoped and whats in our backlog
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

CPE Product Owner Office Hours
* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Every second Tuesday @ 1500 UTC on #centos-meeting (next meeting 23rd June)

Fedora Updates

Data Centre Move

* We are now officially operating under reduced Fedora services until
est 28th July to facilitate the final shipment of hardware to the new
data centre.
* Most recent update to devel-announce is here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/5DNRZ4OUUNGSUJONQLEXXP3CKME43SCE/
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.

AAA Replacement

* The team are working on the changing application code bases to use
the new solution
* The work required to facilitate multiple contributor agreements
under one account is now complete
* The team have also enabled group attributes during this sprint
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Mbbox

* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBS Frontend CRD + documentation
* Staging environment - thanks to CentOS CI team
* MBox shared CRD - certificates and PVCs

* Tasks underway currently
* Staging environment - testing
* MBox shared CRD - tests and polishing

Gitforge

With the data centre move taking most of the teams focus this week,
discussions with gitlab have been quiet. We are still discussing
technical aspects of the project and these are tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
We will keep you up to date with the developments as and when we have
information to share and thank you for your patience.

CentOS Updates

 

CentOS

* CentOS Linux 8.2.2004 Released!!!
https://lists.centos.org/pipermail/centos-announce/2020-June/035756.html

Centos Other

* OCP4 staging cluster is up and running
* CentOS will have a booth at Open Source Summit starting June 29th -
https://events.linuxfoundation.org/open-source-summit-north-america/

### CentOS Stream
* The team are still working on building packages from the RHEL 8.3
nightly composes

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week ahead!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view

 

June 16, 2020

[Video] Neal Gompa Adapting To Appstreams

June 16, 2020 08:26 AM

Adapting to AppStreams: Delivering the Datto Linux Agent for RHEL 8, with Neal Gompa

(A presentation from Red Hat Summit)

CPE IRC Office Hours, June 9th

June 16, 2020 02:43 AM

Last week we had our first CPE/CentOS IRC office hours. This will become a regular thing, every second Thesday at 15:00 UTC. It's an opportunity to ask anything to the CPE - Community Platform Engineering - team about their work with the CentOS (and Fedora!) infrastructure, from the status of ongoing work, to requesting new work.

This meeting ended up mostly being me - Rich Bowen - asking Aoife Moloney questions that I've been hearing from the community over the past months, and so came across as a bit of a one-on-one interview. So I figured I'd just post it as such, for those who missed it.

We hope to see you on Tuesday, June 23, for the next edition of this meeting.

 <amoloney> #startmeeting

 <amoloney> #meetingname CPE PO Office Hours
 <centbot> The meeting name has been set to 'cpe_po_office_hours'
 <amoloney> #info About the CPE team: https://docs.fedoraproject.org/en-US/cpe/
 <amoloney> #topic Open floor and Introductions
 <amoloney> Hi Everyone!
 <amoloney> For those of you who dont know me, my name is Aoife Moloney and I am the Product Owner on the Community Platform Engineering Team
 <rbowen> Hi, amoloney. Thanks for hosting.
 <amoloney> I included a link to the fedora wiki site as I had it in my notes handy to share :)
 <amoloney> And hi Rich! You're welcome! Im happy to be here in the CentOS channel
 <rbowen> Questions, anybody?
 <rbowen> Don't be shy.
 <amoloney> I have not set an agenda or specific topic today so if there are people on this meeting who have them, please feel free to ask
 <amoloney> Even if its how to phonetically pronounce my first name  - there are a lot of vowels in there :)
 <rbowen> I think there is still some confusion about how you - CPE - work in conjunction with the CentOS infra group. Is there some way we can improve that situation?
 <amoloney> I can offer my understanding of how CPE includes CentOS infra if that gives any clarity? The CPE team as a whole is comprised of both Fedora and CentOS sys-admins, developers and release engineers. Both communities rely on our team to maitain and support both infrastructures. We also have projects that our team works on that are outside of the sustainment of the infrastructures and often the communities of the infras
 <amoloney> we support are unaware that the engineers that are doing this work in the infra are actually working across other projects too
 <amoloney> We have been working hard over the last few months to try to lessen these situations, or better yet to mitigate these situations altogether, but it has proven challenging at times!
 <amoloney> What we have done is try to diffretiate between our work request types to two categories: Large initiatives or projects that take a team and time to complete, and responsive, 'lights-on' work
 <amoloney> * What we have done is try to differentiate between our work request types to two categories: Large initiatives or projects that take a team and time to complete, and responsive, 'lights-on' work
 <rbowen> Is this tracked publicly somewhere that the community can inspect and comment on what you're prioritizing?
 <amoloney> This 'lights-on' work probably speaks to the relationship between CentOS infra and CPE as this, to me, would be support and maintenance of the distros infra
 <rbowen> I mean, other than your weekly emails to the centos-devel list, that is.
 <amoloney> Sure!  Not sure I have a CentOS infra tracker as I am not familiar with this side of the house (which is in part why Im here, to talk to the community first hand) :)
 <amoloney> but the initiatives can be found here https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null
 <amoloney> This board and the cards are intended to be view only for drive-by viewing
 <rbowen> Oh, right, and I see that you linked to that in your weekly report email, too.
 <amoloney> This reflects what the CPE team are currently working on per quarter, what we are scoped and/or scoping, and the initiatives that are in our backlog
 <amoloney> Yep I've started to include it in there too
 <rbowen> Latest report is here: https://blog.centos.org/2020/06/cpe-weekly-2020-06-07/
 <amoloney> How are people finding those emails? Do you find them informative? Too long?
 <rbowen> I'm concerned by the fact that there has never been a response to one of those emails. I'm curious who is reading them. I read them every week, but it's kind of my job.
 <amoloney> Im glad to hear someone reads them :)
 <amoloney> No I am more than happy to provide the emails weekly, and I am assuming the format is suitable for people
 <amoloney> however, as I have learned today, 'assuming' makes an ass out of U and ME :)
 <amoloney> Hope Im ok to say that on this channel haha :)
 <amoloney> So please, if there are tips/stricks/suggestions you would like to make in relation to my weekly emails and/or how CPE can engage with CentOS Community & vice versa, please tell me
 <rbowen> Ok, so asking that question the other direction - if community members feel that your priorities should be different - that you should be working on something else - where do we make that request? How do we talk to you?
 <rbowen> I'm kind of asking a question I already know the answer to, but not everyone in the community can pick up the phone and call you or Leigh.
 <rbowen> What's the *public* API?
 <amoloney> You can drop me an email, that is the best way to contact me: amoloney@redhat.com
 <amoloney> Im happy to discuss priorities here in this meeting slot too on a weekly/bi/weekly basis if people want?
 <Evolution> there was a theory that once jira was migrated, some of this would be more open as well, yes?
 <amoloney> I have also started to send emails to the CentOS stakeholders, Rich as Community Architect and Karsten Wade as Board Secretary, leting them know whats coming down the line with CPE in the next quarter
 <Evolution> aside - I miss jira. ADO is...less ideal
 <amoloney> Oh hey Jim!
 <Evolution> Y HALLO THERE
 <amoloney> Yes and I hope it will be more than a theory :)
 <amoloney> With the launch of Red Hat One, the open  version of Jira, I would like to move all CPE initiatives from the taiga instance to a  centralized board there
 <Evolution> wait, Red Hat One?
 <Evolution> that's what they're calling it
 <amoloney> It would be an ideal scenario then that initiatives that CentOS & Fedora want worked on that require a team of people and a lot of time to complete will land on that board for me to pick up and scope with the requestor
 <Evolution> sounds like the name of Paul Cormier's plane.
 <amoloney> Yeah thats what Ive heard it being called but it could be different when released
 <amoloney> 'Launching  RED HAT ONE in T minus..' comes to mind ha
 <amoloney> but Im not really concerned what its called once it helps (selfishly) me interact with the community more directly when prioritizing initiatives, and equally gives visibility on WHY a project is either picked up and proiritized or rejected
 <amoloney> that information is important to share and Im very conscious of that
 <amoloney> Would a public instance of Jira work for the CentOS community to engage with me through?
 <rbowen> Yes, I think it would.
 <amoloney> I have added myself as a watcher to the boards issue tracker too to learn more about the types of requests you see
 <rbowen> It has already helped as a way to engage better with the board.
 <amoloney> oh thats good!
 <rbowen> But that's the difference between executive and operations, I suppose. The requests are going to be different, although many will inevitably be sent to the wrong place.
 <amoloney> In an ideal, and hopefully not too distant future, it would be wonderful to see both CentOS & Fedora project requests that are put forward by the Board and Council/FESCo land on the jira instance for me to pick up, and for operational requests like bug fixes, RAFs to land on a centralized Sustaining Team dashboard. We have some of our team working in this way, but it is currently more tilted to the Fedora side, but
 <amoloney> the end goal is to have CentOS infra requests operate in this way too and keep consistency throughout the team and to the communities we serve
 <rbowen> I think that many of the requests would come from our SIGs, rather than from the board, since they are the primary folks relying on this infrastructure.
 <amoloney> Not RAFS, RFE's :)
 <rbowen> Unfamilar with those TLAs.
 <amoloney> So was I haha - it means, and I did have to ask, its stands for Requested Feature Enhancements
 <amoloney> this is work thats minor to existing applications or services CPE maintain
 <amoloney> they could take a day or three, but would typically not need to have a team put around them
 <rbowen> Oh, this is the project/task distinction that you were making earlier. Got it.
 <amoloney> actually I have a Community Engagement Email drafted that I am waiting to add a graphic design of our initiative workflow to before sending, but Im wondering would this be better as a blog in the CentOS community?
 <rbowen> You're always welcome to post stuff to the blog, then we can promote it to all the various places our community lives.
 <amoloney> Thanks Rich, do you think it would also be equally beneficial to have our teams and the projects they work on listed on the wiki too?
 <rbowen> Having more ways that community members can participate in the work - whether that's packaging, infra, or whatever - would be great. Historically we haven't done much of that.
 <rbowen> That could be helpful, sure, so that people know who they can contact about what.
 <rbowen> Although I wouldn't want to circumvent your management process either.
 <amoloney> Im not sure people realize how much is asked of this team, from both communities, so having the work we do and the people who do that work on the wiki might help peoples awareness and might even offer opportunities to help us too when things get busy around release times
 <amoloney> and data centre moves!
 <rbowen> So, speaking of data center moves, I've heard some grumbling that the people who are most responsible for getting 8.2 out the door have been pulled onto other projects.
 <rbowen> Having a clearer understanding of how all of this interrelates is always going to help allay those kinds of concerns.
 <amoloney> So as the product owner of the entire team and all its projects, both CentOS and Fedora related, I can definitely help answer concerns on this
 <rbowen> (I suppose I should have brought that up earlier. We're almost out of time!)
 <amoloney> Fedoras hardware is moving currently across the USA. Its a well publicised  event and has been in my weekly emails for months (yet another reason to read them if you dont 🙂)
 <amoloney> I'll type fast so people know
 <amoloney> We have x2 sys-admins in Fedora and I think CentOS can emapthize here that that is not enough
 <amoloney> so, the bulk of this project has fallen on two sets of shoulders
 <amoloney> to move an entire distros stuff
 <amoloney> but we are fortunate to be in a collaborative team, CPE, and those sys-admins were able to reach ou to the CentOS sys-admin(s) to help because we are all part of the same team
 <amoloney> and thankfully, they are able to help their fellow teammates, because its the right thing to do
 <amoloney> and its very important this week to have the services we are maintaining for Fedora brought up in time to complete the move safely and on time for the F33 mass rebuild
 <rbowen> So, when people ask "Why is 8.2 taking so long" and we answer "The people working on that have other responsibilities", every single time they come back with "how can I help move that along." I, for one, would like to work towards a world where community members can help build the next release. Do we know what it would take to get there?
 <rbowen> (That's obviously a longer-term goal, but something that people ask me about pretty much every week.)
 <amoloney> Honestly, I have no idea how to get community members to build the next CentOS release, but I am more than willing to help facilitate those conversations if it helps my team feeling pressurized both internally and externally
 <amoloney> Im here to learn about the community and help find better ways of working for my team as their product owner, so if you have suggestions or ideas to make things more open and inclusive that I can help with, please reach out to me
 <rbowen> We're going to try to do this every 2 weeks, right?
 <rbowen> Perhaps next time we'll have more participation.
 <amoloney> Yes Im good with that
 <amoloney> Hopefully! :)
 <amoloney> Rich thanks for asking questions though, I hope i was able to give ok answers to them and anything I missed please let me know and I can follow up on
 <amoloney> Right were at time, thanks for having  me on here and Im looking forward to chatting to you all again!
 <amoloney> #endmeeting

June 15, 2020

CPE Weekly: 2020-06-14

June 15, 2020 02:57 PM

Background:
The Community Platform Engineering group is the Red Hat team combining
IT and release engineering from Fedora and CentOS. Our goal is to keep
core servers and services running and maintained, build releases, and
other strategic tasks that need more dedicated time than volunteers
can give.

See our wiki page here for more
information: https://docs.fedoraproject.org/en-US/cpe/

General Project Updates

 

Please check out our updated initiative timetable for briefing in new
projects to our team
here: https://docs.fedoraproject.org/en-US/cpe/time_tables/
*Note: Initiatives are large pieces of work that require a team of
people and weeks/months to complete. Please continue to open tickets
in the normal way for bugs, issues, etc.

Don't forget to view our taiga board to see the projects we are
currently working on, what we have scoped and whats in our backlog
https://tree.taiga.io/project/amoloney1-cpe-team-projects/kanban?epic=null

CPE Product Owner Office Hours
* Weekly on Thursdays @ 1300 UTC on #fedora-meeting-1
* Every second Tuesday @ 1500 UTC on #centos-meeting (next meeting 23rd June)
* Meeting Logs
* Fedora: https://meetbot.fedoraproject.org/teams/cpe_product_owner_office_hours/cpe_product_owner_office_hours.2020-06-11-13.01.log.html
* CentOS: https://www.centos.org/minutes/2020/June/centos-meeting.2020-06-09-15.00.log.html

Fedora Updates

 

Data Centre Move

* The final Fedora hardware shipment is due to happen on Tuesday 16th June.
* We expect the shipment to arrive in the new data centre the week
beginning 22nd June and the team will begin bringing up services that
are affected by the move.
* A list of affected services is available here
https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view
* Please read the below email sent by kfenzi if you have not already
done so: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/thread/E7HJULW2S76FZCAICURWXX223N5ZXXD7/
* Details on what this move may mean for you can be found here
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/27U6YT73556KYW2RIFJO6J2HYMYVP22U/
* If an application is not working correctly at all, please check this
list https://hackmd.io/hpYYJQRjQy-oHxUS7IonIA?view before opening a
ticket to make sure its not listed as being moved. If it is being
moved, please wait a day or two, then try again.
* Similarly, please be patient when opening tickets for service issues
in general as we have now reached the critical point in this move and
all of our sys-admins and wider teams will be assisting in the
successful bringup of the reduced Fedora service and facilitation of
the final hardware shipment and move.

AAA Replacement

* The team are working on the changing application code bases to use
the new solution
* They are also working on the CentOS account integration and the
solution will see users be able to select the contributor agreements
that are relative to their account.
* They are also testing a script to use in data migration from the
current FAS system to Noggin to allow for very little, if any, changes
to the user
* The team are also completing user documentation that includes a
how-to migration guide for maintainers and a guide to help users
understand the differences between Noggin and the current FAS option
(issue #246 on board)
* Please feel free to check out the team kanban board for more
information on the features the team are working on and have already
completed here https://github.com/orgs/fedora-infra/projects/6

Mbbox

* Project Dashboard here https://github.com/fedora-infra/mbbox/projects/1
* Tasks completed in the project currently
* MBS Backend CRD + documentation
* MBS Frontend CRD - configuration and certificates
* Draft of blog post about MBBox
* Tasks underway currently
* MBS Frontend CRD - deployment config and service
* Staging environment required
* MBox shared CRD

Gitforge

With the data centre move taking most of the teams focus this week,
discussions with gitlab have been quiet. We are still discussing
technical aspects of the project and these are tracked here:
https://gitlab.com/gitlab-org/gitlab/-/issues/217350
We will keep you up to date with the developments as and when we have
information to share and thank you for your patience.

CentOS Updates

 

CentOS

* OCP4 staging cluster in progress
* CentOS Linux 8.2.2004 content is in pre-release with more artifacts to come
* There was also a recent CentOS AMA on Reddit, logs are here so check
it out if you missed it
https://www.reddit.com/r/CentOS/comments/gwnvd2/centos_ama/

CentOS Stream

* The team have been working really hard to get package sources pushed
to keep caught up with RHEL
* We are also working on tooling for more push automation
* And tooling for RealTime and other Variants are coming in Stream soon

As always, feedback is welcome, and we will continue to look at ways
to improve the delivery and readability of this weekly report.

Have a great week ahead!

Aoife

Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view


Powered by Planet!
Last updated: October 29, 2020 04:00 AM