Red Hat Automation of Platform & Cloud

Ansible

How to convert RHEL-like systems to RHEL [for IT Ops]

6 min read

21 Sep, 2020

At the beginning of 2020 (seems so long ago now!), Red Hat announced that it was making a conversion utility available to the public to convert RHEL-like systems in place to RHEL. What this means for customers is that they will be able to convert CentoS/Oracle Linux 6 & 7 to RHEL 6 or 7 on 64bit Intel systems. Reasons for this might include obtaining vendor support, taking advantage of supported major version upgrades, and consolidating ‘nix flavours.

A client approached us post security audit, and asked us if we could assist with one of their requirements: “We need to ensure our Linux fleet is supported by a vendor. We have a large fleet of CentOS servers that we need migrated to Red Hat Enterprise Linux”.

In this article, I will specifically talk about using Ansible & Red Hat Satellite to convert CentOS 7 in place to RHEL 7.

Customer Requirements

After some initial requirements gathering, the customer requires the following:

The ability to revert to a VMWare snapshot
An audit trail with visibility to stakeholders
Notifications of puppet run errors pre- and post-conversion
Change control for production systems

Technical Requirements

The following requirements and pre-requisites were met prior to undertaking conversion:

CentOS patched to version 7.7 with no outstanding reboots
CentOS servers joined to Red Hat Satellite (optional)
Available Red Hat subscriptions on each hypervisor running a target vm
An Activation Key using the Library Environment in Satellite using a Content View containing:
- Red Hat Enterprise Linux 7 Server - Optional RPMs x86_64 7.7
- Red Hat Enterprise Linux 7 Server RPMs x86_64 7.7
- Red Hat Satellite Tools 6.6 for RHEL 7 Server RPMs x86_64
FREE consultation: Understand how to build a modern cloud-based infrastructure with containers using the Red Hat OpenShift platform

Automating it all!

Ansible was the obvious choice to automate server snapshots, send MS Teams notifications, convert to RHEL, and re-join Satellite post-conversion.

But now, we need to automate to run on over 80 servers while meeting customer requirements and tailoring it to the environment.

I decided to split this into 4 re-usable ansible roles and use tags in order to include or exclude them. The playbook will run in this order:

Snapshot server
Send MS Teams notification after snapshots have completed
Run puppet agent and report failure to MS Teams before Migration
Convert 2 RHEL
Send MS Teams notification after conversion completed
Re-join Satellite and reinstall katello, gofer, qpid, insights and update all
Send Teams notification after post conversion completed
Run puppet agent and report failure to MS Teams

Automating CentOS to RHEL conversion

The epel repo is required in order to install the convert2rhel utility, in addition several rpm’s are required including the katello-ca-consumer-latest.noarch.rpm. These can be obtained from the Satellite server itself, the blog mentioned earlier has finer details around how to obtain these files. The activation key that was created in the Technical Requirements stage contains the required repositories in order to successfully replace all of the CentOS signed packages with the Red Hat signed equivalents.

The convert2rhel tool works as follows:
# convert2rhel -k convert2rhel_ak -a -v Server -o Organisation --no-rpm-va -y --enablerepo "*"

‘-k convert2rhel_ak’ refers to the activation key created within Satellite that contains the Content View required by the utility.
‘-a’ with auto-attach an available subscription.
‘-v Server’ specifies that the server is of Server variant.
‘-o Organisation’ refers to the Satellite organisation.
‘--no-rpm-va’ will not verify rpm’s on the server prior to conversion. When enabled this has been known to take up to an extra hour without providing much benefit.
‘-y’ is used to automatically proceed with the conversion (useful for automation).
‘--enablerepo "*"’ enables all repo’s available in the content view. This will ensure that the majority of packages are able to be re-installed.

The following Ansible role will do the following:

Copy a consistent yum.conf, making sure that the distropkg option isn’t set.
Stop and disable puppet to ensure that yum repos are not automatically created during a puppet run.
Remove all yum repos.
Copy the epel repo.
Install the convert2rhel utility.
Copy the subscription manager and katello-ca-consumer rpm’s to the convert2rhel/subscription-manager directory.
Copy and import the GPG key.
Copy the redhat-release-server rpm to the convert2rhel/redhat-release/Server/ directory.
Remove previously installed subscription-manager rpms.
Run convert2rhel.
Log results of the conversion to a variable for use in the main playbook.
Run grub2-mkconfig when the server is using EFI firmware. Otherwise it will fail to boot.
Show the results of the conversion.
Start and enable puppet.
Reboot the server.

---

- name: Copy yum.conf

copy:

src: yum.conf

dest: /etc/yum.conf

owner: root

group: root

mode: 0644

- name: find files

find:

path: /etc/yum.repos.d/

recurse: no

file_type: file

- name: Stop and disable puppet

systemd:

state: stopped

enabled: false

with_items:

- puppet

ignore_errors: yes

- name: Remove all repos

file:

path: "{{item.path}}"

state: absent

with_items:

- ""

- name: Copy epel repo

copy:

src: epel.repo

dest: /etc/yum.repos.d/epel.repo

- name: Install convert2rhel

yum:

state: present

- name: Copy subscription manager rpms

copy:

dest: /usr/share/convert2rhel/subscription-manager/

src: "{{item}}"

with_items:

- subscription-manager-1.24.13-1.el7.x86_64.rpm

- subscription-manager-rhsm-1.24.13-1.el7.x86_64.rpm

- subscription-manager-rhsm-certificates-1.24.13-1.el7.x86_64.rpm

- katello-ca-consumer-latest.noarch.rpm

- name: Copy GPG Key

copy:

dest: /tmp/

src: "{{item}}"

with_items:

- RPM-GPG-KEY-redhat-release

- name: Import GPG key

command: rpm --import /tmp/RPM-GPG-KEY-redhat-release

- name: Copy subscription manager rpms

copy:

dest: /usr/share/convert2rhel/redhat-release/Server/

src: "{{item}}"

with_items:

- redhat-release-server-7.7-10.el7.x86_64.rpm

- name: Remove previously installed subscription-manager rpms

yum:

state: absent

with_items:

- subscription-manager

- subscription-manager-rhsm

- subscription-manager-rhsm-certificates

- name: Covert to RHEL using Satellite

command: convert2rhel -k convert2rhel_ak -a -v Server -o Organisation --no-rpm-va -y --enablerepo "*"

failed_when: conv_result.rc == 1

- name: Build conversion result value

set_fact:

conv_teams: ""

- name: Check if /sys/firmware/efi dir exists

stat:

path=/sys/firmware/efi

- name: run grub2-mkconfig when uefi

command: grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

when: efi.stat.exists

- name:

debug:

var: conv_result.stdout_lines

- name: Start and enable puppet

systemd:

state: stopped

enabled: true

with_items:

- puppet

ignore_errors: yes

- name: Reboot server

reboot:

tags:

- reboot

Where all the magic happens (main playbook)

Extensive use of blocks and tags ensures that logical tasks can remain separated and re-useable.

The following playbook was used to automate and drive the entire conversion process:

---

- name: Migrate CentOS to RHEL

hosts: centos_migration_round_1

become: true

gather_facts: true

vars_files:

- vars/vault.yml

vars:

patching_desc: CentOS to RHEL Migration

vcenter_datacenter: Turino

vcenter_hostname: vc1.example.org

teams_webhook_url: "https://outlook.office.com/webhook/738/IncomingWebhook/738”

tasks:

- name: Snapshot server

block:

- name: Set Datacenter fact

set_fact:

vcenter_datacenter: Alberta

vcenter_hostname: vc2.example.org

when: "'alb' in inventory_hostname"

- include_role:

vars:

vcenter_username: svcSnapShots

vcenter_password: ""

vcenter_snapshot_name: ""

vcenter_snapshot_description: " from Ansible"

- name:

run_once: true

set_fact:

facts_list: ""

loop: ""

become: false

delegate_to: localhost

tags:

- snapshot

- name: Send Teams notification after snapshots have completed

block:

- include_role:

vars:

webhook_url: ""

title: CentOS to RHEL Migration notification from Ansible

text: "CentOS to RHEL VMWare snapshot summary"

color: E81123

section_title: VMWare snapshot summary

section_text: "Snapshots completed with the following results:"

section_facts: ""

become: false

delegate_to: localhost

run_once: true

tags:

- snapshot

- O365

- name: Run puppet agent and report failure to Teams before Migration

block:

- name: Run puppet agent

puppet:

ignore_errors: True

- name: Show puppet results

debug:

var: puppetrc

- include_role:

vars:

webhook_url: ""

title: Pre migration - Puppet failure notification from Ansible

text: "Pre migration - CentOS to RHEL Conversion summary"

color: E81123

section_title: " - Puppet run failed with the following results:"

section_text: ""

when:

- puppetrc.failed

tags:

- puppet1

- name: Convert 2 RHEL

block:

- include_role:

- name:

run_once: true

set_fact:

facts_list: ""

loop: ""

tags:

- convert

- name: Send Teams notification after conversion completed

block:

- include_role:

vars:

webhook_url: ""

title: CentOS to RHEL Migration notification from Ansible

text: "CentOS to RHEL Conversion summary"

color: E81123

section_title: Conversion results summary

section_text: "Conversion completed with the following results:"

section_facts: ""

become: false

delegate_to: localhost

run_once: true

tags:

- convert

- O365

- name: Re-join Satellite and reinstall katello, gofer, qpid, insights and update all

block:

- include_role:

- name:

run_once: true

set_fact:

facts_list: ""

loop: ""

tags:

- rejoin

- name: Send Teams notification after post conversion completed

block:

- include_role:

vars:

webhook_url: ""

title: CentOS to RHEL Migration notification from Ansible

text: "CentOS to RHEL Post Conversion summary"

color: E81123

section_title: Post conversion activity summary

section_text: "Conversion completed with the following results:"

section_facts: ""

become: false

delegate_to: localhost

run_once: true

tags:

- rejoin

- O365

- name: Run puppet agent and report failure to Teams

block:

- pause:

seconds: 45

- name: Run puppet agent

puppet:

ignore_errors: True

- name: Show puppet results

debug:

var: puppetrc

- include_role:

vars:

webhook_url: ""

title: Puppet failure notification from Ansible

text: "CentOS to RHEL Conversion summary"

color: E81123

section_title: " - Puppet run failed with the following results:"

section_text: ""

when:

- puppetrc.failed

tags:

- puppet2

Final outcome: Up to 160 hours saved!

I was able to re-use two of the roles in this conversion and greatly reduce the amount of time required to build automation. The conversion of 88 CentOS servers took 76 hours in total with the majority of time being spent on building automation and staggering after-hours conversions in small numbers.

I would estimate that the amount of time saved would be around 100-160 hours by using Ansible to automate and Red Hat Satellite to manage.