4.9.2. Version-Specific Upgrade Instructions

4.9.2.1. Upgrading from SIMP-6.1.0 to SIMP-6.2.0

Important

It is highly recommended that you read the information in this section in its entirety before upgrading.

4.9.2.1.1. Exclude puppet-agent from the yum update

A bug (SIMP-5021) was identified in SIMP 6.1.0 that will prevent the puppetserver service from starting after the puppet-agent package is upgraded. The issue is fixed in 6.2.0, but special handling is necessary during the upgrade.

Note

  • This problem affects ISO/RPM-based installations of SIMP when upgrading from 6.1.0, unless precautions are taken (detailed below).
  • SIMP installations that are upgraded using r10k and Code Manager are unlikely to be affected, unless the SIMP master’s puppet-agent package is updated independently.
  • These instructions only apply to the SIMP master—no changes are required on clients.

  1. Before upgrading anything, add the following line to /etc/yum.conf:

    exclude=puppet-agent

    Assuming you aren’t already using /etc/yum.conf to exclude other packages, you can add this line automatically by running:

    puppet resource file_line workaround path=/etc/yum.conf line='exclude=puppet-agent'

  2. Proceed with the upgrade as outlined in Incremental Upgrades, up through the yum update:

    # (ISO Installations only:) Unpack the new ISO's RPMs into yum repositories
unpack_dvd </path/to/ISO>

# Make sure yum picks up the new RPMs
yum clean all; yum makecache

# Apply updates to the local master
yum update -y

  3. After running yum update, remove the exclude=puppet-agent line from /etc/yum.conf.

    If you added the line using puppet resource, you can remove it the same way:

    puppet resource file_line workaround path=/etc/yum.conf line='exclude=puppet-agent' ensure=absent

  4. Run puppet agent -t to upgrade the puppet-agent package

  5. Run puppet agent -t again to ensure that everything runs cleanly.

Recovering if the puppeserver fails to restart

If, during the upgrade, the puppetserver service fails to start with the error:

Puppet::Error: Cannot determine basic system flavour

You should be able to recover by running the command:

puppet resource file /opt/puppetlabs/puppet/cache owner=puppet group=puppet

# The service should now restart
puppet resource service puppetserver ensure=running

After you have recovered in this manner, it is safe to proceed re-running the step you had been attempting when the puppetserver failed.

4.9.2.1.2. Update Kickstart Files

The example kickstart file (pupclient_86_64.cfg—see Setting up Kickstart) was updated. Existing kickstart files that are modeled on older versions of this file should backport these changes. New, OS-specific versions of this file are included in the SIMP-6.2.0 ISOs, but may also be found in the simp-core repository under build/distributions/<OS>/<OS Major Version>/<Architecture>/DVD/ks/.

Important

Careful examination of pupclient_86_64.cfg is recommended, if you have customized it for your site beyond token replacement. The latest version contains a few subtle bug fixes, such as fixing an incorrect path that would prevent UEFI systems from booting.

Two major changes were made:

  1. pupclient_86_64.cfg was updated to include instructions regarding what to change in order to accommodate UEFI boot. The associated configuration lines are commented out by default.

  2. pupclient_86_64.cfg was updated to download and use new bootstrap service files. These files, managed by simp::server::kickstart, are:

    • A systemd unit file for CentOS 7 (simp_client_bootstrap.service) or a systemv init script for CentOS 6 (simp_client_bootstrap).
    • A common bootstrap script (bootstrap_simp_client) used by both.

    This pair of files replaces the deprecated runpuppet script. They are required in order to solve two timeout problems on particularly loaded systems, both of which can cause client Puppet bootstrapping to fail and require subsequent manual intervention in order to fix:

    • On CentOS 7, systemd was killing runpuppet, when it ran longer than 5 minutes. This was solved by using an actual systemd unit file (simp_client_bootstrap.service) with a default timeout of 30 minutes.
    • On CentOS 7, if the DHCP lease expired in the middle of bootstrapping, the generated Puppet configuration would erroneously use localhost for the client hostname. This problem was solved by setting the static hostname of the client at the beginning of the client Puppet bootstrap process.

    In addition to solving these specific problems, the new bootstrap scripts use a configurable backoff algorithm in order prevent flooding a heavily-loaded Puppet master with requests. See the simp::server::kickstart::simp_client_bootstrap class for details.

4.9.2.1.3. Update dhcpd.conf

Changes were added to the dhcpd.conf file that enable the DHCP server to determine what mode, BIOS or UEFI, a system is kickstarting in and then to set the appropriate boot loader file on the TFTP server.

On a SIMP server, the example dhcpd.conf file is installed in /var/simp/environments/simp/RedHat/Global/dhcpd/dhcpd.conf via the simp-rsync package. This file may also be found in the simp-rsync-skeleton repository under environments/simp/rsync/RedHat/Global/dhcpd.

Note

When the simp-rsync RPM is upgraded, a message may be displayed that indicates

warning: /var/simp/environments/simp/rsync/RedHat/Global/dhcpd/dhcpd.conf\
created as /var/simp/environments/simp/rsync/RedHat/Global/dhcpd/\
dhcpd.conf.rpmnew

Because the contents of the /var/simp/environments/simp/rsync/ directory are pushed to remote systems, files ending with .rpmnew are deleted from the directory as part of the RPM deployment.

The commands below can be used to extract the new config file from the RPM, which can then be merged with the existing dhcpd.conf file.

  1. Run $ rpm2cpio simp-rsync-6.2.1-0.el7.noarch.rpm |cpio -ir "*dhcpd.conf"

    • rpm2cpio converts the RPM file to cpio archive. In this example, the command is being run from the directory containing the RPM file; if the file is not in the current working directory, the full path to the file must be provided.
    • cpio extracts files from the cpio archive. In this case, the cpio options -i, -r, and “*dhcpd.conf” direct cpio to extract one or more files from the archive, interactively rename the file, and the path and filename of files to be extracted should end with dhcpd.conf.

  2. When prompted rename ./var/simp/environments/simp/rsync/RedHat/Global/ dhcpd/dhcpd.conf ->, enter the desired path and name of the file to be extracted, such as /tmp/dhcpd.conf.rpmnew. Paths can be either absolute or relative, depending whether they start with a / or ./.

4.9.2.1.4. Update the TFTP Root Directory

The default TFTP root directory was changed to /var/lib/tftpboot to conform to DISA STIG standards. To continue using /tftpboot set tftpboot::tftpboot_root_dir in Hiera to /tftpboot. Alternately, to use the new directory, copy any files not managed by the rsync module (i.e., not stored in /var/simp/environments/<environment>/rsync/<OS>/Global/tftpboot) to the new directory. Make sure the permissions, including selinux context, are correct. TFTP boot will fail to find boot files that have the incorrect selinux context.

4.9.2.1.5. Optionally Remove OBE Logrotate Configuration Files

SIMP-managed logrotate rules are now in /etc/logrotate.simp.d instead of /etc/logrotate.d. The rules in /etc/logrotate.d are still applied, but logrotate is configured to read the rules in /etc/logrotate.simp.d, first.

This change was made to ensure SIMP-managed rules take precedence over vendor-supplied rules, because, when there are multiple rules specified for the same file, only the first rule is applied. Any subsequent rules are discarded. In fact, for some versions of logrotate, a rule with a duplicate log file is discarded in its entirety, even if only one of the managed log files is a duplicate. This means the remaining log files specified in that discarded rule will not be rotated!

Because the location of the SIMP-managed logrotate rules has changed, existing, but now OBE, SIMP rules will still reside in /etc/logrotate.d. Although these rules cause no issues with logrotate, they may be confusing to system administrators. So, you may wish to manually remove these rules.

You can easily identify OBE SIMP rules in /etc/logrotate.d by their This file managed by puppet. comment lines.

4.9.2.2. Upgrading from SIMP-6.0.0 to SIMP-6.1.0

Important

It is highly recommended that you read the information in this section in its entirety.

4.9.2.2.1. Upgrade Script

There were several issues found during the SIMP 6.0.0 to 6.1.0 upgrade that necessitated the creation of an upgrade script that is to be run on your SIMP Puppet masters.

Note

No changes are required on your clients for the upgrade to succeed.

The upgrade script, /usr/share/simp/upgrade_scripts/upgrade_simp_6.0.0_to_6.1.0, will assist with the upgrade from 6.0.0 to 6.1.0, taking into account all of the specific issues. This script is available in the simp-utils-6.1.0 package provided by SIMP 6.1.0 or the simp-utils repository.

As always, back up your system prior to upgrading!

Note

This script assumes that you are upgrading from the SIMP RPMs!

If you have chosen some other installation method, you will need to follow the general steps outlined in the script.

To perform the upgrade, with root permissions:

  1. Upgrade the simp-utils package, only, by executing yum update -y simp-utils.
  2. Run the script: /usr/share/simp/upgrade_scripts/upgrade_simp_6.0.0_to_6.1.0

Note

This script will:

  1. Run a yum -y update.
  2. Reinstall simp-gpgkeys and pupmod-simp-timezone due to RPM issues.
  3. Stop and uninstall the PostgresSQL 9.4 server to prevent postgresql upgrade issues.
  4. Restart the puppetserver process.
  5. Run puppet agent -t.
    • Some systems have shown issues with the postgresql upgrade during this step.

4.9.2.2.2. Update auth.conf

The legacy auth.conf, /etc/puppetlabs/puppet/auth.conf, has been deprecated. pupmod-simp-pupmod will back up the legacy auth.conf after the upgrade.

The Puppet master’s auth.conf is now managed by Puppet. You will need to reproduce any custom work done to legacy auth.conf via the new puppet_authorization::rule define. The stock rules are managed in pupmod::master::simp_auth.

4.9.2.2.3. Set up ClamAV DAT Files Updates

Given the wide spacing of SIMP releases, the team determined that it was ineffective for us to maintain the simp-rsync-clamav RPM with upstream ClamAV DAT file updates.

From this point forward, SIMP will not ship with updated ClamAV DAT files and we highly recommend updating your DAT files from the authoritative upstream sources. See the ClamAV Virus Database FAQ for instructions on how to automatically update these files.

4.9.2.2.4. Prepare System for PostgreSQL Upgrade

During the Puppet-managed upgrade, from PostgreSQL 9.4 to PostgreSQL 9.6, the PostgreSQL 9.4 data is not automatically imported into the 9.6 database. If for any reason you need to retain this data, which normally is quite transitory, see Upgrading a PostgreSQL Cluster for detailed instructions.