7.3.2.6.1. Pre-Release Checklist¶
The bulk of the work to release both EL 7 and EL 8 versions of a SIMP ISO is to verify that each ISO is ready for release. Below is the list of verifications that must be executed for each ISO, before proceeding with the release of that ISO. If any of these steps fail, the problem identified must be fixed before you can proceed with the tag and release steps.
7.3.2.6.1.1. Update Policy Evaluation Response Reports¶
Since one of the main goals of SIMP is to assist with compliance of various standards, we should add a response to the latest security scans that we use.
Given that most scanners are only one view on the world and often are not flexible enough to meet all possible solutions to a given policy, it is expected that there will be explanations of both false positives as well as helpful material on why the SIMP framework is compliant for the benefit of our users.
These scans should be added, as applicable, to the Evaluation Artifacts section of the documentation.
7.3.2.6.1.2. Verify RPMs are available on the Download Server¶
This check is to verify that all artifacts used to create the ISO exist as signed RPMs in the appropriate location under https://download.simp-project.com. This will include:
SIMP-owned Puppet modules
Other Puppet modules
SIMP utility RPMs (
rubygem-simp-cli
,simp-adapter
,simp-utils
, etc.)simp-doc
SIMP application RPMs
External vendor application RPMs
OS RPMs
For nearly all the projects listed in Puppetfile.tracking
, you can verify that
the RPMs for those projects exist by executing the pkg:check_published
Rake command:
Checkout the
simp-core
project.git clone https://github.com/simp/simp-core.git cd simp-core
Verify the
Puppetfile.tracking
file contains the component tags for the release.Execute the
pkg:check_published
Rake commandbundle exec rake pkg:check_published > check_published.out
Examine the
check_published.out
content to verify that, except for thesimp-doc
project, no projects listsRPM Publish Required:
orGit Release Tag Required:
. What you should see are lines such as:... Found Existing Remote RPM: pupmod-simp-stunnel-6.1.0-0.noarch.rpm Found Existing Remote RPM: pupmod-simp-sudo-5.0.3-0.noarch.rpm ...
Important
If you see a message like
Warning: Unable to generate build-specific YUM cache
, your results are invalid, as connection to the specified repo under https://download.simp-project.com failed.
#. Manually verify the appropriate simp-doc
RPM exists in the appropriate
location under https://download.simp-project.com.
7.3.2.6.1.2.1. For the external vendor RPMs¶
Upload all vendor RPMs to the proper vendor repository in the relevant version repository under https://download.simp-project.com. Any existing RPMs will not be overwritten.
package_cloud push simp-project/VERSION_Dependencies/el/OS_MAJOR_VERSION /path/to/packages
Warning
DO NOT push any Core Operating System RPMs up to the Download Server, those should be retrieved from official vendor sources.
7.3.2.6.1.3. Verify a valid Puppetfile exists¶
This check is to verify that that Puppetfile.tracking
file for the
simp-core
project is complete and accurate:
It includes all the SIMP-owned Puppet modules, other Puppet modules that are dependencies of SIMP-owned Puppet modules, and utilities to configure the SIMP system when installed from ISO.
The URL for each artifact corresponds to the tag for its signed, published RPM.
7.3.2.6.1.4. Verify the Changelog.rst¶
This check is to verify that the simp-core
Changelog.rst has
been updated:
Manually inspect
7.3.2.6.1.5. Verify the dependencies.yaml¶
This check is to verify that simp-core/build/rpm/dependencies.yaml
contains the correct adjustments to the RPM dependencies, obsoletes,
requires, and/or release fields for any of the components listed
in the Puppetfile.tracking
file.
Manually inspect the file to verify there are entries for
All non-SIMP Puppet modules that have more dependencies listed in their
metadata.json
files than are actually required on a SIMP system. Each entry must list all the relevant dependencies in a:requires
element.Any component that has changed name (e.g.
pupmod-saz-timezone
changing topupmod-simp-timezone
). Each entry must list the package and version obsoleted in an:obsoletes
element.Any component for which for which the RPM release field must be specified (e.g. a component with a RPM-packaging-only change). Each entry must list a
:requires
element.
7.3.2.6.1.6. Verify the simp-core RPMs can be created¶
This check verifies that an RPM can be generated for simp-core
:
git clone https://github.com/simp/simp-core.git
cd simp-core/src/assets/simp
bundle update
bundle exec rake pkg:rpm
Note
This command will build the RPM for the OS of the server on which it was executed.
7.3.2.6.1.7. Verify simp-core tests pass¶
This check verifies that the simp-core
unit and acceptance test
have succeeded.
To verify that the simp-core
unit tests have succeeded, examine
the test results in GitHub Actions.
Navigate to the project’s GitHub Actions results page and verify the tests for the development branch to be tagged and released have passed. For our project, this page is https://github.com/simp/simp-core/actions
Important
If the tests in GitHub Actions fail, you must fix them before proceeding. The automated release procedures will only succeed, if the tests succeed.
To verify that the simp-core
acceptance tests have succeeded
Checkout the
simp-core
project for the last SIMP release.git clone https://github.com/simp/simp-core.git cd simp-core
Run the default
simp-core
acceptance testsbundle update bundle exec rake beaker:suites
Note
If the GitLab instance for simp-core
is current (it is sync’d
every 3 hours), you can look at the latest acceptance test results
run by GitLab, instead. The results will be at
https://gitlab.com/simp/simp-core/-/pipelines.
7.3.2.6.1.8. Verify ISOs can be created¶
This check verifies that SIMP ISOs for CentOS 7 can be
built from the local simp-core
clone and RPMs pushed to the SIMP
Download Server.
For CentOS 7:
Login to a machine that has Docker installed and the
docker
service running.Important
In our development environment, the version of Docker that is available with CentOS works best.
Checkout the
simp-core
project for the last SIMP release.git clone https://github.com/simp/simp-core.git cd simp-core
Populate
simp-core/ISO
directory with CentOS distribution ISOsmkdir ISO <curl the latest EL8 and EL7 ISOs>
Build each ISO for supported CentOS versions. For example,
bundle update SIMP_BUILD_docs=no \ SIMP_BUILD_verbose=yes \ SIMP_PKG_verbose=yes \ bundle exec rake beaker:suites[rpm_docker]
Important
The most reliable way to build each ISO is from a clean checkout of
simp-core
.
Verify none of the RPMs in the ISO that SIMP would have generated are signed by the SIMP development GPG key. For example, for a CentOS 7 build:
cd build/distributions/CentOS/7/x86_64/SIMP/RPMS/noarch # The 7da6f216 key ID may change as the SIMP signing keys get updated over time # The output of this command should be *EMPTY* rpm -q --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SIGPGP:pgpsig} %{SIGGPG:pgpsig}\n' -p * | grep -v 7da6f216
7.3.2.6.1.9. Verify SIMP ISO boot options work¶
This hefty check verifies that a server booted from the SIMP ISO can be bootstrapped for the ‘simp’ scenario and following boot options:
Using default boot option
Using disk encryption boot option
Using FIPS disabled boot option
Using disk encryption and FIPS disabled boot options
Using simp-prompt option
Using simp-prompt and disk encryption boot options
Using simp-prompt and FIPS disabled boot options
Using simp-prompt, disk encryption, and FIPS disabled boot options
Using linux-min boot option
Using linux-min and disk encryption boot options
Using linux-min and FIPS disabled boot options
Using linux-min, disk encryption, and FIPS disabled boot options
For the default boot options with/without encryption and the FIPS disabled boot option with/without encryption test cases, the simp-packer project is the easiest way to verify a SIMP VM can be booted from the ISO and bootstrapped. Otherwise, the check has to be done manually:
Boot a VM with the SIMP ISO
Select the appropriate boot options
Once the server boots, login to the server as root
Bootstrap the system
simp config simp bootstrap reboot
Login to the server as root and run
puppet agent -t
until the results are stableVerify the server is/is not in FIPS mode by inspecting /proc/sys/crypto/fips_enabled
Verify the appropriate disk is/is not encrypted by executing
blkid
Verify the appropriate disk partitioning
lsblk
Important
For the linux-min
test cases, the only verification required is
verification that the server boots up.
7.3.2.6.1.10. Verify component interoperability¶
This check verifies, with simp-core
and pupmod-simp-simp
acceptance tests, that this aggregation of module versions interoperate.
(These tests provide extensive, cross-component, integration tests.)
Note
If simp-core
and pupmod-simp-simp
acceptance tests have
effectively already passed on one of our continuous integration
platforms (e.g., in GitLab), you can skip this painful step.
However, you must be sure that the tests were run with the correct
component versions.
Checkout the
simp-core
project.git clone https://github.com/simp/simp-core.git cd simp-core
Verify the
Puppetfile.tracking
file contains the component tags for the release.Run the default
simp-core
acceptance testsbundle update bundle exec rake beaker:suites
Checkout the version of
pupmod-simp-simp
corresponding to thissimp-core
versionbundle exec rake deps:checkout cd src/puppet/modules/pupmod-simp-simp
Create a
.fixtures.yml
file that sets the version of each dependency to the version contained in thePuppetfile.tracking
file for this ISO release.Run all the functioning acceptance tests with and without FIPS mode enabled
bundle update BEAKER_fips=yes bundle exec rake beaker:suites bundle exec rake beaker:suites BEAKER_fips=yes bundle exec rake beaker:suites[base_apps] bundle exec rake beaker:suites[base_apps] BEAKER_fips=yes bundle exec rake beaker:suites[no_simp_server] bundle exec rake beaker:suites[no_simp_server] BEAKER_fips=yes bundle exec rake beaker:suites[scenario_one_shot] bundle exec rake beaker:suites[scenario_one_shot] BEAKER_fips=yes bundle exec rake beaker:suites[scenario_poss] bundle exec rake beaker:suites[scenario_poss] BEAKER_fips=yes bundle exec rake beaker:suites[scenario_remote_access] bundle exec rake beaker:suites[scenario_remote_access]
7.3.2.6.1.11. Verify otherwise untested capabilities¶
This check verifies that all other major capabilities (not otherwise tested in acceptance/simp-packer tests) do function as advertised:
Note
In order to speed time to market, the goal is to automate as many of these manual tests as possible!
A SIMP client can be PXE booted using the kickstart files from the SIMP ISO
A SIMP client can use the SIMP server for DNS
A ‘simp_lite’ client operates with a SIMP server
login operations (PAM, LDAP, local user)
NFS operations (home directory)
logging operations (rsyslog)
auditing operations
A ‘simp_poss’ client operates with a SIMP server
The SIMP server can be converted from FIPS enabled to FIPS disabled mode.
The SIMP server can be converted from Selinux enforcing to Selinux permissive.
The SIMP server can be converted from Selinux permissive to Selinux enforcing.
A local user with sudo privileges can be created and login to both the SIMP server and client on CentOS 7.
An LDAP user in the
administrators
group can login to both the SIMP server and client on CentOS 7.Local and LDAP users can change their passwords on both the SIMP server and client on CentOS 7.
The Rsyslog rules from
simp_rsyslog
,syslog
and SIMP application modules (aide
,tlog
, etc.) result in application log messages being written to the correct local and remote log files.Note
Although the
simp_rsyslog
andsyslog
modules have excellent acceptance tests, neither has a full-system test to verify integration with actual log producers. The tests for these modules uselogger
as a mock message sender.The compliance map reports for a full SIMP system are accurate.
No reports list non-compliant configuration that is really a parameter mismatches. (Parameter tested differs from parameter that should have been tested; value tested differs from actual values allowed, etc.)
SIMP server and SIMP client reports are generated.
simp-utils
executables that are not tested otherwise work as advertisedunpack_dvd
gen_ldap_update
updaterepos
The SIMP HOWTO Guides are still correct.
7.3.2.6.1.12. Verify SIMP server RPM install¶
This check verifies that CentOS 7 SIMP servers can be installed using the set
of RPMs contained in the SIMP ISOs The verification steps largely follow the
details in Installing SIMP From A Repository. All RPMs except the
simp-core
RPM should be able to be pulled from the appropriate location
under https://download.simp-project.com.
7.3.2.6.1.13. Verify SIMP server RPM upgrade¶
This check verifies that the set of RPMs in the SIMP ISO can upgrade the last full SIMP release.
Bring up a CentOS server that was booted from the appropriate SIMP ISO and for which
simp config
andsimp bootstrap
has been run.Note
If the VirtualBox for the last SIMP ISO was created by the simp-packer project, you can simply setup the appropriate VirtualBox network for that box and then bring up that bootstrapped image with
vagrant up
.Copy the SIMP and system RPMs packaged in the SIMP ISO to the server and install with yum.
FIXME Should put RPMs into appropriate updates repos, run something like the following
cd <updates dir> createrepo . chown -R root.apache ./* find . -type f -exec chmod 640 {} \; find . -type d -exec chmod 750 {} \; yum clean all; yum make cache yum update
Verify
puppet agent -t
runs cleanlyVerify no custom content is removed by the upgrade (e.g.,
environments/production/modules/site/manifests
, content inenvironments/production/data
)
7.3.2.6.1.14. Verify SIMP server R10K install¶
This check verifies that CentOS 7 SIMP servers can be installed via
r10k. Since this capability is already automatically tested in a
simp-core
acceptance test, all verification is handled by Verify simp-core
tests pass.