5.2.13. HOWTO Work with the SIMP Rsync Shares

When we added support for multiple environments, the SIMP rsync space in /var/simp/environments/<environment>/rsync became quite complex.

This will guide you through the new rsync layout as well as providing guidance on setting up new rsync shares for your various components.

This is very SIMP-specific and does not preclude you from using rsync however you like. However, if you want multi-environment support, you will need to replicate something like what we have done for your custom directories.

Warning

All symlinks in the rsync space must be relative links. Do not use absolute paths as this may result in misconfiguration of client nodes.

5.2.13.1. Why SIMP Uses Rsync

Rsync support was introduced in SIMP in the early days due to the fact that the Puppet native file synchronization mechanisms were relatively horrible at syncing large files (too much in memory) and large numbers of files in a directory tree (too many resources and system load).

Rsync neatly solves both of these issues and is present on all SIMP systems by default.

By default, SIMP wraps all rsync connections in a Stunnel connection to provide encrypted connections. Additionally, SIMP adds randomly generated passwords to sensitive shares to prevent unauthorized connections.

You can restrict this as far as necessary in your environment but the defaults should suit most needs.

5.2.13.2. Standard Capabilities

The standard SIMP rsync shares for the production Puppet environment exist at /var/simp/environments/production/rsync. This is an assumed path and changing this path will break some aspects of the system.

Within this directory, you will find a set of files with the name .shares. This file is used by the fact simp_rsync_environments to indicate that all directories at the given location should be added as rsync shared directories.

The data structure in the simp_rsync_environments is based on the lower cased name of the containing directory. This means that there should NEVER be two directories with the same name at the same level of the directory hierarchy. In this case, the last one present alphabetically will win.

As a concrete example, given the following directory structure:

var
└── simp
    └── environments
        └── production
            └── rsync
                ├── Global
                │   ├── .shares
                │   └── clamav
                │       ├── main.cvd
                │       ├── daily.cld
                │       └── bytecode.cld
                ├── .rsync.facl
                ├── README
                └── RedHat
                    ├── Global
                    │   ├── freeradius
                    │   ├── .shares
                    │   ├── tftpboot
                    │   │   └── linux-install
                    │   │       └── README
                    │   ├── dhcpd
                    │   │   ├── dhcpd.conf
                    │   │   └── LICENSE
                    │   ├── snmp
                    │   │   ├── mibs
                    │   │   └── dlmod
                    │   └── apache
                    │       └── www
                    │           ├── cgi-bin
                    │           ├── error
                    │           │   └── include
                    │           ├── icons
                    │           │   └── small
                    │           └── html
                    ├── 6
                    │   ├── bind_dns
                    │   │   └── LICENSE
                    │   └── .shares
                    └── 7
                        ├── bind_dns
                        │   └── LICENSE
                        └── .shares

The following would be returned by the simp_rsync_environments fact:

{
  "production": {
    "id": "production",
    "rsync": {
      "id": "rsync",
      "global": {
        "id": "Global",
        "shares": [
          "clamav"
        ]
      },
      "redhat": {
        "id": "RedHat",
        "6": {
          "id": "6",
          "shares": [
            "bind_dns"
          ]
        },
        "7": {
          "id": "7",
          "shares": [
            "bind_dns"
          ]
        },
        "global": {
          "id": "Global",
          "shares": [
            "freeradius",
            "tftpboot",
            "dhcpd",
            "snmp",
            "apache"
          ]
        }
      }
    }
  }
}

Breaking this down, the following data is shown:

 {
  "downcased_directory_name": {
    "id": "Original_Directory_Name",
    "downcased_subdirectory_name": {
      "id": "Original_Subdirectory_Name",
      "shares": [
        "Directory One",
        "directory two"
      ]
    }
  }
}

Note

The presence of the .shares file in the directory tree tells the simp_rsync_environments fact that all directories at that level are to be exposed as shares in the returned data structure.

That said, it is up to your Puppet logic to actually expose them as such!

See the simp::server::rsync_shares class to see how we do this for the default rsync shares.

5.2.13.3. Supporting Additional Environments

Generally, in a SIMP environment, you are going to want to start with the directory structure that we have and simply copy the entire data structure to a directory with your custom name.

Warning

Be sure not to copy any sensitive information into the space!

For example, if you wanted to create the standard dev/test/prod structure, and the production environment already existed:

cd /var/simp/environments
cp -a production dev
cp -a production test

Alternatively, you can use simp environment new to affect the copy of all or some the SIMP Omni-Environment. (You can also use that command to affect links of the SIMP Secondary Environment or SIMP Writable Environment, which in some circumstances may be more appropriate.)

After this, you will now have an enhanced simp_rsync_environments data structure that holds all of the information for the dev, test, and production environments.

You can then manipulate the contents of the different environments to suit your needs.

Note

The contents of the various rsync directories are not under version control by default. While you may add them to a VCS of your choosing (SVN, Git, etc…), there may be some VERY large files present in these directories.

Make sure your system can handle the load before adding rsync content into a VCS!

5.2.13.4. Disabling Stunnel

If you decide to disable stunnel, you will need to specify your rsync server in Hiera, if it is not already specified.

Warning

If you disable stunnel, your data and any rsync access credentials will be passed in the clear!

---
simp_options::rsync::server: <rsync_server_fqdn>

Additionally, you will need to ensure your firewall is open on the rsync port. Include the following on the node acting as the rsync server.

class site::rsync_iptables (
  Simplib::Netlist $allow      = simplib::lookup('simp_options::trusted_nets'),
  Simplib::Port    $rsync_port = 873
){
  iptables::listen::tcp_stateful { "rsync_shares":
    trusted_nets => $allow,
    dports       => $rsync_port
  }
}