The peadm::upgrade plan can time out when applied to PE XL installs that need to migrate PuppetDB reports [SOLARCH-548]

[Sharpie]

Describe the Bug

When upgrading a PE XL install older than 2019.7.0 the peadm::upgrade plan has to migrate PuppetDB report data on both the primary and replica nodes. The second migration runs under the control of puppet infrastructure upgrade replica which assumes that the reports and resource_events tables have been emptied and that a copy of the freshly migrated data can be obtained from the primary node using pg_basebackup. These assumptions are not met by peadm:upgrade:

• As a result of PR Fix upgrade peconf #129, the upgrade replica step that empties the reporting tables is turned into a no-op. This results in the the upgrade process on the replica hitting timeouts when pe-puppetdb is started with fully loaded tables:
  • The service start occurs during a puppet agent run, which now has a default runtimeout = 1h in Puppet 6.
  • The service start timeout for pe-puppetdb has been lowered back to 900 minutes under the assumption that long-running migrations are done by the puppetdb upgrade command.
• Running pg_basebackup on the replica node and pointing it at the primary node does not have the intended result as the migrated PuppetDB database lives on the primary puppetdb postgres node and needs to be copied to the replica puppetdb postgres node.

Expected Behavior

peadm::upgrade is able to upgrade installations older than PE 2019.7.0 without timing out.

Steps to Reproduce

Steps to reproduce the behavior:

1. Install PE 2018.1 in an XL configuration.
2. Run peadm::convert on the installation.
3. Add a reporting history for about 10,000 nodes generating about 30 million events to the installation.
4. Use peadm::upgrade to upgrade the installation to the latest 2019.8 version.

Environment

• Version: PE 2018.1 or newer.
• Platform: Any OS supported as a primary master platform by PE 2019.8.