Red Hat Satellite Best Practices: Avoiding Common Pitfalls

1. Satellite in One Page #

Red Hat Satellite manages the life cycle of Red Hat Enterprise Linux hosts. It syncs software from Red Hat, freezes it into tested versions, and delivers those versions to hosts. It also provisions new hosts, runs remote jobs, and reports on errata and compliance.

Satellite is a supported build of two open source projects. Foreman handles hosts, provisioning, and remote execution. Katello adds content management and subscriptions on top of Foreman. Every Satellite feature exists upstream, usually under a different name:

The pitfalls below fall into six groups: planning, content, hosts, operations, security, and mindset. The commands use the Satellite names. Upstream users can map most names with the table. Some procedures differ, though. Upgrades are the main example. Follow the Foreman documentation for those.

2. Planning and Installation #

The decisions made before the first satellite-installer run are the hardest to change later. Get these right and the rest of the deployment goes smoothly.

Size storage for Pulp from day one #

Pulp keeps downloaded packages under /var/lib/pulp. The directory grows as you add repositories and as new releases come out. A package is stored once, even when several repositories and content view versions use it. The total still adds up quickly. When the disk fills, syncs fail, tasks stall, and PostgreSQL can stop accepting writes.

Put /var/lib/pulp on a separate logical volume, formatted XFS, with room to grow. Do the same for /var/lib/pgsql. The installation guide lists baseline sizes for these directories. Start from those numbers, adjust for the repositories you plan to sync, and add headroom. Growing an LVM volume later takes minutes. Moving a full Pulp store takes a maintenance window.

Get DNS and the hostname right first #

Satellite issues certificates for its fully qualified domain name. Hosts, Capsules and API clients then trust that name. Forward and reverse lookups must agree before the installer runs, and hostname -f must print the final name.

A rename is possible with satellite-change-hostname. It reissues certificates and every Capsule and client must follow. Treat the name as permanent.

Give Satellite a server of its own #

The installer owns Apache, PostgreSQL, Pulp, Candlepin and their configuration. It rewrites their files on every run, including upgrades. A web application or database added to the same host gets overwritten, or it breaks the next upgrade. Run Satellite on a dedicated server or VM.

Plan Capsules early #

A Capsule is a Satellite proxy placed near the hosts it serves. It caches content, and it can serve DHCP, TFTP, DNS, and remote execution for its site. Content crosses the WAN once per site, not once per host.

Decide which sites and network segments get a Capsule before you register hosts. Moving thousands of registered hosts to a new content source later is real work. Since Satellite 6.17, a Capsule may run up to two minor versions behind its Satellite, which makes upgrades easier to stage.

3. Content Management #

Content is where Satellite earns its keep. It is also where most disk space and most confusion go.

Download on demand #

A repository has a download policy. Immediate downloads all packages at sync time. On demand downloads only the metadata. It fetches a package the first time a host asks for it. A large repository synced as immediate can use hundreds of gigabytes on packages that no host installs.

On demand is the right default for Red Hat repositories, which keep their older packages available. Two cases call for immediate:

• Upstreams that delete old packages. EPEL and some other third-party repositories remove older versions. Suppose a content view version lists a package that Satellite never downloaded. When the upstream deletes it, hosts can no longer get it.
• Disconnected sites, which cannot fetch a package on first request.

Capsules have their own download policy. A Capsule set to immediate makes an on demand Satellite fetch every package anyway, so set the two consistently.

Sync only what hosts use #

A repository on a sync plan and in content views is synced, published and promoted over and over. That costs time, database work and disk. Enabling every architecture, every minor release, and every debug and source repository “just in case” turns a small Satellite into a slow one. Enable what your hosts install from today. Add more when a host needs it.

Remove old content view versions #

Publishing a content view creates a new, frozen version. An old version stays until you delete it, along with its claim on the packages it lists. That is true even after no lifecycle environment uses it.

Delete unused versions as part of routine maintenance. hammer content-view purge removes old versions that no environment uses. It keeps the newest few, three by default:

$ hammer content-view purge --organization "Example" --name Server --versions-to-keep 2

Older Hammer releases call that option --count. Deleting a version does not free disk at once. Packages still used elsewhere stay, and Satellite removes the rest in its scheduled orphan cleanup.

Test what a filter actually does #

Content view filters include or exclude packages, errata, and module streams. They are powerful and easy to get wrong:

• An exclude filter on an RPM name also drops the errata that ship that package. A security fix can quietly disappear from a view. Package filters work on ordinary RPMs. For modular content, use module stream filters.
• An errata date filter uses the Issued On date or the Updated On date. Take an advisory issued before your cutoff and revised after it. With Issued On it is in the view. With Updated On it is out.
• Dependency solving changes the result. With it off, an update can arrive without a dependency it needs. With it on, Satellite can add back packages a filter excluded.
• Filters apply at publish time. Changing a filter changes nothing until the next publish.

After each publish, check the result. Compare package and errata counts between versions, and test an update on a host in the first environment before you promote.

Mind the order of composite content views #

A composite content view bundles other content views. Publishing the composite leaves its components alone. It uses the component versions you selected. Either publish components first, then the composite, or set each component to “Always use latest” and enable auto-publish on the composite.

Promote on a schedule #

Lifecycle environments let a change reach a few hosts first. Promoting a new version straight to Production skips that test. Set a rhythm, for example Development on patch day, QA a week later, Production after that, and keep to it. Emergency fixes can still move fast with an incremental update of the version already in Production.

4. Hosts and Registration #

Enable repositories on the activation key #

An activation key decides what a host gets when it registers. Red Hat repositories follow the subscription defaults. Custom repositories, such as third-party or internal software, are disabled on a key by default. A host registered with that key sees the repository but cannot install from it.

Open the key’s Repository Sets tab, select the repositories, and choose Override to Enabled. Then register a test host and run subscription-manager repos --list-enabled to confirm.

An activation key applies only at registration. Editing the key later leaves hosts that already registered as they were. Update their repository overrides on the host, or for many hosts at once with a bulk action on the host list.

Treat registration commands like passwords #

Hosts > Register Host generates a curl command with a bearer token in it. Anyone who has that command can register hosts until the token expires. Generating a new command does not cancel the old one. Keep the token lifetime short, and never paste the command into a ticket, chat, or wiki. If a command leaks, invalidate the user’s registration tokens in Satellite.

Disable the old repositories #

Registration adds the Satellite repositories. It does not remove the ones the host already had. A host that still pulls from a public mirror or an old internal server gets updates nobody approved, and the content view stops meaning anything. Disable the old repository files during registration, or right after it.

Clean up retired and rebuilt hosts #

A host retired or rebuilt outside Satellite leaves a record behind. Stale records clutter reports, skew errata counts, and inflate subscription usage figures. Delete the host in Satellite when you retire it. Rebuilds through Satellite keep the record for you.

Never clone a registered host #

A cloned VM starts with its source’s identity. That includes the consumer certificate and UUID, the machine ID and the SSH host keys. The two hosts then overwrite each other’s facts and status in Satellite.

Build clones from a generalized template. virt-sysprep resets these identities. For a clone that already exists, run subscription-manager clean on it and register it again. Avoid subscription-manager unregister there. The clone has the source’s identity, so that command unregisters the source host.

The machine ID also sets a host’s IPv6 link-local addresses on RHEL 9 and later, and its DHCP client identity. Two hosts that share one clash on the network, whatever their MAC addresses.

We hit this in a GL650 class with no cloning at all. On a KVM guest, systemd fills an empty /etc/machine-id from the SMBIOS UUID at first boot. VMs with the same SMBIOS UUID get the same machine ID. Give each VM its own.

5. Operations #

Back up with satellite-maintain, and test the restore #

Copying /var/lib/pulp and a database dump by hand does not give a consistent backup. Use the tool built for it:

# satellite-maintain backup offline /var/backup

An offline backup stops the services for its duration. An online backup keeps most services running and takes longer. Store backups off the Satellite host. They contain certificates and keys, so encrypt them or keep them somewhere secure.

A restore needs a server with the original hostname and configuration, running the same minor version. Practice a restore once, on a spare VM on an isolated network. The restored copy has production’s identity and keys. Keep it away from production clients while the original runs. A backup you have never restored is a hope, not a plan.

Upgrade one release at a time #

A Satellite minor release ships database and Pulp migrations that must run in order. So Satellite upgrades one minor version at a time: 6.17 to 6.18, then 6.18 to 6.19. Every hop follows this routine:

1. Take a backup.
2. Enable the target release’s maintenance repository and run satellite-maintain self-upgrade.
3. Run satellite-maintain upgrade check and fix what it reports.
4. Run satellite-maintain upgrade run.
5. Sync the new Capsule, maintenance and client repositories. Then upgrade the Capsules as your schedule allows.

Read the upgrade guide for each hop. It lists the changes that need action.

Take the Red Hat Satellite Upgrade Checklist with you: every step above, plus what to check before and after each hop.

Change configuration through the installer #

The installer writes the configuration of every service it manages. A change made with an editor lasts until the next installer run, and an upgrade always runs the installer.

Make changes with satellite-installer options. They are saved in the answers file and reapplied every time. For a setting the installer has no option for, add it to /etc/foreman-installer/custom-hiera.yaml.

This rule also covers the operating system packages on the Satellite server. Satellite locks them against plain dnf updates. Update the server with satellite-maintain packages update. It reruns the installer afterward.

Watch the task queue #

Syncs, publishes, promotions, and remote jobs all run as tasks. A stuck or paused task can hold up everything queued behind it. When Satellite feels slow or an action never finishes, open the task list under Monitor first. Look for paused tasks and long-running ones, and read their errors before you restart anything.

Keep the manifest current #

The subscription manifest connects Satellite to the Red Hat Customer Portal. When it or its certificate expires, Red Hat content stops syncing. Refresh the manifest on a schedule, and put the expiry date in the team calendar.

6. Security and Access #

Retire the shared admin login #

The admin account can do everything, everywhere. When the whole team uses it, the audit log cannot tell who did what. Create named accounts, and connect Satellite to Red Hat Identity Management or another LDAP directory so accounts follow your existing joiner and leaver process.

Grant access with roles. A role is a set of filters. A filter grants permissions on one kind of resource, and a search can narrow it further. Limit users to the organizations and locations they manage.

Plan the certificate story early #

Every client and Capsule trusts the Satellite certificate authority. A host that does not trust it yet needs that CA before it registers. Deliver it through a channel the host already trusts: the base image, a kickstart, or your configuration management.

The Insecure registration option skips certificate checks for the first request. That request downloads the CA. The option is handy in a lab. On a hostile network, an attacker could answer that first request. They could hand the host a false CA, steal the registration token or plant an SSH key. Later checks would then trust the attacker. Keep the option out of production.

Track certificate expiry dates. When a certificate expires, connections that use it fail. Renew certificates through the installer. Replacing the CA takes a planned transition, because all clients must learn to trust the new one.

7. Lessons from the Lab Floor #

These come straight from our classrooms. Each one cost a student an afternoon.

A failed install can leave a spent enrollment password #

When Satellite provisions a host into Identity Management, the realm proxy gets a one-time password and the kickstart uses it. Satellite clears that password when the host reports a finished build, and the next build gets a fresh one. An install that enrolls the host but dies before it reports back leaves the spent password in place. The next install tries it again, and enrollment fails with SASL Bind failed. Identity Management keeps the first install’s SSH host key, so tools that trust host keys from Identity Management reject the new host. On the host, ipa-client-install --force-join with an administrator login enrolls it again and uploads its current keys.

Invisible characters break kickstart templates #

Provisioning templates turn into shell scripts. One trailing space after a here-document marker, or a stray line copied from a document footer, and the %pre or %post section fails. With --erroronfail, the whole install stops. Preview the rendered template in Satellite before you boot a host, and keep templates in version control rather than retyping them.

Truncated values fail far from where they were typed #

Satellite saves a host group parameter even when the value is cut short. One example is an installer image URL that lost its last path segment. The mistake shows up minutes later, when a network boot cannot find the installer. After you set a parameter, display it and read the value to the end.

8. The Mindset That Ties It Together #

Satellite is not a tool you install and forget. It is infrastructure, with a database, a message bus, a content store, and dozens of services. Treat it like any production system:

• Monitor it. Watch disk space, the task queue, and service health.
• Plan its capacity. Content and hosts grow, so storage must too.
• Change it deliberately. Use the installer, record the change, and test it before production.
• Practice recovery. Restore a backup before you need one.

Teams that run Satellite this way rarely fight it. Teams that do not tend to rebuild it.

Learn it hands-on #

Guru Labs teaches these practices in GL650: Red Hat Satellite v6 Administration (Foreman/Katello). Students build a Satellite from scratch and run it end to end. The labs cover content views and lifecycles, host registration, remote execution, Ansible, Identity Management integration, provisioning, image mode hosts, and Capsules. Every lab runs on real systems in a live classroom.