Troubleshooting and Debugging

This tutorial covers the different ways of debugging and troubleshooting the in-place upgrade process.

Debug logs

The leapp framework, as well as actors in the in-place upgrade repositories log important events during the upgrade. Debug logs are written to /var/log/leapp/leapp-preupgrade.log or /var/log/leapp/leapp-upgrade.log.

Note

Debug logs are not enabled by default, run leapp with the --debug option to enable debug logging.

Troubleshooting with leapp-inspector

The leapp-inspector tool is unofficial and unstable tool helping with debugging based on data stored inside the leapp.db audit database.

In addition to extensive logging, leapp writes a lot of information into it’s database at /var/lib/leapp/leapp.db, such as messages and their contents, executed actors, logs and messages produced by individual actors and more. To view data in the database you can use the leapp-inspector tool, see the docs for installation instructions.

Note

leapp-inspector does not have to be installed on the system being upgraded. You can copy leapp.db to another system and use leapp-inspector there. In such case specify path to leapp.db using the --db option.

Warning

The leapp-inspector tool is not official and it is unstable. However, we are using it on daily bases as it’s very useful for debugging. So we are sharing the information about the tool with you. Just be aware that everything around the tool can be changed without any warnings.

Usage

See leapp-inspector --help for all options, it’s well documented.

Inspect Specific Execution

By default, leapp-inspector shows information about the most recent leapp execution. To inspect a specific execution:

1. List the executions and find the ID of the desired execution.

   [root@-20250129142956 ~]$ leapp-inspector executions
   ######################################################################
                            Executions of Leapp
   ######################################################################
   Execution                            | Timestamp
   ------------------------------------ | ---------------------------
   58c2167d-d747-4a98-8660-af457e32711e | 2025-01-29T13:48:12.119698Z
   42340943-e06c-4ff7-a2e6-9601eb76bbe0 | 2025-01-29T15:10:10.892776Z
   

2. Add the --context <execution id> parameter in subsequent commands.

Debugging Code

The Python Debugger - pdb can be used for standard debugging.

1. Before running leapp insert the following line to set a breakpoint in the code:

   import pdb; pdb.set_trace()
   

2. Run leapp and wait for pdb prompt to appear.

3. Debug. See the pdb docs on how to use it.

4. To continue the upgrade use the continue (or c) command.

Inspecting the Target Userspace Container During an Upgrade

Sometimes it’s useful to interactively inspect the target userspace container during the upgrade, e.g. to ensure required files are present. The following steps describe how to enter the container:

1. Set a breakpoint at the desired point in the code as described in the previous section

2. Open another terminal session on the machine being upgraded

3. Run the upgrade in the first terminal

4. When the breakpoint is reached (pdb prompt is displayed), enter the container in the second terminal. As the container needs a lot of variables passed in, the easiest way to do this is to find the systemd-nspawn command that leapp used to enter the container in the debug output in the first terminal. In the following example the first line is the command:

   2025-01-29 13:49:24.691 DEBUG    PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['systemd-nspawn', '--register=no', '--quiet', '--keep-unit', '--capability=all', '-D', '/var/lib/leapp/scratch/mounts/root_/system_overlay', '--setenv=LEAPP_DEVEL_USE_PERSISTENT_PACKAGE_CACHE=1', '--setenv=LEAPP_NO_RHSM=1', '--setenv=LEAPP_UNSUPPORTED=1', '--setenv=LEAPP_HOSTNAME=leapp-20250129142956', '--setenv=LEAPP_EXPERIMENTAL=0', '--setenv=LEAPP_NO_RHSM_FACTS=1', '--setenv=LEAPP_TARGET_PRODUCT_CHANNEL=ga', '--setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.6', '--setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default', '--setenv=LEAPP_IPU_IN_PROGRESS=8to9', '--setenv=LEAPP_EXECUTION_ID=58c2167d-d747-4a98-8660-af457e32711e', '--setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools', '--setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files', '--setenv=LEAPP_DEVEL_DATABASE_SYNC_OFF=1', 'dnf', 'install', '-y', '--setopt=module_platform_id=platform:el9', '--setopt=keepcache=1', '--releasever', '9.6', '--installroot', '/el9target', '--disablerepo', '*', '--enablerepo', 'BASEOS', '--enablerepo', 'APPSTREAM', 'util-linux', 'kpatch-dnf', 'dnf', 'dnf-command(config-manager)', '-v', '--disableplugin', 'subscription-manager']
   2025-01-29 13:49:24.692 DEBUG    PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['umount', '-fl', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
   2025-01-29 13:49:24.703 DEBUG    PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['umount', '-fl', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
   2025-01-29 13:49:24.704 DEBUG    PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has started: ['rm', '-rf', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
   2025-01-29 13:49:24.709 DEBUG    PID: 40673 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['rm', '-rf', '/var/lib/leapp/scratch/mounts/root_/system_overlay/el9target']
   > /etc/leapp/repos.d/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py(569)_copy_certificates()
   -> files_owned_by_rpms = _get_files_owned_by_rpms(target_context, '/etc/pki', recursive=True)
   (Pdb)
   

   Command will contain another shell command at the end (dnf install ... in this example) that needs to be stripped. Then run the command in the second terminal to enter and inspect the container:

   # WARNING: DO NOT BLINDLY COPY THIS COMMAND, THE VARIABLE VALUES ARE UNIQUE FOR AN UPGRADE
   systemd-nspawn --register=no --quiet --keep-unit --capability=all -D /var/lib/leapp/scratch/mounts/root_/system_overlay --setenv=LEAPP_DEVEL_USE_PERSISTENT_PACKAGE_CACHE=1 --setenv=LEAPP_NO_RHSM=1 --setenv=LEAPP_UNSUPPORTED=1 --setenv=LEAPP_HOSTNAME=leapp-20250129142956 --setenv=LEAPP_EXPERIMENTAL=0 --setenv=LEAPP_NO_RHSM_FACTS=1 --setenv=LEAPP_TARGET_PRODUCT_CHANNEL=ga --setenv=LEAPP_UPGRADE_PATH_TARGET_RELEASE=9.6 --setenv=LEAPP_UPGRADE_PATH_FLAVOUR=default --setenv=LEAPP_IPU_IN_PROGRESS=8to9 --setenv=LEAPP_EXECUTION_ID=58c2167d-d747-4a98-8660-af457e32711e --setenv=LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/common/tools:/etc/leapp/repos.d/system_upgrade/el8toel9/tools --setenv=LEAPP_COMMON_FILES=:/etc/leapp/repos.d/system_upgrade/common/files:/etc/leapp/repos.d/system_upgrade/el8toel9/files --setenv=LEAPP_DEVEL_DATABASE_SYNC_OFF=1
   

5. To continue the upgrade, exit the container and resume the debugger (continue or c).

Debugging Inside Initramfs

One of the biggest debugging challenges is exploring something in initramfs stage, as currently there is no network connectivity.

Note

When working in initramfs stage you will need a serial console. There is not network connectivity, SSH does not work.

1. If not already in the dracut emergency console, such as in case of an error, a breakpoint can be set to force leapp to enter the console. The following breakpoints are available:

   Breakpoint	Description
   leapp-pre-upgrade	Break right before running leapp in initramfs
   leapp-upgrade	Break right after LEAPP upgrade, before post-upgrade leapp run
   leapp-finish	Break after LEAPP save_journal (upgrade initramfs end)

   The breakpoint has to be passed using the rd.upgrade.break= argument on the kernel command line. There are (at least) 2 ways to do this:

   • Modify the cmdline in GRUB, press Ctrl-X when the Upgrade initramfs is selected and add the argument at the end of the line starting with LINUX=

   • Modify the addupgradebootentry actor to add the argument

2. Once in the emergency shell, you may want to use some of the utilities provided in $CWD/leapp_debug_tools.sh (should be automatically sourced):

   • leapp_dbg_mount - mounts /sysroot as read/write

   • leapp_dbg_source - the set of commands available in the emergency shell is very limited, this sets PATH and LD_LIBRARY_PATH to be able to run commands from /sysroot

   • leapp_dbg_chroot - change root into /sysroot

3. Inspect/modify /sysroot as needed

4. To continue the upgrade, exit the shell