Description of problem:
The org.osbuild.rpm stage fails with the below error:
FileNotFoundError: [Errno 2] No such file or directory: '/run/osbuild/bin/org.osbuild.rpm': '/run/osbuild/bin/org.osbuild.rpm'

The execve() syscall returns an ENOENT, not because the file in question does not exist, but because the binary corresponding to shebang of the stage does not exist.

1. head -n1 /usr/lib/osbuild/stages/org.osbuild.rpm
   #!/usr/bin/python3

Version-Release number of selected component (if applicable):
osbuild-53-2.el8.noarch
osbuild-composer-46.3-1.el8_6.x86_64

How reproducible:
Always

Steps to Reproduce:
1. dnf remove python36
2. Keep the python36 module enabled or the composer will warn you of the missing module
3. Try to compose an image

Actual results:
Stage org.osbuild.rpm
Output:
[/usr/lib/tmpfiles.d/frr.conf:1] Unknown user 'frr'.
[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping
All rules containing unresolvable specifiers will be skipped.
[/usr/lib/tmpfiles.d/radvd.conf:1] Unknown user 'radvd'.
[/usr/lib/tmpfiles.d/rpcbind.conf:2] Unknown user 'rpc'.
Traceback (most recent call last):
File "/run/osbuild/lib/osbuild/api.py", line 190, in exception_handler
yield
File "/run/osbuild/lib/runners/org.osbuild.rhel86", line 79, in main
check=False)
File "/usr/lib64/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib64/python3.6/subprocess.py", line 729, in _init_
restore_signals, start_new_session)
File "/usr/lib64/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/run/osbuild/bin/org.osbuild.rpm': '/run/osbuild/bin/org.osbuild.rpm'

Expected results:
The stages are properly executed.

Additional info:

• The customer installed python39 and removed the python36 package but left the module of the same name enabled. Hence /etc/alternatives/python3 points to python3.9.

• I tried to workaround the issue by modifying the symlink done by the rhel86 runner (overwriting /usr/bin/python3 to point to /usr/libexec/platform-python3.6) but it cannot work as the content of /usr/bin has been RO bind-mounted. Reinstalling python36 is hence the only workaround to move forward.

• I think having a dependency against python36 or modifying the osbuild's stages shebangs (to use directly /usr/libexec/platform-python3.6) are possible tracks to fix the issue permanently.

• Find below an excerpt of the relevant calls of the worker process.

After the depsolve step and the org.osbuild.files stage, a bubblewrap container is executed:
9223 12:52:50.665090 execve("/usr/bin/bwrap", ["bwrap", <...> "--", "/run/osbuild/lib/runners/org.osbuild.rhel86", "/run/osbuild/bin/org.osbuild.rpm"], ["container=bwrap-osbuild", "LC_CTYPE=C.UTF-8", "PATH=/usr/sbin:/usr/bin", "PYTHONPATH=/run/osbuild/lib", "PYTHONUNBUFFERED=1", "TERM=dumb"]) = 0 <0.000735>

The org.osbuild.rpm python file is successfully bind-mounted into the container, and lstat() confirms a correct size (st_size=9468):
9224 12:52:50.694994 mkdir("/newroot/run/osbuild/bin", 0755) = 0 <0.000021>
9224 12:52:50.695082 creat("/newroot/run/osbuild/bin/org.osbuild.rpm", 0666) = 5</newroot/run/osbuild/bin/org.osbuild.rpm> <0.000021>
9224 12:52:50.695139 close(5</newroot/run/osbuild/bin/org.osbuild.rpm>) = 0 <0.000015>
9224 12:52:50.695182 mount("/oldroot/usr/lib/osbuild/stages/org.osbuild.rpm", "/newroot/run/osbuild/bin/org.osbuild.rpm", NULL, MS_BIND|MS_REC, NULL) = 0 <0.000018>
:
9224 12:52:50.695436 lstat("/newroot/run/osbuild/bin/org.osbuild.rpm",

{st_dev=makedev(0xfd, 0), st_ino=452213, st_mode=S_IFREG|0755, st_nlink=1, st_uid=0, st_gid=0, st_blksize=4096, st_blocks=24, st_size=9468, st_atime=1661446370 /* 2022-08-25T12:52:50.044685577-0400 */, st_atime_nsec=44685577, st_mtime=1648076732 /* 2022-03-23T19:05:32-0400 */, st_mtime_nsec=0, st_ctime=1661347883 /* 2022-08-24T09:31:23.265159724-0400 */, st_ctime_nsec=265159724}

) = 0 <0.000015>

Just after the pivot_root() to the container, the runner `org.osbuild.rhel86` is properly executed through its shebang (hence platform-python binary and all the python stack is there):
9224 12:52:50.705193 chdir("/newroot") = 0 <0.000014>
9224 12:52:50.705231 pivot_root(".", ".") = 0 <0.000050>
:
9227 12:52:50.710591 execve("/run/osbuild/lib/runners/org.osbuild.rhel86", ["/run/osbuild/lib/runners/org.osbuild.rhel86", "/run/osbuild/bin/org.osbuild.rpm"], ["container=bwrap-osbuild", "LC_CTYPE=C.UTF-8", "PATH=/usr/sbin:/usr/bin", "PYTHONPATH=/run/osbuild/lib", "PYTHONUNBUFFERED=1", "TERM=dumb", "PWD=/"]) = 0 <0.000115>
:
9227 12:52:51.106337 mkdir("/etc/alternatives", 0777) = 0 <0.000016>
9227 12:52:51.106388 symlink("/usr/bin/python3.6", "/etc/alternatives/python3") = 0 <0.000016>

But then the stage failed to get executed:
9242 12:52:51.107114 execve("/run/osbuild/bin/org.osbuild.rpm", ["/run/osbuild/bin/org.osbuild.rpm"], ["container=bwrap-osbuild", "LC_CTYPE=C.UTF-8", "PATH=/usr/sbin:/usr/bin", "PYTHONPATH=/run/osbuild/lib", "PYTHONUNBUFFERED=1", "TERM=dumb", "PWD=/"] <unfinished ...>
9242 12:52:51.107177 <... execve resumed>) = -1 ENOENT (No such file or directory) <0.000044>
:
9242 12:52:51.107317 exit_group(255 <unfinished ...>
9242 12:52:51.107337 <... exit_group resumed>) = ?
9242 12:52:51.107560 +++ exited with 255 +++