Description

OCP 4.16.5 on Azure,  Have also seen on AWS

Installed kataconfig with peer-pods-enabled

Updated worker nodes  to kata-containers-3.7.0-1.rhaos4.16.el9_4.x86_64.rpm

Launching kata-remote workloads fails

Actual result

The pods never get created

There are no errors in the controller-manger

The peerpodconfig-ctrl-caa-daemon has a log like:

2024/08/01 15:12:53 [adaptor/proxy]         mount_point:/run/kata-containers/3079be1aafcbeeb454280af1df5d11750a65fbfaaa1cbef05654c1f2e627d2d9/rootfs source:quay.io/openshift/origin-hello-openshift:latest fstype:overlay driver:image_guest_pull
2024/08/01 15:12:54 [adaptor/proxy] CreateContainer fails: rpc error: code = Internal desc = ttrpc client init failed

caa-daemon1.logcaa-daemon2.log

Impact

<How badly does this interfere with using the software?>

Env

<Where was the bug found, i.e. OCP build, operator build, kata-containers build, cluster infra, test case id>

Additional helpful info

<logs, screenshot, doc links, etc.>