Loading...

Type: Bug
Resolution: Unresolved
Priority: Major
Fix Version/s: ACM 2.15.2, MCE 2.10.2
Affects Version/s: ACM 2.14.z, acm 2.13.z, acm 2.15.z
Component/s: Infrastructure Operator
Labels:
- qe-core

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Release Note Type:
Release Note Not Required
Release Note Status:
In Progress
Intelligence Requested:
Market:

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

PX Impact Score:

The issue raised its described under the below section:
```
apiVersion: v1
items:

apiVersion: agent-install.openshift.io/v1beta1
kind: InfraEnv
metadata:
annotations:
argocd.argoproj.io/sync-wave: "1"
argocd.argoproj.io/tracking-id: clusters:agent-install.openshift.io/InfraEnv:cwl/cwl
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"agent-install.openshift.io/v1beta1","kind":"InfraEnv","metadata":{"annotations":{"argocd.argoproj.io/sync-wave":"1","argocd.argoproj.io/tracking-id":"clusters:agent-install.openshift.io/InfraEnv:cwl/cwl","ran.openshift.io/ztp-gitops-generated":"{}"},"name":"cwl","namespace":"cwl"},"spec":{"additionalNTPSources":["192.168.120.11"],"clusterRef": {"name":"cwl","namespace":"cwl"}
,"nmStateConfigLabelSelector":{"matchLabels":{"nmstate-label":"cwl"}},"pullSecretRef":{"name":"disconnected-quay-registry-pull-secret"},"sshAuthorizedKey":"ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC5qMofrBHyrb+0AsDqteB8TIUXbyLnwnnNU+Aq2TAih7EtLBhvuUaUnwcUPPPrgYWBoYQO5GPzaXwYuIMQ+dHo8EvA2T2CUCHvYGUhXYXvP/aGHJ1ie1IgHZoT/8Ebmvtc+g+IH0/Nluc1ZthE7mpzLi0c3G6ik2v7Mk18bkZUJSOkK98JkxQnvSqs0OxS9fllVgveZ4RR+7CM6oiaO1Is9fh/230ajDJHs1+LUNdRsoaH0wAGuDtGZTFqw0ARTe5HJPwMRI7LbHIg5BDyS3ucOsTLV9piwGdK4E3MDb6deCWB00Y3FNSZBYLtHGdmnzxkcq22KrY6CDJ3ftzq9pC+3vxRZRqZooYLPEY1V+oMylLFFYJaOzvfhtbW5OCNf64UhGMSgDJG7plwAMFIJIwpSz2NaWNYSafp5qP/OqdmjAl4lviwYDqukXsi4XUQ1+Uupx9NWJOJiClvU5r+iUaixrs0K2ky1W/pjqSdvaJ7m2gaf2NgDpTKeBMm8XHUoM= root@jumphost-integration-nokia"}}
ran.openshift.io/ztp-gitops-generated: '{}'
creationTimestamp: "2026-01-28T10:51:44Z"
finalizers:
- infraenv.agent-install.openshift.io/ai-deprovision
generation: 1
name: cwl
namespace: cwl
resourceVersion: "311196253"
uid: 06a72c0a-e122-4bce-9ed0-ccaec7bbdb32
spec:
additionalNTPSources:
- 192.168.120.11
clusterRef:
name: cwl
namespace: cwl
cpuArchitecture: x86_64
ipxeScriptType: DiscoveryImageAlways
nmStateConfigLabelSelector:
matchLabels:
nmstate-label: cwl
pullSecretRef:
name: disconnected-quay-registry-pull-secret
sshAuthorizedKey: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDC5qMofrBHyrb+0AsDqteB8TIUXbyLnwnnNU+Aq2TAih7EtLBhvuUaUnwcUPPPrgYWBoYQO5GPzaXwYuIMQ+dHo8EvA2T2CUCHvYGUhXYXvP/aGHJ1ie1IgHZoT/8Ebmvtc+g+IH0/Nluc1ZthE7mpzLi0c3G6ik2v7Mk18bkZUJSOkK98JkxQnvSqs0OxS9fllVgveZ4RR+7CM6oiaO1Is9fh/230ajDJHs1+LUNdRsoaH0wAGuDtGZTFqw0ARTe5HJPwMRI7LbHIg5BDyS3ucOsTLV9piwGdK4E3MDb6deCWB00Y3FNSZBYLtHGdmnzxkcq22KrY6CDJ3ftzq9pC+3vxRZRqZooYLPEY1V+oMylLFFYJaOzvfhtbW5OCNf64UhGMSgDJG7plwAMFIJIwpSz2NaWNYSafp5qP/OqdmjAl4lviwYDqukXsi4XUQ1+Uupx9NWJOJiClvU5r+iUaixrs0K2ky1W/pjqSdvaJ7m2gaf2NgDpTKeBMm8XHUoM=
root@jumphost-integration-nokia
status:
agentLabelSelector:
matchLabels:
infraenvs.agent-install.openshift.io: cwl
bootArtifacts:
discoveryIgnitionURL: ""
initrd: ""
ipxeScript: ""
kernel: ""
rootfs: ""
conditions:
- lastTransitionTime: "2026-01-28T10:51:44Z"
message: "Failed to create image: 1 error occurred:\n\t* failed to validate
network yaml for host 0, nmstate GenerateConfiguration failed, error: failed
when generating the configuration {\"dns-resolver\":{\"config\":{\"search\":[\"cwl.integration.core.bos2.lab\"],\"server\":[\"192.168.120.11\"]}},\"interfaces\":[{\"ethernet\":
{\"auto-negotiation\":true,\"duplex\":\"full\",\"speed\":25000}
,\"ipv4\":{\"dhcp\":false,\"enabled\":false},\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9126,\"name\":\"ens15f0np0\",\"state\":\"up\",\"type\":\"ethernet\"},{\"ethernet\":
{\"auto-negotiation\":true,\"duplex\":\"full\",\"speed\":25000}
,\"ipv4\":{\"dhcp\":false,\"enabled\":false},\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9126,\"name\":\"ens15f1np1\",\"state\":\"up\",\"type\":\"ethernet\"},{\"ethernet\":
{\"auto-negotiation\":true,\"duplex\":\"full\",\"speed\":25000}
,\"ipv4\":{\"dhcp\":false,\"enabled\":false},\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9126,\"name\":\"ens1f0np0\",\"state\":\"up\",\"type\":\"ethernet\"},{\"ethernet\":
{\"auto-negotiation\":true,\"duplex\":\"full\",\"speed\":25000}
,\"ipv4\":{\"dhcp\":false,\"enabled\":false},\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9126,\"name\":\"ens1f1np1\",\"state\":\"up\",\"type\":\"ethernet\"},{\"ethernet\":
{\"auto-negotiation\":true,\"duplex\":\"full\",\"speed\":25000}
,\"ipv4\":{\"dhcp\":false,\"enabled\":false},\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9126,\"name\":\"ens3f0np0\",\"state\":\"up\",\"type\":\"ethernet\"},{\"ethernet\":
{\"auto-negotiation\":true,\"duplex\":\"full\",\"speed\":25000}
,\"ipv4\":{\"dhcp\":false,\"enabled\":false},\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9126,\"name\":\"ens3f1np1\",\"state\":\"up\",\"type\":\"ethernet\"},{\"ipv4\":
{\"dhcp\":false,\"enabled\":false}
,\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"link-aggregation\":{\"mode\":\"active-backup\",\"options\":
{\"miimon\":\"100\"}
,\"port\":[\"ens15f0np0\",\"ens15f1np1\"]},\"mtu\":9126,\"name\":\"infra-bond\",\"state\":\"up\",\"type\":\"bond\"},{\"ipv4\":

Unknown macro: {"address"}

,\"ipv6\":{\"dhcp\":false,\"enabled\":false},\"mtu\":9000,\"name\":\"infra-bond.121\",\"state\":\"up\",\"type\":\"vlan\",\"vlan\":{\"base-iface\":\"infra-bond\",\"id\":121}},{\"link-aggregation\":

Unknown macro: {"mode"}

,\"mtu\":9126,\"name\":\"tenant-bond1\",\"state\":\"up\",\"type\":\"bond\"},{\"link-aggregation\":

Unknown macro: {"mode"}

,\"mtu\":9126,\"name\":\"tenant-bond2\",\"state\":\"up\",\"type\":\"bond\"}],\"routes\":{\"config\":[
{\"advmss\":1024,\"destination\":\"0.0.0.0/0\",\"next-hop-address\":\"192.168.121.1\",\"next-hop-interface\":\"infra-bond.121\",\"table-id\":254}
]}}
with rc: 1, err_msg: Error on converting string to rust NetworkState: InvalidArgument:
Invalid YAML string: routes.config[0]: unknown field `advmss`, expected one
of `state`, `destination`, `next-hop-interface`, `next-hop-address`, `metric`,
`table-id`, `weight`, `route-type`, `cwnd`, `source`, `initcwnd`, `initrwnd`,
`mtu`, `quickack` at line 1 column 2187, err_kind: InvalidArgument\n\n"
reason: ImageCreationError
status: "False"
type: ImageCreated
debugInfo:
eventsURL: ""
kind: List
metadata:
resourceVersion: ""
```

The issue gets encountered 100% when the environment its using the following software stack:
MCEv2.8.4 - RHACMv2.13.z - OCPv2.18.z
MCEv2.9.z - RHACMv2.14.z - OCPv2.19.z
MCEv2.10.z - RHACMv2.15.z - OCPv2.20.z

As far as it was empirically determined from the beginning the MCEv2.8.3 was a version for which this the deployment was working as expected.

The validation workflow of the NMstate configuration its the following:

```
InfraEnv YAML (with advmss)
↓
nmstate.GenerateConfiguration() (Go)
↓
C.nmstate_generate_configurations() (CGo)
↓
rust/src/clib/gen_conf.rs: nmstate_generate_configurations()
↓
rust/src/clib/state.rs: c_str_to_net_state() ← ERROR MESSAGE CONSTRUCTED HERE (line 29)
↓
rust/src/lib/net_state.rs: NetworkState::new_from_yaml()
↓
serde_yaml::from_str() ← SERDE ERROR GENERATED HERE
↓
Deserializes RouteEntry
↓
serde encounters "advmss" → unknown field error
```

Based on the workflow presented above, the validation its passing if the `advmss` parameter exists under the [route.rs](https://github.com/nmstate/nmstate/blob/v2.2.48/rust/src/lib/route.rs#L279) RouteEntry structure definition exists. This parameter has been added for the first time under the nmstatev2.2.46 [route.rs](https://github.com/nmstate/nmstate/blob/6d01ecf6c3823a5b380c8971ad01d9f6ac15b1ba/rust/src/lib/route.rs#L279) .

By installing the MCEv2.8.3:
```
multicluster-engine multicluster-engine.v2.8.3 multicluster engine for Kubernetes 2.8.3 multicluster-engine.v2.8.2 Succeeded
multicluster-engine
```

and checking the `assisted-service` pod under `multicluster-engine` namespace:
```
[root@nokia-blueprint-jumphost acm]# oc exec -n multicluster-engine assisted-service-57494cc77b-2r4rq – ls -la /usr/lib64/ | grep nmst
Defaulted container "assisted-service" out of: assisted-service, postgres
lrwxrwxrwx. 1 root root 20 Jul 24 2025 libnmstate.so -> libnmstate.so.2.2.48
lrwxrwxrwx. 1 root root 20 Jul 24 2025 libnmstate.so.2 -> libnmstate.so.2.2.48
-rwxr-xr-x. 1 root root 10259896 Jul 24 2025 libnmstate.so.2.2.48
```

By installing the MCEv2.8.4:

```
midu@midu-thinkpadp16vgen1:~/NGC-499$ oc get csv -n multicluster-engine
NAME DISPLAY VERSION REPLACES PHASE
multicluster-engine.v2.8.4 multicluster engine for Kubernetes 2.8.4 multicluster-engine.v2.8.3 Succeeded
openshift-gitops-operator.v1.19.1 Red Hat OpenShift GitOps 1.19.1 openshift-gitops-operator.v1.19.0 Succeeded
topology-aware-lifecycle-manager.v4.18.2 Topology Aware Lifecycle Manager 4.18.2 topology-aware-lifecycle-manager.v4.18.1 Succeeded

```

and checking the `assisted-service` pod under `multicluster-engine` namespace:
```
midu@midu-thinkpadp16vgen1:~/NGC-499$ oc exec -n multicluster-engine assisted-service-9bfc96db4-gbmlh – ls -la /usr/lib64/ | grep nmst
Defaulted container "assisted-service" out of: assisted-service, postgres
lrwxrwxrwx. 1 root root 20 May 28 2025 libnmstate.so -> libnmstate.so.2.2.45
lrwxrwxrwx. 1 root root 20 May 28 2025 libnmstate.so.2 -> libnmstate.so.2.2.45
-rwxr-xr-x. 1 root root 10097368 May 28 2025 libnmstate.so.2.2.45
```

By installing the MCEv2.10.1:
```
midu@midu-thinkpadp16vgen1:~/NGC-499$ oc get csv -n multicluster-engine
NAME DISPLAY VERSION REPLACES PHASE
multicluster-engine.v2.10.1 multicluster engine for Kubernetes 2.10.1 multicluster-engine.v2.10.0 Succeeded
```

and checking the `assisted-service` pod under `multicluster-engine` namespace:
```
midu@midu-thinkpadp16vgen1:~/NGC-499$ oc exec -n multicluster-engine assisted-service-5c6bf66bbb-tdtzf – ls -la /usr/lib64/libnmstate | grep nmst
Defaulted container "assisted-service" out of: assisted-service, postgres
-rwxr-xr-x. 1 root root 10097368 May 28 2025 /usr/lib64/libnmstate.so.2.2.45
```

The initial issue reported on MCEv2.8.4:

![[Pasted image 20260128200832.png]]
Its replicate on the MCEv2.10.1.

The demo config used its the following:

```
interfaces:
- name: enp1s0
type: ethernet
state: up
ipv4:
address:
- ip: 172.16.30.30
prefix-length: 24
enabled: true
dns-resolver:
config:
server:
- 172.16.30.1
routes:
config:
- destination: 0.0.0.0/0
next-hop-address: 172.16.30.1
next-hop-interface: enp1s0
advmss: 10
```

Looking into the [assisted-service](https://github.com/openshift/assisted-service/blob/c313528fddb88cb401226ac1fdb149aff06b9461/go.mod#L49) project, the go.mod nmstate:
```
github.com/nmstate/nmstate/rust/src/go/nmstate v0.0.0-20220811151154-801022633c42
```

points to a even older nmstate version 1.3.3, where the [router.rs](https://github.com/nmstate/nmstate/blob/v1.3.3/rust/src/lib/route.rs) doesnt include the advmss .

There seems to be a regression within the `assisted-service` libnmstate.so library, between the more recent MCE version and the MCEv2.8.3 .

The open questions we need to sort are:
What its the build workflow of `assisted-service` , hence multicluster-operator?
Was the library been overwritten by a different layer within the same OCI ?
-

is cloned by

ACM-29346 [ACM-2.13] InfraEnv image creation failure due to libnmstate regression (advmss)

Testing

ACM-29345 [ACM-2.14] InfraEnv image creation failure due to libnmstate regression (advmss)

Resolved

links to

openshift/assisted-service#8811: ACM-29344: Bump RPM versions, specifically nmstate-lib, to get advmss support in the updated nmstate version

openshift/assisted-service#8813: OCPBUGS-74634: Bump RPM versions, specifically nmstate-lib, to get advmss support in the updated nmstate version

openshift/assisted-service#8815: OCPBUGS-74634: Bump RPM versions, specifically nmstate-lib, to get advmss support in the updated nmstate version

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates