Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: None
Component/s: infra-operator
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
?
PM Approval:
?
QE Approval:
?
Regression:
None
Intelligence Requested:
Market:

Severity:
Moderate

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Deploying RHOSO beta on OCP 4.14.30 (dual stack cluster). After applying the OpenStackControlPlane CR, the memcached pod failed to start. Pod log shows the following error:

Invalid IPV6 address: "[10.131.0.91"failed to listen on TCP port 11212: Success

While the OCP cluster, is dual stack, the OpenStack configuration is being built with only IPv4 addresses.

[cwl@bastion Ali]$ oc -n openstack get pods -owide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES

memcached-0 0/1 CrashLoopBackOff 21 (3m40s ago) 86m 10.131.0.91 worker1.cwl-site2.npss.bos2.lab <none> <none>

openstack-cell1-galera-0 1/1 Running 0 86m 10.131.0.92 worker1.cwl-site2.npss.bos2.lab <none> <none>

openstack-cell1-galera-1 1/1 Running 0 86m 10.128.2.4 master1.cwl-site2.npss.bos2.lab <none> <none>

openstack-cell1-galera-2 1/1 Running 0 86m 10.128.0.6 master2.cwl-site2.npss.bos2.lab <none> <none>

openstack-galera-0 1/1 Running 0 86m 10.131.0.90 worker1.cwl-site2.npss.bos2.lab <none> <none>

openstack-galera-1 1/1 Running 0 86m 10.128.2.3 master1.cwl-site2.npss.bos2.lab <none> <none>

openstack-galera-2 1/1 Running 0 86m 10.128.0.5 master2.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-dlhxf 1/1 Running 0 85m 10.128.0.7 master2.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-hmd9h 1/1 Running 0 85m 10.128.2.6 master1.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-l9bgt 1/1 Running 0 85m 10.131.0.94 worker1.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-ovs-5lzf4 2/2 Running 0 85m 10.131.0.93 worker1.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-ovs-dn7wj 0/2 ContainerCreating 0 85m <none> worker2.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-ovs-j5q7f 2/2 Running 0 85m 10.129.0.6 master3.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-ovs-ql8ln 2/2 Running 0 85m 10.128.0.8 master2.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-ovs-tnjsp 2/2 Running 0 85m 10.128.2.8 master1.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-pjlrh 1/1 Running 0 85m 10.130.0.32 worker2.cwl-site2.npss.bos2.lab <none> <none>

ovn-controller-wzw44 1/1 Running 0 85m 10.129.0.5 master3.cwl-site2.npss.bos2.lab <none> <none>

ovn-northd-74b897b864-9jb2d 1/1 Running 0 76m 10.131.0.103 worker1.cwl-site2.npss.bos2.lab <none> <none>

ovsdbserver-nb-0 1/1 Running 0 85m 10.131.0.95 worker1.cwl-site2.npss.bos2.lab <none> <none>

ovsdbserver-nb-1 1/1 Running 0 85m 10.128.2.12 master1.cwl-site2.npss.bos2.lab <none> <none>

ovsdbserver-nb-2 1/1 Running 0 85m 10.128.0.16 master2.cwl-site2.npss.bos2.lab <none> <none>

ovsdbserver-sb-0 1/1 Running 0 85m 10.131.0.102 worker1.cwl-site2.npss.bos2.lab <none> <none>

rabbitmq-server-0 1/1 Running 0 86m 10.131.0.87 worker1.cwl-site2.npss.bos2.lab <none> <none>

rabbitmq-server-1 1/1 Running 0 86m 10.131.0.89 worker1.cwl-site2.npss.bos2.lab <none> <none>

rabbitmq-server-2 1/1 Running 0 86m 10.131.0.88 worker1.cwl-site2.npss.bos2.lab <none> <none>

[cwl@bastion Ali]$

[cwl@bastion Ali]$ oc -n openstack logs memcached-0

+ sudo -E kolla_set_configs

sudo: unable to send audit message: Operation not permitted

INFO:_main_:Loading config file at /var/lib/kolla/config_files/config.json

INFO:_main_:Validating config file

INFO:_main_:Kolla config strategy set to: COPY_ALWAYS

INFO:_main_:Copying service configuration files

INFO:_main_:Deleting /etc/sysconfig/memcached

INFO:_main_:Copying /var/lib/kolla/config_files/src/etc/sysconfig/memcached to /etc/sysconfig/memcached

INFO:_main_:Copying /var/lib/config-data/tls/private/memcached.key to /etc/pki/tls/private/memcached.key

INFO:_main_:Setting permission for /etc/pki/tls/private/memcached.key

INFO:_main_:Copying /var/lib/config-data/tls/certs/memcached.crt to /etc/pki/tls/certs/memcached.crt

INFO:_main_:Setting permission for /etc/pki/tls/certs/memcached.crt

INFO:_main_:Writing out command to execute

++ cat /run_command

+ CMD='bash -c $* – eval source /etc/sysconfig/memcached; exec /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS'

+ ARGS=

+ sudo kolla_copy_cacerts

sudo: unable to send audit message: Operation not permitted

+ [[ ! -n '' ]]

+ . kolla_extend_start

+ echo 'Running command: '\''bash -c $* – eval source /etc/sysconfig/memcached; exec /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS'\'''

Running command: 'bash -c $* – eval source /etc/sysconfig/memcached; exec /usr/bin/memcached -p ${PORT} -u ${USER} -m ${CACHESIZE} -c ${MAXCONN} $OPTIONS'

+ umask 0022

+ exec bash -c '$*' – eval source '/etc/sysconfig/memcached;' exec /usr/bin/memcached -p '${PORT}' -u '${USER}' -m '${CACHESIZE}' -c '${MAXCONN}' '$OPTIONS'

slab class 1: chunk size 96 perslab 10922

slab class 2: chunk size 120 perslab 8738

slab class 3: chunk size 152 perslab 6898

slab class 4: chunk size 192 perslab 5461

slab class 5: chunk size 240 perslab 4369

slab class 6: chunk size 304 perslab 3449

slab class 7: chunk size 384 perslab 2730

slab class 8: chunk size 480 perslab 2184

slab class 9: chunk size 600 perslab 1747

slab class 10: chunk size 752 perslab 1394

slab class 11: chunk size 944 perslab 1110

slab class 12: chunk size 1184 perslab 885

slab class 13: chunk size 1480 perslab 708

slab class 14: chunk size 1856 perslab 564

slab class 15: chunk size 2320 perslab 451

slab class 16: chunk size 2904 perslab 361

slab class 17: chunk size 3632 perslab 288

slab class 18: chunk size 4544 perslab 230

slab class 19: chunk size 5680 perslab 184

slab class 20: chunk size 7104 perslab 147

slab class 21: chunk size 8880 perslab 118

slab class 22: chunk size 11104 perslab 94

slab class 23: chunk size 13880 perslab 75

slab class 24: chunk size 17352 perslab 60

slab class 25: chunk size 21696 perslab 48

slab class 26: chunk size 27120 perslab 38

slab class 27: chunk size 33904 perslab 30

slab class 28: chunk size 42384 perslab 24

slab class 29: chunk size 52984 perslab 19

slab class 30: chunk size 66232 perslab 15

slab class 31: chunk size 82792 perslab 12

slab class 32: chunk size 103496 perslab 10

slab class 33: chunk size 129376 perslab 8

slab class 34: chunk size 161720 perslab 6

slab class 35: chunk size 202152 perslab 5

slab class 36: chunk size 252696 perslab 4

slab class 37: chunk size 315872 perslab 3

slab class 38: chunk size 394840 perslab 2

slab class 39: chunk size 524288 perslab 2

<26 server listening (auto-negotiate)

<27 server listening (auto-negotiate)

<28 server listening (auto-negotiate)

<29 server listening (auto-negotiate)

Invalid IPV6 address: "[10.131.0.91"failed to listen on TCP port 11212: Success

[cwl@bastion Ali]$ oc -n openstack get pods memcached-0 -oyaml

apiVersion: v1

kind: Pod

metadata:

annotations:

k8s.ovn.org/pod-networks: '{"default":{"ip_addresses":["10.131.0.91/23","1001:1:0:4::5b/64"],"mac_address":"0a:58:0a:83:00:5b","gateway_ips":["10.131.0.1","1001:1:0:4::1"],"routes":[

{"dest":"10.128.0.0/14","nextHop":"10.131.0.1"}

,{"dest":"172.30.0.0/16","nextHop":"10.131.0.1"},{"dest":"100.64.0.0/16","nextHop":"10.131.0.1"},{"dest":"1001:1::/48","nextHop":"1001:1:0:4::1"},{"dest":"1001:2::/112","nextHop":"1001:1:0:4::1"},{"dest":"fd98::/64","nextHop":"1001:1:0:4::1"}]}}'

k8s.v1.cni.cncf.io/network-status: |-

[{

"name": "ovn-kubernetes",

"interface": "eth0",

"ips": [

"10.131.0.91",

"1001:1:0:4::5b"

],

"mac": "0a:58:0a:83:00:5b",

"default": true,

"dns": {}

}]

openshift.io/scc: anyuid

creationTimestamp: "2024-07-09T17:18:01Z"

generateName: memcached-

labels:

app: memcached

controller-revision-hash: memcached-75b86cb678

cr: memcached

memcached/name: memcached

memcached/namespace: openstack

memcached/uid: fda9e86c-7fb8-4c4b-9280-a75c19c4f0bd

owner: infra-operator

statefulset.kubernetes.io/pod-name: memcached-0

namespace: openstack

ownerReferences:

- apiVersion: apps/v1

blockOwnerDeletion: true

controller: true

kind: StatefulSet

uid: 08d2a6e3-e433-4e0f-833d-1ad7ea101c77

resourceVersion: "1089858"

uid: 3cf3f1ee-0e24-4734-8be8-440c636f7f8d

spec:

containers:

- command:

- /usr/bin/dumb-init

- –

- /usr/local/bin/kolla_start

env:

- name: KOLLA_CONFIG_STRATEGY

value: COPY_ALWAYS

- name: POD_IPS

valueFrom:

fieldRef:

apiVersion: v1

fieldPath: status.podIPs

image: registry.redhat.io/rhoso-beta/openstack-memcached-rhel9:18.0.0

imagePullPolicy: IfNotPresent

livenessProbe:

failureThreshold: 3

initialDelaySeconds: 3

periodSeconds: 3

successThreshold: 1

tcpSocket:

port: 11211

timeoutSeconds: 5

ports:

- containerPort: 11211

protocol: TCP

- containerPort: 11212

protocol: TCP

readinessProbe:

failureThreshold: 3

initialDelaySeconds: 5

periodSeconds: 5

successThreshold: 1

tcpSocket:

port: 11211

timeoutSeconds: 5

resources: {}

securityContext:

capabilities:

drop:

- MKNOD

runAsUser: 0

terminationMessagePath: /dev/termination-log

terminationMessagePolicy: File

volumeMounts:

- mountPath: /var/lib/kolla/config_files/src

readOnly: true

- mountPath: /var/lib/kolla/config_files

readOnly: true

- mountPath: /var/lib/config-data/tls/certs/memcached.crt

readOnly: true

subPath: tls.crt

- mountPath: /var/lib/config-data/tls/private/memcached.key

readOnly: true

subPath: tls.key

- mountPath: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

readOnly: true

subPath: tls-ca-bundle.pem

- mountPath: /var/run/secrets/kubernetes.io/serviceaccount

readOnly: true

dnsPolicy: ClusterFirst

enableServiceLinks: true

hostname: memcached-0

imagePullSecrets:

- name: memcached-memcached-dockercfg-mrt6z

nodeName: worker1.cwl-site2.npss.bos2.lab

preemptionPolicy: PreemptLowerPriority

priority: 0

restartPolicy: Always

schedulerName: default-scheduler

securityContext:

seLinuxOptions:

level: s0:c31,c10

serviceAccount: memcached-memcached

serviceAccountName: memcached-memcached

subdomain: memcached

terminationGracePeriodSeconds: 30

tolerations:

- effect: NoExecute

key: node.kubernetes.io/not-ready

operator: Exists

tolerationSeconds: 300

- effect: NoExecute

key: node.kubernetes.io/unreachable

operator: Exists

tolerationSeconds: 300

volumes:

- configMap:

defaultMode: 420

items:

- key: config.json

path: config.json

- configMap:

defaultMode: 420

items:

- key: memcached

path: etc/sysconfig/memcached

- name: memcached-tls-certs

secret:

defaultMode: 256

secretName: cert-memcached-svc

- name: combined-ca-bundle

secret:

defaultMode: 292

secretName: combined-ca-bundle

- name: kube-api-access-xkhv9

projected:

defaultMode: 420

sources:

- serviceAccountToken:

expirationSeconds: 3607

path: token

- configMap:

items:

- key: ca.crt

path: ca.crt

- downwardAPI:

items:

- fieldRef:

apiVersion: v1

fieldPath: metadata.namespace

path: namespace

- configMap:

items:

- key: service-ca.crt

path: service-ca.crt

status:

conditions:

- lastProbeTime: null

lastTransitionTime: "2024-07-09T17:18:01Z"

status: "True"

type: Initialized

- lastProbeTime: null

lastTransitionTime: "2024-07-09T17:18:01Z"

message: 'containers with unready status: [memcached]'

reason: ContainersNotReady

status: "False"

type: Ready

- lastProbeTime: null

lastTransitionTime: "2024-07-09T17:18:01Z"

message: 'containers with unready status: [memcached]'

reason: ContainersNotReady

status: "False"

type: ContainersReady

- lastProbeTime: null

lastTransitionTime: "2024-07-09T17:18:01Z"

status: "True"

type: PodScheduled

containerStatuses:

- containerID: cri-o://331e387cc7ff0286d7e04bbfdd8242065ae754816417eeb2d30f234bf373c8ff

image: registry.redhat.io/rhoso-beta/openstack-memcached-rhel9:18.0.0

imageID: registry.redhat.io/rhoso-beta/openstack-memcached-rhel9@sha256:7f4e5e72c23a2d3d045ea3aefd03b36c4803d38d5697e689d3d5f73cd7b5c38e

lastState:

terminated:

containerID: cri-o://331e387cc7ff0286d7e04bbfdd8242065ae754816417eeb2d30f234bf373c8ff

exitCode: 71

finishedAt: "2024-07-09T18:45:25Z"

reason: Error

startedAt: "2024-07-09T18:45:24Z"

ready: false

restartCount: 22

started: false

state:

waiting:

message: back-off 5m0s restarting failed container=memcached pod=memcached-0_openstack(3cf3f1ee-0e24-4734-8be8-440c636f7f8d)

reason: CrashLoopBackOff

hostIP: 172.31.61.9

phase: Running

podIP: 10.131.0.91

podIPs:

- ip: 10.131.0.91

- ip: 1001:1:0:4::5b

qosClass: BestEffort

startTime: "2024-07-09T17:18:01Z"

Details

Description

Attachments

Activity

People

Dates

PagerDuty