Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-3738

galara does not start on IPv6 single-stack

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • rhos-18.0 Dev Preview 3
    • rhos-18.0 Dev Preview 3
    • None
    • None
    • Important

      ERROR

      + exec /usr/libexec/mysqld --wsrep-cluster-address=gcomm://
2024-01-17 11:09:02 0 [Note] /usr/libexec/mysqld (mysqld 10.5.16-MariaDB) starting as process 7 ...
2024-01-17 11:09:02 0 [Note] WSREP: Loading provider /usr/lib64/galera/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
2024-01-17 11:09:02 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib64/galera/libgalera_smm.so'
2024-01-17 11:09:02 0 [Note] WSREP: wsrep_load(): Galera 26.4.11(rXXXX) by Codership Oy <info@codership.com> loaded successfully.
2024-01-17 11:09:02 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2024-01-17 11:09:02 0 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1, safe_to_bootstrap: 1
2024-01-17 11:09:02 0 [Note] WSREP: GCache DEBUG: opened preamble:
Version: 2
UUID: 00000000-0000-0000-0000-000000000000
Seqno: -1 - -1
Offset: -1
Synced: 1
2024-01-17 11:09:02 0 [Note] WSREP: Skipped GCache ring buffer recovery: could not determine history UUID.
2024-01-17 11:09:02 0 [Warning] WSREP: Option 'gcs.fc_master_slave' is deprecated and will be removed in the future versions, please use 'gcs.fc_single_primary' instead. 
2024-01-17 11:09:02 0 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/mysql/; base_host = openstack-galera-0.openstack-galera; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.keep_plaintext_size = 128M; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.fc_single_primary = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = 
2024-01-17 11:09:02 0 [Note] WSREP: Start replication
2024-01-17 11:09:02 0 [Note] WSREP: Connecting with bootstrap option: 1
2024-01-17 11:09:02 0 [Note] WSREP: Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
2024-01-17 11:09:02 0 [Note] WSREP: protonet asio version 0
2024-01-17 11:09:02 0 [Note] WSREP: Using CRC-32C for message checksums.
2024-01-17 11:09:02 0 [Note] WSREP: backend: asio
2024-01-17 11:09:02 0 [Note] WSREP: gcomm thread scheduling priority set to other:0 
2024-01-17 11:09:02 0 [Warning] WSREP: access file(/var/lib/mysql//gvwstate.dat) failed(No such file or directory)
2024-01-17 11:09:02 0 [Note] WSREP: restore pc from disk failed
2024-01-17 11:09:02 0 [Note] WSREP: GMCast version 0
2024-01-17 11:09:02 0 [Note] WSREP: (d21be057-aae7, 'tcp://[fd00:abcd:0:1:0:af:4567:4567]:0') listening at tcp://[fd00:abcd:0:1:0:af:4567:4567]:0
2024-01-17 11:09:02 0 [Note] WSREP: (d21be057-aae7, 'tcp://[fd00:abcd:0:1:0:af:4567:4567]:0') multicast: , ttl: 1
2024-01-17 11:09:02 0 [Note] WSREP: EVS version 1
2024-01-17 11:09:02 0 [Note] WSREP: gcomm: bootstrapping new group 'galera_cluster'
2024-01-17 11:09:02 0 [Note] WSREP: start_prim is enabled, turn off pc_recovery
2024-01-17 11:09:02 0 [ERROR] WSREP: failed to open gcomm backend connection: 99: Failed to listen: bind: Cannot assign requested address: 99 (Cannot assign requested address)
	 at /builddir/build/BUILD/galera-26.4.11/galerautils/src/gu_asio_stream_react.cpp:listen():746
2024-01-17 11:09:02 0 [ERROR] WSREP: /builddir/build/BUILD/galera-26.4.11/gcs/src/gcs_core.cpp:gcs_core_open():221: Failed to open backend connection: -99 (Cannot assign requested address)
2024-01-17 11:09:02 0 [ERROR] WSREP: /builddir/build/BUILD/galera-26.4.11/gcs/src/gcs.cpp:gcs_open():1669: Failed to open channel 'galera_cluster' at 'gcomm://': -99 (Cannot assign requested address)
2024-01-17 11:09:02 0 [ERROR] WSREP: gcs connect failed: Cannot assign requested address
2024-01-17 11:09:02 0 [ERROR] WSREP: wsrep::connect(gcomm://) failed: 7
2024-01-17 11:09:02 0 [ERROR] Aborting

      Describe pod

      $ oc describe pod/openstack-galera-0 
Name:             openstack-galera-0
Namespace:        openstack
Priority:         0
Service Account:  galera-openstack
Node:             api.sno.lab.example.com/fd00:abcd:abcd:fc00::11
Start Time:       Wed, 17 Jan 2024 12:05:26 +0100
Labels:           app=galera
                  controller-revision-hash=openstack-galera-5b7959d4cc
                  cr=galera-openstack
                  galera/name=openstack
                  galera/namespace=openstack
                  galera/uid=15be8f08-f7bc-48de-80df-b38228b416ed
                  owner=mariadb-operator
                  statefulset.kubernetes.io/pod-name=openstack-galera-0
Annotations:      k8s.ovn.org/pod-networks:
                    {"default":{"ip_addresses":["fd00:abcd:0:1::af/64"],"mac_address":"0a:58:53:e3:51:d4","gateway_ips":["fd00:abcd:0:1::1"],"ip_address":"fd0...
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "ovn-kubernetes",
                        "interface": "eth0",
                        "ips": [
                            "fd00:abcd:0:1::af"
                        ],
                        "mac": "0a:58:53:e3:51:d4",
                        "default": true,
                        "dns": {}
                    }]
                  openshift.io/scc: anyuid
Status:           Running
IP:               fd00:abcd:0:1::af
IPs:
  IP:           fd00:abcd:0:1::af
Controlled By:  StatefulSet/openstack-galera
Init Containers:
  mysql-bootstrap:
    Container ID:  cri-o://7d35203038c8cfcb4d6e8f31b976734102c55bc43593f32d3b58565de5799777
    Image:         quay.io/podified-antelope-centos9/openstack-mariadb@sha256:0625afedc508312534a6f1738257c0bcbe5c263461efe497dce03c8b7d7bc05d
    Image ID:      quay.io/podified-antelope-centos9/openstack-mariadb@sha256:0625afedc508312534a6f1738257c0bcbe5c263461efe497dce03c8b7d7bc05d
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      /var/lib/operator-scripts/mysql_bootstrap.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 17 Jan 2024 12:08:25 +0100
      Finished:     Wed, 17 Jan 2024 12:08:34 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      KOLLA_BOOTSTRAP:        True
      KOLLA_CONFIG_STRATEGY:  COPY_ALWAYS
      DB_ROOT_PASSWORD:       <set to the key 'DbRootPassword' in secret 'osp-secret'>  Optional: false
    Mounts:
      /var/lib/config-data from config-data (ro)
      /var/lib/kolla/config_files from kolla-config (ro)
      /var/lib/mysql from mysql-db (rw)
      /var/lib/operator-scripts from operator-scripts (ro)
      /var/lib/pod-config-data from pod-config-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dfd6r (ro)
Containers:
  galera:
    Container ID:  cri-o://1ba405e1658ebe98c7590891a789857c81f9903a0fa1c4d55ef5e9f4a80e958f
    Image:         quay.io/podified-antelope-centos9/openstack-mariadb@sha256:0625afedc508312534a6f1738257c0bcbe5c263461efe497dce03c8b7d7bc05d
    Image ID:      quay.io/podified-antelope-centos9/openstack-mariadb@sha256:0625afedc508312534a6f1738257c0bcbe5c263461efe497dce03c8b7d7bc05d
    Ports:         3306/TCP, 4567/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /usr/bin/dumb-init
      --
      /usr/local/bin/kolla_start
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 17 Jan 2024 15:46:59 +0100
      Finished:     Wed, 17 Jan 2024 15:47:03 +0100
    Ready:          False
    Restart Count:  47
    Liveness:       exec [/bin/bash /var/lib/operator-scripts/mysql_probe.sh liveness] delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      exec [/bin/bash /var/lib/operator-scripts/mysql_probe.sh readiness] delay=0s timeout=1s period=10s #success=1 #failure=3
    Startup:        exec [/bin/bash /var/lib/operator-scripts/mysql_probe.sh startup] delay=0s timeout=1s period=10s #success=1 #failure=30
    Environment:
      CR_CONFIG_HASH:         n55fhb9hbbhd4h5ddh644h598h67dh5h5d9h67fh654hfch669h5f9h548h57bh89h5dchbdhc6h5d5h655h7bhdbh58fh89h67dh5cfh58fh68ch5d9q
      KOLLA_CONFIG_STRATEGY:  COPY_ALWAYS
      DB_ROOT_PASSWORD:       <set to the key 'DbRootPassword' in secret 'osp-secret'>  Optional: false
    Mounts:
      /var/lib/config-data from config-data (ro)
      /var/lib/kolla/config_files from kolla-config (ro)
      /var/lib/mysql from mysql-db (rw)
      /var/lib/operator-scripts from operator-scripts (ro)
      /var/lib/pod-config-data from pod-config-data (rw)
      /var/lib/secrets from secrets (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-dfd6r (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  mysql-db:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  mysql-db-openstack-galera-0
    ReadOnly:   false
  secrets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  osp-secret
    Optional:    false
  kolla-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openstack-config-data
    Optional:  false
  pod-config-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  config-data:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openstack-config-data
    Optional:  false
  operator-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openstack-scripts
    Optional:  false
  kube-api-access-dfd6r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                       From     Message
  ----     ------   ----                      ----     -------
  Normal   Pulled   39m (x41 over 3h41m)      kubelet  Container image "quay.io/podified-antelope-centos9/openstack-mariadb@sha256:0625afedc508312534a6f1738257c0bcbe5c263461efe497dce03c8b7d7bc05d" already present on machine
  Warning  BackOff  4m42s (x1067 over 3h41m)  kubelet  Back-off restarting failed container galera in pod openstack-galera-0_openstack(bc27168d-b104-44c2-a282-2eadf303922c)

      Reproduce

      #### Create the IPv6 LAB

# Export vars:
export NETWORK_ISOLATION_NET_NAME=net-iso
export NETWORK_ISOLATION_IPV4=false
export NETWORK_ISOLATION_IPV6=true
export NETWORK_ISOLATION_INSTANCE_NAME=sno
export NETWORK_ISOLATION_IP_ADDRESS=fd00:aaaa::10
export NNCP_INTERFACE=enp7s0

# Change to the devsetup directory:
cd <install_yamls_root_path>/devsetup

# Set up the networking using NAT64 and SNO Single-node-Openshift:
make ipv6_lab

# Create the network-isolation network with IPv6 enabled
make network_isolation_bridge

# Attach the network-isolation bridge to SNO (Single-node-Openshift):
make attach_default_interface

# Login to the cluster:
source $HOME/.ipv6lab/sno_env
oc login -u admin -p 12345678 https://api.sno.lab.example.com:6443

# Change to the install_yamls directory:
cd <install_yamls_root_path>


# Deploy the Openstack Controlplane
make namespace
make input
make crc_storage
make openstack
make openstack_deploy

       

              rhn-engineering-dciabrin Damien Ciabrini
              rhn-gps-hjensas Harald Jensas
              rhos-dfg-hardprov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: