Uploaded image for project: 'AMQ Streams'
  1. AMQ Streams
  2. ENTMQST-6201

Replacing CA certificates and private keys doesn't work anymore

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • 2.7.0.GA
    • cluster-operator
    • None
    • True
    • Prevents installation in a production environment.
    • False
    • Hide

      Detailed procedure starting from a new cluster OCP (CRC) 4.14 with Streams 2.7-2 operator:

      • Create a cluster with 00-my-cluster.yaml
      • Wait cluster up and running
      • Create ca for cluster and client
      • `oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"`
      • `oc describe Kafka my-cluster|grep ReconciliationPaused`
      • oc edit secrets/my-cluster-cluster-ca-cert
      • cat cluster.crt| base64 -w 0|xclip
      • rename ca.crt to `ca-20240719T14354402.crt`
      • create ca.crt with clipboard as value
      • increased ca-cert-generation ("1")
      • same as before for secrets/my-cluster-clients-ca-cert
      • update the key for cluster and client from clipboard and increase ca-cert-generation ("1")
      • `oc annotate --overwrite Kafka my-cluster strimzi.io/pause-reconciliation="false"`

      Operator restarts all PODs two times (I suppose for cluster CA then for client CA)

      • remove updated certificates
      • manual rolling update with
      • `oc annotate strimzipodset my-cluster-kafka strimzi.io/manual-rolling-update="true"`
      • `oc annotate strimzipodset my-cluster-zookeeper strimzi.io/manual-rolling-update="true"`
      • my-cluster-kafka-0 in Crashloop
        NAME                                                   READY   STATUS    RESTARTS   AGE
        amq-streams-cluster-operator-v2.7.0-2-cbd78884-m6zzj   1/1     Running   0          4h23m
        my-cluster-entity-operator-7974b8bb5c-kr5db            2/2     Running   0          7m20s
        my-cluster-kafka-0                                     0/1     Running   0          2s
        my-cluster-kafka-1                                     1/1     Running   0          8m36s
        my-cluster-kafka-2                                     1/1     Running   0          8m13s
        my-cluster-zookeeper-0                                 1/1     Running   0          5m13s
        my-cluster-zookeeper-1                                 1/1     Running   0          4m51s
        my-cluster-zookeeper-2                                 1/1     Running   0          4m28s
        my-cluster-kafka-0                                     0/1     Error     0          3s
        my-cluster-kafka-0                                     0/1     Error     1 (2s ago)   4s
        my-cluster-kafka-0                                     0/1     CrashLoopBackOff   1 (1s ago)   5s
        
      • oc logs my-cluster-kafka-0
      removed directory '/tmp/hsperfdata_1000660000'
      removed '/tmp/kafka/cluster.truststore.p12'
      removed directory '/tmp/kafka'
      STRIMZI_BROKER_ID=0
      Preparing truststore for replication listener
      Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca
      Certificate was added to keystore
      Preparing truststore for replication listener is complete
      Looking for the right CA
      No CA found. Thus exiting.
      

      In attach yaml of each changes (00,01,02).

      Show
      Detailed procedure starting from a new cluster OCP (CRC) 4.14 with Streams 2.7-2 operator: Create a cluster with 00-my-cluster.yaml Wait cluster up and running Create ca for cluster and client `oc annotate Kafka my-cluster strimzi.io/pause-reconciliation="true"` `oc describe Kafka my-cluster|grep ReconciliationPaused` oc edit secrets/my-cluster-cluster-ca-cert cat cluster.crt| base64 -w 0|xclip rename ca.crt to `ca-20240719T14354402.crt` create ca.crt with clipboard as value increased ca-cert-generation ("1") same as before for secrets/my-cluster-clients-ca-cert update the key for cluster and client from clipboard and increase ca-cert-generation ("1") `oc annotate --overwrite Kafka my-cluster strimzi.io/pause-reconciliation="false"` Operator restarts all PODs two times (I suppose for cluster CA then for client CA) remove updated certificates manual rolling update with `oc annotate strimzipodset my-cluster-kafka strimzi.io/manual-rolling-update="true"` `oc annotate strimzipodset my-cluster-zookeeper strimzi.io/manual-rolling-update="true"` my-cluster-kafka-0 in Crashloop NAME READY STATUS RESTARTS AGE amq-streams-cluster- operator -v2.7.0-2-cbd78884-m6zzj 1/1 Running 0 4h23m my-cluster-entity- operator -7974b8bb5c-kr5db 2/2 Running 0 7m20s my-cluster-kafka-0 0/1 Running 0 2s my-cluster-kafka-1 1/1 Running 0 8m36s my-cluster-kafka-2 1/1 Running 0 8m13s my-cluster-zookeeper-0 1/1 Running 0 5m13s my-cluster-zookeeper-1 1/1 Running 0 4m51s my-cluster-zookeeper-2 1/1 Running 0 4m28s my-cluster-kafka-0 0/1 Error 0 3s my-cluster-kafka-0 0/1 Error 1 (2s ago) 4s my-cluster-kafka-0 0/1 CrashLoopBackOff 1 (1s ago) 5s oc logs my-cluster-kafka-0 removed directory '/tmp/hsperfdata_1000660000' removed '/tmp/kafka/cluster.truststore.p12' removed directory '/tmp/kafka' STRIMZI_BROKER_ID=0 Preparing truststore for replication listener Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore /tmp/kafka/cluster.truststore.p12 with alias ca Certificate was added to keystore Preparing truststore for replication listener is complete Looking for the right CA No CA found. Thus exiting. In attach yaml of each changes (00,01,02).
    • Critical

      Following the documented procedure, the Zookeeper 0 POD fails in crashloop with the error `Looking for the right CA, No CA found. Thus exiting`

        1. cluster.key
          3 kB
        2. cluster.crt.dump.txt
          5 kB
        3. cluster.crt
          2 kB
        4. client.key
          3 kB
        5. client.crt.dump.txt
          5 kB
        6. client.crt
          2 kB
        7. 02-my-cluster-cluster-ca-cert.yaml
          6 kB
        8. 02-my-cluster-clients-ca-cert.yaml
          6 kB
        9. 01-my-cluster-cluster-ca-cert.yaml
          8 kB
        10. 01-my-cluster-cluster-ca.yaml
          5 kB
        11. 01-my-cluster-clients-ca-cert.yaml
          8 kB
        12. 01-my-cluster-clients-ca.yaml
          5 kB
        13. 01-my-cluster.yaml
          1 kB
        14. 00-my-cluster-cluster-ca-cert.yaml
          6 kB
        15. 00-my-cluster-cluster-ca.yaml
          5 kB
        16. 00-my-cluster-clients-ca-cert.yaml
          6 kB
        17. 00-my-cluster-clients-ca.yaml
          5 kB
        18. 00-my-cluster.yaml
          0.8 kB

              Unassigned Unassigned
              rhn-support-agagliar Antonio Gagliardi
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: