Uploaded image for project: 'AMQ Streams'
  1. AMQ Streams
  2. ENTMQST-3918

Rolling update force-rolls pods during cluster startup

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 2.2.0.GA
    • None
    • None
    • None

    Description

      When using StrimziPodSets and deploying a new cluster, it looks like the KafkaRoller force-rolls the pods during their startup because it cannot connect to them.

      What seems to happen is the following:

      • KafkaAssemblyOperator creates the Kafka StrimziPodSet
      • PodSetController starts the pods
      • While the containers are still in the Pending or ContainerCreating phase, the KafkaRolller terminates them
      • The PodSet controller starts them again and the pod is eventually started

       

      my-cluster-kafka-0                          0/1     Pending             0             0s
my-cluster-kafka-0                          0/1     Pending             0             0s
my-cluster-kafka-1                          0/1     Pending             0             0s
my-cluster-kafka-1                          0/1     Pending             0             0s
my-cluster-kafka-2                          0/1     Pending             0             0s
my-cluster-kafka-0                          0/1     ContainerCreating   0             0s
my-cluster-kafka-2                          0/1     Pending             0             0s
my-cluster-kafka-1                          0/1     ContainerCreating   0             0s
my-cluster-kafka-1                          0/1     Terminating         0             0s
my-cluster-kafka-1                          0/1     Terminating         0             1s
my-cluster-kafka-2                          0/1     ContainerCreating   0             1s
my-cluster-kafka-2                          0/1     ContainerCreating   0             2s
my-cluster-kafka-0                          0/1     ContainerCreating   0             2s
my-cluster-kafka-1                          0/1     Terminating         0             3s
my-cluster-kafka-2                          0/1     Running             0             3s
my-cluster-kafka-1                          1/1     Terminating         0             4s
my-cluster-kafka-0                          0/1     Running             0             4s
my-cluster-kafka-0                          1/1     Running             0             20s
my-cluster-kafka-2                          1/1     Running             0             20s
my-cluster-kafka-1                          1/1     Terminating         0             34s
my-cluster-kafka-1                          0/1     Terminating         0             35s
my-cluster-kafka-1                          0/1     Terminating         0             35s
my-cluster-kafka-1                          0/1     Terminating         0             35s
my-cluster-kafka-1                          0/1     Pending             0             0s
my-cluster-kafka-1                          0/1     Pending             0             0s
my-cluster-kafka-1                          0/1     ContainerCreating   0             0s
my-cluster-kafka-1                          0/1     ContainerCreating   0             1s
my-cluster-kafka-1                          0/1     Running             0             2s

      The log from the KafkaRoller suggests it cannot connect to them due to the DNS not being up yet:

       

      2022-04-01 21:01:08 INFO  OperatorWatcher:38 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Kafka my-cluster in namespace myproject was ADDED
2022-04-01 21:01:08 INFO  AbstractOperator:226 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Kafka my-cluster will be checked for creation or modification
2022-04-01 21:01:08 INFO  CrdOperator:113 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Status of Kafka my-cluster in namespace myproject has been updated
2022-04-01 21:01:08 INFO  OperatorWatcher:38 - Reconciliation #2(watch) Kafka(myproject/my-cluster): Kafka my-cluster in namespace myproject was MODIFIED
2022-04-01 21:01:11 INFO  StrimziPodSetController:247 - Reconciliation #3(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:11 INFO  StrimziPodSetController:281 - Reconciliation #3(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:11 INFO  StrimziPodSetController:247 - Reconciliation #4(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:11 INFO  StrimziPodSetController:281 - Reconciliation #4(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:11 INFO  StrimziPodSetController:247 - Reconciliation #5(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:11 INFO  StrimziPodSetController:281 - Reconciliation #5(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:12 INFO  StrimziPodSetController:247 - Reconciliation #6(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:12 INFO  StrimziPodSetController:281 - Reconciliation #6(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:12 INFO  StrimziPodSetController:247 - Reconciliation #7(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:12 INFO  StrimziPodSetController:281 - Reconciliation #7(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:12 INFO  StrimziPodSetController:247 - Reconciliation #8(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:12 INFO  StrimziPodSetController:281 - Reconciliation #8(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:13 INFO  StrimziPodSetController:247 - Reconciliation #9(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:13 INFO  StrimziPodSetController:281 - Reconciliation #9(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:36 INFO  StrimziPodSetController:247 - Reconciliation #10(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:36 INFO  StrimziPodSetController:281 - Reconciliation #10(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:36 INFO  StrimziPodSetController:247 - Reconciliation #11(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:36 INFO  StrimziPodSetController:281 - Reconciliation #11(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:37 INFO  StrimziPodSetController:247 - Reconciliation #12(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:37 INFO  StrimziPodSetController:281 - Reconciliation #12(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:42 INFO  StrimziPodSetController:247 - Reconciliation #13(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:42 INFO  StrimziPodSetController:281 - Reconciliation #13(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:43 INFO  StrimziPodSetController:247 - Reconciliation #14(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:43 INFO  StrimziPodSetController:281 - Reconciliation #14(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:247 - Reconciliation #15(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:281 - Reconciliation #15(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:247 - Reconciliation #16(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:281 - Reconciliation #16(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:247 - Reconciliation #17(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:281 - Reconciliation #17(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:247 - Reconciliation #18(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:01:51 INFO  StrimziPodSetController:281 - Reconciliation #18(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:01:56 INFO  ClusterOperator:128 - Triggering periodic reconciliation for namespace myproject
2022-04-01 21:02:01 INFO  StrimziPodSetController:247 - Reconciliation #20(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:02:01 INFO  StrimziPodSetController:281 - Reconciliation #20(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:02:01 INFO  StrimziPodSetController:247 - Reconciliation #21(watch) StrimziPodSet(myproject/my-cluster-zookeeper): StrimziPodSet will be reconciled
2022-04-01 21:02:01 INFO  StrimziPodSetController:281 - Reconciliation #21(watch) StrimziPodSet(myproject/my-cluster-zookeeper): reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:247 - Reconciliation #22(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:281 - Reconciliation #22(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:247 - Reconciliation #23(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:281 - Reconciliation #23(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:247 - Reconciliation #24(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:281 - Reconciliation #24(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:03 WARN  ClientUtils:75 - Couldn't resolve server my-cluster-kafka-0.my-cluster-kafka-brokers.myproject.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for my-cluster-kafka-0.my-cluster-kafka-brokers.myproject.svc.cluster.local
2022-04-01 21:02:03 WARN  ClientUtils:75 - Couldn't resolve server my-cluster-kafka-1.my-cluster-kafka-brokers.myproject.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for my-cluster-kafka-1.my-cluster-kafka-brokers.myproject.svc.cluster.local
2022-04-01 21:02:03 WARN  ClientUtils:75 - Couldn't resolve server my-cluster-kafka-2.my-cluster-kafka-brokers.myproject.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for my-cluster-kafka-2.my-cluster-kafka-brokers.myproject.svc.cluster.local
2022-04-01 21:02:03 INFO  KafkaRoller:472 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Pod {my-cluster-kafka-2} needs to be restarted, because it does not seem to responding to connection attempts
2022-04-01 21:02:03 INFO  KafkaRoller:290 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Could not roll pod {my-cluster-kafka-2} due to java.lang.NullPointerException, retrying after at least 250ms
2022-04-01 21:02:03 WARN  ClientUtils:75 - Couldn't resolve server my-cluster-kafka-0.my-cluster-kafka-brokers.myproject.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for my-cluster-kafka-0.my-cluster-kafka-brokers.myproject.svc.cluster.local
2022-04-01 21:02:03 WARN  ClientUtils:75 - Couldn't resolve server my-cluster-kafka-1.my-cluster-kafka-brokers.myproject.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for my-cluster-kafka-1.my-cluster-kafka-brokers.myproject.svc.cluster.local
2022-04-01 21:02:03 WARN  ClientUtils:75 - Couldn't resolve server my-cluster-kafka-2.my-cluster-kafka-brokers.myproject.svc.cluster.local:9091 from bootstrap.servers as DNS resolution failed for my-cluster-kafka-2.my-cluster-kafka-brokers.myproject.svc.cluster.local
2022-04-01 21:02:03 INFO  KafkaRoller:472 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Pod {my-cluster-kafka-1} needs to be restarted, because it does not seem to responding to connection attempts
2022-04-01 21:02:03 INFO  PodOperator:54 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Rolling pod my-cluster-kafka-1
2022-04-01 21:02:03 INFO  StrimziPodSetController:247 - Reconciliation #25(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:03 INFO  StrimziPodSetController:281 - Reconciliation #25(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:04 INFO  StrimziPodSetController:247 - Reconciliation #26(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:04 INFO  StrimziPodSetController:281 - Reconciliation #26(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:04 INFO  StrimziPodSetController:247 - Reconciliation #27(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:04 INFO  StrimziPodSetController:281 - Reconciliation #27(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:05 INFO  StrimziPodSetController:247 - Reconciliation #28(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:05 INFO  StrimziPodSetController:281 - Reconciliation #28(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:05 INFO  StrimziPodSetController:247 - Reconciliation #29(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:05 INFO  StrimziPodSetController:281 - Reconciliation #29(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:06 INFO  StrimziPodSetController:247 - Reconciliation #30(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:06 INFO  StrimziPodSetController:281 - Reconciliation #30(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:06 INFO  StrimziPodSetController:247 - Reconciliation #31(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:06 INFO  StrimziPodSetController:281 - Reconciliation #31(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:07 INFO  StrimziPodSetController:247 - Reconciliation #32(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:07 INFO  StrimziPodSetController:281 - Reconciliation #32(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:07 INFO  StrimziPodSetController:247 - Reconciliation #33(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:07 INFO  StrimziPodSetController:281 - Reconciliation #33(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:07 INFO  StrimziPodSetController:247 - Reconciliation #34(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:07 INFO  StrimziPodSetController:281 - Reconciliation #34(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:08 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Reconciliation is in progress
2022-04-01 21:02:23 INFO  StrimziPodSetController:247 - Reconciliation #35(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:23 INFO  StrimziPodSetController:281 - Reconciliation #35(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:23 INFO  StrimziPodSetController:247 - Reconciliation #36(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:23 INFO  StrimziPodSetController:281 - Reconciliation #36(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:23 INFO  StrimziPodSetController:247 - Reconciliation #37(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:23 INFO  StrimziPodSetController:281 - Reconciliation #37(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:37 INFO  StrimziPodSetController:247 - Reconciliation #38(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:37 INFO  StrimziPodSetController:281 - Reconciliation #38(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:247 - Reconciliation #39(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:281 - Reconciliation #39(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:247 - Reconciliation #40(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:281 - Reconciliation #40(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:247 - Reconciliation #41(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:281 - Reconciliation #41(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:247 - Reconciliation #42(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:281 - Reconciliation #42(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:247 - Reconciliation #43(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:38 INFO  StrimziPodSetController:281 - Reconciliation #43(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:39 INFO  StrimziPodSetController:247 - Reconciliation #44(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:39 INFO  StrimziPodSetController:281 - Reconciliation #44(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:40 INFO  StrimziPodSetController:247 - Reconciliation #45(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:40 INFO  StrimziPodSetController:281 - Reconciliation #45(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:45 INFO  StrimziPodSetController:247 - Reconciliation #46(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:45 INFO  StrimziPodSetController:281 - Reconciliation #46(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:46 INFO  StrimziPodSetController:247 - Reconciliation #47(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:46 INFO  StrimziPodSetController:281 - Reconciliation #47(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:51 INFO  StrimziPodSetController:247 - Reconciliation #48(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:51 INFO  StrimziPodSetController:281 - Reconciliation #48(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:02:58 INFO  StrimziPodSetController:247 - Reconciliation #49(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:02:58 INFO  StrimziPodSetController:281 - Reconciliation #49(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled
2022-04-01 21:03:08 INFO  AbstractOperator:373 - Reconciliation #1(watch) Kafka(myproject/my-cluster): Reconciliation is in progress
2022-04-01 21:03:09 INFO  StrimziPodSetController:247 - Reconciliation #50(watch) StrimziPodSet(myproject/my-cluster-kafka): StrimziPodSet will be reconciled
2022-04-01 21:03:09 INFO  StrimziPodSetController:281 - Reconciliation #50(watch) StrimziPodSet(myproject/my-cluster-kafka): reconciled

      While the cluster eventually starts fine, it is a bit annoying and means that cluster startup takes longer than needed.

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            scholzj JAkub Scholz
            Lukas Kral Lukas Kral
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: