All data in the Kafka Cluster is lost by "(Preview) Migrating existing Kafka clusters to use Kafka node pools" procedure

• All data in the Kafka Cluster is lost by "(Preview) Migrating existing Kafka clusters to use Kafka node pools(1)" procedure
  • For example:

    # Before KafkaNodePool migration
    $ oc exec -it ${KAFKA_CLUSTER}-kafka-0 -c kafka -- env - bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic ${TOPIC_NAME}
    my-topic:0:1773427
    my-topic:1:1239981
    my-topic:2:1239727
    my-topic:3:1767828
    my-topic:4:1252663
    my-topic:5:1273168
    my-topic:6:1790543
    my-topic:7:1211590
    
    # After KafkaNodePool migration
    $ oc exec -it ${KAFKA_CLUSTER}-pool-a-0 -c kafka -- env - bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic ${TOPIC_NAME}
    my-topic:0:0
    my-topic:1:0
    my-topic:2:0
    my-topic:3:0
    my-topic:4:0
    my-topic:5:0
    my-topic:6:0
    my-topic:7:0
    

• As far as I could find out
  • All the old Kafka pods will be terminated suddenly when to enable the KafkaNodePool feature.
  • After that, all new each Kafka pod by KafkaNodePool will be created with new PVCs and PVs.
    • The new PVCs are created with new names: "data-id-cluster-name-kafka-nodepool-name-idx"
  • ==> So all data is lost from the new Kafka Cluster with KafkaNodePool
  • ==> The data itself is not lost, but only exists in old PVCs and PVs and is no longer used from the new Kafka Cluster.

    data-0-my-cluster-kafka-0     Bound    pvc-39d9cff1-fbdc-4066-a98b-c486c903cb18   10Gi       RWO            gp2            32m
    data-0-my-cluster-kafka-1     Bound    pvc-37f5a5a6-84d8-4bd6-99ff-45cdbabe4781   10Gi       RWO            gp2            32m
    data-0-my-cluster-kafka-2     Bound    pvc-169e0621-9abe-4f4a-a73e-ae6648cba190   10Gi       RWO            gp2            32m
    data-0-my-cluster-pool-a-0    Bound    pvc-3c536de8-8023-4e79-9d69-dc60c5ffdf76   100Gi      RWO            gp2            23m
    data-0-my-cluster-pool-a-1    Bound    pvc-8e644aed-e56e-489b-b86b-51cd2f9e6c0c   100Gi      RWO            gp2            23m
    data-0-my-cluster-pool-a-2    Bound    pvc-4bc6cfab-27be-4cf9-a2ad-7fe8d5a4091e   100Gi      RWO            gp2            23m
    data-0-my-cluster-pool-b-3    Bound    pvc-77ce02a9-5daf-4afa-9cee-d31cf72263c4   200Gi      RWO            gp2            23m
    data-0-my-cluster-pool-b-4    Bound    pvc-773bc017-44fe-4e20-9a8a-517c08abff73   200Gi      RWO            gp2            23m
    data-0-my-cluster-pool-b-5    Bound    pvc-8ff8f963-ec2a-4365-9503-97c241e40e1a   200Gi      RWO            gp2            23m
    data-my-cluster-zookeeper-0   Bound    pvc-cf7941e3-38cc-4f56-bbf3-4b171da85ee6   10Gi       RWO            gp2            32m
    data-my-cluster-zookeeper-1   Bound    pvc-5ebb52bd-1b56-47d0-b4c9-7121e52b6bf5   10Gi       RWO            gp2            32m
    data-my-cluster-zookeeper-2   Bound    pvc-b2f6d28a-7806-4a34-8d7e-8f8a10014eff   10Gi       RWO            gp2            32m
    

• We will need to provide the users with a way to migrate the existing Kafka cluster to use Kafka node pools.
  • I can't think of an easy way by manual procedure, but it may be possible to rename the PVCs after shut down the Kafka cluster in advance, although the Kafka id used for the new pod with KafkaNodePool must be identified in advance. 
  • Or either the PVC name format("data-id-cluster-name-kafka-idx") should be the same on the old and new on so that the same PVCs/PVs can be used, or a data migration tool may be needed.

(1) https://access.redhat.com/documentation/en-us/red_hat_amq_streams/2.5/html-single/deploying_and_managing_amq_streams_on_openshift/index#proc-migrating-clusters-node-pools-str