Dear Team, 
 
I wanted to let you know that we have recently tested a software update on one of our Kubernetes/Openshift sites and it appears that this update needs to be implemented on all of our sites in order for it to work properly. 
 
Before rolling out this update to all sites, we are conducting a confirmation test to ensure that it will not disrupt any current processes or cause any issues. 
 
Once this confirmation test has been completed, we will provide an update on the next steps for implementing this software update across all of our sites. 
 
Test performed:

1. OC411 AND OCP311 (non-admin user)
2. Skupper previous version: skupper-cli-1.0.2-2.el8.x86_64
3. Example: https://github.com/skupperproject/skupper-example-hello-world
4. Update to the CR2 version: First the backend (OCP411)
5. Frontend wasn't updated and the application connection did not work
6. The only way to get the app back to work was to update the Frontend site to CR2. 
    
   Here are more details:

Using the same approach, but with the TCP example, I got the same results, the client only was able to get a response when both sites were updated:
In this example and also for the hello-world, the update first was made on the server side.

• Logs:
  • Client-side:

    ○ → skupper version                                                                                                                                                                                                                                                                      
    client version                 1.0.2-redhat-2                                                                                                                                                                                                                                             
    transport version              registry.redhat.io/application-interconnect/skupper-router-rhel8:2.0.2 (sha256:00b220ad21ed)                                                                                                                                                              
    controller version             registry.redhat.io/application-interconnect/skupper-service-controller-rhel8:1.0.2 (sha256:a18fea07b665)                                                                                                                                                  
    config-sync version            registry.redhat.io/application-interconnect/skupper-config-sync-rhel8:1.0.2 (sha256:6c04ee807543)
    

• • • Router Logs:

      2023-01-09 17:59:10.064686 +0000 TCP_ADAPTOR (info) PN_LISTENER_ACCEPT Accepting TCP connection to 0.0.0.0:1024                                                                                                                                                                          
      2023-01-09 17:59:10.064960 +0000 ROUTER_CORE (info) [C26] Connection Opened: dir=in host=10.131.0.58:42568 encrypted=no auth=no user= container_id=TcpAdaptor props={:"qd.adaptor"="tcp"}                                                                                                
      2023-01-09 17:59:10.065030 +0000 TCP_ADAPTOR (info) [C26] PN_RAW_CONNECTION_CONNECTED Listener ingress accepted to 0.0.0.0:1024 from 10.131.0.58:42568 (global_id=10.131.0.58:42568@816578d9-2bba-491e-a24e-7dc309351828)                                                                
      2023-01-09 17:59:10.065061 +0000 ROUTER_CORE (info) [C26][L98] Link attached: dir=out source={(dyn)<none> expire:link} target={<none> expire:link}                                                                                                                                        
      2023-01-09 17:59:10.065093 +0000 ROUTER_CORE (info) [C26][L99] Link attached: dir=in source={<none> expire:link} target={tcp-server:9090 expire:link}                                                                                                                                    
      2023-01-09 17:59:10.292164 +0000 ROUTER_CORE (info) [C13] Endpoint attach forbidden on inter-router connection                                                                                                                                                                            
      2023-01-09 17:59:10.518529 +0000 ROUTER_CORE (info) [C13][L100] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no                                                                                                                            
      2023-01-09 17:59:10.518653 +0000 TCP_ADAPTOR (info) [C26] PN_RAW_CONNECTION_DISCONNECTED listener                                                                                                                                                                                        
      2023-01-09 17:59:10.518693 +0000 ROUTER_CORE (info) [C26][L99] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no                                                                                                                                  
      2023-01-09 17:59:10.518741 +0000 ROUTER_CORE (info) [C26][L98] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no                                                                                                                                  
      2023-01-09 17:59:10.518753 +0000 ROUTER_CORE (info) [C26] Connection Closed
      

• • Server-side:

    ○ → skupper version
    client version                 1.2.0.redhat.1.1.0-5
    transport version              registry-proxy.engineering.redhat.com/rh-osbs/application-interconnect-skupper-router-rhel8:2.2.0-3 (sha256:c4a684a510fa)
    controller version             registry-proxy.engineering.redhat.com/rh-osbs/application-interconnect-service-controller-rhel8:1.2.0-2 (sha256:405c52701a00)
    config-sync version            registry-proxy.engineering.redhat.com/rh-osbs/application-interconnect-config-sync-rhel8:1.2.0-2 (sha256:db56cf2d8a45)
    

• • • Router Logs:

      2023-01-09 17:59:10.176950 +0000 FLOW_LOG (warning) Reference ID cannot be parsed from the received field                                                                                                                                                                                
      2023-01-09 17:59:10.176974 +0000 TCP_ADAPTOR (info) [C14] qdr_tcp_connection_egress call pn_proactor_raw_connect(). Egress connecting to: 10.129.2.207:9090                                                    
      2023-01-09 17:59:10.177399 +0000 TCP_ADAPTOR (info) [C14] PN_RAW_CONNECTION_CONNECTED Connector egress connected to 10.129.2.207:9090                                                                                                                                                    
      2023-01-09 17:59:10.177417 +0000 TCP_ADAPTOR (info) [C14] Opening server-side core connection 10.129.2.207:9090                                                                                                                                                                          
      2023-01-09 17:59:10.177467 +0000 ROUTER_CORE (info) [C14] Connection Opened: dir=out host=10.129.2.207:9090 encrypted=no auth=no user= container_id=TcpAdaptor props={:"qd.adaptor"="tcp"}                                                                                                
      2023-01-09 17:59:10.177520 +0000 ROUTER_CORE (info) [C14][L50] Link attached: dir=out source={tcp-server:9090 expire:link} target={<none> expire:link}                                    
      2023-01-09 17:59:10.177557 +0000 ROUTER_CORE (info) [C14][L51] Link attached: dir=in source={<none> expire:link} target={amqp:/_topo/0/west-tcp-skupper-router-84bfdfb8bc-q6nhw/temp.9IPKKSehQwYDGB+ expire:link}
      2023-01-09 17:59:10.177591 +0000 ROUTER_CORE (info) [C3][L52] Link attached: dir=out source={<none> expire:link} target={<none> expire:link}                                                                                                                                              
      2023-01-09 17:59:10.403691 +0000 TCP_ADAPTOR (info) [C14] PN_RAW_CONNECTION_DISCONNECTED connector, drained_buffers=0                                                                                                                                                                     2023-01-09 17:59:10.403769 +0000 ROUTER_CORE (info) [C3][L52] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no                                                                                            
      2023-01-09 17:59:10.403780 +0000 ROUTER_CORE (info) [C14][L51] Link lost: del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no                                                                                                                                  2023-01-09 17:59:10.403784 +0000 ROUTER_CORE (info) [C14][L50] Link lost: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=1 delay1=0 delay10=0 blocked=no  
      2023-01-09 17:59:10.403819 +0000 ROUTER_CORE (info) [C14] Connection Closed   
      

• • • TCP Server logs:

      ○ → oc logs deploy/tcp-server -f
      Listening on port 9090
      Accepted connection 1
      Received from connection 1: hello
      Closed connection 1
      Accepted connection 2
      Received from connection 2: hello
      Closed connection 2
      Accepted connection 3
      Received from connection 3: hello
      Closed connection 3
      Accepted connection 4
      Closed connection 4
      --- From this part, the client was updated and the server not
      Accepted connection 5
      Closed connection 5
      Accepted connection 6
      Closed connection 6
      Accepted connection 7
      Closed connection 7
      --- From this part both sites were updated
      Accepted connection 8
      Received from connection 8: hello
      Closed connection 8
      Accepted connection 9
      Received from connection 9: hello
      Closed connection 9
      

---------------------------------------------------------------------------------

The communication between both sites was successfully re-established ONLY AFTER UPDATE on both sites.

The dumps are attached.