Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-16899

OCP 4.1 - EAP 72 migration pod for transactions fails due to: "ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc',..."

    Details

    • Type: Bug
    • Status: Verified (View Workflow)
    • Priority: Blocker
    • Resolution: Done
    • Affects Version/s: 7.2.1.GA
    • Fix Version/s: 7.2.1.GA
    • Component/s: OpenShift
    • Labels:
      None
    • Target Release:
    • Steps to Reproduce:
      Hide

      Use following job to provision OCP 4.1 instance on AWS:
      https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-create-openshift-4-cluster/

      Follow those steps to run test against OCP 4.1 cluster:

      git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/openshift-eap-tests.git
      cd openshift-eap-tests
      git checkout ocp_41
      # create test.properties file and set correct *.url and *.token
      xtf.openshift.url=https://api.eap-qe-ocp41-cluster7.eap-qe-ocp41-cluster7.fw.rhcloud.com:6443
      xtf.openshift.namespace=test-namespace
      xtf.openshift.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo
      xtf.openshift.admin.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo
      xtf.openshift.version=4.1.0-201904090034.git.0.1016bda.el7
      
      Start test by:
      mvn clean test -P72 -Dtest=JtaCrashRecQuickstartTest#testJTAXaRecovery -Dxtf.version=0.12-SNAPSHOT
      
      Show
      Use following job to provision OCP 4.1 instance on AWS: https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-create-openshift-4-cluster/ Follow those steps to run test against OCP 4.1 cluster: git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/openshift-eap-tests.git cd openshift-eap-tests git checkout ocp_41 # create test.properties file and set correct *.url and *.token xtf.openshift.url=https: //api.eap-qe-ocp41-cluster7.eap-qe-ocp41-cluster7.fw.rhcloud.com:6443 xtf.openshift.namespace=test-namespace xtf.openshift.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo xtf.openshift.admin.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo xtf.openshift.version=4.1.0-201904090034.git.0.1016bda.el7 Start test by: mvn clean test -P72 -Dtest=JtaCrashRecQuickstartTest#testJTAXaRecovery -Dxtf.version=0.12-SNAPSHOT

      Description

      This issue was hit during testing of eap72 image on OCP 4.1.

      EAP migration pod for recovery of XA transactions fails on OCP 4.1 with following error:

      INFO [Tue May 14 08:37:50 UTC 2019] Examining existence of living pod 'eap-app-1-6jdxn'
      CRITICAL:__main__:Cannot query OpenShift API for "https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods"
      Traceback (most recent call last):
        File "/opt/partition/queryosapi.py", line 151, in <module>
          queryResult = getLivingPods()
        File "/opt/partition/queryosapi.py", line 109, in getLivingPods
          jsonPodsData = getPodsJsonData()
        File "/opt/partition/queryosapi.py", line 97, in getPodsJsonData
          jsonText = OpenShiftQuery.queryApi('/api/v1/namespaces/{}/pods'.format(OpenShiftQuery.getNameSpace()))
        File "/opt/partition/queryosapi.py", line 89, in queryApi
          return urllib2.urlopen(request, cafile = OpenShiftQuery.CERT_FILE_PATH).read()
        File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
          return opener.open(url, data, timeout)
        File "/usr/lib64/python2.7/urllib2.py", line 431, in open
          response = self._open(req, data)
        File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
          '_open', req)
        File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
          result = func(*args)
        File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
          context=self._context, check_hostname=self._check_hostname)
        File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
          h.request(req.get_method(), req.get_selector(), req.data, headers)
        File "/usr/lib64/python2.7/httplib.py", line 1041, in request
          self._send_request(method, url, body, headers)
        File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
          self.endheaders(body)
        File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
          self._send_output(message_body)
        File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
          self.send(msg)
        File "/usr/lib64/python2.7/httplib.py", line 843, in send
          self.connect()
        File "/usr/lib64/python2.7/httplib.py", line 1263, in connect
          ssl.match_hostname(self.sock.getpeercert(), server_hostname)
        File "/usr/lib64/python2.7/ssl.py", line 267, in match_hostname
          % (hostname, ', '.join(map(repr, dnsnames))))
      ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '172.30.0.1'
      [0;33mWARN [Tue May 14 08:37:55 UTC 2019] Can't get list of living pods[0m
      INFO [Tue May 14 08:37:55 UTC 2019] Finished migration check cycle, pausing for 30 seconds before resuming
      CRITICAL:__main__:Cannot query OpenShift API for "https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods/eap-app-migration-1-rln94/log?timestamps=true&tailLines=1"
      Traceback (most recent call last):
        File "/opt/partition/queryosapi.py", line 159, in <module>
          queryResult = getLog(podName, sinceTime, tailLine)
        File "/opt/partition/queryosapi.py", line 122, in getLog
          .format(OpenShiftQuery.getNameSpace(), podName, sinceTimeParam, tailLineParam))
        File "/opt/partition/queryosapi.py", line 89, in queryApi
          return urllib2.urlopen(request, cafile = OpenShiftQuery.CERT_FILE_PATH).read()
        File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
          return opener.open(url, data, timeout)
        File "/usr/lib64/python2.7/urllib2.py", line 431, in open
          response = self._open(req, data)
        File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
          '_open', req)
        File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
          result = func(*args)
        File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
          context=self._context, check_hostname=self._check_hostname)
        File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
          h.request(req.get_method(), req.get_selector(), req.data, headers)
        File "/usr/lib64/python2.7/httplib.py", line 1041, in request
          self._send_request(method, url, body, headers)
        File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
          self.endheaders(body)
        File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
          self._send_output(message_body)
        File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
          self.send(msg)
        File "/usr/lib64/python2.7/httplib.py", line 843, in send
          self.connect()
        File "/usr/lib64/python2.7/httplib.py", line 1263, in connect
          ssl.match_hostname(self.sock.getpeercert(), server_hostname)
        File "/usr/lib64/python2.7/ssl.py", line 267, in match_hostname
          % (hostname, ', '.join(map(repr, dnsnames))))
      ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '172.30.0.1'
      

      Problem is that domain openshift.default.svc is not present in {{/var/run/secrets/kubernetes.io/serviceaccount/ca.crt }} file. This file is passed to migration pod and used for authentication in API calls in python script.

      So as following command fails:
      curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo" 'https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods'

      this on passes:
      curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo" 'https://kubernetes.default.svc/api/v1/namespaces/mnovak-mynamespace/pods'

      This appears to be more OCP 4.1 issue than problem in migration pod as ca.crt does no longer contains openshift.default.svc.

        Gliffy Diagrams

          Attachments

            Activity

              People

              • Assignee:
                brian.stansberry Brian Stansberry
                Reporter:
                mnovak Miroslav Novak
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: