Loading...

Details

Type: Bug
Resolution: Done
Priority: Blocker
Fix Version/s: 7.2.1.GA
Affects Version/s: 7.2.1.GA
Component/s: OpenShift
Labels:
None

Bugzilla References:
https://bugzilla.redhat.com/show_bug.cgi?id=1709869
Target Release:

7.2.z.GA
Steps to Reproduce:
Hide

Use following job to provision OCP 4.1 instance on AWS:
https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-create-openshift-4-cluster/

Follow those steps to run test against OCP 4.1 cluster:

git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/openshift-eap-tests.git cd openshift-eap-tests git checkout ocp_41 # create test.properties file and set correct *.url and *.token xtf.openshift.url=https://api.eap-qe-ocp41-cluster7.eap-qe-ocp41-cluster7.fw.rhcloud.com:6443 xtf.openshift.namespace=test-namespace xtf.openshift.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo xtf.openshift.admin.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo xtf.openshift.version=4.1.0-201904090034.git.0.1016bda.el7 Start test by: mvn clean test -P72 -Dtest=JtaCrashRecQuickstartTest#testJTAXaRecovery -Dxtf.version=0.12-SNAPSHOT
Show
Use following job to provision OCP 4.1 instance on AWS: https://eap-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/eap-7.x-create-openshift-4-cluster/ Follow those steps to run test against OCP 4.1 cluster: git clone git@gitlab.mw.lab.eng.bos.redhat.com:jbossqe-eap/openshift-eap-tests.git cd openshift-eap-tests git checkout ocp_41 # create test.properties file and set correct *.url and *.token xtf.openshift.url=https: //api.eap-qe-ocp41-cluster7.eap-qe-ocp41-cluster7.fw.rhcloud.com:6443 xtf.openshift.namespace=test-namespace xtf.openshift.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo xtf.openshift.admin.token=dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo xtf.openshift.version=4.1.0-201904090034.git.0.1016bda.el7 Start test by: mvn clean test -P72 -Dtest=JtaCrashRecQuickstartTest#testJTAXaRecovery -Dxtf.version=0.12-SNAPSHOT

SFDC Cases Counter:
SFDC Cases Links:

Description

This issue was hit during testing of eap72 image on OCP 4.1.

EAP migration pod for recovery of XA transactions fails on OCP 4.1 with following error:

INFO [Tue May 14 08:37:50 UTC 2019] Examining existence of living pod 'eap-app-1-6jdxn'
CRITICAL:__main__:Cannot query OpenShift API for "https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods"
Traceback (most recent call last):
  File "/opt/partition/queryosapi.py", line 151, in <module>
    queryResult = getLivingPods()
  File "/opt/partition/queryosapi.py", line 109, in getLivingPods
    jsonPodsData = getPodsJsonData()
  File "/opt/partition/queryosapi.py", line 97, in getPodsJsonData
    jsonText = OpenShiftQuery.queryApi('/api/v1/namespaces/{}/pods'.format(OpenShiftQuery.getNameSpace()))
  File "/opt/partition/queryosapi.py", line 89, in queryApi
    return urllib2.urlopen(request, cafile = OpenShiftQuery.CERT_FILE_PATH).read()
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1263, in connect
    ssl.match_hostname(self.sock.getpeercert(), server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 267, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '172.30.0.1'
[0;33mWARN [Tue May 14 08:37:55 UTC 2019] Can't get list of living pods[0m
INFO [Tue May 14 08:37:55 UTC 2019] Finished migration check cycle, pausing for 30 seconds before resuming
CRITICAL:__main__:Cannot query OpenShift API for "https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods/eap-app-migration-1-rln94/log?timestamps=true&tailLines=1"
Traceback (most recent call last):
  File "/opt/partition/queryosapi.py", line 159, in <module>
    queryResult = getLog(podName, sinceTime, tailLine)
  File "/opt/partition/queryosapi.py", line 122, in getLog
    .format(OpenShiftQuery.getNameSpace(), podName, sinceTimeParam, tailLineParam))
  File "/opt/partition/queryosapi.py", line 89, in queryApi
    return urllib2.urlopen(request, cafile = OpenShiftQuery.CERT_FILE_PATH).read()
  File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1211, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1041, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1075, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1037, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 881, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 843, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 1263, in connect
    ssl.match_hostname(self.sock.getpeercert(), server_hostname)
  File "/usr/lib64/python2.7/ssl.py", line 267, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '172.30.0.1'

Problem is that domain openshift.default.svc is not present in {{/var/run/secrets/kubernetes.io/serviceaccount/ca.crt }} file. This file is passed to migration pod and used for authentication in API calls in python script.

So as following command fails:
curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo" 'https://openshift.default.svc/api/v1/namespaces/mnovak-mynamespace/pods'

this on passes:
curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer dBNsUYunIES1nIB0yfL-NNSE51Nk_cRXcDSzKz2dEuo" 'https://kubernetes.default.svc/api/v1/namespaces/mnovak-mynamespace/pods'

This appears to be more OCP 4.1 issue than problem in migration pod as ca.crt does no longer contains openshift.default.svc.

OCP 4.1 - EAP 72 migration pod for transactions fails due to: "ssl.CertificateError: hostname 'openshift.default.svc' doesn't match either of 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc',..."

Details

Description

Attachments

Activity

People

Dates