Uploaded image for project: 'Cloud Enablement'
  1. Cloud Enablement
  2. CLOUD-3172

readinessProve / livenessProbe fails with "'ascii' codec can't encode characters" when the locale supports non-ascii characters

XMLWordPrintable

    • Hide

      Specify the following system property. This can output log messages in English. Hence, "failure description" in DMR is also written only in English:

      -Dorg.jboss.logging.locale=en-US
      

      Note that the specified value should be in the BCP 47 format: https://tools.ietf.org/html/bcp47

      Show
      Specify the following system property. This can output log messages in English. Hence, "failure description" in DMR is also written only in English: -Dorg.jboss.logging.locale=en-US Note that the specified value should be in the BCP 47 format: https://tools.ietf.org/html/bcp47
    • Hide

      1. Deploy eap7.1 pod using eap71-basic-s2i template.

      $ oc new-project reproduce
      $ oc new-app --template=eap71-basic-s2i -p SOURCE_REPOSITORY_URL="https://github.com/nagetsum/eap-debug.git" -p APPLICATION_NAME="echo" -p CONTEXT_DIR="echo" -p SOURCE_REPOSITORY_REF="master"
      

      2. Set non-acsii locale to just created DeploymentConfig.

      $ oc set env dc/echo JAVA_OPTS_APPEND="-Duser.language=ja -Duser.country=JP -Dfile.encoding=UTF8"
      

      3. Check pod "echo-2-w72vh" status, READY 0/1 and state is "Unhealthy" because readinessProbe always failed.

      $ oc get pod
      NAME            READY     STATUS      RESTARTS   AGE
      echo-1-build    0/1       Completed   0          10m
      echo-2-deploy   1/1       Running     0          8m
      echo-2-w72vh    0/1       Running     0          7m
      
      $ oc describe pod echo-1-rq9tv
      (omitted)
      Events:
        Type     Reason     Age               From                                   Message
        ----     ------     ----              ----                                   -------
        Normal   Scheduled  2m                default-scheduler                      Successfully assigned test/echo-1-rq9tv to node01.openshift.example.com
        Normal   Pulling    2m                kubelet, node01.openshift.example.com  pulling image "docker-registry.default.svc:5000/test/echo@sha256:b621c27162a66a8410b204a0a0f0aa0077101c8615a7edb7a872dc46244e4123"
        Normal   Pulled     2m                kubelet, node01.openshift.example.com  Successfully pulled image "docker-registry.default.svc:5000/test/echo@sha256:b621c27162a66a8410b204a0a0f0aa0077101c8615a7edb7a872dc46244e4123"
        Normal   Created    2m                kubelet, node01.openshift.example.com  Created container
        Normal   Started    2m                kubelet, node01.openshift.example.com  Started container
        Warning  Unhealthy  1m (x2 over 1m)   kubelet, node01.openshift.example.com  Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
        Warning  Unhealthy  8s (x5 over 48s)  kubelet, node01.openshift.example.com  Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1
      
      Show
      1. Deploy eap7.1 pod using eap71-basic-s2i template. $ oc new -project reproduce $ oc new -app --template=eap71-basic-s2i -p SOURCE_REPOSITORY_URL= "https: //github.com/nagetsum/eap-debug.git" -p APPLICATION_NAME= "echo" -p CONTEXT_DIR= "echo" -p SOURCE_REPOSITORY_REF= "master" 2. Set non-acsii locale to just created DeploymentConfig. $ oc set env dc/echo JAVA_OPTS_APPEND= "-Duser.language=ja -Duser.country=JP -Dfile.encoding=UTF8" 3. Check pod "echo-2-w72vh" status, READY 0/1 and state is "Unhealthy" because readinessProbe always failed. $ oc get pod NAME READY STATUS RESTARTS AGE echo-1-build 0/1 Completed 0 10m echo-2-deploy 1/1 Running 0 8m echo-2-w72vh 0/1 Running 0 7m $ oc describe pod echo-1-rq9tv (omitted) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default -scheduler Successfully assigned test/echo-1-rq9tv to node01.openshift.example.com Normal Pulling 2m kubelet, node01.openshift.example.com pulling image "docker-registry. default .svc:5000/test/echo@sha256:b621c27162a66a8410b204a0a0f0aa0077101c8615a7edb7a872dc46244e4123" Normal Pulled 2m kubelet, node01.openshift.example.com Successfully pulled image "docker-registry. default .svc:5000/test/echo@sha256:b621c27162a66a8410b204a0a0f0aa0077101c8615a7edb7a872dc46244e4123" Normal Created 2m kubelet, node01.openshift.example.com Created container Normal Started 2m kubelet, node01.openshift.example.com Started container Warning Unhealthy 1m (x2 over 1m) kubelet, node01.openshift.example.com Readiness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1 Warning Unhealthy 8s (x5 over 48s) kubelet, node01.openshift.example.com Liveness probe errored: rpc error: code = Unknown desc = command error: command timed out, stdout: , stderr: , exit code -1

      When EAP 7.1 image running with the locale like ja_JP which outputs non-ascii characters, HealthCheckProbe in readinessProve / livenessProbe fails like the following example:

      sh-4.2$ /bin/bash -c /opt/eap/bin/readinessProbe.sh
      {
          "probe.eap.dmr.EapProbe": {
              "probe.eap.dmr.ServerStatusTest": "running",
              "probe.eap.dmr.DeploymentTest": {
                  "example.war": "OK"
              },
              "probe.eap.dmr.BootErrorsTest": "No boot errors"
          },
          "probe.eap.dmr.HealthCheckProbe": {
              "probe.eap.dmr.HealthCheckTest": "Exception executing test: 'ascii' codec can't encode characters in position 13-16: ordinal not in range(128)"
          }
      }
      

      HealthCheckTest checks the existence of "microprofile-health-smallrye" subsystem and check the healthiness from it:

      https://github.com/jboss-container-images/jboss-eap-modules/blob/master/os-eap-probes/added/probes/probe/eap/dmr.py#L211

      163 class HealthCheckTest(Test):
      164     """
      165     Checks the state of the Health Check subsystem, if installed.
      166     We use a composite with a first step that does a simple read-resource
      167     and a second step that reads the health check status.
      168     A failure in the first step means the subsystem is not present and any
      169     failure in the second step should be ignored as meaningless.
      170     """
      171
      172     def __init__(self):
      173         super(HealthCheckTest, self).__init__(
      174             {
      175                 "operation": "composite",
      176                 "address": [],
      177                 "steps": [
      178                     {
      179                         "operation": "read-resource",
      180                         "address": {
      181                             "subsystem": "microprofile-health-smallrye"
      182                         },
      183                         "recursive" : False
      184                     },
      185                     {
      186                         "operation": "check",
      187                         "address": {
      188                             "subsystem": "microprofile-health-smallrye"
      189                         }
      190                     }
      191                 ]
      192             }
      193         )
       :
      211         if results.get("failure-description") and re.compile("JBAS014883|WFLYCTL0030").search(str(results.get("failure-description"))):
      212             return (Status.READY, "Health Check not configured")
       :
      

      EAP 7.1 does not have "microprofile-health-smallrye" subsystem, so it returns "failure-description". However, when EAP 7.1 starts with "LANG=ja_JP.utf8" (or LANG=zn_CN.utf8), it returns non-ascii characters in "failure-description". For example:

      • With the locale en_US.utf8:
      {
          "outcome" : "failed",
          "result" : {
              "step-1" : null,
              "step-2" : null
          },
          "failure-description" : "WFLYCTL0030: No resource definition is registered for address [(\"subsystem\" => \"microprofile-health-smallrye\")]",
          "rolled-back" : true
      }
      
      • With the locale ja_JP.utf8 (Japanese):
      {
          "outcome" : "failed",
          "result" : {
              "step-1" : null,
              "step-2" : null
          },
          "failure-description" : "WFLYCTL0030: アドレス [(\"subsystem\" => \"microprofile-health-smallrye\")] に対して登録されたリソース定義はありません",
          "rolled-back" : true
      }
      
      • With the locale zn_CN.utf8 (Chinese):
      {
          "outcome" : "failed",
          "result" : {
              "step-1" : null,
              "step-2" : null
          },
          "failure-description" : "WFLYCTL0030: 在地址 [(\"subsystem\" => \"microprofile-health-smallrye\")] 上没有注册资源定义",
          "rolled-back" : true
      }
      

      I would suggest using "encode('utf-8')" instead of "str" in dmr.py. See also https://docs.python.org/2.7/howto/unicode.html#the-unicode-type for details.

      Additional comment:

      $ ipython
      
      In [1]: u1 = u'WFLYCTL0030: No resource definition is registered for address [("subsystem" => "microprofile-health-smallrye")]'
      
      In [2]: str(u1)
      Out[2]: 'WFLYCTL0030: No resource definition is registered for address [("subsystem" => "microprofile-health-smallrye")]'
      
      In [3]: u2 = u'WFLYCTL0030: アドレス [("subsystem" => "microprofile-health-smallrye")] に対して登録されたリソース定義はありません'
      
      In [4]: str(u2)
      ---------------------------------------------------------------------------
      UnicodeEncodeError                        Traceback (most recent call last)
      <ipython-input-4-399d8867784e> in <module>()
      ----> 1 str(u2)
      
      UnicodeEncodeError: 'ascii' codec can't encode characters in position 13-16: ordinal not in range(128)
      
      In [5]: u2.encode('utf-8')
      Out[5]: 'WFLYCTL0030: \xe3\x82\xa2\xe3\x83\x89\xe3\x83\xac\xe3\x82\xb9 [("subsystem" => "microprofile-health-smallrye")] \xe3\x81\xab\xe5\xaf\xbe\xe3\x81\x97\xe3\x81\xa6\xe7\x99\xbb\xe9\x8c\xb2\xe3\x81\x95\xe3\x82\x8c\xe3\x81\x9f\xe3\x83\xaa\xe3\x82\xbd\xe3\x83\xbc\xe3\x82\xb9\xe5\xae\x9a\xe7\xbe\xa9\xe3\x81\xaf\xe3\x81\x82\xe3\x82\x8a\xe3\x81\xbe\xe3\x81\x9b\xe3\x82\x93'
      
      In [6]: type(u2.encode('utf-8'))
      Out[6]: str
      
      In [7]: u3 = u'WFLYCTL0030: 在地址 [(\"subsystem\" => \"microprofile-health-smallrye\")] 上没有注册资源定义'
      
      In [8]: str(u3)
      ---------------------------------------------------------------------------
      UnicodeEncodeError                        Traceback (most recent call last)
      <ipython-input-8-1be681bc93b8> in <module>()
      ----> 1 str(u3)
      
      UnicodeEncodeError: 'ascii' codec can't encode characters in position 13-15: ordinal not in range(128)
      
      In [9]: u3.encode('utf-8')
      Out[9]: 'WFLYCTL0030: \xe5\x9c\xa8\xe5\x9c\xb0\xe5\x9d\x80 [("subsystem" => "microprofile-health-smallrye")] \xe4\xb8\x8a\xe6\xb2\xa1\xe6\x9c\x89\xe6\xb3\xa8\xe5\x86\x8c\xe8\xb5\x84\xe6\xba\x90\xe5\xae\x9a\xe4\xb9\x89'
      
      In [10]: type(u3.encode('utf-8'))
      Out[10]: str
      

              rhn-engineering-lgao Lin Gao
              rhn-support-nagetsum Norito Agetsuma
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: