+++ This bug was initially created as a clone of Bug #2242214 +++

Description of problem:

After upgrading to 6.13, this warning message started happening on Candlepin.

The error.log has below traceback:

2023-09-27 00:00:00,786 [thread=https-jsse-nio-127.0.0.1-23443-exec-93] [req=d8b0e463-dadc-4521-8da1-856ef5e22509, org=UBCITServices, csid=] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error Text 'Sun, 11 Jun 2023 17:51:29 GMT' could not be parsed at index 0 at java.time.format.DateTimeFormatter.parseResolved0:2,046
java.time.format.DateTimeParseException: Text 'Sun, 11 Jun 2023 17:51:29 GMT' could not be parsed at index 0
        at java.base/java.time.format.DateTimeFormatter.parseResolved0(DateTimeFormatter.java:2046)
        at java.base/java.time.format.DateTimeFormatter.parse(DateTimeFormatter.java:1948)
        at java.base/java.time.ZonedDateTime.parse(ZonedDateTime.java:598)
        at org.candlepin.resource.ConsumerResource.getContentAccessBody(ConsumerResource.java:2137)
        at jdk.internal.reflect.GeneratedMethodAccessor212.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:151)
        at org.jboss.resteasy.core.MethodInjectorImpl.lambda$invoke$3(MethodInjectorImpl.java:122)
        at java.base/java.util.concurrent.CompletableFuture.uniApplyNow(CompletableFuture.java:680)
        at java.base/java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:658)
        at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:2094)
        at java.base/java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:143)
 


Version-Release number of selected component (if applicable):

candlepin-4.2.15-1.el8sat.noarch

How reproducible:

Hard

Steps to Reproduce:

Only in some of the customer environments.

Actual results:

This warning message is filling up rhsm.log on the candlepin error.log as well as all the content hosts rhsm.log files.

Expected results:

Should have no such warning

Additional info:

--- Additional comment from  on 2023-10-05T01:17:49Z 

Could be related to this commit?

commit 96567c879d4b9309ea3634304eb046126349fee8
Author: sbakaj <sbakaj@redhat.com>
Date:   Tue Jul 12 15:34:47 2022 +0000

    ENT-4809: Enable bean validation.

    - Added reasteasy validator
    - Downgrade hibernate validator
    - Fix tests

diff --git a/api/candlepin-api-spec.yaml b/api/candlepin-api-spec.yaml
index 68ccdadf2..732b3a326 100644
--- a/api/candlepin-api-spec.yaml
+++ b/api/candlepin-api-spec.yaml
@@ -1021,8 +1021,9 @@ paths:
           description: Modified date. Accepted format EEE, dd MMM yyyy HH:mm:ss z
           schema:
             type: string
-            format: date-time
-            pattern: EEE, dd MMM yyyy HH:mm:ss z
+#            FIXME It wont work on older version resteasy validation.
+#            format: date-time
+#            pattern: EEE, dd MMM yyyy HH:mm:ss z

The weird part is that I can't reproduce it..

--- Additional comment from  on 2023-10-05T11:51:06Z 

The customer has shared the backup of his Satellite over support case 03627694 which can be downloaded from : https://access.redhat.com/support/cases/#/case/03627694

--- Additional comment from  on 2023-10-05T14:07:49Z 

I tried to build a reproducer and below are the machine details:

IP Address: 10.74.195.69
Root Password: redhat

The backup shared by customer is copied into "/var/satellite_backup/satellite-backup-2023-10-05-12-17-51" directory. I tried to restore the backup but it is getting failed:

~~~~~~~~~~
# satellite-maintain restore /var/satellite_backup/satellite-backup-2023-10-05-12-17-51
Are you sure you want to continue? This will drop the databases, reset all configurations that you have made and bring all application data back to a fresh install. [y/n]
Failed to start services: Job for postgresql.service failed because the control process exited with error code.
See "systemctl status postgresql.service" and "journalctl -xe" for details.
2023-10-05 18:49:41 [ERROR ] [root] Failed to start services: Job for postgresql.service failed because the control process exited with error code.
See "systemctl status postgresql.service" and "journalctl -xe" for details.
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.
~~~~~~~~~~~

We will be further working on it to fix it.

--- Additional comment from  on 2023-10-06T01:32:29Z 

(In reply to Jayant Bhatia from comment #3)
> I tried to build a reproducer and below are the machine details:
> 
> IP Address: 10.74.195.69
> Root Password: redhat
> 

Hmm, I can't reproduce the issue here.

--- Additional comment from  on 2023-10-06T06:23:21Z 

This is also working:

~~~
curl -k -u "admin:redhat" -H "If-Modified-Since: Wed, 04 Oct 2023 12:55:48 GMT" https://$(hostname)/rhsm/consumers/1d95fac6-e9cf-4574-8641-bc499ae1c791/accessible_content
{"lastUpdate":"2023-10-06T06:17:09+0000","contentListing":{"6346998154475823378":["-----BEGIN CERTIFICATE-----\nMIIGGjCCBAKgAwIBAgIIWBURMZOXYRIwDQYJKoZIhvcNAQELBQAwfzELMAkGA1UE\nBhMCVVMxFzAVBgNVBAgMDk5vcnRoIENhcm9saW5hMRAwDgYDVQQHDAdSYWxlaWdo\nMRAwDgYDVQQKDAdLYXRlbGxvMRQwEgYDVQQLDAtTb21lT3JnVW5pdDEdMBsGA1UE\nAwwUY2FyZG9uLnF1YWxpZi5jZWEuZnIwHhcNMjMxMDA0MTE1NjA2WhcNMjQxMDA0\nMTE1NjA2WjA9MQwwCgYDVQQKDANEU0kxLTArBgNVBAMMJDFkOTVmYWM2LWU5Y2Yt\nNDU3NC04NjQxLWJjNDk5YWUxYzc5MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCC\nAgoCggIBAKzyWskjqosbvjAB23uULapEz32IF9aqBUca9V0upScQcyRTeYfyWQn0\nvrlIZazSS/i6ys3IZfAcdZZ+yPlggl7SW2SmfzjcBmCf0ojFyy+ArVAp6SHzKNPc\nFLyoSq9zHpcZPCrA/kd2+hCqI7TBnPOgBYL0uIY2LrXHMys28puPDjJa+xk/4wDF\nCmVqpBbeUobn0OnIIRTCOQifeYQz8WvRX/oStut1fWJrMRxbd4LX7T1BCnvwCPep\nMA2eH+VcApEJlUY2b0UVIkcq/iD8jqWEmiIQhhtlQwZ8tkfZD4RkwGJ8QdbkS8ox\nlPweaDQ1WjIbV6jl57Gich9nIZuJk7x5WiAd2UDM6dztIzoI6R0+Gn92yU49t8/c\nuxyBVhIzCjQlto9m0BN3LiCcJZwyQIHOxFkf9+1KnurSi171WNChuVRYO27IDNDy\n0qGdRMaZmpW3I9rKR8ll+zIr6Ob7DoF/Qknz/KFrDpIQ4V7wMGkzgagZQhaAELU0\nGci4SUYBOq+Jp3IMntY8kp4ZZSG3GWrsnKhI0PMtymQ0znFSCEL0V26hEshjGxKO\nbtHzlHXQkAlh70rltNwGIvEQq0XpqiqRW3eeA4c1uakEXQMwTOR9lG+2haaquY9A\nsUAgRJLpr/0Sip1TXUwlUya74cfLvcxXbxVW8KlwE4z1/yJp13KfAgMBAAGjgdsw\ngdgwDgYDVR0PAQH/BAQDAgSwMBMGA1UdJQQMMAoGCCsGAQUFBwMCMAkGA1UdEwQC\nMAAwEQYJYIZIAYb4QgEBBAQDAgWgMB0GA1UdDgQWBBSI+aHRA+JwgkjpeKRwsn1j\nA+cTRDAfBgNVHSMEGDAWgBQFxQf16GUYSMN3WZSI3DEAEi6aGDASBgkrBgEEAZII\nCQYEBQwDMy40MBcGCSsGAQQBkggJCAQKDAhPcmdMZXZlbDAmBgkrBgEEAZIICQcE\nGQQXeNpzCfZk8MlMKkosqmQAABc4A7YDpgAwDQYJKoZIhvcNAQELBQADggIBAOV+\nnQds/jDQesx9t2q1tuCzW0GSPCvZfhQKK6LOBhIQ581UnH50C2CmUBI+Lb3cXpBf\nYDPXQeMzady+pNPxWQN8ebNeJuCX8QLdPY0tooAAY67j+IHo6lyA/KOhn/cJRHlt\nBXg7aoMMGhG0o2oGszDoCrMcm38mhxuF9de6cNnkaJ3XIR+UAoARWUBNq3MRFGFg\n+ttm+gwnSa1kak5SenDdW+bdAqpSRrF7ZV47RkAW0Sbo2dDyxfD73LFF6yUxH0M+\np2nFAgRDR+MsyNY4gMRRPQ6zYhxDcK+ID7vJOMmuBK/qwV9IQnbRrxB2dfTuAIfI\nSTNu6j2+eWNW0V0Z5ub7HksJn1n4zPezvyuKJkulp2zCHE8dizC1rTmdxdLgY+u3\nhHjji60T7erp5R7U3cobLfHvHq4c35ZSdNJuQe5x8ZSgU/Tpw0teEF0lGnuLcZg3\nT5HolpjS9pD6YJaDpvAKbyf9D8IznB03+VbEgVklImSOgSFRXgcpS49mcw2rSmSo\nsKg5vRTK/TZsoajNGiOOQQ109UCXk4Mc3Ong81YhSBpebsW8RBJ4QuOfG6zFmdqa\nBe6E2THKV+QFcHNWjxB4eChT2mrre2P2eL5JxMQ7txeSh1RAiZZVDCYjzzacDNfX\nEMkXgzd61Yjr0ADlHJrqK1lAExRok+/zJPt1iQnJ\n-----END CERTIFICATE-----\n","-----BEGIN ENTITLEMENT DATA-----\neJzVVd9P2zAQ/lcsiweQajm/E/dtYwimMQ3RvWwTitzkSiPSJLOdiarq/75L0tAy\nWlABCaa+pGffd999d75b0KQsdD0DRYfUToU/kUnAQCQT5vmhx6LAs9k48YSQYCeh\nsOmA6nqsE5VVJisLOlxQfVOjM+IYKEwskwS0xmuFnAHajzs7+dDZlwNaqrQJh45G\nKoNXHMtxmW0xK/huBUM7HFriJwJAke46RJRKlWmdGE2HvxY0Sx9hQP6hMKB/QOmW\nPMU/UiXTzEBiagUN2tWgR1pD207guHjXzKsGcV7P1vCXkJIzachIGshzRCLHedaE\nC8ikVOTy7OScROQ2CuLAI4eXF1/1ETrncgw5euveiyWtFwsYejE1hZxFrPNiqpp1\ntIu0VOuQaKqkmaKBfxp95ufZWEk15yv2PM204bmcg4KUN4AR7/A4Bl2F4wEvdau1\nHOeAqU5krqETpVWDdi4UVbmuruNaNaQnWQ5DzjmYhFc3GUd+DE85JsdOL07Zl5Mf\nDINOMYyCHKQGDDEDI1NpZAy3VaZQOXtAFfyu8TuNjbxuo93Lm14tB30BbMsTTxTg\nBNNWlco0kPOsqG+JaAuwUp6Rj8jj2+hhCdqYopV9pfcYb5b6JbI3mIIfrLLHfuul\n76A70d9UZLFFZOzy4AVdLp7V5eKVu1z8J12+rQCBCDzHi2zb8ZwdhTCgjbMhLWoS\nN7a4P7jT77jWppztkq895I0T7z3v5Ljalsc9lr7vh65ji11DsYF0d7F0n83S3Y+l\nZ3s4t33XeoTlLpLP5rgHRccL/P3f2+HdgwvJCBQOl6MdQy1kuj1nW97fC4cb75B5\nyMOOAz9oRluT+Xt/eL0qGw8v9OynBt+D7RLtsV2i198u0bveLltWuOP5e2vcN/hT\n/f1qzbyp6bqh317PjZ7FH6ZVlhh1sVz+Bb3Txns=\n-----END ENTITLEMENT DATA-----\n-----BEGIN RSA SIGNATURE-----\nVIQLgMlVTEPiNoB4Bl5VNBNEuHzfeKSKB5LHa8+xBf91zPktftOY6byvdMwP+zHc\nn9ruX+y+qNNMohQw+HXYk/pKGchMZ3OEycJe+cWwZ1kevGYGtOhYiG0pfQN1FRZ3\n11LnKCg8h9DB0hiWaYBBR5VV5+5L5P1ReZGBd+YBYM50KceJGx3rdJ19yTD5TH3p\nZG772E8Sb3jUpCvrMImtQHMi9Q7+QetVYbAJweIR95HWQl4DilqBo85b1/E7Rron\n6U/cBm2MT3MMODO7qzlBukoubg12j6LWSRD0RYp2nF6VV80iS+p85KfcKK97gNG2\n1V6sMbCHiN0H+567BkB+zbcmGks2pe2ZB58X6prt7/VlFw1h6nCJyl7BYJ0XjLoK\nv+vgxcIXrZv9bhLYD8AYISNqopCdZvdz4u08j8Fpjm0MEljZ6cv4rZLsJdeaOsLv\nc2MlpQx51eaFc6HEkdKotetvKwHSU92u1d0V9GdQMeOyxPnIG+Ph5ol0URkoksn6\nUmhcEQEr+LBZBqN7SrwB4+u66+Bfjp4GxZXf9QIS1riUnoWHQCiEYvdEq3H1i0GU\n2HPq7b3B6OBEZKT4G5VzPZdBK8dG3kPmEoQyFUIpdPaspFiTJaarf+46XvlQksnD\nsxSF6+u4/+PiFU9awh0vF8WDWkGs4XXpJMVhW3wYhFs=\n-----END RSA SIGNATURE-----\n"]}}
`~~

The cert includes the new custom repository too. So what could cause this error?

~~~
Error Text 'Sun, 11 Jun 2023 17:51:29 GMT' could not be parsed at index 0 at java.time.format.DateTimeFormatter.parseResolved0:2,046
~~

is system locale related?

~~~
cat etc/locale.conf
LANG="fr_FR.UTF-8"
~~~

--- Additional comment from  on 2023-10-23T14:08:05Z 

*** Bug 2244861 has been marked as a duplicate of this bug. ***

--- Additional comment from  on 2023-10-24T14:28:03Z 

Thanks for those release. Do you know the time required for the rpm to be available ? Or we need to do the upgrade to 6.13.5, 6.14 or 7 (both seems ok for me). No time pressure we have a workaround applied.

QE Tracker for https://issues.redhat.com/browse/SAT-21389
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=2250347