Uploaded image for project: 'Project Quay'
  1. Project Quay
  2. PROJQUAY-3671

Quay 3.7.0 APP POD was crashed when OCP is FIPS enabled

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • quay-v3.7.0
    • quay-v3.7.0
    • quay
    • False
    • None
    • False
    • 0

    Description

      Description:

      This is an issue found when deploy quay 3.7.0 on OCP with FIPS enabled, after created QuayRegistry CR, Quay APP POD was crashed with following error message, see attached logs quayregistry-quay-app-84567d8db5-8wchp-quay-app.log 

      "boringcrypto: unexpected code execution in config-tool
      panic: boringcrypto: invalid code execution"

      Quay Image: quay-operator-bundle-container-v3.7.0-94

      oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.10.0-0.nightly-2022-04-24-083512   True        False         6h25m   Cluster version is 4.10.0-0.nightly-2022-04-24-083512
      
      sh-4.4# fips-mode-setup --check
      FIPS mode is enabled.
      
      oc get pod
      NAME                                               READY   STATUS                  RESTARTS         AGE
      quay-operator.v3.7.0-69fdcfc55d-bglw9              1/1     Running                 0                6h7m
      quayregistry-clair-app-86dc648865-24t77            1/1     Running                 0                56s
      quayregistry-clair-app-86dc648865-nszn6            1/1     Running                 0                7m11s
      quayregistry-clair-app-86dc648865-vqdz4            1/1     Running                 0                56s
      quayregistry-clair-app-86dc648865-vqlsl            1/1     Running                 0                6h3m
      quayregistry-clair-postgres-65f47d8965-xtqj5       1/1     Running                 1 (6h3m ago)     6h3m
      quayregistry-quay-app-84567d8db5-8wchp             0/1     CrashLoopBackOff        77 (5m6s ago)    6h2m
      quayregistry-quay-app-84567d8db5-ps446             0/1     CrashLoopBackOff        76 (5m3s ago)    6h2m
      quayregistry-quay-app-upgrade-nz9jr                0/1     Completed               0                6h3m
      quayregistry-quay-config-editor-5547c9759b-5n2fg   1/1     Running                 0                6h3m
      quayregistry-quay-database-8c6dd4969-k9vjw         1/1     Running                 0                6h3m
      quayregistry-quay-mirror-6cf6bcd695-x65dv          0/1     Init:CrashLoopBackOff   53 (28s ago)     6h3m
      quayregistry-quay-mirror-6cf6bcd695-xpqc2          0/1     Init:0/1                53 (6m52s ago)   6h3m
      quayregistry-quay-redis-766bb9d589-89xz7           1/1     Running                 0                6h3m 

      Quay APP POD logs:

      Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh'
      Validating Configuration
      plpgsql
      pg_trgm
      boringcrypto: unexpected code execution in config-tool
      panic: boringcrypto: invalid code executiongoroutine 1 [running]:
      crypto/internal/boring.UnreachableExceptTests()
          /usr/lib/golang/src/crypto/internal/boring/boring.go:118 +0x14b
      crypto/hmac.New(0x106bbd0, 0xc0005aa0c0, 0x2c, 0x30, 0x8, 0x8)
          /usr/lib/golang/src/crypto/hmac/hmac.go:137 +0x38d
      github.com/minio/minio-go/v7/pkg/signer.sumHMAC(0xc0005aa0c0, 0x2c, 0x30, 0xc0004cf6f8, 0x8, 0x8, 0x8, 0x40, 0xc0005d5c20)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/pkg/signer/utils.go:40 +0x4f
      github.com/minio/minio-go/v7/pkg/signer.getSigningKey(0xc00012c270, 0x28, 0xc000485f8b, 0x9, 0x2e9429f7, 0xed9f862a2, 0x0, 0x100b1e4, 0x2, 0xc0005d5c20, ...)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/pkg/signer/request-signature-v4.go:68 +0x13d
      github.com/minio/minio-go/v7/pkg/signer.signV4(0x100bbdc, 0x4, 0xc000721320, 0x10109c1, 0x8, 0x1, 0x1, 0xc0004b1140, 0x0, 0x0, ...)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/pkg/signer/request-signature-v4.go:289 +0x4f4
      github.com/minio/minio-go/v7/pkg/signer.SignV4(...)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/pkg/signer/request-signature-v4.go:317
      github.com/minio/minio-go/v7.Client.newRequest(0xc000720fc0, 0xc0005e06c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0004b1050, 0xc00059fbc0, ...)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/api.go:803 +0x97f
      github.com/minio/minio-go/v7.Client.executeMethod(0xc000720fc0, 0xc0005e06c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0004b1050, 0xc00059fbc0, ...)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/api.go:565 +0x298
      github.com/minio/minio-go/v7.Client.BucketExists(0xc000720fc0, 0xc0005e06c0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc0004b1050, 0xc00059fbc0, ...)
          /remote-source/config-tool/app/vendor/github.com/minio/minio-go/v7/api-stat.go:37 +0x1b8
      github.com/quay/config-tool/pkg/lib/shared.validateMinioGateway(0x7ffc039fdbe4, 0x6, 0xc000591890, 0xc00042e870, 0x7, 0xc000500780, 0x1a, 0xc000400630, 0x14, 0xc00012c270, ...)
          /remote-source/config-tool/app/pkg/lib/shared/storage_validators.go:353 +0x3d4
      github.com/quay/config-tool/pkg/lib/shared.ValidateStorage(0x7ffc039fdbe4, 0x6, 0xc000591890, 0xc00042e870, 0x7, 0xc00042e880, 0x9, 0xc000102c00, 0x101b25f, 0x12, ...)
          /remote-source/config-tool/app/pkg/lib/shared/storage_validators.go:143 +0x829
      github.com/quay/config-tool/pkg/lib/fieldgroups/distributedstorage.(*DistributedStorageFieldGroup).Validate(0xc000128c40, 0x7ffc039fdbe4, 0x6, 0xc000591890, 0xc000159c70, 0x1, 0x1)
          /remote-source/config-tool/app/pkg/lib/fieldgroups/distributedstorage/distributedstorage_validator.go:42 +0x2c5
      github.com/quay/config-tool/commands.glob..func3(0x2031fc0, 0xc000128a40, 0x0, 0x4)
          /remote-source/config-tool/app/commands/validate.go:102 +0x5a3
      github.com/spf13/cobra.(*Command).execute(0x2031fc0, 0xc000128a00, 0x4, 0x4, 0x2031fc0, 0xc000128a00)
          /remote-source/config-tool/app/vendor/github.com/spf13/cobra/command.go:846 +0x2c2
      github.com/spf13/cobra.(*Command).ExecuteC(0x2031d20, 0x40c7c5, 0xc000092058, 0x405ca0)
          /remote-source/config-tool/app/vendor/github.com/spf13/cobra/command.go:950 +0x375
      github.com/spf13/cobra.(*Command).Execute(...)
          /remote-source/config-tool/app/vendor/github.com/spf13/cobra/command.go:887
      github.com/quay/config-tool/commands.Execute()
          /remote-source/config-tool/app/commands/root.go:37 +0x2d
      main.main()
          /remote-source/config-tool/app/cmd/config-tool/main.go:21 +0x25 

      Attachments

        Issue Links

          Activity

            People

              doconnor@redhat.com Dave O'Connor
              lzha1981 luffy zhang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: