Loading...

Type: Bug
Resolution: Not a Bug
Priority: Critical
Fix Version/s: None
Affects Version/s: rhel-9.6
Component/s: mod_md
Labels:
None

Regression:
No
Severity:
Important

AssignedTeam:
rhel-stacks-web-servers

Story Points:
0
Blocked:
False
Ready:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None

Preliminary Testing:
None
Test Coverage:
None

ProdDocsReview-CCS:
Unspecified
ProdDocsReview-Dev:
Unspecified
ProdDocsReview-QE:
Unspecified

Experience:

PX Impact Score:
PX Technical Impact:
PX Impact Range:
PX Review Complete:
SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

A customer is getting the below messages when using the md httpd module (and with SELinux in permissive mode):

Mon Jul 07 08:22:20.742584 2025] [md:trace2] [pid 93414:tid 93415] md_crypt.c(143): initializing RAND
[Mon Jul 07 08:22:20.742734 2025] [md:trace3] [pid 93414:tid 93415] md_store_fs.c(464): (13)Permission denied: loading type 1 from /var/lib/httpd/md/staging/mdcphd.mdc-berlin.de/job.json
[Mon Jul 07 08:22:20.742744 2025] [md:debug] [pid 93414:tid 93415] mod_md_drive.c(108): AH10052: md(mdcphd.mdc-berlin.de): state=1, driving
[Mon Jul 07 08:22:20.742776 2025] [md:trace1] [pid 93414:tid 93415] md_acme_drive.c(531): mdcphd.mdc-berlin.de: init_base driver
[Mon Jul 07 08:22:20.742797 2025] [md:trace3] [pid 93414:tid 93415] md_store_fs.c(526): (13)Permission denied: mk_group_dir /var/lib/httpd/md/staging perm set
[Mon Jul 07 08:22:20.742801 2025] [md:error] [pid 93414:tid 93415] (13)Permission denied: mk_group_dir 4 /var/lib/httpd/md/staging
[Mon Jul 07 08:22:20.742805 2025] [md:trace1] [pid 93414:tid 93415] md_acme_drive.c(613): mdcphd.mdc-berlin.de: init driver
[Mon Jul 07 08:22:20.742809 2025] [md:debug] [pid 93414:tid 93415] md_reg.c(1113): mdcphd.mdc-berlin.de: init done
[Mon Jul 07 08:22:20.742812 2025] [md:debug] [pid 93414:tid 93415] md_reg.c(1158): mdcphd.mdc-berlin.de: run staging
[Mon Jul 07 08:22:20.742819 2025] [md:trace3] [pid 93414:tid 93415] md_store_fs.c(464): (13)Permission denied: loading type 1 from /var/lib/httpd/md/staging/mdcphd.mdc-berlin.de/md.json
[Mon Jul 07 08:22:20.742824 2025] [md:debug] [pid 93414:tid 93415] md_acme_drive.c(714): mdcphd.mdc-berlin.de: staging started, state=1, attempt=0, acme=https://acme-v02.harica.gr/acme/f9a596ca-2a8d-4253-b0bb-c02b007e3f20/directory, challenges='http-01'
[Mon Jul 07 08:22:20.742830 2025] [md:debug] [pid 93414:tid 93415] md_acme_drive.c(752): mdcphd.mdc-berlin.de: setup staging
[Mon Jul 07 08:22:20.742837 2025] [md:trace2] [pid 93414:tid 93415] md_store_fs.c(758): (13)Permission denied: purge staging/mdcphd.mdc-berlin.de (/var/lib/httpd/md/staging/mdcphd.mdc-berlin.de)
[Mon Jul 07 08:22:20.742857 2025] [md:trace3] [pid 93414:tid 93415] md_store_fs.c(526): (13)Permission denied: mk_group_dir /var/lib/httpd/md/staging perm set
[Mon Jul 07 08:22:20.742861 2025] [md:error] [pid 93414:tid 93415] (13)Permission denied: mk_group_dir 4 /var/lib/httpd/md/staging
[Mon Jul 07 08:22:20.742871 2025] [md:error] [pid 93414:tid 93415] (13)Permission denied: md[mdcphd.mdc-berlin.de] while[Resetting staging for mdcphd.mdc-berlin.de] detail[Saving MD information in staging area.]
[Mon Jul 07 08:22:20.742875 2025] [md:debug] [pid 93414:tid 93415] md_result.c(254): (13)Permission denied: md[mdcphd.mdc-berlin.de] while[Resetting staging for mdcphd.mdc-berlin.de] detail[Saving MD information in staging area.]
[Mon Jul 07 08:22:20.742879 2025] [md:debug] [pid 93414:tid 93415] md_reg.c(1164): (13)Permission denied: mdcphd.mdc-berlin.de: staging done
[Mon Jul 07 08:22:20.742922 2025] [md:error] [pid 93414:tid 93415] (13)Permission denied: AH10056: processing mdcphd.mdc-berlin.de: Saving MD information in staging area.
[Mon Jul 07 08:22:20.742932 2025] [md:info] [pid 93414:tid 93415] AH10057: mdcphd.mdc-berlin.de: encountered error for the 1. time, next run in 6 seconds
[Mon Jul 07 08:22:20.742941 2025] [md:trace3] [pid 93414:tid 93415] md_store_fs.c(526): (13)Permission denied: mk_group_dir /var/lib/httpd/md/staging perm set
[Mon Jul 07 08:22:20.742954 2025] [md:error] [pid 93414:tid 93415] (13)Permission denied: mk_group_dir 4 /var/lib/httpd/md/staging
[Mon Jul 07 08:22:20.742958 2025] [md:trace1] [pid 93414:tid 93415] mod_md_drive.c(173): (13)Permission denied: mdcphd.mdc-berlin.de: saving job props
[Mon Jul 07 08:22:20.742961 2025] [md:debug] [pid 93414:tid 93415] mod_md_drive.c(237): AH10107: next run in 6 seconds

They said that after updating to RHEL 9.6 the error started to occur (I asked more info about the "working" env but they didn't respond, but if necessary I will ask it again).

This is the customer configuration (let me know if you need the sosreport):

LogLevel debug md:trace5
MDStoreDir "/var/lib/httpd/md"
MDNotifyCmd /opt/mdc/bin/apache_acme
<MDomain mdcphd.mdc-berlin.de mdcphd-test.mdc-berlin.net sl-mdc-p-phd1.mdc-berlin.de>
  MDExternalAccountBinding /etc/acme_key
  MDCertificateAuthority https://acme-v02.harica.gr/acme/f9a596ca-2a8d-4253-b0bb-c02b007e3f20/directory 
  MDCertificateAgreement accepted
  # Muss niemand lesen.
  MDCertificateStatus Off
  # X448 oder X25519 derzeit nicht implementiert
  MDPrivateKeys rsa4096
  MDRenewWindow 50%
  MDContactEmail webmaster@mdc-berlin.de
  MDStapling On
  MDMustStaple On
</MDomain>

Following the folder structure after asking to backup and delete the "/var/lib/httpd/md" folder:

$ tree -pug /var/lib/httpd/md -L 3
/var/lib/httpd/md
├── [drwxr-x--- root     root    ]  accounts
├── [drwxr-x--- apache   root    ]  challenges
├── [drwx------ root     root    ]  domains
│   └── [drwx------ root     root    ]  mdcphd.mdc-berlin.de
│       ├── [-rw------- root     root    ]  fallback-privkey.pem
│       ├── [-rw------- root     root    ]  fallback-pubcert.pem
│       └── [-rw------- root     root    ]  md.json
├── [-rw------- root     root    ]  md_store.json
├── [drwxr-x--- apache   root    ]  ocsp
└── [drwxr-x--- apache   root    ]  staging

Previously they had:

sl-mdc-t-phd1 [test] ~ # tree -pug /var/lib/httpd/md -L 3
/var/lib/httpd/md
├── [drwxr-xr-x root     root    ]  accounts
│   └── [drwxr-xr-x root     root    ]  ACME-me-v02.harica.gr-0000
│       ├── [-rw-r--r-- root     root    ]  account.json
│       └── [-rw-r--r-- root     root    ]  account.pem
├── [drwx------ root     root    ]  archive
│   └── [drwx------ root     root    ]  mdcphd.mdc-berlin.de.1
│       ├── [-rw------- root     root    ]  fallback-privkey.pem
│       ├── [-rw------- root     root    ]  fallback-pubcert.pem
│       └── [-rw------- root     root    ]  md.json
├── [drwxr-x--- apache   root    ]  challenges
├── [drwx------ root     root    ]  domains
│   └── [drwx------ root     root    ]  mdcphd.mdc-berlin.de
│       ├── [-rw------- root     root    ]  job.json
│       ├── [-rw------- root     root    ]  md.json
│       ├── [-rw------- root     root    ]  privkey.pem
│       └── [-rw------- root     root    ]  pubcert.pem
├── [-rw------- root     root    ]  md_store.json
├── [drwxr-xr-x apache   root    ]  ocsp
│   └── [drwxr-xr-x apache   apache  ]  mdcphd.mdc-berlin.de
│       ├── [-rw-r--r-- apache   apache  ]  job.json
│       └── [-rw-r--r-- apache   apache  ]  ocsp-c65e3a4e50576c61d84d2a64c5211e6fa77b588d.json
├── [drwxr-xr-x apache   root    ]  staging
│   └── [drwxr-xr-x apache   apache  ]  mdcphd.mdc-berlin.de
│       └── [-rw-r--r-- apache   apache  ]  md.json
└── [drwx------ root     root    ]  tmp
    └── [drwx------ root     root    ]  mdcphd.mdc-berlin.de
        └── [-rw------- root     root    ]  job.json

the httpd packages:

httpd.x86_64 2.4.62-4.el9 @rhel-9-for-x86_64-appstream-rpms httpd-core.x86_64 2.4.62-4.el9 @rhel-9-for-x86_64-appstream-rpms httpd-filesystem.noarch 2.4.62-4.el9 @rhel-9-for-x86_64-appstream-rpms httpd-tools.x86_64 2.4.62-4.el9 @rhel-9-for-x86_64-appstream-rpms
mod_md.x86_64 1:2.4.26-1.el9 @rhel-9-for-x86_64-appstream-rpms mod_ssl.x86_64 1:2.4.62-4.el9 @rhel-9-for-x86_64-appstream-rpms

Let me know if you need other info.

Kind Regards

Luca

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

httpd-04186459.tar.xz
13 kB
2025/07/07 9:44 AM

Details

Description

Attachments

Attachments

Easy Agile Planning Poker

Activity

People

Dates