Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: rhos-18.0.0
Affects Version/s: None
Component/s: cinder-operator, glance-operator
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
?
Fixed in Build:
glance-operator-bundle-container-1.0.0-44
PM Approval:
?
QE Approval:
?
Regression:
None
Release Note Type:
Release Note Not Required
Intelligence Requested:
Market:

Severity:
Critical

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

From the cinder-volume-nfs-0 pod there occur failure in mounting NFS share from NetApp:

Command: mount -t nfs 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a
Exit code: 32
Stdout: ''
stderr: 'mount.nfs: access denied by server while mounting 10.46.246.61:/cinder_nfs'

Surprisingly, this attempting to mount the same from other place, e.g. local laptop, controler-0 or even different pods, it works:

sudo mount -t nfs 10.46.246.61:/cinder_nfs /mnt

However, the netapp policy should allow any client IP:

ntap-rhv-qe-cl::*> vserver export-policy rule show -policyname cinder_nfs
             Policy          Rule    Access   Client                RO
Vserver      Name            Index   Protocol Match                 Rule
------------ --------------- ------  -------- --------------------- ---------
vserver-rhos-qe cinder_nfs   1       any      0.0.0.0/0             any

This needs investigation.

Below is log from service:

2024-05-30 08:53:40.988 2070734 DEBUG oslo_concurrency.processutils [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] 'mount -t nfs 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a' failed. Not Retrying. execute /usr/lib/python3.9/site-packages/oslo_concurrency/processutils.py:473
2024-05-30 08:53:40.988 2070734 ERROR os_brick.remotefs.remotefs [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Failed to mount 10.46.246.61:/cinder_nfs, reason: mount.nfs: access denied by server while mounting 10.46.246.61:/cinder_nfs
: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-05-30 08:53:40.989 2070734 DEBUG os_brick.remotefs.remotefs [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Failed to do nfs mount. _mount_nfs /usr/lib/python3.9/site-packages/os_brick/remotefs/remotefs.py:157
2024-05-30 08:53:40.989 2070734 ERROR cinder.volume.drivers.nfs [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Mount failure for 10.46.246.61:/cinder_nfs after 3 attempts.: os_brick.exception.BrickException: NFS mount failed for share 10.46.246.61:/cinder_nfs. Error - {'pnfs': "Unexpected error while running command.\nCommand: mount -t nfs -o vers=4,minorversion=1 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: Operation not permitted\\n'", 'nfs': "Unexpected error while running command.\nCommand: mount -t nfs 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: access denied by server while mounting 10.46.246.61:/cinder_nfs\\n'"}
2024-05-30 08:53:40.992 2070734 ERROR cinder.volume.drivers.remotefs [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Exception during mounting NFS mount failed for share 10.46.246.61:/cinder_nfs. Error - {'pnfs': "Unexpected
error while running command.\nCommand: mount -t nfs -o vers=4,minorversion=1 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: Operation not permitted\\n'", 'nfs': "Unexpected error while running command.\nCommand: mount -t nfs 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: access denied by server while mounting 10.46.246.61:/cinder_nfs\\n'"}: cinder.exception.NfsException: NFS mount failed for share 10.46.246.61:/cinder_nfs. Error - {'pnfs': "Unexpected error while running command.\nCommand: mount -t nfs -o vers=4,minorversion=1 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: Operation not permitted\\n'", 'nfs': "Unexpected error while running command.\nCommand: mount -t nfs 10.46.246.61:/cinder_nfs /var/lib/cinder/mnt/00d8a1b433f8e321067dc8cfea5fcf7a\nExit code: 32\nStdout: ''\nStderr: 'mount.nfs: access denied by server while mounting 10.46.246.61:/cinder_nfs\\n'"}
2024-05-30 08:53:40.993 2070734 DEBUG cinder.volume.drivers.remotefs [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Available shares [] _ensure_shares_mounted /usr/lib/python3.9/site-packages/cinder/volume/drivers/remotefs.py:358
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Failed to initialize driver.: cinder.exception.NfsNoSharesMounted: No mounted NFS shares found
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager Traceback (most recent call last):
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager   File "/usr/lib/python3.9/site-packages/cinder/volume/manager.py", line 524, in _init_host
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager     self.driver.do_setup(ctxt)
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager   File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/nfs.py", line 212, in do_setup
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager     self.set_nas_security_options(self._is_voldb_empty_at_startup)
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager   File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/nfs.py", line 423, in set_nas_security_options
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager     raise exception.NfsNoSharesMounted()
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager cinder.exception.NfsNoSharesMounted: No mounted NFS shares found
2024-05-30 08:53:40.994 2070734 ERROR cinder.volume.manager
2024-05-30 08:53:40.994 2070734 DEBUG cinder.service [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Creating RPC server for service cinder-volume start /usr/lib/python3.9/site-packages/cinder/service.py:238
2024-05-30 08:53:41.041 2070734 DEBUG oslo_db.sqlalchemy.engines [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python3.9/site-packages/oslo_db/sqlalchemy/engines.py:335
2024-05-30 08:53:41.054 2070734 DEBUG cinder.service [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Pinning object versions for RPC server serializer to 1.39 start /usr/lib/python3.9/site-packages/cinder/service.py:244
2024-05-30 08:53:41.265 2070734 INFO cinder.volume.manager [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Initializing RPC dependent components of volume driver NfsDriver (1.4.0)
2024-05-30 08:53:41.266 2070734 ERROR cinder.volume.volume_utils [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Volume driver NfsDriver not initialized
2024-05-30 08:53:41.266 2070734 ERROR cinder.volume.manager [None req-917bf11c-1caf-4eee-9eb6-5dc7ab2eb2c6 - - - - - -] Cannot complete RPC initialization because driver isn't initialized properly.: cinder.exception.DriverNotInitialized: Volume driver not ready.
2024-05-30 08:53:51.272 2070734 ERROR cinder.service [-] Manager for service cinder-volume cinder-volume-nfs-0@nfs is reporting problems, not sending heartbeat. Service will appear "down".
2024-05-30 08:54:01.274 2070734 ERROR cinder.service [-] Manager for service cinder-volume cinder-volume-nfs-0@nfs is reporting problems, not sending heartbeat. Service will appear "down".
2024-05-30 08:54:11.278 2070734 ERROR cinder.service [-] Manager for service cinder-volume cinder-volume-nfs-0@nfs is reporting problems, not sending heartbeat. Service will appear "down".