Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-8777

Terratorch and Mmseg import fails for RHAIIS 3.3 collections on ROCM and CUDA env

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • Development Platform
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • DP Sprint 24

      Steps to reproduce the behavior:

      1. podman login quay.io
      2. podman pull quay.io/aipcc/base-images/rocm-6.4-el9.6:3.3.0-1768562816
      3. Create podman container

      podman run -dit \
        --name aipcclatest \
        --userns=keep-id:uid=1001 \
        --shm-size=4g \
        --device=/dev/kfd \
        --device=/dev/dri \
        --group-add=keep-groups \
        -v /home:/home \
        quay.io/aipcc/base-images/rocm-6.4-el9.6:3.3.0-1768562816
      

      4. Create Requirement file from RHAIIS rocm collections. → https://gitlab.com/redhat/rhel-ai/rhaiis/pipeline/-/blob/main/collections/rhaiis/rocm-ubi9/requirements.txt?ref_type=heads

      5. Pip install RHAIIS rocm wheels

      NETRC=.netrc pip install --only-binary :all: --index-url https://gitlab.com/api/v4/projects/75894209/packages/pypi/simple/ --trusted-host gitlab.com -r req.txt
      Logs:

      (.venv) (app-root) /opt/app-root$ python 
      Python 3.12.9 (main, Nov 21 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] on linux
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import terratorch
      /opt/app-root/.venv/lib64/python3.12/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'repr' attribute with value False was provided to the `Field()` function, which has no effect in the context it was used. 'repr' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
        warnings.warn(
      /opt/app-root/.venv/lib64/python3.12/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'frozen' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'frozen' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
        warnings.warn(
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/terratorch/__init__.py", line 7, in <module>
          import terratorch.models  # noqa: F401
          ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/terratorch/models/__init__.py", line 6, in <module>
          import terratorch.models.necks  # register necks  # noqa: F401
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/terratorch/models/necks.py", line 14, in <module>
          from terratorch.registry import NECK_REGISTRY, TERRATORCH_NECK_REGISTRY
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/terratorch/registry/__init__.py", line 17, in <module>
          import terratorch.registry.mmseg_registry
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/terratorch/registry/mmseg_registry.py", line 101, in <module>
          MMSEG_DECODER_REGISTRY = MMSegRegistry()
                                   ^^^^^^^^^^^^^^^
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/terratorch/registry/mmseg_registry.py", line 50, in __init__
          self.mmseg_reg = importlib.import_module("mmseg.models.decode_heads")
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/lib64/python3.12/importlib/__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/mmseg/__init__.py", line 4, in <module>
          import mmcv
      ModuleNotFoundError: No module named 'mmcv'
      >>> 
      (.venv) (app-root) /opt/app-root$ pip list show | grep "terratorch"
      terratorch                         1.2.1
      (.venv) (app-root) /opt/app-root$ pip list show | grep "mmseg"
      mmsegmentation                     1.2.2
      (.venv) (app-root) /opt/app-root$ python 
      Python 3.12.9 (main, Nov 21 2025, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)] on linux
      Type "help", "copyright", "credits" or "license" for more information.
      >>> import mmseg
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/opt/app-root/.venv/lib64/python3.12/site-packages/mmseg/__init__.py", line 4, in <module>
          import mmcv
      ModuleNotFoundError: No module named 'mmcv'
      >>>
      

      OS and ROCM version:

      (.venv) (app-root) /opt/app-root$ amd-smi version
      AMDSMI Tool: 25.5.1+41065ee6 | AMDSMI Library version: 25.5.1 | ROCm version: 6.4.3 | amdgpu version: 6.16.6 | amd_hsmp version: N/A
      (.venv) (app-root) /opt/app-root$ cat /etc/os-release 
      NAME="Red Hat Enterprise Linux"
      VERSION="9.6 (Plow)"
      ID="rhel"
      ID_LIKE="fedora"
      VERSION_ID="9.6"
      PLATFORM_ID="platform:el9"
      PRETTY_NAME="Red Hat Enterprise Linux 9.6 (Plow)"
      ANSI_COLOR="0;31"
      LOGO="fedora-logo-icon"
      CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
      HOME_URL="https://www.redhat.com/"
      DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
      BUG_REPORT_URL="https://issues.redhat.com/"
      
      REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
      REDHAT_BUGZILLA_PRODUCT_VERSION=9.6
      REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
      REDHAT_SUPPORT_PRODUCT_VERSION="9.6"
      (.venv) (app-root) /opt/app-root$
      

              rh-ee-rdevasth Rohan Devasthale
              rh-ee-konagara Koushik Nagaraj
              Antonio's Team
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: