Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-8018

[QA][PyTorch UT][CPU][Latest PyTorch Dependency versions]test_unary_ufuncs tests are failing because of AssertionError

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • PyTorch
    • False
    • Hide

      None

      Show
      None
    • False
    • PyTorch Sprint 21, PyTorch Sprint 22, PyTorch Sprint 23

      test_unary_ufuncs tests are failing because of AssertionError: Tensor-likes are not close! (Numerical mismatch in mish activation function with float16) on main branch

      Tests Failing:
      test_reference_numerics_largerefs_nn_functional_mish_cpu_float16,
      test_reference_numerics_large_nn_functional_mish_cpu_float16,
      test_reference_numerics_large_refs_nn_functional_mish_cuda_float16,
      test_reference_numerics_large_nn_functional_mish_cuda_float16

      Env details:
      PyTorch version: 2.10.0
      Branch: main
      OS: RHEL 9.6
      CPU: Intel
      python version: 3.12
      commit id : 6de6685797cabc6256df76803f3a5f772d5275a7 (tag: trunk/6de6685797cabc6256df76803f3a5f772d5275a7, origin/main, origin/HEAD)

      Steps to repro:
      Log in to H200.
      Login to quay.io: podman login quay.io
      Pull base image: podman pull quay.io/aipcc/pytorch:rhel_cuda_build_without_pins
      Run the image and specify the GPU to be used: podman run -it <IMAGE_NAME>
      Run the PyTorch UT: TEST_CONFIG=cpu python3 test/run_test.py -i test_unary_ufuncs

      Expected result: UTs should run fine.
      Actual result:
      AssertionError: Tensor-likes are not close! - The mish activation function tests fail consistently with numerical precision issues when using float16 dtype. The error shows: Mismatched elements: 12 / 36 (33.3%), Greatest absolute difference: 1.5 at index (18,) (up to 0.01 allowed), Greatest relative difference: 0.0015010833740234375 at index (18,) (up to 0.0012 allowed). The actual output values differ from the expected reference values beyond the allowed tolerance thresholds, indicating precision loss or numerical instability in the mish activation function implementation for float16 dtype on both CPU and CUDA backends.

      The same tests are passing when run in docker quay.io/aipcc/pytorch:rhel9_6_pytorch_main_git6de6685_cuda12_8 which has the same commit base but was built with pinned versions as per the upstream requirements.

      Logs are attached below

              rh-ee-ktanmay Kumar Tanmay
              rh-ee-nkangana Nayan Bhushan Kanganahalli Nagabhushana
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: