test_unary_ufuncs tests are failing because of AssertionError: Tensor-likes are not close! (Numerical mismatch in mish activation function with float16) on main branch
Tests Failing:
test_reference_numerics_largerefs_nn_functional_mish_cpu_float16,
test_reference_numerics_large_nn_functional_mish_cpu_float16,
test_reference_numerics_large_refs_nn_functional_mish_cuda_float16,
test_reference_numerics_large_nn_functional_mish_cuda_float16
Env details:
PyTorch version: 2.10.0
Branch: main
OS: RHEL 9.6
CPU: Intel
python version: 3.12
commit id : 6de6685797cabc6256df76803f3a5f772d5275a7 (tag: trunk/6de6685797cabc6256df76803f3a5f772d5275a7, origin/main, origin/HEAD)
Steps to repro:
Log in to H200.
Login to quay.io: podman login quay.io
Pull base image: podman pull quay.io/aipcc/pytorch:rhel_cuda_build_without_pins
Run the image and specify the GPU to be used: podman run -it <IMAGE_NAME>
Run the PyTorch UT: TEST_CONFIG=cpu python3 test/run_test.py -i test_unary_ufuncs
Expected result: UTs should run fine.
Actual result:
AssertionError: Tensor-likes are not close! - The mish activation function tests fail consistently with numerical precision issues when using float16 dtype. The error shows: Mismatched elements: 12 / 36 (33.3%), Greatest absolute difference: 1.5 at index (18,) (up to 0.01 allowed), Greatest relative difference: 0.0015010833740234375 at index (18,) (up to 0.0012 allowed). The actual output values differ from the expected reference values beyond the allowed tolerance thresholds, indicating precision loss or numerical instability in the mish activation function implementation for float16 dtype on both CPU and CUDA backends.
The same tests are passing when run in docker quay.io/aipcc/pytorch:rhel9_6_pytorch_main_git6de6685_cuda12_8 which has the same commit base but was built with pinned versions as per the upstream requirements.
Logs are attached below
- blocks
-
AIPCC-7777 Update PyTorch Upstream CI Requirements
-
- In Progress
-
-
AIPCC-4643 transition torch build ownership from AE to torch team
-
- In Progress
-