inductor.test_select_algorithm tests are failing because of InductorError (AssertionError): Incorrect result from choice TritonTemplateCaller on main branch
Tests Failing:
test_convolution1
Env details:
PyTorch version: 2.10.0
Branch: main
OS: RHEL 9.6
CPU: Intel
python version: 3.12
commit id : 6de6685797cabc6256df76803f3a5f772d5275a7 (tag: trunk/6de6685797cabc6256df76803f3a5f772d5275a7, origin/main, origin/HEAD)
Steps to repro:
Log in to H200.
Login to quay.io: podman login quay.io
Pull base image: podman pull quay.io/aipcc/pytorch:rhel_cuda_build_without_pins
Run the image and specify the GPU to be used: podman run -it <IMAGE_NAME>
Run the PyTorch UT: TEST_CONFIG=cpu python3 test/run_test.py -i inductor/test_select_algorithm
Expected result: UTs should run fine.
Actual result:
InductorError (AssertionError): Incorrect result from choice TritonTemplateCaller. The selected TritonTemplateCaller algorithm produces incorrect results with numerical mismatch (84.7% of elements differ, greatest absolute difference: 132.32, greatest relative difference: inf), indicating the algorithm selection is choosing an incorrect implementation for the convolution operation.
Logs are attached below