The CUDA builds are failing on requirements-pipeline 1.5 branch because we are unable to pick the right version for protobuf.

Failed job: https://gitlab.com/redhat/rhel-ai/wheels/requirements-pipeline/-/jobs/9906052498

Here is the debug output for why protobuf is used:

protobuf: no single version meets all requirements
finding why protobuf was being used
ERROR: Could not produce a pip compatible constraints file. Please review /mnt/work-dir/constraints.txt for more details

• is an install dependency of googleapis-common-protos==1.70.0 with req protobuf!=4.21.1,Unable to render embedded object: File (=4.21.2,) not found.=4.21.3,Unable to render embedded object: File (=4.21.4,) not found.=4.21.5,<7.0.0,>=3.20.2
• is an install dependency of opentelemetry-exporter-otlp-proto-grpc==1.32.1 with req googleapis-common-protos~=1.52
• is an install dependency of opentelemetry-exporter-otlp==1.32.1 with req opentelemetry-exporter-otlp-proto-grpc==1.32.1
• is an install dependency of vllm==0.8.4 with req opentelemetry-exporter-otlp>=1.26.0
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of opentelemetry-exporter-otlp-proto-http==1.32.1 with req googleapis-common-protos~=1.52
• is an install dependency of opentelemetry-exporter-otlp==1.32.1 with req opentelemetry-exporter-otlp-proto-http==1.32.1
• is an install dependency of vllm==0.8.4 with req opentelemetry-exporter-otlp>=1.26.0
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of vllm==0.8.4 with req protobuf
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of opentelemetry-proto==1.32.1 with req protobuf<6.0,>=5.0
• is an install dependency of opentelemetry-exporter-otlp-proto-common==1.32.1 with req opentelemetry-proto==1.32.1
• is an install dependency of opentelemetry-exporter-otlp-proto-grpc==1.32.1 with req opentelemetry-exporter-otlp-proto-common==1.32.1
• is an install dependency of opentelemetry-exporter-otlp==1.32.1 with req opentelemetry-exporter-otlp-proto-grpc==1.32.1
• is an install dependency of vllm==0.8.4 with req opentelemetry-exporter-otlp>=1.26.0
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of opentelemetry-exporter-otlp-proto-http==1.32.1 with req opentelemetry-exporter-otlp-proto-common==1.32.1
• is an install dependency of opentelemetry-exporter-otlp==1.32.1 with req opentelemetry-exporter-otlp-proto-http==1.32.1
• is an install dependency of vllm==0.8.4 with req opentelemetry-exporter-otlp>=1.26.0
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of opentelemetry-exporter-otlp-proto-grpc==1.32.1 with req opentelemetry-proto==1.32.1
• is an install dependency of opentelemetry-exporter-otlp==1.32.1 with req opentelemetry-exporter-otlp-proto-grpc==1.32.1
• is an install dependency of vllm==0.8.4 with req opentelemetry-exporter-otlp>=1.26.0
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of opentelemetry-exporter-otlp-proto-http==1.32.1 with req opentelemetry-proto==1.32.1
• is an install dependency of opentelemetry-exporter-otlp==1.32.1 with req opentelemetry-exporter-otlp-proto-http==1.32.1
• is an install dependency of vllm==0.8.4 with req opentelemetry-exporter-otlp>=1.26.0
• vllm==0.8.4 is a toplevel dependency
• is an install dependency of wandb==0.16.6 with req protobuf!=4.21.0,<5,>=3.19.0; python_version > "3.9" and sys_platform == "linux"
• is an install dependency of instructlab==0.26.0 with req wandb>=0.16.4
• instructlab==0.26.0 is a toplevel dependency

The fromager explain-duplicates command shows us versions of protobuf we can pick

protobuf
4.25.7
protobuf!=4.21.0,<5,>=3.19.0; python_version > "3.9" and sys_platform == "linux" matches ['4.25.7']
wandb==0.16.6 <-- required because of golang compiler version
5.29.4
protobuf<6.0,>=5.0 matches ['5.29.4']
opentelemetry-proto==1.32.1
6.30.2
protobuf matches ['6.30.2', '5.29.4', '4.25.7']
vllm==0.8.4
protobuf!=4.21.1,Unable to render embedded object: File (=4.21.2,) not found.=4.21.3,Unable to render embedded object: File (=4.21.4,) not found.=4.21.5,<7.0.0,>=3.20.2 matches ['6.30.2', '5.29.4', '4.25.7']
googleapis-common-protos==1.70.0

• No single version of protobuf meets all requirements