Uploaded image for project: 'AI Platform Core Components'
  1. AI Platform Core Components
  2. AIPCC-3633

Transformers is not working with llmcompressor in 3.2 image

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • RHAIIS-3.2
    • RHAIIS-3.2
    • Development Platform
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      llmcompressor 0.6.0 needs to work with transformers 4.52.4.

       

      The 3.2 image installs transformers 4.53.2 which caused following error:

       

      >>> oneshot(
      ...     model=model,
      ...     dataset=ds,
      ...     recipe=recipe,
      ...     max_seq_length=MAX_SEQUENCE_LENGTH,
      ...     num_calibration_samples=NUM_CALIBRATION_SAMPLES,
      ... )
      <stdin>:1: DeprecationWarning: `from llmcompressor.transformers import oneshot` is deprecated, please use `from llmcompressor import oneshot`.
      2025-07-17T13:13:32.750911+0000 | reset | INFO - Compression lifecycle reset
      2025-07-17T13:13:32.752952+0000 | from_modifiers | INFO - Creating recipe from modifiers
      2025-07-17T13:13:32.870933+0000 | initialize | INFO - Compression lifecycle initialized for 1 modifiers
      2025-07-17T13:13:32.871136+0000 | IndependentPipeline | INFO - Inferred `SequentialPipeline` for `GPTQModifier`
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/opt/app-root/lib64/python3.12/site-packages/compressed_tensors/utils/helpers.py", line 193, in wrapped
          return func(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/transformers/finetune/text_generation.py", line 33, in oneshot
          oneshot(**kwargs)
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/entrypoints/oneshot.py", line 311, in oneshot
          one_shot()
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/entrypoints/oneshot.py", line 149, in __call__
          self.apply_recipe_modifiers(
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/entrypoints/oneshot.py", line 193, in apply_recipe_modifiers
          pipeline(self.model, calibration_dataloader, self.dataset_args)
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/pipelines/independent/pipeline.py", line 49, in __call__
          pipeline(model, dataloader, dataset_args)
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/pipelines/sequential/pipeline.py", line 66, in __call__
          subgraphs = trace_subgraphs(model, sample_input, sequential_targets, ignore)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/llmcompressor/pipelines/sequential/helpers.py", line 126, in trace_subgraphs
          tracer.trace(
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/utils/fx.py", line 1315, in trace
          self.graph = super().trace(root, concrete_args=concrete_args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 838, in _fn
          return fn(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/fx/_symbolic_trace.py", line 838, in trace
          (self.create_arg(fn(*args)),),
                           ^^^^^^^^^
        File "LlamaForCausalLM_8782366938750_autowrapped", line -1, in forward
        File "/opt/app-root/lib64/python3.12/site-packages/torch/fx/_symbolic_trace.py", line 813, in module_call_wrapper
          return self.call_module(mod, forward, args, kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/utils/fx.py", line 1179, in call_module
          return super().call_module(m, forward, args, kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/fx/_symbolic_trace.py", line 531, in call_module
          ret_val = forward(*args, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/fx/_symbolic_trace.py", line 806, in forward
          return _orig_module_call(mod, *args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
          return self._call_impl(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
          return forward_call(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "LlamaModel_8782366938903_autowrapped", line 50, in forward
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/masking_utils.py", line 785, in create_causal_mask
          causal_mask = mask_interface(
                        ^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/masking_utils.py", line 481, in eager_mask
          mask = sdpa_mask(
                 ^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/masking_utils.py", line 344, in sdpa_mask_recent_torch
          padding_mask = prepare_padding_mask(attention_mask, kv_length, kv_offset, _slice=False)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/masking_utils.py", line 187, in prepare_padding_mask
          if (padding_length := kv_length + kv_offset - attention_mask.shape[-1]) > 0:
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/transformers/utils/fx.py", line 670, in __bool__
          return super().__bool__()
                 ^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/fx/proxy.py", line 555, in __bool__
          return self.tracer.to_bool(self)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/app-root/lib64/python3.12/site-packages/torch/fx/proxy.py", line 366, in to_bool
          raise TraceError(
      torch.fx.proxy.TraceError: symbolically traced variables cannot be used as inputs to control flow 

              cheimes@redhat.com Christian Heimes
              rh-ee-dahuang Dan Huang
              Antonio's Team
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: