Uploaded image for project: 'Red Hat Enterprise Linux AI'
  1. Red Hat Enterprise Linux AI
  2. RHELAI-2336

[python-wheel-build/fromager] use zipfile instead of wheel command to extract build requirements

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • None
    • Fromager
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      [2676902576] Upstream Reporter: Shubh Bapna
      Upstream description:

      Currently the cache wheel feature in bootstrap uses the wheel command to unpack the wheel and extract the build requirements. Using the wheel command requires spawning a child process which is expensive compared to using the zipfile library. There is almost a 90% difference when running this script:

      ```python3
      import pathlib
      import shutil
      import time
      import zipfile

      import dependencies
      import external_commands
      import wheels

      whl = "e2e-output/wheels-repo/downloads/setuptools-75.6.0-0-py3-none-any.whl"
      unpack_dir = pathlib.Path("unpack_dir")
      unpack_dir.mkdir()

      dist_name, dist_version, build_tag, _ = wheels.extract_info_from_wheel_file(
      None, pathlib.Path(whl)
      )
      dist_filename = f"

      Unknown macro: {dist_name}

      -

      Unknown macro: {dist_version}

      "
      metadata_dir = pathlib.Path(f"

      Unknown macro: {dist_filename}

      .dist-info")

      start = time.perf_counter()
      external_commands.run(
      ["wheel", "unpack", whl, "--dest", str(unpack_dir)],
      )
      shutil.copy(
      unpack_dir
      / dist_filename
      / metadata_dir
      / f"

      Unknown macro: {wheels.FROMAGER_BUILD_REQ_PREFIX}

      -

      Unknown macro: {dependencies.BUILD_SYSTEM_REQ_FILE_NAME}

      ",
      unpack_dir / dependencies.BUILD_SYSTEM_REQ_FILE_NAME,
      )
      shutil.copy(
      unpack_dir
      / dist_filename
      / metadata_dir
      / f"

      -

      Unknown macro: {dependencies.BUILD_BACKEND_REQ_FILE_NAME}

      ",
      unpack_dir / dependencies.BUILD_BACKEND_REQ_FILE_NAME,
      )
      shutil.copy(
      unpack_dir
      / dist_filename
      / metadata_dir
      / f"

      Unknown macro: {wheels.FROMAGER_BUILD_REQ_PREFIX}

      -

      Unknown macro: {dependencies.BUILD_SDIST_REQ_FILE_NAME}

      ",
      unpack_dir / dependencies.BUILD_SDIST_REQ_FILE_NAME,
      )
      end = time.perf_counter()
      external = end - start

      shutil.rmtree(str(unpack_dir))
      unpack_dir.mkdir()

      start = time.perf_counter()
      archive = zipfile.ZipFile(whl)
      archive.extract(
      str(
      metadata_dir
      / f"

      -

      Unknown macro: {dependencies.BUILD_SYSTEM_REQ_FILE_NAME}

      "
      ),
      unpack_dir,
      )
      archive.extract(
      str(
      metadata_dir
      / f"

      Unknown macro: {wheels.FROMAGER_BUILD_REQ_PREFIX}

      -

      Unknown macro: {dependencies.BUILD_SDIST_REQ_FILE_NAME}

      "
      ),
      unpack_dir,
      )
      archive.extract(
      str(
      metadata_dir
      / f"

      -

      "
      ),
      unpack_dir,
      )
      end = time.perf_counter()
      zip = end - start

      print(f"external command:

      Unknown macro: {external}

      ")
      print(f"zipfile:

      Unknown macro: {zip}

      ")
      print(f"zipfile is faster by:

      Unknown macro: {(external - zip) / external * 100}

      ")

      ```

              rh-ee-sbapna Shubh Bapna
              upstream-sync Upstream Sync
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: