Uploaded image for project: 'Red Hat Enterprise Linux AI'
  1. Red Hat Enterprise Linux AI
  2. RHELAI-2628

RHEL AI 1.3 Release notes - Benchmark - unclear wording

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • None
    • Documentation
    • None
    • False
    • Hide

      None

      Show
      None
    • False

      Section "1.2.4.3. Benchmark evaluation" of RHEL AI 1.3 Release notes:
      https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.3/html-single/release_notes/index#benchmark-evaluation

      The second sentence creates confusion about what's being evaluated:

      "Red Hat Enterprise Linux AI includes the ability to run benchmark evaluations on the newly trained models. On your trained model, you can evaluate how well the model knows the model you added with the MMLU_BRANCH benchmark. For more details on benchmark evaluation, see Evaluating your new model."

      The illogical part is in the second sentence: "On your trained model, you can evaluate how well the model knows the model you added". It sounds somewhat circular and implies model does "know" itself. It might misrepresent what MMLU benchmarks actually test.

      Something like this could bring more clarity:

      "On your trained model, you can evaluate how well the model knows the knowledge/skills you added"

              kelbrown@redhat.com Kelly Brown
              rhn-support-pzimek1 Pepa Zimek
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: