In the light of moving from old (rhts/restraint) to new (FMF files for TMT/TF) testing infrastructure, the transition phase might take different time periods between different SSTs and can be quite confusing, even more when the test metadata for both infrastructures are present in the same test directory, for instance:

$ tree stress/stress-ng/
stress/stress-ng/
├── ...
├── main.fmf                # <-- TMT metadata
├── metadata               # <-- restraint metadata
├── ...
├── runtest.sh              # <-- test entry point
├── stress-ng-dccp.te
└── unique-stressors.sh

With that, it's important to create an automated check (e.g. GitLab pipeline?) in our test repositories (e.g. CentOS kernel-tests to warn test code contributors when their changes update test metadata for the old infra, but does not update the metadata for the new infra in the same way, possibly creating a requirement gap/breakage between the two. At first, the automation should not block contributor's merge request, since some changes might indeed be specific for one infrastructure only.

Initial thoughts about the check:

• If MR changes touch restraint metadata file and:
  • FMF file is also present, but was not updated: warn the user
  • FMF file is not present, warn the user about converting the test
• If MR changes touch an FMF file and:
  • restraint metadata file is also present, but was not updated: warn the user
  • restraint metadata file is not present, pass the check