Enable AWS EFA for OSD/ROSA

User Story

As a customer of OSD or ROSA, I'd like to utilise AWS Elastic Fabric Adapter (Network performance related) in my ROSA/OSD cluster(s) at install time and day2 in clusters.

Acceptance Criteria

• All the things that have to be done for the feature to be ready to
  release.

• We have a tested and fully functional implementation of AWS EFA in a ROSA STS cluster, based on the OCP 4.11 feature
• We have a tested and fully functional implementation of AWS EFA in a OSD cluster, based on the OCP 4.11 feature
• Red Hat / SRE can successfully manage the offering for customer clusters
• The solution is properly documented, including how to enable/disable and any concerns

Default Done Criteria

• All existing/affected SOPs have been updated.
• New SOPs have been written.
• Internal training has been developed and delivered.
• The feature has both unit and end to end tests passing in all test
  pipelines and through upgrades.
• If the feature requires QE involvement, QE has signed off.
• The feature exposes metrics necessary to manage it (VALET/RED).
• The feature has had a security review.* Contract impact assessment.
• Service Definition is updated if needed.* Documentation is complete.
• Product Manager signed off on staging/beta implementation.

Dates

Integration Testing:
Beta:
GA:

Current Status

GREEN | YELLOW | RED
GREEN = On track, minimal risk to target date.
YELLOW = Moderate risk to target date.
RED = High risk to target date, or blocked and need to highlight potential
risk to stakeholders.

References

Links to Gdocs, github, and any other relevant information about this epic.