Description of problem:

When a user project is created, the modelmesh-enabled: true annotation is automatically added to the project, even if the user never uses model-mesh related features.

This can create scale issues, as this NetworkPolicy relies on this label to define network flow rules:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels: ...
  name: etcd
  namespace: redhat-ods-applications
spec:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          modelmesh-enabled: "true"
    ports:
    - port: 2379
      protocol: TCP
  podSelector:
    matchLabels:
      app: model-mesh
      app.kubernetes.io/part-of: model-mesh
      component: model-mesh-etcd
  policyTypes:
  - Ingress

Prerequisites (if any, like setup, operators/versions):

Steps to Reproduce

1. create a Project from the Dashboard

Actual results:

see the label modelmesh-enabled: true in the project

with many such namespaces (tested with 1500), see this warning in redhat-ods-applications events:

25m         Warning   NetworkPolicySize   networkpolicy/etcd                     TooManyFlows%!(EXTRA string=This NetworkPolicy generates a very large number of OVS flows (%d) and may degrade network performance., int=1042)

Expected results:

the label modelmesh-enabled: true should be set to the Project only if ModelMesh components are created

Reproducibility (Always/Intermittent/Only Once):

Build Details:

Workaround:

Additional info: