1. Proposed title of this feature request
Enable persistent volume for prometheus deployed with servicemesh

2. What is the nature and description of the request?
One of the customers is asking the procedure to attach persistent storage for prometheus which is deployed by servicemesh to keep metrics data persistent.

I checked in my test lab environment and followed the procedure below to attach PV-PVC to the prometheus pod.

Prometheus pod related to service mesh is referring to the deployment.
In the deployment, I could not see any volumeMount section for /prometheus where metrics data is stored by pod and volumes section for prometheus-db.
So, to have that information in place, edit the prometheus deployment to add below sections which are highlighted with arrows.

kind: Deployment
metadata:
  [..]
  name: prometheus
  [..]
spec:
  [..]
  template:
    [..]
    spec:
      containers:
      [..]
      - args:
        [..]
        - --storage.tsdb.path=/prometheus
	[..]
        name: prometheus
        [..]
        volumeMounts:
	[..]
        - mountPath: /prometheus              <====
          name: prometheus-k8s-db             <====
      [..]
      volumes:
      [..]
      - emptyDir:                             <====
        name: prometheus-db                   <====      

After exiting save the changes.

Let the new prometheus pod be into running state with emptyDir first.
Check in to describe the prometheus pod.

# oc describe pod <prometheus-pod-name>

After this, you can set the PVC with `oc set volume` command.

# oc set volume deployment/prometheus --add --name=prometheus-db -t pvc --claim-name=<pvc-name> --claim-size=<size> --overwrite

Note: Replace below parameter as per your need. This command will create the PVC automatically and seek for any storageclass which is set as default.
<pvc-name>: set the PVC name which you want for prometheus.
<size>: pvc size e.g. 1Gi

There are more parameters which can be added to this command according to your use cases.
--claim-class='': StorageClass to use for the persistent volume claim.
--claim-mode='ReadWriteOnce': Set the access mode of the claim to be created. Valid values are ReadWriteOnce (rwo), ReadWriteMany (rwm), or ReadOnlyMany (rom).

After following the above procedure, PVC got automatically created and the default storageclass is used to create PV. Also, the changes were persistent and operator is not reverting those changes.
Note: I have tried this by deploying servicemesh v2.

But it is not guaranteed that the operator will not revert the changes now or in the future. It's certainly possible that this works today, but not buy design and my concern is that any changes made to the reconciliation code could break this in the future.

So, prometheus deployed in relation with servicemesh should be having some definite procedure for addition of persistent volume.

3. Why does the customer need this? (List the business requirements here)
Customer needs to attach persistent volume to prometheus to make the metrics data persistent. It should not lose historic metrics data.

4. List any affected packages or components.
Prometheus deployed in istio-system project.

Customer Account Name: NEC Corporation