-
Feature Request
-
Resolution: Unresolved
-
Undefined
-
None
-
None
-
None
-
Product / Portfolio Work
-
None
-
False
-
-
None
-
None
-
None
-
-
None
-
None
-
None
-
None
-
None
1. Proposed title of this feature request
Archive ephemeral batch job resources
2. What is the nature and description of the request?
Provide a way for customers to query for historical records about their short-lived batch workloads. Make a copy of short-lived resources from the cluster and store them in durable storage (rds, etc). Expose a kube-like API to query the history. If pod logs were handled by cluster log forwarder, provide an API for users to access logs of old pods associated with a batch workload.
3. Why does the customer need this? (List the business requirements here)
Whenever a user has an batch workload, of course they may want to check on its status from time to time, or the historical record of runs. However, keeping all resources on cluster results in etcd pressure. Users are forced to prune resources with a cronjob, but then - how can they see what yesterday's jobs were?
https://github.com/kubearchive/kubearchive is the solution.
It is a controller that makes a copy of resources to a postgres cluster before they are deleted. It exposes a kube-like API for users to query for historical records and gain access to pod logs that have already been reaped from the cluster.
It was created as a part of the konflux project, and was inspired by its predecessor "tekton-results" which did the same thing, but in a more narrow way targetted at only the tekton community.
4. List any affected packages or components.
kueue is related here, but not directly affected.
If kueue is applicable to a workload, kubearchive is also applicable to that workload.