• Bulk import from Git
  • • git providers to support by priority
      • github.com & github enterprise
      • gitlab
      • azure gitops
      • Bitbucket
    • where the generated catalog-info should be stored? (need feedback from customers)
      • PR against source
      • only in Backstage db
      • separate git repo
    • RBAC for github repos
      • extending github auth provider to get required info?
      • querying LDAP or AD to get required info?
      • Special role (like admin) that will allow only users with that role to do import (this will allow the role access to all repositories accessible by the integration) 
        • This look to be the chosen option at the moment
    • how to generate catalog-info
    • who should be able to see and access the plugin?
      • Platform engineers / admins only
    • service now ticketing for approvers 
      • This option should be toggleable if ServiceNow is not used or available
      • Would we want to keep track of the status of the tickets (or pull/merge requests for the repositories if ServiceNow is not used) to show progress of the bulk import?
        • This would require periodic querying of ServiceNow API or Git provider API for statuses
        • How to get the Github (or other git provider) pull request status from ServiceNow ticket?
        • Can we just trust that the pull request is merged once the ServiceNow ticket is resolved?
      • Ex: import statuses such as 
        • Queued (waiting for the task to do the ingestion/pull request to start)
        • Waiting for PR Approval
        • Waiting for ServiceNow Ticket Resolution (if ServiceNow is used)
        • PR Rejected
        • ServiceNow Ticket Rejected (if ServiceNow is used)
        • Finished and Ingested
        • Ingestion Error (if there’s some error like illegal name, entity name already exists in catalog and such)
      • Do we want to allow users to retry ingestion for failed tickets/PRs/ingestions?
        • create another service now ticket or pull request
        • If the issue is an ingestion failure due to incorrect configurations, then potentially have them manually fix it or allow fixing the `catalog-info.yaml` via the plugin with a YAML editor (and create PR or ServiceNow ticket to apply the change?)
    • UX mockups
    • can we use orchestrator for parts of this?
    • running it per-org is probably ok
      • How do we determine what organizations/repositories to show?
        • Does the end user provide of a list of organizations to import from or do we need to infer somehow?
      • Need a way to automate the repository selection process (something like a YAML configuration or Regex?)
      • For Github, it should be possible to use a github app for each org and configure the integrations section of the app-config.yaml with the apps' credentials, to gain access to all the organizations 
        • Alternatively, have a public github app that is installed across multiple organizations?
    • dealing with monorepo (multiple components in one repository)
      • Need customer feedback on how many monorepos they own
      • If number of monorepos are low, then potentially allowing them to exclude them (and have users manually import them) could be an option
        General Flow (Subject to Change)

• • • Prompt user for an organization or user (support for multiple can be added in the future)
      • List all repositories in the given organization or user that the github apps have access to.
      • Allow the user to select a subset of the repositories
    • Generate a catalog-info.yaml for each repository based on provided information and open a PR to add it into the repository
      • We will want the admin user to be able to customize this catalog-info.yaml with things like annotations, owners, etc.
      • We should have a default catalog-info.yaml if no additional configurations were provided
      • A dryrun catalog ingestion should occur to determine if any entity name collisions occur.
        • Import shouldn't fail, so something like a -(number) may need to be appended to the end of the entity name to ensure uniqueness
        • Sanitation of inputs will also be required to follow the backstage naming schemas. 
    • Initial implementation won't do tracking of import status (ex: Waiting for PR Approval) and will operate on a Fire and Forget basis
      • We will still want to track whether the PR was created properly and if the generated catalog-info.yaml was valid and track those failures if they occur
• Discovery
  • OpenShift
  • Tekton
  • ArgoCD
  • etc

Acceptance Criteria

1. As a platform engineer, I want to give permission to specific users/groups to use the bulk import plugin using RBAC
2. As a platform engineer, I want to provide a template of the `catalog-info.yaml` that will be used by the bulk import plugin
3. As a developer, I want to be able to bulk import my existing applications available in Git (GitHub, GitLab, etc.).
4. As a developer, I should import only the repositories I have access to. I cannot import someone else repo.
5. As a developer, when I import a Git repo, it should create a PR with the catalog-info.yaml and add the component to the catalog.
6. It would be great that the process is using a software template. The platform engineer will be able to provide the catalog-info file and define the various steps. Create the PR, submit a ServiceNow ticket, send a notification, add to the catalog, or anything else.