We need to refactor the tangerine-backend database schema to allow for document de-deduplication. Today a doc is ingested for an agent. If the doc should be available to multiple agents the document is duplicated. The definition of done here is to have a new concept a knowledge base that docs can be ingested into and then agents can reference a doc in a knowledge base. This has a number of follow on effects including exposing the knowledge bases in the tangerine-frontend, new REST API routes for working with KBs, and more. Here's a full breakdown from bsquizza@redhat.com 

1. a set of documents should now be uploaded as a "set" known as a knowledge base
2. So we need a representation in the DB for a knowledgebase with things like: uuid, name, document list
3. We need a CRUD API in the backend for a knowledgebase
4. You should be able to add docs to a knowledgebase or remove docs from a knowledgebase. This would re-use a lot of the code we already have to add docs to an agent or remove them.
5. You should be able to assign or unassign a knowledge base to 1 or more agents... so there's going to need to be a field in 'agent' that references KB uuid's. Something in the agent database needs to indicate that "agent 1" has access to "kb uuid1" and "kb uuid2" and so on.
6. Deleting a knowledgebase is prohibited until it has been "unassociated" with all agents. Either that or ... when you delete it, it gets auto "un-associated" from all agents.
7. When you ask an agent a question, we currently use metadata filters in the search query to limit the documents that we look for. Currently we look for "agent=1" or whatever agent ID you were talking to. Now the metadata filters for the question are no longer going to filter for "agent 1" or "agent 2". They are now going to filter for the knowledgebase ID's that are associated with that agent.
8. the s3sync .yaml config will need a refactor to be able to express this new layout... so instead of assigning document sets to agents, you'll now need a config section in that file for "knowledgebases" and a config for "agents".

Please feel free to reach out to and work with bsquizza@redhat.com for help or clarification.