Digital Ocean Gradient Knowledge Bases: Basic Flaw

First of all, I love the Digital Ocean Gradient platform. It’s really well done and simplifies the tooling for AI alot. However, there is one major flaw in the entire platform: Knowledge bases. Right now, you can only create Knowledge bases by dumping your files in a Spaces bucket and then with some “magic” background process DO processes these files and puts them in an OpenSearch database. But, managing files in Spaces is not scaleable and the data for the AI is already stored in another format, like in a Postgres database or Redis, or even Open Search itself. So this entire process is duplicative and prone to error. Why do we want to duplicate all our data into Spaces, to then have you put it into opensearch? Actually, the solution is simple, you already offer OpenSearch hosting. Why can’t we just connect to the opensearch instance ourselves and get our data from there, instead of having to duplicate all our data in Spaces, and then you have you put it into the Opensearch database for us and vectorize it? Opensearch already does this. It’s helpful that you have a process to vectorize everything in Open Search, so why not just have a tool that points to the opensearch database and then you say what you want to vectorize and use the data in opensearch as the data for AI instead of forcing the use of Files in Spaces. Please fix this. This is the major bottleneck and since good data is essential for AI, the current set up is not useful for production grade AI.

And actually what is interesting is that I signed up for an OpenSearch database and dumped our data into it. But then I couldn’t use it for a knowledge base and had to export it all to Spaces and run it again. That’s not feasible. So I just deleted the Opensearch database and am not even using a knowledge base now.

Thanks.