PineconeChatGPT and other generative AI programs often produce false information, referred to as “hallucinations,” because they lack actual knowledge and are designed to generate plausible responses based on input. Edo Liberty, the CEO of Pinecone, a New York-based software company, acknowledges this issue and aims to enhance the reliability of AI output by incorporating a vector database, which has received $138 million in funding.
The vector database serves as a component of the “retrieval-augmented generation” (RAG) approach, where AI models seek external input to enhance their outputs. Among various RAG approaches, the vector database stands out due to its extensive research and industry background. Liberty, who previously worked at tech giants like Yahoo! and Amazon, played a key role in developing vector databases for applications such as shopping recommendations and ad targeting.
For many years, vector databases remained relatively unknown within the database community, requiring companies to build their own solutions. Liberty recognized the potential of using vectors in AI models and understood that it required a separate architecture and database. He anticipated the rise of AI language models like Google’s BERT and realized the need to develop foundational database layers ahead of time to meet the growing demand.
In a vector database, each piece of data is represented by a vector embedding, which places the data in an abstract space based on similarity. For example, cities like London and Paris would be closer in the embedding space compared to New York. Vector embeddings allow for efficient representation and comparison of various data types, such as text, images, and program codes. By converting queries into vectors and performing similarity searches, the vector database can provide relevant outputs.
The vector database’s relevance is evident in recommender systems, where it can match similar items based on their vector embeddings. By adjusting the query, users can narrow or broaden the search for similarity throughout the embedding space. However, similarity search alone is insufficient for a complete database. A vector database requires a management system to handle storage, scalability, and updates.
Pinecone’s vector database aims to address these challenges and provide a reliable and efficient solution for AI models. With substantial funding and Liberty’s expertise, Pinecone is poised to make significant contributions to the field of generative AI and retrieval-augmented generation.