Pinecone is a vector database that allows you to store and query high-dimensional vectors. It is a great tool for building recommendation systems, search engines, and more. In this tutorial, we'll show you how to integrate Pinecone into your Appwrite project.
Inside an Appwrite Function, we'll create a method to that indexes an Appwrite collection into Pinecone. We'll also create a method to query the Pinecone index and return the results.
Prerequisites
An Appwrite project
An Appwrite collection
A Pinecone index
Create new function
Head to the Appwrite Console then click on Functions in the left sidebar and then click on the Create Function button.
In the Appwrite Console's sidebar, click Functions.
Click Create function.
Under Connect Git repository, select your provider.
After connecting to GitHub, under Quick start, select the Node.js starter template.
In the Variables step, add the PINECONE_API_KEY, generate it here. Add the OPENAI_API_KEY, generate it here.For the APPWRITE_API_KEY, tick the box to Generate API key on completion.
Follow the step-by-step wizard and create the function.
Add dependencies
Once the function is created, navigate to the freshly created repository and clone it to your local machine.
Install the @pinecone-database/pinecone package to simplify the process of interacting with the Pinecone API. We'll also install the openai package to interact with the OpenAI API.
npm install @pinecone-database/pinecone openai
Create utility function
For this example, the function will be able to take both GET and POST requests.
Create a new src/utils.js file with the following code:
import path from 'path';
import { fileURLToPath } from 'url';
import fs from 'fs';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const staticFolder = path.join(__dirname, '../static');
export function getStaticFile(fileName) {
return fs.readFileSync(path.join(staticFolder, fileName)).toString();
}
Handle GET request
Write the GET request handler in the src/main.js file.
import { getStaticFile } from './utils.js';
export default async ({ req, res, error }) => {
if (req.method === 'GET') {
const html = getStaticFile('index.html');
return res.send(html, 200, { 'Content-Type': 'text/html; charset=utf-8' });
}
};
The code checks if all required environment variables are present and then returns the static HTML page when a GET request is made.
Create web page
Create a HTML web page that the function will serve. Create a new file at static/index.html with some HTML boilerplate:
<!doctype html>
<html lang="en">
</html>
Within the <html> tag, Add a <head> tag that will define the style and scripts.
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Pinecone Demo</title>
<script src="https://unpkg.com/meilisearch@0.34.1"></script>
<script src="https://unpkg.com/alpinejs" defer></script>
<link rel="stylesheet" href="https://unpkg.com/@appwrite.io/pink" />
<link rel="stylesheet" href="https://unpkg.com/@appwrite.io/pink-icons" />
</head>
And after the </head> tag add this <body> which will contain the actual form:
<body>
<main class="main-content">
<div class="top-cover u-padding-block-end-56">
<div class="container">
<div
class="u-flex u-gap-16 u-flex-justify-center u-margin-block-start-16"
>
<h1 class="heading-level-1">Pinecone Demo</h1>
<code class="u-un-break-text"></code>
</div>
<p
class="body-text-1 u-normal u-margin-block-start-8"
style="max-width: 50rem"
>
Use this demo to verify that the sync between Appwrite Databases and
Pinecone was successful. Search your Pinecone vector database using
the input below.
</p>
</div>
</div>
<div
class="container u-margin-block-start-negative-56"
x-data="{ search: '', results: [ ] }"
x-init="$watch('search', async (value) => { results = await onSearch(value) })"
>
<div class="card u-flex u-gap-24 u-flex-vertical">
<div id="searchbox">
<div
class="input-text-wrapper is-with-end-button u-width-full-line"
>
<input x-model="search" type="search" placeholder="Search" />
<div class="icon-search" aria-hidden="true"></div>
</div>
</div>
<div id="hits" class="u-flex u-flex-vertical u-gap-12">
<template x-for="result in results">
<div class="card">
<pre x-text="JSON.stringify(result, null, '\t')"></pre>
</div>
</template>
</div>
</div>
</div>
</main>
<script>
window.onSearch = async function (prompt) {
const response = await fetch('/search', {
method: 'POST',
body: JSON.stringify({ prompt }),
headers: {
'Content-Type': 'application/json',
},
});
return response.matches;
};
</script>
</body>
This will render a form that will submit your search query to the function and display the results.
Setup SDKs
Add methods necessary to integrate with the OpenAI and Pinecone APIs
Import openai and @pinecone-database/pinecone at the top of the main.js file:
import { Pinecone } from '@pinecone-database/pinecone';
import { OpenAI } from 'openai';
Add the following code at the end of request handler in the main.js file:
const openai = new OpenAI();
const pinecone = new Pinecone();
const pineconeIndex = pinecone.index(process.env.PINECONE_INDEX_ID);
The functions checks the request method, and then initializes the OpenAI and Pinecone SDKs.
Handle search requests
To handle the search requests, add the following code to the end of the request handler in the main.js file:
if (req.path === '/search') {
const queryEmbedding = await openai.embeddings.create({
model: 'text-embedding-ada-002',
input: req.body.prompt,
});
const searchResults = await pineconeIndex.query({
vector: queryEmbedding.data[0].embedding,
topK: 5,
});
return res.json(searchResults);
}
For all requests with the path /search, the function sends the search query to the OpenAI API to get the embedding. The function then queries the Pinecone index with the embedding and returns the results.
Handle indexing requests
The Appwrite collection needs to be indexed into the Pinecone index. Create a new file at src/appwrite.js with the following code:
import { Client, Databases, Query } from 'node-appwrite';
export default class AppwriteService {
constructor() {
const client = new Client();
client
.setEndpoint(
process.env.APPWRITE_ENDPOINT ?? 'https://cloud.appwrite.io/v1'
)
.setProject(process.env.APPWRITE_FUNCTION_PROJECT_ID)
.setKey(process.env.APPWRITE_API_KEY);
this.databases = new Databases(client);
}
async getAllDocuments(databaseId, collectionId) {
const cumulative = [];
let cursor = null;
do {
const queries = [Query.limit(100)];
if (cursor) {
queries.push(Query.cursorAfter(cursor));
}
const { documents } = await this.databases.listDocuments(
databaseId,
collectionId,
queries
);
if (documents.length === 0) {
break;
}
cursor = documents[documents.length - 1].$id;
cumulative.push(...documents);
} while (cursor);
return cumulative;
}
}
The service provides a method to iterate the documents contained within an entire collection, fetching the limit of 100 documents per request.
const appwrite = new AppwriteService();
const documents = await appwrite.getAllDocuments(
process.env.APPWRITE_DATABASE_ID,
process.env.APPWRITE_COLLECTION_ID
);
const embeddings = await Promise.all(
documents.map(async (document) => {
const record = await openai.embeddings.create({
model: 'text-embedding-ada-002',
input: JSON.stringify(document),
});
return {
id: document.$id,
values: record.data[0].embedding,
metadata: document,
};
})
);
await pineconeIndex.upsert(embeddings);
The code fetches all documents from the Appwrite collection, then sends each document to the OpenAI API to get the embedding. The embeddings are then uploaded to the Pinecone index.
Test the function
Now that the function is deployed, test it by visiting the function URL in your browser. This should show the UI created earlier and to test it, write a search query and click the submit button. After a brief moment you should see the matched results.