Back

this code runs too slow

  • 1
  • Self Hosted
  • Flutter
  • Functions
jesus
12 Feb, 2024, 17:35

Hi, this a function in dart that subtract a 1 from the LMMS attribute for 6000 documents then update every document with the new subtracted value, the issue is, it takes so long (11 minutes)😭

var response = await database.listDocuments( databaseId: databaseId, collectionId: collectionId, queries: [Query.limit(6000)]); var documents = response.documents;

try { for (var doc = 0; doc < documents.length; doc++) { String documentId = documents[doc].$id; final data = documents[doc].data;

TypeScript
  // Subtract 1 from LMMS first
  int LMMS = data['LMMS'] - 1;

  // Update the document with the initially subtracted LMMS

  await database.updateDocument(
      databaseId: databaseId,
      collectionId: collectionId,
      documentId: documentId,
      data: {'LMMS': LMMS});
}
return context.res.send(200);

} catch (e) { print('Error updating document: $e'); return context.res.send(401); }

Could any one help me make it faster.

TL;DR
Title: Dart function takes too long to update 6000 documents The developer is frustrated that their function takes 11 minutes to update 6000 documents. They are looking for help to make it faster. Solution: One suggestion is to use pagination by incorporating `Query.cursorAfter(cursor)` to process documents in smaller batches. This can be implemented using a do-while loop. Another suggestion is to use asynchronous operations and `Promise.all` in the JavaScript code, which can improve performance.
Kenny
12 Feb, 2024, 17:52

This is a function I've used in the past to update a lot of existing documents, you might see if it helps out any.

TypeScript
import { Client, Databases, Query } from 'node-appwrite';

export default async ({ req, res, log, error }) => {
  const client = new Client()
    .setEndpoint(process.env.APPWRITE_FUNCTION_ENDPOINT)
    .setProject(process.env.APPWRITE_FUNCTION_PROJECT_ID)
    .setKey(process.env.APPWRITE_FUNCTION_API_KEY);

  const database = new Databases(client);

  let response;

  const queries = [Query.limit(25), Query.select("$id")];
  let cursor = null;

  do {
    const newQueries = [...queries];

    if (cursor) {
      newQueries.push(Query.cursorAfter(cursor));
    }

    response = await database.listDocuments(
      DATABASE_ID,
      COLLECTION_ID,
      newQueries
    );

    cursor = response.documents[response.documents.length - 1].$id;

    await Promise.all(
      response.documents.map((document) => {
        database.updateDocument(
          DATABASE_ID,
          COLLECTION_ID,
          DOCUMENT_ID,
          data
        )
       })
    );
  } while (response.documents.length >= 25);

  return res.send('Complete!');
};
jesus
12 Feb, 2024, 17:54
Kenny
12 Feb, 2024, 17:55

How often are you needing to run this function? Is it a one and done or ?

jesus
12 Feb, 2024, 17:56

It will run a lot

jesus
12 Feb, 2024, 17:57

That's why I'm frustrated it takes 11 minutes to complete for only 6000 document

Reply

Reply to this thread by joining our Discord

Reply on Discord

Need support?

Join our Discord

Get community support by joining our Discord server.

Join Discord

Get premium support

Join Appwrite Pro and get email support from our team.

Learn more