ResolvedFunctions

[SOLVED] An Internal CURL Error Has Occurred Within the Executor

Dec 27, 2023, 17:22

View on Discord

Votes

Replies

Participants

Unknown

Messages

ZachHandleyOriginal post

Moderator

Oct 31, 2023, 22:45

you know how long I was going 1 by 1? lmao

Summary

User is experiencing an internal CURL error within the executor when calling it in their code. They are confused about what the internal CURL error is and why their code isn't working. They are using promise.all to batch delete documents, with each function creation only handling 1000 documents at a time. However, they are still receiving the internal CURL error. Other users suggest lowering the batch size and using similar features in other languages like Dart. The user also mentions needing "tricks" or best practices for handling large amounts of data and suggests updating the documentation with common helper functions. The user realizes that their previous implementation was slow and inefficient.

ZachHandley

Moderator

Oct 31, 2023, 22:46

Docs should be updated with common "helper" functions IMO, stuff like that isn't that common knowledge, it makes sense, but I feel like most won't think of it

Drake

Oct 31, 2023, 22:46

the promise.all part is the:

insert into Appwrite M CVEs at a time

part

ZachHandley

Moderator

Oct 31, 2023, 22:46

though I imagine there's something in the works

ZachHandley

Moderator

Oct 31, 2023, 22:46

yeah that's what I figured

Drake

Oct 31, 2023, 22:46

that M is what you would tune

ZachHandley

Moderator

Oct 31, 2023, 22:46

I thought it would be cool to spawn X child functions hahaha

Drake

Oct 31, 2023, 22:46

ya...that sounds complicated

ZachHandley

Moderator

Oct 31, 2023, 22:46

Technically it was easy, but yeah the actual implementation with Functions is not

ZachHandley

Moderator

Oct 31, 2023, 22:47

cause it was waiting for the execution to finish before returning resolution, and if I did allSettled then nothing would happen

ZachHandley

Moderator

Oct 31, 2023, 22:47

which makes sense, docker container gets destroyed

Drake

Oct 31, 2023, 22:51

stuff like what?

Drake

Oct 31, 2023, 22:52

i was doing 150/s...so i think each execution took about 15s to finish

Drake

Oct 31, 2023, 22:57

so since it took 15s, i had the function run every minute

ZachHandley

Moderator

Oct 31, 2023, 23:10

neat, smart, that makes sense

ZachHandley

Moderator

Oct 31, 2023, 23:11

Like "tricks" sort of that aren't commonly known, such as inserting large amounts of data using promise.all because I def didn't know for a minute, mostly because I love Python and Dart and such

Drake

Oct 31, 2023, 23:18

dart has something like promise.all too 😉

ZachHandley

Moderator

Nov 2, 2023, 01:36

Hey I'm trying to batch delete and I made it dead simple, I'm only giving it 1000 documents at a time for each function creation and every single one is getting internal curl error

Drake

Nov 2, 2023, 02:29

Maybe it's too much? Try lowering it?

Drake

Nov 2, 2023, 02:30

What's your code?

ZachHandley

Moderator

Nov 2, 2023, 04:59

TypeScript

const batch_size = 1000;

export default async ({ req, res, log, error }: any) => {
  const client = new Client()
    .setEndpoint('https://appwrite.pva.ai/v1')
    .setProject(Bun.env["APPWRITE_FUNCTION_PROJECT_ID"])
    .setKey(Bun.env["APPWRITE_API_KEY"]);
  const db = new Databases(client);
  log("Spawned child function to delete data");
  const databaseId = req.body.databaseId;
  const collectionId = req.body.collectionId;
  const ids = req.body.ids;

  const chunkedIds = Array.from({ length: Math.ceil(ids.length / batch_size) }, (_, i) => ids.slice(i * batch_size, (i + 1) * batch_size));
  
  const childPromises: Promise<any>[] = chunkedIds.map(chunk => {
    log(`Deleting ${chunk.length} documents`);
    return Promise.all(chunk.map(id => db.deleteDocument(databaseId, collectionId, id).catch((e: any) => log(`Error deleting document: ${e}`))));
  });

  await Promise.all(childPromises);
  return res.empty();
};

ZachHandley

Moderator

Nov 2, 2023, 04:59

I'm only seeing 2-3 second runtimes before internal curl

ZachHandley

Moderator

Nov 2, 2023, 05:00

but tbh what I don't understand is what the heck is the internal curl error that's happening? Shouldn't code that works just... work?

ZachHandley

Moderator

Nov 2, 2023, 17:32

Yeah this is still broken

ZachHandley

Moderator

Nov 2, 2023, 17:41

@Steven so is this an Executor error when calling it? I imagine you guys have some idea of what the issue is

[SOLVED] An Internal CURL Error Has Occurred Within the Executor

Need support?_