Internal curl error, Http invalid protocol\nError Code: 500

I am running into an issue where a datebase collection refuses to go beyond 5004 documents.

I have a function that fetches a test file, daily updated, from the web and processes and creates documents per record in the file.

Function checks is a document already exists and if not creates it essentially only updating new records.

That function works for 2 minutes, reaches 5004 documents then fails with the internal curl error. Any subsequent run the function crashes in a couple for secs with the same error.

Is there some configuration that defines max records in a collection or what ??

I've seen the invalid protocol error when developers try to log too much data.

Are you paginating using offset pagination? What's your code?

Log meaning "context.log()" ? .. Not paginating. The function is dead simple, its literally

1. Fetch file from github,
2. Loop through records,
3. Fetch document with id from DB,
4. If request returns 404,
5. create the document

I will share the code if that will help

But more bizzare is 5004 record wall I seem to have hit.

A simple recreation would be to create a for loop from 1 to 10000 and create documents will Id being the number. Gonna try and report

@Drake it is impossible to validate if it works. The GUI apparently has a limitation where after 5000 documents in a collection it just says 5000+ with no way, that I could work out, to know the actual document count in a collection. Is that so ? No where is the actual count mentioned ?

Also mean that my initial assumption of ~5000 documents total might be faulty. Maybe everything works as intended and I just can't tell because the UI is limiting visibility. The function does fail still with the same error but it runs for a couple of minutes so maybe it does everything before 500.. Can't be certain ..

yes, please share the code

yes, the total part of list documents caps out at 5000. We don't count further because it gets slower as the number gets higher

''' try { final response = await http.get(Uri.parse(url)); context.log(response.statusCode);

if (response.statusCode == 200) {
  String content = utf8.decode(response.bodyBytes);
  List<String> records = LineSplitter.split(content).toList();

  for (String record in records) {
 
      try {
        await database.createDocument(
            databaseId: envVars['APPWRITE_DATABASE_ID']!,
            collectionId: 'CollectionID',
            documentId: record,
            data: {'data': record});
      } catch (e) {
         context.log("${e.toString}");
      } 
    }
} else {
  context.log('Failed to fetch $url. Status code: ${response.statusCode}');
}

} catch (e) { context.log('Error fetching $url: $e'); }

1. Are you logging anything else?
2. How many records are there?
3. What else happens after this?

I removed all context.log statements to validate but function works for 2 mins and fails with 500 ... currently about 30000 and after this the function just "return context.res.json({'ok': true});"

There aren't any print statements, right?

Also, that's a very high number of records to process serially in one function execution 👀

how would you suggest I go about processing them. Currently to accommodate for the constant 500's, I have added a random statement which randomly picks up a record and check / adds it to DB. So its at least is incrementally adding records.. No other print statements.

1. Don't await every create document. Do 100-200 concurrently.
2. Break down the task into smaller chunks and have the function run more frequently to handle the smaller chunks

Smaller takes that can be retired are typically better than 1 big task

understood. but from your reading can the "invalid protocol" in such situations be addressed ? the most annoying bit about internal curl error is no logs on the console : /

I'm not sure why you're getting the error