Concurrent database calls / high CPU usage / Internal curl errors [...] Error Number: 52.

I have a logic that fetches around 5000 data entries from the Facebook API. I use batch calls that can pack up to 50 requests, so it's around 100 single API calls. The 5000 received data objects are then saved to the database. For which I used a batched parallel execution with 100 database calls concurrently. So: the only activity the logic does is fetching data from the API and interacting with the Appwrite database (reading/writing).

When running it for those 5000 entries with 100 database calls in parallel, i get two different problematic behaviors:

Behavior 1: Function fails with Internal curl errors has occurred within the executor! Error Number: 52. Error Msg: Empty reply from server\nError Code: 500, and it always has a duration of 1m (screenshot 1). Function timeout is set to 900 seconds and it is executed async.

When this happens that the function is labeled "failed", CPU usage stays at around 30% (screenshot 2).

Behavior 2: Here the function runs for >1m, and i can see in the Sentry logs that some hundred of the data entries are processed. CPU usage is high, fluctuating between 40% and 90%. After 1-2 minutes it jumps to 100% on all 4 cores and stays there (see screenshot 3). Seems to be the app/http.php process. But before this constant 100%, there is another process at the top, node --max_old_space_size=8192 server.js (see screenshot 4). To get out of the 100% blockage, i need to restart Appwrite with sudo docker compose down and sudo docker compose up -d .

When I fetch with a lower volume of 200 entries at that 100 calls concurrency it all works fine. When I fetch that high volume of 5000 entries with a concurrency of 10 database requests, the CPU usage is a bit higher but it also works fine. CPU stays around 20-30%. But fetching those 5000 objects with 100 database requests concurrency results in those two behaviors above.

Rest of text see below ⬇️

Setup:

• Version 1.6.1, self-hosted on VPS with 4 dedicated vCPU and 16GB
• _APP_FUNCTIONS_INACTIVE_THRESHOLD=86400 (to avoid cold start every couple minutes)
• _APP_FUNCTIONS_MAINTENANCE_INTERVAL=3600
• _APP_WORKERS_NUM=4 (for parallel exeuction of async executions) -- also tried without this parameter but didn't change the behavior
• _APP_WORKER_PER_CORE=12 (for pre-1.6.1 to reduce occurrence of the function workers deadlock problem) -- also tried with 5 but didn't change the behavior

Questions

1. To behavior 1 -- That Error Number: 52 error seems to be related to too many concurrent database operations. But can someone confirm that too many database operations can cause this error? Is there some more documentation on this somewhere?
2. To behavior 1 -- Why does that Error Number: 52 error always comes with exactly 1m duration?
3. To behavior 1 -- Why did the CPU usage stay high (20-30%) after the function failed with that Error Number: 52 at 1m duration? What exactly is using up resources still? Does it have to do with the increased _APP_FUNCTIONS_INACTIVE_THRESHOLD?
4. To behavior 2 -- Even if those concurrent calls brick a function, wasn't the resource usage of a container limited so it doesn't freeze the entire host machine?

From an earlier debugging session a couple weeks ago on that Internal curl errors has occurred within the executor! Error Number: 52 Error Code: 500 I had added this comment to https://github.com/appwrite/appwrite/issues/8907 (at that time i was still on 1.6.0):

The Sentry error log (which is connected to Appwrite) says this:

Internal curl errors has occurred within the executor! Error Number: 52. Error Msg: Empty reply from server File /usr/src/code/src/Appwrite/Platform/Workers/Functions.php line 608

#0 /usr/src/code/src/Appwrite/Platform/Workers/Functions.php(170): Appwrite\Platform\Workers\Functions->execute(Object(Utopia\Logger\Log), Object(Utopia\Database\Database), Object(Appwrite\Event\Func), Object(Appwrite\Event\Usage), Object(Appwrite\Event\Event), Object(Utopia\Database\Document), Object(Utopia\Database\Document), 'http', '/', 'POST', Array, '{"action":"conn...', Object(Utopia\Database\Document), 'eyJ0eXAiOiJKV1Q...', NULL, NULL, '678684f2e467915...')
#1 /usr/src/code/src/Appwrite/Platform/Workers/Functions.php(50): Appwrite\Platform\Workers\Functions->action(Object(Utopia\Queue\Message), Object(Utopia\Database\Database), Object(Appwrite\Event\Func), Object(Appwrite\Event\Event), Object(Appwrite\Event\Usage), Object(Utopia\Logger\Log))
#2 /usr/src/code/vendor/utopia-php/queue/src/Queue/Server.php(248): Appwrite\Platform\Workers\Functions->Appwrite\Platform\Workers{closure}(Object(Utopia\Queue\Message), Object(Utopia\Database\Database), Object(Appwrite\Event\Func), Object(Appwrite\Event\Event), Object(Appwrite\Event\Usage), Object(Utopia\Logger\Log))
#3 [internal function]: Utopia\Queue\Server->Utopia\Queue{closure}('0')
#4 /usr/src/code/vendor/utopia-php/queue/src/Queue/Adapter/Swoole.php(37): call_user_func(Object(Closure), '0')
#5 [internal function]: Utopia\Queue\Adapter\Swoole->Utopia\Queue\Adapter{closure}(Object(Swoole\Process\Pool), '0')
#6 {main}

And the code in question attached to here. The main function is the pullCampaigns function, above it i added the database helper methods.