Rank 2: Process
Hey, thought I'd report some finding from today here.
Setup:
- Appwrite 2.0.0 self-hosted (combined worker topology, default coroutine API)
- 16 cores / 30 GB, _APP_WORKER_PER_CORE=4 (64 API workers)
- openruntimes/executor 0.29.0
- node-22 runtime
- node-appwrite 21 inside the function.
Upgraded 1.8.1 -> 1.9.6 -> 2.0.0; the problem started with 2.0.0.
Symptom: synchronous executions of a function that calls the Appwrite API from inside (users.get, databases.listDocuments, via the server SDK at http://appwrite/v1) fail with 408 "Synchronous function execution timed out" after 30 s. The function itself is fast (under 1 s). Inside the function, its first API call is what hangs for about 28 s. Async executions of the same function never fail.
Reproduction (from inside the appwrite container, session auth): 16 parallel loops, each doing 6 sequential POST /v1/functions//executions with async=false, where makes about 5 API calls back to http://appwrite/v1.
- 64 API workers: about 45 % of executions time out
- 32 workers: about 90 %
- 96 workers: about 27 %
- same load, same function, but a body that returns before any API call: 0 of 96
Failure rate scales with (API calls per execution x concurrent sync executions) / API workers.
Reading: a sync execution occupies an API coroutine waiting on the executor, while the function's own callbacks need a free API coroutine on the same server. Under parallel load they wait on each other until the 30 s executor timeout.The same app under the same load ran fine on 1.8.1 for months.
Fix: the second-main-container setup from https://github.com/appwrite/appwrite/issues/5629 (joined to runtimes network and used by function code as endpoint http://appwrite-functions/v1). Result: 0 of 192 timeouts at 16 parallel, 0 of 48 through the public path.
This setup I had active in the 1.6.x days, as there was also an issue with worker deadlocks... Seems a similar case again?
(investigation above was done using Claude Fable 5.1)