Async createExecution returns 202 with an id that getExecution can 404 until ClickHouse persistence

👟 Reproduction

1. Self-hosted Appwrite 2.5.x (ClickHouse executions store).
2. POST /v1/functions/{functionId}/executions with async: true → 202 and an execution.$id.
3. Immediately (or within ~1–2s) GET /v1/functions/{functionId}/executions/{executionId} with the same project/auth.
4. Intermittently get 404 execution_not_found.
5. Retrying the same GET shortly after often succeeds; the function itself may still run normally.

Observed behavior

Create builds the execution document in memory, dispatches ExecutionScheduled, enqueues the function, and returns 202 without waiting for the executions worker to executionStore.upsert() into ClickHouse.

getExecution reads only via executionStore (ClickHouse), not MariaDB. So the id in the create response is not always GET-visible yet.

API logs show listener.log.event = ExecutionScheduled / success = true on the POST, then 404 on the GET for the same id.

Expected behavior

Either:

• Persist (or acknowledge) a GET-visible “waiting” execution before returning 202, so getExecution is read-after-write safe for the returned id; or
• Document that early 404 on poll is expected and specify a retry window / Retry-After.

Returning an id that the poll endpoint cannot find violates the create → poll flow described in the docs.

Appwrite version

2.5.x (self-hosted). Same architecture since ~2.2 “executions live in ClickHouse.”

-# debugged with AI for the reason for the create write lag

Also the reason is AI generation that takes more than 30s sometimes, so i run it async and poll the execution to know when it's finished (i get the output from a document with the same execution id)

will be checking this

can u monitor how long does the sync take, is it too big?

also whats the particular use case ur facing issue with hre, i would think clickhouse taking a bit longer is fine since its just logging data and thats not a critical path

apparently sometimes more than 30s, i will check more in details today

since 2.2 clickhouse handle executions as well, they aren't in the db anymore apparently

yep but execution document is just a logging data point right?

clickhouse is extremely efficient at logging data, hence the reason we moved audits, and then now executions to it

but ofc it doesnt guarantee efficient delivery

no i poll the execution to know if it's completed

because i can't run the function sync as it's not a fixed time (or jsut AI lag), so async + poll, then once it's complete i look at the output row in a database

(2+ only, mean it checked for execution where it found the create + update in the worker logs, as sometimes, it apparently just save on complete, but possibly for scheduled or others types of functions)

i look at the output row in a database

why not use the sync execution output?

max sync are 30s right ? so if my function is more than that i need to run async and fetch the output from somewhere else than the execution output itself

which is often the case with image generation

hmm interesting 🤔 i can see the issue a bit, ig a valid way to poll can be nice solution for ur usecase

did some digging, the lag is because clickhouse upsert is called in the worker, not in the critical path which is the api call

now doing so makes sense to me, clickhouse request can fail / takes time. even if low. so having it off critical path ensures execution is always made and is efficient. even if we somehow lose the actual record of the execution, the side effect is the only thing that we lose

will see if the clickhouse call is efficient enough that it can warrant potentially being in api path

as a tmp workaround, i'm allowing small polls with 404 status at the start so it can wait for the worker to upsert

https://github.com/appwrite/appwrite/pull/14652