Issue after migrating storage from local to S3

We migrated our storage from local storage to S3, and the S3 integration itself appears to be working correctly. Builds are successfully uploaded to S3, and newly created files in the bucket can also be accessed/used normally.

However, all newly deployed Functions and Sites are failing to load and return:

Timed out waiting for runtime.
Error Code: 400

This appears to be specifically affecting the runtime initialization/execution of newly deployed Functions and Sites, rather than S3 file upload or file access itself.

Hey, i do have s3 setup on my instance, and it's working fine, so i'm more thinking about the configuration that you have.

e.g. the executor have a OPR_EXECUTOR_CONNECTION_STORAGE which is a s3:// URL, it's usualy missed on S3 setup

if you like, hope in DM and paste your s3 config, so i can see why it's it's working (remove the secrets before)
You can also check this thread talking about S3 : https://discord.com/channels/564160730845151244/1507320206119141407 (it was made for 1.x, tho i don't think s3 conf changed since then so should work)

we solved it. There was two problems, the first was the env OPR_EXECUTOR_CONNECTION_STORAGE that where still pointing to local, and the second was :

$ docker exec exc1 php -r 'require "/usr/local/vendor/autoload.php"; $d=\OpenRuntimes\Executor\StorageFactory::getDevice("/", getenv("OPR_EXECUTOR_CONNECTION_STORAGE")); try { $r=$d->listFiles("", 5); echo "OK\n"; var_export($r); echo "\n"; } catch (Throwable $e) { fwrite(STDERR, "FAIL: ".$e->getMessage()."\n"); exit(1);} '

> Fatal error: Uncaught InvalidArgumentException: Storage DSN scheme '3' has no device in /usr/local/src/Executor/StorageFactory.php:49
> Stack trace:
> #0 Command line code(1): OpenRuntimes\Executor\StorageFactory::getDevice('/', '3://LWB4WCLJGVD...')
> #1 {main}
>   thrown in /usr/local/src/Executor/StorageFactory.php on line 49

adding quotes to the env fixed the issue. (before : OPR_EXECUTOR_CONNECTION_STORAGE=s3://... | after : OPR_EXECUTOR_CONNECTION_STORAGE="s3://...")

also when i say OPR_EXECUTOR_CONNECTION_STORAGE, it's in the docker-compose, in the .env you should edit the _APP_EXECUTOR_CONNECTION_STORAGE

which also apparently silently crash, according to the logs i asked there was nothing after the ImagePuller says it pulled everything

seems to be broken again in 2.4.0 or is it just me?

what isn't working ?
functions execution ?
functions build ?
images ?

functions build

claude claims this is what broke it on 2.4.0
https://github.com/appwrite/appwrite/blob/2.4.0/packages/storage/src/Device/S3.php#L910-L915

hmm possible, will check with garage

hmm, it did work for me, do you have any logs (from the build or from the containers) ?
is your function big ?

I'm using Cloudflare R2 if that matters,

build logs:

Internal server error. Please try again.

container logs:

  level                  info
  http.method            POST
  listener.usage.event   Appwrite\Bus\Events\RequestCompleted
--
  http.path          /v1/functions/:functionId/deployments/duplicate
  http.response.code 403

  Utopia\Storage\Exception\RemoteException: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
    at /usr/src/code/packages/storage/src/Device/S3.php:1094
    at /usr/src/code/packages/storage/src/Device/S3.php:956
    at /usr/src/code/packages/storage/src/Device/S3.php:331
    at /usr/src/code/packages/storage/src/Device.php:186
    ... 11 more

────────────────────────────────────────────────────────────
http.request · 1.7ms · fb960906

  level              info
  http.method        GET
  http.path          /v1/health/version
--
  http.path          /v1/functions/:functionId/deployments/duplicate
  http.response.code 403

  Utopia\Storage\Exception\RemoteException: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.
    at /usr/src/code/packages/storage/src/Device/S3.php:1094
    at /usr/src/code/packages/storage/src/Device/S3.php:956
    at /usr/src/code/packages/storage/src/Device/S3.php:331
    at /usr/src/code/packages/storage/src/Device.php:186
    ... 11 more````

no, the function is not big at all

then it might be an R2 only issue, i went deeper on the issue, and ye some S3 providers do more checks check, including on the content-length signature.

cc @Arnab Chatterjee can u take a look? not sure if its the actual issue but worth an investigation if we can recreate

ok in prod i do get the error with garage, i'm checking, i will hand my analysis here once i'm done, but it does look like it's because of the new content-length header

since my setup is self-hosted S3, it's apparently because traefik remove the content-length header.
So the signature includes the content-length header, but when the S3 server receive it, it doesn't see the header, and so complains the signing is broken.

HEAD is defined as a no-body method. For that, Traefik / Go’s HTTP stack routinely drops Content-Length (especially Content-Length: 0) before talking to Garage. Utopia already put content-length in the SigV4 signed-header list, so Garage sees: signature claims that header, request doesn’t have it → signed header content-length is not present.

For R2 it seems different tho :

function deployment duplicate fails with SignatureDoesNotMatch when storage is Cloudflare R2.

What happens :

• Duplicate calls $device->copy(source → dest).
• With a custom endpoint (*.r2.cloudflarestorage.com), getDevice() builds new S3(...) without the $bucket argument → server-side CopyObject is skipped.
• It falls back to read() then write() — a streamed GET.
• In 2.4.0, every S3 request signs Content-Length (even empty GETs). R2 rejects that request as a bad signature.
• Stack lands in S3.php around read / call, not upload.

Why it worked before: older utopia did not always put content-length in SignedHeaders, so R2 never disagreed with it.

yep sure, I will take a look

thanks for the minute details

@user can you check this one

now another issue bro

appwrite | level error
appwrite | http.method GET
appwrite | http.hostname jmkin.top
appwrite | http.locale en
appwrite | project.id printa4
appwrite | user.id 6ac8be9b41d360061fd7
appwrite | database.host database_db_main
appwrite | http.path /v1/tablesdb/:databaseId/tables/:tableId/columns
appwrite | http.response.code 404
appwrite |
appwrite | Appwrite\Extend\Exception: Table with the requested ID 'shopprotected' could not be found.
appwrite | at /usr/src/code/src/Appwrite/Platform/Modules/Databases/Http/Databases/Collections/Attributes/XList.php:81
appwrite | at /usr/src/code/src/Appwrite/Platform/Modules/Databases/Http/TablesDB/Tables/Columns/XList.php:65
appwrite | at /usr/src/code/packages/http/src/Http.php:649
appwrite | at /usr/src/code/packages/http/src/Http.php:877
appwrite | ... 4 more
appwrite |
appwrite | ────────────────────────────────────────────────────────────

is it on cloud/self hosted?
Its a metadata or table not found error
have you manually edited the db or something if on self hosted?

selfhosted