[1.9.5] Functions building indefinitely and cause server to freeze

I try to deploy some function, it show "Building..." during minutes and freeze the entire server.

Here is the result of docker logs appwrite-worker-builds

TypeScript

worker.v1-builds · 295.44s · 4e97ac22

  level                            info
  project.id                       xxx-prod
  build.type                       deployment
  resource.id                      db-synchronization
  resource.type                    functions
  deployment.id                    6a5f67e6c5b4152f8b34
  build.timeout                    900
  build.runtime                    dart-3.5
  build.version                    v3
  build.cpus                       1
  build.memory                     512
  deployment.status                failed
  build.source_size                18801
  build.node_modules_cache.enabled false
  build.node_modules_cache.key     
  build.logs.finished              true
  build.runtime.error_type         Executor\Exception
  build.runtime.error_message      Internal server error.
  build.error.stage                deployment
  build.error.type                 Executor\Exception
  build.error.message              Internal server error.
  build.error.file                 /usr/src/code/src/Executor/Executor.php
  build.error.line                 109
  build.duration                   295

Maybe it's related to some .env variables wrongly set?

TypeScript

_APP_FUNCTIONS_SIZE_LIMIT="30000000"
_APP_COMPUTE_SIZE_LIMIT="30000000"
_APP_FUNCTIONS_BUILD_SIZE_LIMIT="2000000000"
_APP_FUNCTIONS_TIMEOUT="900"
_APP_FUNCTIONS_BUILD_TIMEOUT="900"
_APP_COMPUTE_BUILD_TIMEOUT="900"
_APP_FUNCTIONS_CONTAINERS="10"
_APP_FUNCTIONS_CPUS="0"
_APP_COMPUTE_CPUS="0"
_APP_FUNCTIONS_MEMORY="0"
_APP_COMPUTE_MEMORY="0"
_APP_FUNCTIONS_MEMORY_SWAP="0"
_APP_FUNCTIONS_RUNTIMES="node-18.0,php-8.0,python-3.9,python-3.12,ruby-3.0,dart-2.17,dart-3.0,dart-3.5"

TL;DR

Functions building indefinitely and causing server to freeze. Need to verify and run specific SQL patch to fix the issue for Appwrite version 1.9.5.Ensure to backup before running the patch and consider potential problems with future migrations. Additional options include using a new Appwrite instance or applying an SQL script.Upgrading to version 2.0.0 will require running migrations for version 24 first.可能的解决方案：运行特定的SQL修补程序来解决问题。

is there any build logs in the UI ?

does it only happen to Builds ? (if you have one already builded from an older version, does running it works ?)

between 1.9.0 and 1.9.5 the executor have changed, maybe your still using the old one ? (check if your using 0.25.2 and not 0.11.x, also check if the docker compose environment part includes OPR_EXECUTOR_CONNECTION_STORAGE which is a new env)

also if you want (hoping it works on your PC too, here is all the changes of the .env between 1.9.0 and 1.9.5 : https://github.com/appwrite/appwrite/compare/1.9.0...1.9.5 (go to "Files Changed" and search for ".env")

also question, i don't remember if it's you, but do you use S3 with the executor ?

<@493852865907916800>

👉 is there any build logs in the UI ? Yes when forcing server to restart I can see someting like:

TypeScript

[12:36:59] [open-runtimes] Build cache miss. 
[12:36:59] [open-runtimes] Environment preparation started. 
Resolving dependencies...
Downloading packages...
+ async 2.13.1
+ uuid 4.6.0
+ web 1.1.1
Changed 20 dependencies!
Compiling ...

Internal server error.

👉 does it only happen to Builds ? (if you have one already builded from an older version, does running it works ?)

Running works well on old yes

👉 between 1.9.0 and 1.9.5 the executor have changed, maybe your still using the old one ? (check if your using 0.25.2 and not 0.11.x, also check if the docker compose environment part includes OPR_EXECUTOR_CONNECTION_STORAGE which is a new env)

TypeScript

openruntimes-executor:
    container_name: exc1
    hostname: exc1
    logging:
      driver: json-file
      options:
        max-file: "5"
        max-size: 10m
    stop_signal: SIGINT
    image: openruntimes/executor:0.25.1
    restart: unless-stopped
    networks:
    - appwrite
    volumes:
    - /var/run/docker.sock:/var/run/docker.sock

No it's no me for S3 with executor 🙂

hmmm, you don't show the environment for the executor, is it normal ? did you strip it ? (there shouldn't be anything sensible in that part)

from the GH repo (main shows 0.25.2, and tag 1.9.5 shows 0.25.1, maybe there was a bug and got fixed in 0.25.2, you can try that as well if you want)

In .env I have:

TypeScript

_APP_EXECUTOR_IMAGES="openruntimes/node:v5-18.0,openruntimes/php:v5-8.0,openruntimes/python:v5-3.9,openruntimes/python:v5-3.12,openruntimes/ruby:v5-3.0,openruntimes/dart:v5-2.17,openruntimes/dart:v5-3.0,openruntimes/dart:v5-3.5,openruntimes/static:v5-1,openruntimes/node:v5-22,openruntimes/flutter:v5-3.29"

Maybe it's wrong no?

It's because it was too big :p

don't think, in mine i have :

TypeScript

_APP_EXECUTOR_IMAGES=openruntimes/node:v5-22,openruntimes/static:v5-1

but the most important one is "OPR_EXECUTOR_RUNTIMES" which concat your functions and sites runtimes (the values looks like this : node-22,node-24,python-3.12,php-8.3,static-1,node-22,node-24)

but it seams like the executor works, but is not able to compile/build your app

currently the guess is that for some reason, your app can't build and crash before the API/workers retreive the logs.

A similar thing happend to me with Angular, where the node version was too old to build it in Appwrite.

As a test for build, can you try to use a template ?

for the template since the template page is not working, click "Create function" and use one of the "Quick start" element

I will check thanks

Well ^^

TypeScript

http.request · 11.9ms · 315128e1

  level              error
  http.method        POST
  http.path          /v1/functions
  http.response.code 500

  Utopia\Database\Exception\Structure: Invalid document structure: Unknown attribute: "providerBranches"
    at /usr/src/code/vendor/utopia-php/database/src/Database/Database.php:5654
    at /usr/src/code/src/Appwrite/Platform/Modules/Functions/Http/Functions/Create.php:230
    at /usr/src/code/vendor/utopia-php/http/src/Http/Http.php:645
    at /usr/src/code/vendor/utopia-php/http/src/Http/Http.php:864
    ... 4 more

I think it's an another issue XD

oh ye provider thingy is an SQL bug from 1.9.0 to 1.9.5 upgrade

full thread on that one : https://discord.com/channels/564160730845151244/1522114393179947018

Will check that thanks. In addition I have some "zombies" docker container:

When I logs on of it there is something like this:

TypeScript

[08:28:19] [open-runtimes] Code extraction started. 
[08:28:21] [open-runtimes] Code extraction finished. 
[08:28:21] [open-runtimes] Environment preparation started. 
[08:28:21] [open-runtimes] Environment preparation finished. 
[08:28:21] [open-runtimes] Runtime started. 
HTTP server successfully started!
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory
cp: can't stat '/tmp/code.tar.gz': No such file or directory

What is this? :p

oh that's strange, and the open-executor don't have logs ? (as it's the one downloading to /tmp)

and does the /tmp on your host have any files showing (when you build)

it seams like the function code, isn't sent to the executor and/or runtime which cause the issue

Nop, I removed theses openruntime zombie containers will see if it was the issue ...

also as a last resort, other people have downgrade the executor to use the one from 1.9.0, and seems to work

Reply to this thread by joining our Discord

Reply on Discord

Need support?

Join our Discord

Get community support by joining our Discord server.

Join Discord

Get premium support

Join Appwrite Pro and get email support from our team.

Learn more

Recommended threads

Need support?

Join our Discord

Get premium support