Functions

Function failing with Operation timed out after 9000x

Dec 27, 2023, 19:19

View on Discord

Votes

Replies

Participants

Unknown

Messages

krunchyOriginal post

Rank 2: Process

Sep 14, 2023, 19:39

Same result.

Summary

The function is failing with an operation timed out error after 9000x milliseconds. The user suggests increasing the `_APP_FUNCTIONS_BUILD_TIMEOUT` environment variable to solve the issue. The logs show that the function is running successfully, but there may be an exception happening that is not being logged. The user also checks the appwrite container and finds unrelated errors. The dart:v3-3.0 container logs do not provide any relevant information. The user provides the docker logs for the openruntimes-executor container, but it does not show any useful information. The user also mentions the `_APP_FUNCTIONS_TIMEOUT` environment variable, but it

Drake

Sep 14, 2023, 19:42

oh wait...an hour? holy crap...

krunchy

Rank 2: Process

Sep 14, 2023, 19:43

the message was the same with operation timed out after 9000061 miliseconds, but the execution time shown in the console was 15 minutes

Drake

Sep 14, 2023, 19:44

what's your _APP_FUNCTIONS_TIMEOUT env var set to?

krunchy

Rank 2: Process

Sep 14, 2023, 19:44

450000

Drake

Sep 14, 2023, 19:44

and _APP_FUNCTIONS_BUILD_TIMEOUT?

krunchy

Rank 2: Process

Sep 14, 2023, 19:44

900

Drake

Sep 14, 2023, 19:45

what's in your docker logs for the openruntimes-executor container?

krunchy

Rank 2: Process

Sep 14, 2023, 19:45

Removing orphan runtimes...
Cleaning up containers...
No containers found to clean up.
Cleanup finished.
Orphan runtimes removal finished.
Pulling v3 images...
Warming up Dart 2.17 environment...
Warming up Dart 3.0 environment...
Successfully Warmed up Dart 3.0!
Successfully Warmed up Dart 2.17!
Image pulling finished.
Starting maintenance interval...
Maintenance interval started.
Starting stats interval...
Stats interval started.
Executor is ready.```

krunchy

Rank 2: Process

Sep 14, 2023, 19:45

that's the entire thing

krunchy

Rank 2: Process

Sep 14, 2023, 19:47

the dart:v3-3.0 container does't have anything but ```f 17f7e506e2738f471e030fd44fcfa402af74b28ef07614aacf22bce8bd59c2f4

Preparing for start ...
Starting ...

Drake

Sep 14, 2023, 19:47

that's weird...there should be something about your function...these are the logs from when the timeout occurred?

krunchy

Rank 2: Process

Sep 14, 2023, 19:48

yea. I haven't restarted it

Drake

Sep 14, 2023, 19:50

anything in the appwrite container?

krunchy

Rank 2: Process

Sep 14, 2023, 19:52

[Error] Timestamp: 2023-09-14T19:43:43+00:00
[Error] Method: GET
[Error] URL: /v1/databases/:databaseId/collections/:collectionId/documents
[Error] Type: Appwrite\Extend\Exception
[Error] Message: Database not found
[Error] File: /usr/src/code/app/controllers/api/databases.php
[Error] Line: 2963
[Error] Timestamp: 2023-09-14T19:46:48+00:00
[Error] Method: GET
[Error] URL: /v1/databases/:databaseId/collections/:collectionId/documents
[Error] Type: Appwrite\Extend\Exception
[Error] Message: Database not found
[Error] File: /usr/src/code/app/controllers/api/databases.php
[Error] Line: 2963``` doesn't appear related

krunchy

Rank 2: Process

Sep 14, 2023, 19:53

or well.. maybe there's an exception happening here and it's not making it into the logs.

krunchy

Rank 2: Process

Sep 14, 2023, 19:53

looks like it happened after though

krunchy

Rank 2: Process

Sep 14, 2023, 20:12

It seems that it actually completed this time though. The last part of the function is deleting some files from the bucket, and they were removed, along with a record that prevents it from being run twice. There doesn't seem to be anything wrong with the function. If I run it with a small bit of data so the execution is shorter, the information is available in the consoles execution logs.

Drake

Sep 14, 2023, 20:42

how about logs for the appwrite-worker-functions container?

krunchy

Rank 2: Process

Sep 14, 2023, 20:51

It does have where the exception happened: ```
Fetched 6 functions...
[Job] (65035fc32762f1.01852600) successfully run.
[Job] Received Job (65035fc3530782.64248465).
[Job] (65035c3f2b0620.52912777) failed to run.
[Job] (65035c3f2b0620.52912777) Operation timed out after 900067 milliseconds with 0 bytes received with status code 0
[Error] Type: Exception
[Error] Message: Operation timed out after 900067 milliseconds with 0 bytes received with status code 0
[Error] File: /usr/src/code/app/workers/functions.php
[Error] Line: 284

krunchy

Rank 2: Process

Sep 14, 2023, 20:51

but otherwise it's just a lot of ```
Fetched 6 functions...
[Job] (6503661f64e133.46487641) successfully run.
[Job] Received Job (6503661f7638a6.82070204).
[Job] Received Job (6503661f786c51.57933980).
Fetched 6 functions...
[Job] (6503661f7638a6.82070204) successfully run.
[Job] (6503661e56b184.03639233) successfully run.
Fetched 6 functions...
[Job] (6503661f786c51.57933980) successfully run.

Plain text

Drake

Sep 14, 2023, 23:17

Ok I think it is _APP_FUNCTIONS_BUILD_TIMEOUT. Would you please increase that to whatever you had the other timeout env var set to?

krunchy

Rank 2: Process

Sep 15, 2023, 15:07

Thanks. I will give that a try here shortly.

krunchy

Rank 2: Process

Sep 15, 2023, 15:44

That seemed to do the trick. Thanks

Drake

Sep 15, 2023, 16:48

FYI, I've created an issue for this: https://github.com/appwrite/appwrite/issues/6244

Function failing with Operation timed out after 9000x

Need support?_