Function failing with Operation timed out after 9000x
Same result.
oh wait...an hour? holy crap...
the message was the same with operation timed out after 9000061 miliseconds, but the execution time shown in the console was 15 minutes
what's your _APP_FUNCTIONS_TIMEOUT env var set to?
450000
and _APP_FUNCTIONS_BUILD_TIMEOUT?
900
what's in your docker logs for the openruntimes-executor container?
Removing orphan runtimes...
Cleaning up containers...
No containers found to clean up.
Cleanup finished.
Orphan runtimes removal finished.
Pulling v3 images...
Warming up Dart 2.17 environment...
Warming up Dart 3.0 environment...
Successfully Warmed up Dart 3.0!
Successfully Warmed up Dart 2.17!
Image pulling finished.
Starting maintenance interval...
Maintenance interval started.
Starting stats interval...
Stats interval started.
Executor is ready.```
that's the entire thing
the dart:v3-3.0 container does't have anything but ```f 17f7e506e2738f471e030fd44fcfa402af74b28ef07614aacf22bce8bd59c2f4
Preparing for start ... Starting ...
that's weird...there should be something about your function...these are the logs from when the timeout occurred?
yea. I haven't restarted it
anything in the appwrite container?
[Error] Timestamp: 2023-09-14T19:43:43+00:00
[Error] Method: GET
[Error] URL: /v1/databases/:databaseId/collections/:collectionId/documents
[Error] Type: Appwrite\Extend\Exception
[Error] Message: Database not found
[Error] File: /usr/src/code/app/controllers/api/databases.php
[Error] Line: 2963
[Error] Timestamp: 2023-09-14T19:46:48+00:00
[Error] Method: GET
[Error] URL: /v1/databases/:databaseId/collections/:collectionId/documents
[Error] Type: Appwrite\Extend\Exception
[Error] Message: Database not found
[Error] File: /usr/src/code/app/controllers/api/databases.php
[Error] Line: 2963``` doesn't appear related
or well.. maybe there's an exception happening here and it's not making it into the logs.
looks like it happened after though
It seems that it actually completed this time though. The last part of the function is deleting some files from the bucket, and they were removed, along with a record that prevents it from being run twice. There doesn't seem to be anything wrong with the function. If I run it with a small bit of data so the execution is shorter, the information is available in the consoles execution logs.
how about logs for the appwrite-worker-functions container?
It does have where the exception happened:
Fetched 6 functions...
[Job] (65035fc32762f1.01852600) successfully run.
[Job] Received Job (65035fc3530782.64248465).
[Job] (65035c3f2b0620.52912777) failed to run.
[Job] (65035c3f2b0620.52912777) Operation timed out after 900067 milliseconds with 0 bytes received with status code 0
[Error] Type: Exception
[Error] Message: Operation timed out after 900067 milliseconds with 0 bytes received with status code 0
[Error] File: /usr/src/code/app/workers/functions.php
[Error] Line: 284
but otherwise it's just a lot of
Fetched 6 functions...
[Job] (6503661f64e133.46487641) successfully run.
[Job] Received Job (6503661f7638a6.82070204).
[Job] Received Job (6503661f786c51.57933980).
Fetched 6 functions...
[Job] (6503661f7638a6.82070204) successfully run.
[Job] (6503661e56b184.03639233) successfully run.
Fetched 6 functions...
[Job] (6503661f786c51.57933980) successfully run.
over and over..
Ok I think it is _APP_FUNCTIONS_BUILD_TIMEOUT. Would you please increase that to whatever you had the other timeout env var set to?
Thanks. I will give that a try here shortly.
That seemed to do the trick. Thanks
FYI, I've created an issue for this: https://github.com/appwrite/appwrite/issues/6244
