Migration from 1.7.4 to 1.8.0 deadlocked database due to OOM error

After having updated to 1.8.0 yesterday I tested the setup with no issues. Today I got message that our application is unreachable. Investigation showed that the database and the swoole worker pool got deadlocked by queries run during the migration.

Server: 16GB/8CPU
Worker configuration: _APP_WORKER_PER_CORE=10, 8 CPU cores = 80 Swoole workers

This is the information I found:

Deadlocked queries:

[see detailed info in the thread below, due to Discord character limit]

Hope this gives some insights. I'm wondering, is there a misconfiguration involved on my side? I'm managing another application too (still on 1.7.4), and am a little worried about updating it at the moment.

Restarting the appwrite container and killing the stuck queries using docker exec appwrite-mariadb mysql -u user -ppassword -e 'KILL 110; KILL 112; KILL 150; KILL 151; KILL 154; KILL 156; KILL 158; KILL 65; KILL 12;' got the backend back running, but I guess this is something to investigate nevertheless?

Please let me know if there is more information I can provide.

Database Status at Detection:

MariaDB Configuration:

Table Impact Analysis:

Logs:
appwrite
[Error] URL: /v1/migrations
Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453
[2025-11-03 12:18:15 *81.64] ERROR php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453
[Error] URL: /v1/migrations
[Error] URL: /v1/migrations

[Error] Type: Appwrite\Extend\Exception
[Error] Message: Document with the requested ID could not be found.
[Error] File: /usr/src/code/src/Appwrite/Platform/Modules/Databases/Http/Databases/Collections/Documents/Get.php
[Error] Line: 116

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453
[2025-11-03 12:18:15 *81.64] ERROR php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453
[2025-11-03 12:18:15 $7.0] WARNING Server::check_worker_exit_status(): worker(pid=81, id=64) abnormal exit, status=255, signal=0
Worker 65 started successfully
Using deprecated logging configuration. Please update your configuration to use DSN format.Unable to parse DSN: sentry://:@
[Error] Timestamp: 2025-11-03T12:21:49+00:00
[Error] Method: GET

[2025-11-03 12:18:15 *81.64] ERROR php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453

What the AI says about this:
Exact Failure Sequence:

1. Migration process attempted to read/transform _1_audit table (2.28 GB)
2. Utopia Database Document class built array in memory (line 453)
3. Memory exhausted at 512 MB trying to allocate final 12 KB
4. Worker #64 (PID 81) crashed with exit status 255
5. Swoole restarted worker #65, but did not release database locks from crashed worker #64
6. Database metadata locks remained indefinitely:
   • LOCK TABLES _1__metadata READ LOCAL (from migration backup/read process)
   • UPDATE on _console_project (waiting for metadata lock)
   • INSERT into _2_stats (waiting for metadata lock)
     Code Context (Document.php:453):

if (empty($value)) {
    $output[$key] = $value;
} else {
    $output[$key] = $value;  // Line 453 - memory exhaustion here
}```

Root Cause:
Migration attempted to load entire 2.28 GB audit table into memory for processing. When worker crashed due to OOM, database connection was not properly closed, leaving table metadata locks active. All subsequent queries requiring metadata access deadlocked.

what's this migration you're doing? Was this successful before upgrading?

I did the upgrade process according to the docs, so it's just the docker compose exec appwrite migrate.

And actually i saw my fix from yesterday (restarting appwrite container, killing the locked queries) didn't work, the database got bricked again

Wait i'm wrong, it's not only this migration. I also have a recurring backup process creating a database dump every 4 hours using mysqldump. Seems like with the size of the audit table it caused this deadlock again.

But it only happened since the upgrade to 1.8.0, that this backup command kept running into this deadlock. Before it never made any problems.

So i guess it has to do with the errors during migration (see logs above). Perhaps it left a corrupted state. ALthough I killed the stuck queries, so it should have also cleared the locks. So i don't know why the mysqldump would repeatedly result in the deadlocked state now.

I added the --single-transaction flag to mysqldump and hope it will mitigate the issue.

My server also froze with a timeout. The cause was this: for some reason, the backup code from version 1.8 onwards was freezing MariaDB.