Migration from 1.7.4 to 1.8.0 deadlocked database due to OOM error

After having updated to 1.8.0 yesterday I tested the setup with no issues. Today I got message that our application is unreachable. Investigation showed that the database and the swoole worker pool got deadlocked by queries run during the migration.

Server: 16GB/8CPU Worker configuration: _APP_WORKER_PER_CORE=10, 8 CPU cores = 80 Swoole workers

This is the information I found:

Deadlocked queries:

-- Query ID: 12, Runtime: 45,413 seconds (12.6 hours)
-- State: Waiting for table metadata lock
-- Origin: console project (host: c9607d771422)
UPDATE `appwrite`.`_console_project` ...

-- Query ID: 65, Runtime: 78,330 seconds (21.7 hours)  
-- State: Waiting for table metadata lock
INSERT INTO `appwrite`.`_2_stats` (`_createdAt`, `_permissions`, `_uid`, `_updatedAt`, `metric`, `pe...

-- Query IDs: 110, 112, 150, 151, 154, 156, 158
-- Runtime: 13,012 - 85,013 seconds (3.6 - 23.6 hours)
-- State: Waiting for table metadata lock
-- Origin: localhost (likely migration or backup process)
LOCK TABLES `_1__metadata` READ /*!32311 LOCAL */,
            `_1__metadata_perms` READ /*!32311 LOCAL */,
            `_1_ab... [truncated]

[see detailed info in the thread below, due to Discord character limit]

Hope this gives some insights. I'm wondering, is there a misconfiguration involved on my side? I'm managing another application too (still on 1.7.4), and am a little worried about updating it at the moment.

Restarting the appwrite container and killing the stuck queries using docker exec appwrite-mariadb mysql -u user -ppassword -e 'KILL 110; KILL 112; KILL 150; KILL 151; KILL 154; KILL 156; KILL 158; KILL 65; KILL 12;' got the backend back running, but I guess this is something to investigate nevertheless?

Please let me know if there is more information I can provide.

Database Status at Detection:

Active connections: 88
Threads running: 10
Locked queries: 9 major queries blocking all metadata access
Primary blocked tables: _1__metadata, _console_project, _2_stats

MariaDB Configuration:

innodb_lock_wait_timeout: 50 seconds
wait_timeout: 28800 seconds (8 hours)
net_read_timeout: 30 seconds
lock_wait_timeout: 86400 seconds (24 hours)

Table Impact Analysis:

_1__metadata - 2.13 MB, 92 rows, InnoDB | Lock: LOCK TABLES READ LOCAL never released after migration crash
_1__metadata_perms - 0.05 MB, 37 rows | Locked together with _1__metadata
_1_audit - 2,279.97 MB (2.23 GB), 3,257,881 rows | Impact: Migration attempted in-memory load → OOM crash
_console_project - Query stuck: UPDATE waiting for metadata lock (12.6 hours)
_2_stats - 6.41 MB, 10,583 rows | Query stuck: INSERT waiting for metadata lock (21.7 hours)
_1_stats - 7.73 MB, 19,535 rows

Logs: appwrite [Error] URL: /v1/migrations Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453 [2025-11-03 12:18:15 *81.64] ERROR php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453 [Error] URL: /v1/migrations [Error] URL: /v1/migrations

[Error] Type: Appwrite\Extend\Exception [Error] Message: Document with the requested ID could not be found. [Error] File: /usr/src/code/src/Appwrite/Platform/Modules/Databases/Http/Databases/Collections/Documents/Get.php [Error] Line: 116

Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453 [2025-11-03 12:18:15 *81.64] ERROR php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453 [2025-11-03 12:18:15 $7.0] WARNING Server::check_worker_exit_status(): worker(pid=81, id=64) abnormal exit, status=255, signal=0 Worker 65 started successfully Using deprecated logging configuration. Please update your configuration to use DSN format.Unable to parse DSN: sentry://:@ [Error] Timestamp: 2025-11-03T12:21:49+00:00 [Error] Method: GET

[2025-11-03 12:18:15 *81.64] ERROR php_swoole_server_rshutdown() (ERRNO 503): Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 12288 bytes) in /usr/src/code/vendor/utopia-php/database/src/Database/Document.php on line 453

What the AI says about this: Exact Failure Sequence:

1. Migration process attempted to read/transform _1_audit table (2.28 GB)
2. Utopia Database Document class built array in memory (line 453)
3. Memory exhausted at 512 MB trying to allocate final 12 KB
4. Worker #64 (PID 81) crashed with exit status 255
5. Swoole restarted worker #65, but did not release database locks from crashed worker #64
6. Database metadata locks remained indefinitely:
   • LOCK TABLES _1__metadata READ LOCAL (from migration backup/read process)
   • UPDATE on _console_project (waiting for metadata lock)
   • INSERT into _2_stats (waiting for metadata lock) Code Context (Document.php:453):

if (empty($value)) {
    $output[$key] = $value;
} else {
    $output[$key] = $value;  // Line 453 - memory exhaustion here
}```

Root Cause:
Migration attempted to load entire 2.28 GB audit table into memory for processing. When worker crashed due to OOM, database connection was not properly closed, leaving table metadata locks active. All subsequent queries requiring metadata access deadlocked.