DatabasesSelf Hosted

Attributes stuck in processing

Dec 27, 2023, 17:13

View on Discord

Votes

Replies

Participants

Unknown

Messages

DrakeOriginal post

Oct 17, 2023, 17:18

i wonder if that's related 🧐

Summary

Title: Stuck attributes and MariaDB connection issues Solution: The user increased the connection limit in MariaDB, which resolved the issue with too many connections. The stuck attributes seemed to have cleared up on their own. There is an error in the mariadb container that may or may not be related. To clear stuck attributes, manually delete the column from the table and update the row in the metadata table. The root cause of why the attributes became stuck is unknown. The problem with creating attributes is unrelated.

Drake

Oct 17, 2023, 17:18

can you share details about all the collections?

Ryan

Moderator

Oct 17, 2023, 17:22

Yeah, was there anything specific you wanted to know?

Drake

Oct 17, 2023, 17:23

all the details to see if there are any attributes that might conflict

Ryan

Moderator

Oct 17, 2023, 17:26

These are all of the attributes in the collection, there are 908 documents for this collection, about 1M reads/month

Ryan

Moderator

Oct 17, 2023, 17:26

These are the attributes of the collection being used for the relationship

IsItBathTime

Rank 3: Container

Oct 17, 2023, 23:32

I've got the same issue too on appwrite cloud 😦

Drake

Oct 17, 2023, 23:49

Your problem is unrelated

Drake

Oct 17, 2023, 23:50

I'm pretty sure it became stuck because you tried to delete it...

Drake

Oct 17, 2023, 23:50

These logs are from trying to delete it. The root cause for why it was processing is unknown

Drake

Oct 17, 2023, 23:51

You would need to find the logs for when it was processing

Ryan

Moderator

Oct 17, 2023, 23:54

The only logs from when it was processing is the original one in this thread, that was all that was showing in the logs when I tried creating the attribute

Drake

Oct 18, 2023, 00:05

maybe your server was overloaded, then, and couldn't handle creating the attribute? sounds really odd, though...

Ryan

Moderator

Oct 18, 2023, 00:06

I’ll take a look at our server metrics to see how it was managing at the time

Ryan

Moderator

Oct 18, 2023, 09:08

I've had a look thought my server metrics and I've not broken 50% usage in any kind. Is there a way to clear the attributes that are stuck so I can have a look at trying again?

Drake

Oct 18, 2023, 12:44

You would have to manually go into the database and delete things. Delete the column from the table and update the row in the metadata table

Ryan

Moderator

Oct 18, 2023, 12:45

Okay, thank you!

Ryan

Moderator

Oct 19, 2023, 12:41

So this has gotten weirder, I'm looking in the database container and both the table and the metadata have no record of the stuck attributes even though they're still appearing in the Appwrite console

Ryan

Moderator

Oct 19, 2023, 16:44

I'm also getting an error when trying to create any attributes on any collections, this is what I'm seeing in worker-databases:

Stack trace:
#0 /usr/src/code/app/init.php(700): PDO->__construct('mysql:host=mari...', 'user', '1be2e4156b08898...', Array)
#1 @swoole-src/library/core/Database/PDOProxy.php(82): {closure}()
#2 @swoole-src/library/core/Database/PDOStatementProxy.php(71): Swoole\Database\PDOProxy->reconnect()
#3 /usr/src/code/vendor/utopia-php/database/src/Database/Adapter/SQL.php(122): Swoole\Database\PDOStatementProxy->__call('execute', Array)
#4 /usr/src/code/vendor/utopia-php/database/src/Database/Database.php(2265): Utopia\Database\Adapter\SQL->getDocument('attributes', '1_4_typeOfArtis...', Array)
#5 /usr/src/code/app/workers/databases.php(83): Utopia\Database\Database->getDocument(Object(Utopia\Database\Document), '1_4_typeOfArtis...')
#6 /usr/src/code/app/workers/databases.php(41): DatabaseV1->createAttribute(Object(Utopia\Database\Document), Object(Utopia\Database\Document), Object(Utopia\Database\Document), Object(Utopia\Database\Document))
#7 /usr/src/code/src/Appwrite/Resque/Worker.php(125): DatabaseV1->run()
#8 /usr/src/code/vendor/resque/php-resque/lib/Resque/Job.php(201): Appwrite\Resque\Worker->perform()
#9 /usr/src/code/vendor/resque/php-resque/lib/Resque/Worker.php(303): Resque_Job->perform()
#10 /usr/src/code/vendor/resque/php-resque/lib/Resque/Worker.php(242): Resque_Worker->perform(Object(Resque_Job))
#11 /usr/src/code/vendor/resque/php-resque/bin/resque(185): Resque_Worker->work('0.1', false)
#12 {main}

Drake

Oct 19, 2023, 18:14

can you share more of the logs before and after?

Ryan

Moderator

Oct 19, 2023, 19:10

[critical] (Job{v1-database} | ID: 55033e8bc68ed558c4d5dd5f654d1eb7 | DatabaseV1 | {there's a lot of JSON here, can send it if you want} has failed PDOException: SQLSTATE[HY000] [1040] Too many connections in /usr/src/code/app/init.php:700

I'm also seeing an error in the mariadb container, so not sure if it's related?

2023-10-19 14:56:49 0 [Warning] Aborted connection 0 to db: 'unconnected' user: 'unauthenticated' host: 'connecting host' (Too many connections)
2023-10-19 15:50:07 1877 [Warning] Aborted connection 1877 to db: 'appwrite' user: 'user' host: '172.24.0.11' (Got timeout reading communication packets)

Ryan

Moderator

Oct 20, 2023, 14:46

I've managed to make progress on the problem, it was just that the connection limit in MariaDB was maxing out with my traffic, I increased it, restarted the containers and it's working again. Now my only problem is getting rid of the existing attributes that are either stuck or processing. They're not appearing in the tables or metadata table when looking directing in MariaDB so I'm not sure where else the console would be finding them? The attributes seem to have cleared up now, all looks good again

Drake

Oct 20, 2023, 18:04

increased it, restarted the containers and it's working again

How did you do that?

Drake

Oct 20, 2023, 18:05

Related Github issue: https://github.com/appwrite/appwrite/issues/6968

Ryan

Moderator

Oct 20, 2023, 18:09

I changed this line to add --max-connections=500 https://github.com/appwrite/appwrite/blob/fc35e0f6ffd5ce7dab863515983e1495027e410c/docker-compose.yml#L808 and then just did docker-compose restart and it seems to have worked

Attributes stuck in processing

Need support?_