Large File Upload Issue with S3/RustFS

Hi, we are seeing a reproducible large upload failure with Appwrite 1.8.0 using S3-compatible storage through RustFS.

A file upload of about 10.7 GB consistently fails around the 1651st 5 MB chunk with a 500 error. In the logs we also saw: Invalid document structure: Attribute "metadata" has invalid type. Value must be a valid string and no longer than 75000 chars.

Our current understanding is that Appwrite stores multipart upload state in the internal files.metadata field, and that this field becomes too large when many chunks are uploaded. We also checked the current state and found:

Appwrite 1.9.0 is out, but we could not find a clear fix for this in the release notes
in the 1.9.0 source, files.metadata still seems to be limited to 75000
the 1.9.0 migration even appears to set bucket metadata to 65534
So our question is: is this understanding actually correct, or are we missing something? If the upload logic is still the same, it looks like this issue would happen earlier in 1.9.0 rather than later.

Is there an official fix or recommended workaround for this?

thanks and best regards
Matthias

Hey, we are investigatin some issues right now actually regarding upload. One of the issues are OOM operations when uploading large files.

You should expect a hotfix in the next days 🙂

Hello, thanks for the reply. Does this also apply to S3 adapters, or only to Appwrite's internal storage?

Appwrite already acknowledged this exact bug class in issue #7439 for S3-compatible storage, where multipart upload metadata became too large. Since we are still seeing the same metadata failure on 1.8.0, it looks like the old fix may have raised the ceiling rather than removing the underlying limit. https://github.com/appwrite/appwrite/issues/7439

Only affects S3