DatabasesSelf HostedStorage

Can't upload large files with `InputFile.from_bytes()` in the Python SDK

Jan 24, 2024, 01:00

View on Discord

Votes

Replies

Participants

Unknown

Messages

ideclonOriginal post

Rank 6: Server

Jan 24, 2024, 00:04

Uploading any file which is chunked with the Python SDK (using InputFile.from_bytes()) fails.

I wasn't entirely sure whether from_bytes() is supposed to take a bytesarray or an io.BytesIO object, but since the SDK calls len() on the input, I'm assuming it takes a bytesarray.

The error is:

Traceback (most recent call last):
  File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/client.py", line 100, in call
    response.raise_for_status()
  File "[PATH_TO_PYTHON_REDACTED]/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://[APPWRITE_ENDPOINT]/v1/storage/buckets/[BUCKET_ID]/files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "[MY_SCRIPT_PATH].py", line 40, in <module>
    file_storage = storage.create_file('[BUCKET_ID]', 'unique()', input_file)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/services/storage.py", line 156, in create_file
    return self.client.chunked_upload(api_path, {
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/client.py", line 175, in chunked_upload
    result = self.call(
             ^^^^^^^^^^
  File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/client.py", line 112, in call
    raise AppwriteException(response.json()['message'], response.status_code, response.json().get('type'), response.json())
appwrite.exception.AppwriteException: Server Error

Summary

Unable to upload large files using `InputFile.from_bytes()` in the Python SDK. The error message indicates that the chunked upload fails. It's unclear whether `from_bytes()` should take a `bytesarray` or an `io.BytesIO` object, but the code assumes it takes a `bytesarray`. The error traceback suggests a server error. Unfortunately, there is no solution provided in the support thread.

ideclon

Rank 6: Server

Jan 24, 2024, 00:04

And on the server:

appwrite  | [Error] Timestamp: [TIMESTAMP]
appwrite  | [Error] Method: POST
appwrite  | [Error] URL: /v1/storage/buckets/:bucketId/files
appwrite  | [Error] Type: Exception
appwrite  | [Error] Message: Failed to read chunk /storage/uploads/app-[PROJECT_ID]/[BUCKET_ID]/tmp_65b052e0e175a7c00935.txt/65b052e0e175a7c00935.part.2
appwrite  | [Error] File: /usr/src/code/vendor/utopia-php/storage/src/Storage/Device/Local.php
appwrite  | [Error] Line: 183

Here's my code:

Python

import io
from appwrite.client import Client
from appwrite.services.storage import Storage
from appwrite.input_file import InputFile

client = (
    Client()
    .set_endpoint("[ENDPOINT_URL]")
    .set_project("[PROJECT_ID]")
    .set_key("[API_KEY]")
)

storage = Storage(client)

input = open("./big_file.txt", 'rb')
input = input.read().decode("utf-8")

bytes = io.BytesIO(bytearray(input, "UTF-8"))

input_file = InputFile.from_bytes(bytes.getvalue(), "test_file.txt", mime_type=("text/plain", None))

file_storage = storage.create_file('[BUCKET_ID]', 'unique()', input_file)

print(file_storage)

big_file.txt is just generated with this:

Bash

#!/bin/bash

touch big_file.txt

for i in {1..500000}
do
    echo "a bunch of text" >> big_file.txt
done

ideclon

Rank 6: Server

Jan 24, 2024, 00:11

Oh, and here's what shows up in the Console

Join the discussion

Reply to this thread by joining our Discord.

Reply on Discord

Need support?_

Join Discord Get premium support