Back
Can't upload large files with `InputFile.from_bytes()` in the Python SDK
Uploading any file which is chunked with the Python SDK (using InputFile.from_bytes()) fails.
I wasn't entirely sure whether from_bytes() is supposed to take a bytesarray or an io.BytesIO object, but since the SDK calls len() on the input, I'm assuming it takes a bytesarray.
The error is:
TypeScript
Traceback (most recent call last):
File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/client.py", line 100, in call
response.raise_for_status()
File "[PATH_TO_PYTHON_REDACTED]/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://[APPWRITE_ENDPOINT]/v1/storage/buckets/[BUCKET_ID]/files
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[MY_SCRIPT_PATH].py", line 40, in <module>
file_storage = storage.create_file('[BUCKET_ID]', 'unique()', input_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/services/storage.py", line 156, in create_file
return self.client.chunked_upload(api_path, {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/client.py", line 175, in chunked_upload
result = self.call(
^^^^^^^^^^
File "[PATH_TO_PYTHON_REDACTED]/site-packages/appwrite/client.py", line 112, in call
raise AppwriteException(response.json()['message'], response.status_code, response.json().get('type'), response.json())
appwrite.exception.AppwriteException: Server Error
TL;DR
Unable to upload large files using `InputFile.from_bytes()` in the Python SDK. The error message indicates that the chunked upload fails. It's unclear whether `from_bytes()` should take a `bytesarray` or an `io.BytesIO` object, but the code assumes it takes a `bytesarray`. The error traceback suggests a server error.
Unfortunately, there is no solution provided in the support thread.And on the server:
TypeScript
appwrite | [Error] Timestamp: [TIMESTAMP]
appwrite | [Error] Method: POST
appwrite | [Error] URL: /v1/storage/buckets/:bucketId/files
appwrite | [Error] Type: Exception
appwrite | [Error] Message: Failed to read chunk /storage/uploads/app-[PROJECT_ID]/[BUCKET_ID]/tmp_65b052e0e175a7c00935.txt/65b052e0e175a7c00935.part.2
appwrite | [Error] File: /usr/src/code/vendor/utopia-php/storage/src/Storage/Device/Local.php
appwrite | [Error] Line: 183
Here's my code:
TypeScript
import io
from appwrite.client import Client
from appwrite.services.storage import Storage
from appwrite.input_file import InputFile
client = (
Client()
.set_endpoint("[ENDPOINT_URL]")
.set_project("[PROJECT_ID]")
.set_key("[API_KEY]")
)
storage = Storage(client)
input = open("./big_file.txt", 'rb')
input = input.read().decode("utf-8")
bytes = io.BytesIO(bytearray(input, "UTF-8"))
input_file = InputFile.from_bytes(bytes.getvalue(), "test_file.txt", mime_type=("text/plain", None))
file_storage = storage.create_file('[BUCKET_ID]', 'unique()', input_file)
print(file_storage)
big_file.txt is just generated with this:
TypeScript
#!/bin/bash
touch big_file.txt
for i in {1..500000}
do
echo "a bunch of text" >> big_file.txt
done
Oh, and here's what shows up in the Console
