Hi, I'm trying to, figure out why fetching a large amount of documents is so long. I suspect a network bottleneck as using the CLI locally is way faster. Doing so I checked my Chrome dev tools and I can't find any Content-encoding: zip or so. So my question: is there any compression done in Appwrite http responses? If not, is there a way to enable it? Thx
Summary
The user is trying to figure out why fetching a large amount of documents is taking a long time. They suspect a network bottleneck and are wondering if there is compression being done in Appwrite http responses and if there is a way to enable it. The thread discusses the limit for compressed http responses and suggests tweaking request settings to keep them under the limit. The solution is to reduce the limit and use pagination instead of fetching a large number of documents at once. The user mentions they need offline functionality for their PWA, so pagination may not be suitable for their use case. The thread also mentions the usage of brotli compression and the absence
Drake
Admin
Dec 1, 2023, 18:22
we use brotli: 🧐
Drake
Admin
Dec 1, 2023, 18:22
are you fetching in your app or Appwrite Console?
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:23
In my app which is a PWA, so Chrome involved in the process
Drake
Admin
Dec 1, 2023, 18:23
Here's another test from my (react) web app:
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:26
Hmm, strange. My response contains no "Content-encoding"...
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:27
while others do present this "Content-encoding: br"
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:29
The requests missing this field are big queries with limit() and offset()
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:39
I reduced the limit() from 5000 down to 100 and I've the Content-encoding: br field. Any clue?
Drake
Admin
Dec 1, 2023, 18:40
lol you shouldn't be fetching so many things at once 😜
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:42
Ah? This is bad news for my app...
Drake
Admin
Dec 1, 2023, 18:42
why can't you paginate?
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:43
No, I need offline feature, this is a PWA and my users may be offline and still be able to use the app.
Drake
Admin
Dec 1, 2023, 18:43
still....why can't you paginate?
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:43
Except the compression issue, all is performing well like this.
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:44
Paginate? The fetch of my 40K documents?
Gildas Quiniou
Rank 2: Process
Dec 1, 2023, 18:45
I'm currently fetching 100 at a time... so 400 requests. Takes way longer.
Drake
Admin
Dec 1, 2023, 18:52
if you're using limit + offset, can you do them concurrently?
What's the time difference between 1 request of limit(5000) and 50 request of limit (100)? You might even be able to turn that for a better result