Best practices for loading large data sets in a Next.js App using Appwrite Cloud?

Hi everyone!

I’m building a CRM-like application using Next.js and Hono with Appwrite Cloud as the backend to manage my documents and databases. The app is designed for workspaces, and each workspace typically has around 1,000 clients stored as documents in the database.

I’m currently struggling to figure out the best practice for loading these client documents efficiently, especially because they’re used in many features across the app, such as:

• Searching for clients
• Managing attendance for events
• Various other interactions

Given that I need to frequently access this client data across the app, I’m concerned about:

1. Performance issues when loading large data sets
2. Excessive query requests to Appwrite, which could slow down the app or hit rate limits

Could someone please suggest the best way to load and manage this data efficiently? Should I consider some kind of caching mechanism, server-side fetching, batch loading, load them all at ones, or even something like infinite scrolling to manage the client list?

I’m open to any suggestions or best practices for handling large data sets in Appwrite within a Next.js environment.

Thanks in advance! 😊