Skip to content
Start project
Back

Optimal data import and update 400k+ documents

  • 0
  • Self Hosted
  • Functions
  • Databases
  • Web
View on Discord
taavisaft
18 Jan, 2024, 16:42

I have a collection of over 400,000 documents that I need to update daily using a CSV file. Most of the time, only about 1% of the data changes, so I don't want to update everything. This collection is essential for our clients, mainly for full-text searches, so I want to keep downtime to a minimum. If you have any ideas on how to do this efficiently, please share them.

TL;DR
Title: Efficiently importing and updating 400k+ documents Message: I am a developer seeking help with efficiently importing and updating over 400,000 documents daily using a CSV file. The majority of the data remains the same, and I want to minimize downtime as this collection is vital for our clients' full-text searches. Any suggestions for an efficient solution?
Reply

Reply to this thread by joining our Discord

Reply on Discord

Recommended threads

Need support?

Join our Discord

Get community support by joining our Discord server.

Join Discord

Get premium support

Join Appwrite Pro and get email support from our team.

Learn more