Back
database rate limit
Hi guys, I need to import 20millions documents into the database, I am currently running some scripts that add the documents one by one. Its has been a week and I only managed to import 1million due to hitting rate limits very often.
What are the options to populate my collections a bit faster? I've seen some mentions that says the rate limit only applies for 1 connection and that I could have different connections for increasing it? in practice, how you guys implement that?
TL;DR
Developers are struggling with hitting rate limits while trying to import 20 million documents into a database at a slow pace. Possible solution could be to explore concurrent insertions with different connections to bypass rate limits.
On the roadmap for 1.7 is bulk inserting, other than that I think this is all you can do. Depending on how you're inserting them you can look at doing concurrent insertions in your code.
