[SOLVED] Best way to create 10000 records?

Title.

Trying to import records from another DB. It fails part of the way through and I have no idea why. How to do?

TypeScript

Converted Contact!
Converted Contact!
Converted Contact!
Converted Contact!
Finished converting Contacts.json, created 10082 new contacts
Operation failed: AppwriteException [Error]: Server Error
    at Client.call (/Users/zachhandley/Documents/GitHub/aba-app-astro/website/node_modules/node-appwrite/lib/client.js:174:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Databases.createDocument (/Users/zachhandley/Documents/GitHub/aba-app-astro/website/node_modules/node-appwrite/lib/services/databases.js:1691:16)
    at async initOrUpdateCollections (file:///Users/zachhandley/Documents/GitHub/aba-app-astro/website/src/api/migrations/collections.ts:1:9611)
    at async setupDatabases (file:///Users/zachhandley/Documents/GitHub/aba-app-astro/website/src/api/migrations/setupDatabase.ts:1:839)
    at async Object.runMigrations (file:///Users/zachhandley/Documents/GitHub/aba-app-astro/website/src/api/server.ts:1:887)
    at async applyMigrations (file:///Users/zachhandley/Documents/GitHub/aba-app-astro/website/migrate.ts:1:490)
    at async runCommand (file:///Users/zachhandley/Documents/GitHub/aba-app-astro/website/migrate.ts:1:1175) {
  code: 500,
  type: 'general_unknown',
  response: {
    message: 'Server Error',
    code: 500,
    type: 'general_unknown',
    version: '1.4.13'
  }
}

TL;DR

Developers discussing the best way to create 10,000 records in a database. Suggestions include using functions, setting limits, iterating in batches, and creating import functionalities. A snippet of code is shared for filtering and checking existing documents. Advice includes using a Python script to import data and checking Docker logs for 500 errors.

that's after meh

5088 contacts or so

so like 8.33 minutes

in theory if my _NUM_WORKERS ENV var works

then I can be creating functions and mass creating them

honestly, if it's a one-time, functions isn't worth it

personal opinion, function also times out too much to be sure it works well

@Steven

success!

gracias for the idea

also sidebar

there should totally be an equals Query for an object that JSON stringifies it and then does a comparison between all the keys found in the collection that match the keys in the object, and then compares the values for X values to determine if any are a match

cause it's really useful when importing data lol

I was nervous about how many queries though (max 25) so I only used 5

it's funny for how many things you can use a simple table in a database

Ikr

It’s hard to gauge what kinds of limits to set e.g. like

It’s hard to tell how much of some data will be over X characters and what not