[SOLVED] How to use the Health endpoints

So i wanted to monitor everything on my appwrite instance tho it seams like there is no API scope available for thoses.

Looking at : https://appwrite.io/docs/references/1.8.x/server-nodejs/health I should be able to create an API key and use it to healthcheck if appwrite is still online. But i can't create an API key with the scope health.read which is required for thoses endpoint.

So now i would have multiple question. Is it possible to run a healthcheck ? on Cloud ? and on Self-Hosted ? If yes, how should we setup the keys ? (what scope are needed and if not in the UI how to enable them)

Why unable to create api key with health?

ok i'm holy blind then xpp. it's like the 10th time i check if it's on the API keys but never saw it. (either got patched in like 1.8.X where i didn't often checked that, or i was just blind as fuck)

also is there a reason why the healthcheck are not public ? (or at least one endpoint to know if the container is running for self-hosted)

[SOLVED] How to use the Health endpoints

Why should it be public?

for example when you want to put a healthcheck on the container (so it restart when appwrite/redis/mysql get disconnected), since you just put a command it's better to just do a curl localhost:80/health than trying to setup the healthcheck and find a way to protect that API key. + that mean that by default containers will not be healthy unless i create my API token and then add it to the healthcheck

also for self hosted most service propose a public API endpoint tho true some doesn't but more rare.

I dont know about self hosting or containers. But, I assume one container contain multiple projects? If so, one project may malfunction. So that api key has only that project related checks. Ignore if what i said does not make any sense 🥲.

yes the self hosted appwrite stack use multiple container (api, realtime, workers) which run multiple projects and orgs inside (it's kinda like we deploy a new region on self hosted) but we should have a health check to check if for example appwrite still can access it's database or redis.

With the current healthcheck you can't really check if the functions-worker that run all your self hosted function (from multiple projects) is still working or not.

The example i have is that recently my cluster had a network issue and, after everything was back, appwrite was throwing error 500 because it couldn't reach the database anymore. Restarting the container solved the issue, but i needed to do that manually.

You could make a feature request ig. But can you confirm, if this is related the appwrite/console sdk? Cause I am building the native version of appwrite which could in theory manage the cloud and selfhosted instances. I would be happy to add anything you can suggest.

there was one here : https://github.com/appwrite/appwrite/issues/2908 tho got closed

i don't think it's related to any sdk but directly to the appwrite source code (like adding a new local only endpoint forr healthcheck) and the docker-compose/Dockerfile to run the healthcheck for the coontainers

also when you say native version, are you talking of the appwrite stack ? or the server/client SDK ?

The appwrite console website Aka The dashboard

so, no it's not related to the console part. only the APIs but could be nice to have a healthcheck to see if the console is still running (checking if for example /console path still returns 200) but the console is less important as it's not connecting to anything server-side (the API request are made from the browser so as long as the console returns 200 it should be fine)

Hi Sorry to disturb again could you clarify if this was the requirement?

In self-hosted Appwrite setups, "Container Uptime" does not equal "System Health." You can have a situation where: The API Container is "Up": It responds to curl and passes standard HTTP health checks. The System is "Down": Internally, the connection to MariaDB is broken, or the Functions Worker is dead. The Consequence: You don't know something is wrong until users start reporting 500 errors or missed emails. Why is this hard to fix? To monitor these specific internal components (DB connectivity, Worker Queues), you must use the Appwrite Health API (health.getDB, health.getQueue*). However, these endpoints usually require a specific API Key with health.read scope. Most admins skip setting up custom monitoring scripts because managing these keys and setting up external cron jobs is a hassle. The Solution: Appwrite Native How I plan to solve it

1. Deep Internal Inspection (The "Pulse") Instead of just pinging the server, the app uses your authenticated Admin Session to query the privileged Health services directly: Database/Cache Check: It confirms the API can actually read/write to MariaDB and Redis. Worker Queue Audit: It checks the size of specific queues (Functions, Mails, Webhooks). If functions-queue > 500, it knows the worker is stuck, even if the container is running.
2. "Sentinel" Background Monitoring You don't need to keep the app open. Background Fetch: The app registers a background task (via expo-background-fetch) that wakes up periodically (e.g., every 15 mins) even when the app is closed. Silent Polling: It performs the "Health Pulse" check in the background.
3. Active Alerting If the Sentinel detects a failure (e.g., "DB Disconnected" or "Queue Spiking"), it triggers a Critical Local Notification.

so yes this was the issue i was having .. container running but only error 500 due to redis and/or maria that went away.

For the solutions :

1. I was wondering which admin session ? the server is not authenticated which is why we can't use the health endpoint so how would it have the admin session ?

For the check if possible yes it would be on all workers, tasks, the API, realtime and executor (and also maybe assistant and browser but don't know if they use cache and/or db) Also on what i saw workers have reconnect capabilities for the db and redis, but the API doesn't.

2. How could that work ? which app ?
3. is the Critical Notifiation a new thing ? can it restart the failed container ?

So my implementation is for self hosters I think you do also have a web console dashboard where you login. and that session as admin level access. without any setup needed by a toggle you can just enable monitoring the containers in background about 15 min duration. which will check the container health and notify you if anything goes wrong. the app is this https://discord.com/channels/564160730845151244/1468260333071110276

And it cannot restart the failed container cause the methods that appwrites console sdk provides are read only. to implement the restarting thing I need to experiment to include a secure SSH Client.instead of an API Key, the user provides their server SSH credentials (Host, User, Private Key/Password). These are stored securely in the phone's Hardware Keychain/Keystore. when you want to restart , the app connects via SSH and runs the "Power User" command: docker compose -f /path/to/appwrite/docker-compose.yml restart. I dont know if this is the way, cause I will experiement by self hosting my self.

then no it's not what i asked

what i asekd is something inside the appwrite stack, not a client somewhere else ...

basicly this :

appwrite:
    container_name: appwrite
    <<: *x-logging
    image: appwrite/appwrite
    ports:
      - 9501:80
    networks:
      - appwrite
    labels:
      - "traefik.enable=true"
      - ....
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock # Only needed for tests
      - ....
    depends_on:
      - mariadb
      - redis
      - coredns
      # - clamav
    entrypoint:
      - php
      - -e
      - app/http.php
    environment:
      - _APP_ENV
      - ...
    extra_hosts:
      - "host.docker.internal:host-gateway"
    # ADD THIS
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost/v1/health"]
      interval: 1m30s
      timeout: 10s
      retries: 3
      start_period: 40s
      start_interval: 5s

just adding this health check could make :

1. the orchestrator know when a container is not working proprely
2. restart them if needed
3. avoid routing on errored container

ok thanks for the discussion.