LetsEncrypt Certificate Renewal is not working

I followed the instructions on the docs on how to set up Appwrite certificates properly, but even though I did all the steps I can't get them to work!
WHAT I DID:

• double-checked that port 80 and 443 are opened
• double-checked my server is accessible using the domain I've chosen (let's say mydomain.com)
• double-checked that the _APP_ENV variable is set to production and that _APP_DOMAIN and _APP_SYSTEM_SECURITY_EMAIL_ADDRESS are correct in the .env file of
• tried to create a test file in /storage/certificates/.well-known/acme-challenge/ (inside the appwrite-worker-certificates container) and access it typing https://mydomain.com/.well-known/acme-challenge/test in my browser

WHAT I GET:
When I try to generate the certificate using docker compose exec appwrite ssl, in the appwrite-worker-certificates logs I see:

And inside the letsencrypt log file there is this error: https://privatebin.devol.it/?52af7f3c1d748f3c#GJFHv5iDnNNzhh1aUEUqDHnaFvKQBx4AhM4zjyyqdeFt
How can I fix it?

Do you have any other reverse proxy in front of Appwrite? Do you use cloudflare proxy?

Nope, I'm just using the traefik that is provided with appwrite, modified a little bit to host a static website using caddy as well... do you think it could be the problem? I can share my traefik configuration if it could be useful

Caddy? Isn't that a reverse proxy? Is that only in front of your static website?

It can be used also as a web server, I'm using it to provide my Flutter Web App...
To do so I added the following service to appwrite's docker-compose.yaml:

appwrite_hosting:
    image: caddy:latest
    container_name: appwrite-hosting
    restart: unless-stopped
    volumes:
      - ${AW_HOSTING_DATA_VOLUME_PATH}:/data
      - ${AW_HOSTING_SOURCE_VOLUME_PATH}:/src
      - ${AW_HOSTING_CADDYFILE}:/etc/caddy/Caddyfile
    networks:
      - appwrite
    labels:
      - "traefik.enable=true"
      - "traefik.constraint-label-stack=appwrite"
      - "traefik.docker.network=appwrite"
      - "traefik.http.services.appwrite_hosting.loadbalancer.server.port=80"
      # http
      - traefik.http.routers.appwrite_hosting_http.entrypoints=appwrite_web
      - traefik.http.routers.appwrite_hosting_http.rule=Host(`${AW_HOSTING_FQDN}`)
      - traefik.http.routers.appwrite_hosting_http.service=appwrite_hosting
      # https
      - traefik.http.routers.appwrite_hosting_https.entrypoints=appwrite_websecure
      - traefik.http.routers.appwrite_hosting_https.rule=Host(`${AW_HOSTING_FQDN}`)
      - traefik.http.routers.appwrite_hosting_https.service=appwrite_hosting
      - traefik.http.routers.appwrite_hosting_https.tls=true
      - traefik.http.routers.appwrite_hosting_https.tls.certresolver=awhosting_certresolver

And added the following commands below the existing ones in the Traefik service

# Configuration of Certresolver for Appwrite Hosting
      - --certificatesresolvers.awhosting_certresolver.acme.httpchallenge=true
      - --certificatesresolvers.awhosting_certresolver.acme.httpchallenge.entrypoint=appwrite_web
      - --certificatesresolvers.awhosting_certresolver.acme.email=${_APP_SYSTEM_SECURITY_EMAIL_ADDRESS}
      - --certificatesresolvers.awhosting_certresolver.acme.storage=/letsencrypt/acme.json

Also, I don't know if it could be of any use for you but here is the content of the Caddyfile:

I'm not good with traefik so I'm not sure, but maybe the new traefik stuff is causing a problem?

Honestly, I would not modify the original docker-compose.yml file because upgraded wipe the file. I'd suggest using a docker-compose.override.yml file.

Personally, to host multiple apps on the same server, I put a Nginx Proxy Manager (NPM) in front of Appwrite because it's easy to use and I don't need to manually do too much nginx config edits

Oh ok, so you’d suggest to put another reverse proxy (NPM) in front of the one shipped with appwrite (Traefik?) Could you please share how you did that? Like at the moment appwrite’s Traefik is using ports 80 and 443 so I can’t spin up NPM in a standalone container, can I?

(Sorry if these are dumb questions but it’s the first time I use two reverse proxies one in front of the other 😅)

Yes, you'd run appwrite on non-standard ports. Maybe this will help: https://medium.com/@stnguyen90/how-to-run-appwrite-behind-nginx-19348ed34243

not a dumb question at all!

Thank you! I'll check it out then

I think you did the same as me. Having another app at the root domain (or subdomain) instead of the default Appwrite Console.
I made it work in my case by instructing Traefik proxy to also redirect the Acme challenge request to the Appwrite console, instead of my frontend app, like this:

  appwrite:
    container_name: appwrite
    <<: *x-logging
    image: appwrite-dev
    // some configs omitted
    labels:
      - "traefik.enable=true"
      - "traefik.constraint-label-stack=appwrite"
      - "traefik.docker.network=appwrite_appwrite"
      - "traefik.http.services.appwrite_api.loadbalancer.server.port=80"
      # http
      - traefik.http.routers.appwrite_api_http.entrypoints=appwrite_web
-     - traefik.http.routers.appwrite_api_http.rule=Host(`console.myapp.mydomain.tld`) && PathPrefix(`/`)
+     - traefik.http.routers.appwrite_api_http.rule=(Host(`console.myapp.mydomain.tld`) && PathPrefix(`/`)) || (Host(`myapp.mydomain.tld`) && PathPrefix(`/.well-known/acme-challenge/`))
      - traefik.http.routers.appwrite_api_http.service=appwrite_api