schedule-functions (and sibling schedulers) silent hang after Redis restart

Description

On self-hosted Appwrite 2.5.0, all schedule-triggered function executions stopped after a Redis/Valkey pod restart. The schedule-functions process stayed alive and Ready, but the tick loop stopped (no further schedule.functions.enqueue logs, no errors). HTTP-triggered functions continued to work.

Environment

• Version: Appwrite 2.5.0 (appwrite/appwrite:2.5.0)
• Deploy: Kubernetes / Helm (not official one, but official don't have probes either)
• Queue / cache: Valkey (valkey.valkey.svc.cluster.local:6379)
• Executions store: ClickHouse enabled
• Affected workloads: task-scheduler-functions, task-scheduler-executions, task-scheduler-messages (1 replica each)

Steps to reproduce

Not reliably reproducible on demand. Observed correlation:

1. Schedulers running healthy for days (pods started 2026-10-14).
2. Valkey primary pod restarted (valkey-0 start 2026-10-18T12:50:48Z, ready ~12:52:18Z).
3. After that, schedulers never enqueued again until (planned) process restart.

Expected behavior

• Scheduler either reconnects and resumes ticks after Redis/Valkey failover, or exits so Kubernetes restarts the pod.
• Operators can detect a stuck tick loop (health/metrics), not only that PID 1 exists.

Actual behavior

• Last schedule.functions.enqueue at 2026-10-18T12:45:01Z, then complete log silence (no exception, no schedule.functions.stopped).
• ClickHouse: last trigger = 'schedule' row at 2026-10-18 12:45:01 (484 schedule runs that day, zero after).
• Process still Running: wchan=ep_poll, State S, CPU jiffies frozen over multi-second samples.
• TCP to Valkey from inside the scheduler pods succeeds again after Valkey recovered.
• Same freeze on executions and messages schedulers.
• worker-functions kept processing HTTP jobs — only the schedule path died.
• No livenessProbe / readinessProbe on these Deployments, so K8s never restarted them.

Evidence

Last enqueue (scheduler logs):

2026-10-18T12:45:01Z schedule.functions.enqueue project.id 688228a50003cf2546fa function.id 6883cff4000d4eae152a schedule.id 6883e94d6e9b792b466c schedule.cron */5 * * * * occurrence.due 2026-10-18T12:45:00+00:00

Valkey:

valkey-0 started 2026-10-18T12:50:48Z Ready to accept connections ~2026-10-18T12:52:18Z replicas resynced shortly after

Process while hung:

PID 1 {php} schedule-functions wchan=ep_poll State=S /proc/1 utime+stime unchanged across 3s samples nc to Valkey: open

Asks / questions

1. Should the scheduler fail closed (exit or reconnect+resume) when Redis/cache connections drop during failover?
2. Can you expose a tick heartbeat (HTTP health or a Prometheus counter that advances every tick) so a real livenessProbe is possible? Process-existence probes would not have caught this.
3. Is this a known issue in 2.5.0 / utopia-php/schedule after Redis reconnect?

Happy to capture more dumps (strace, open FDs, Swoole coroutine dump) before restart if you have a preferred recipe.