schedule-functions (and sibling schedulers) silent hang after Redis restart

Rank 5: Hypervisor

Oct 19, 2026, 07:42

Description

On self-hosted Appwrite 2.5.0, all schedule-triggered function executions stopped after a Redis/Valkey pod restart. The schedule-functions process stayed alive and Ready, but the tick loop stopped (no further schedule.functions.enqueue logs, no errors). HTTP-triggered functions continued to work.

Environment

Version: Appwrite 2.5.0 (appwrite/appwrite:2.5.0)
Deploy: Kubernetes / Helm (not official one, but official don't have probes either)
Queue / cache: Valkey (valkey.valkey.svc.cluster.local:6379)
Executions store: ClickHouse enabled
Affected workloads: task-scheduler-functions, task-scheduler-executions, task-scheduler-messages (1 replica each)

Steps to reproduce

Not reliably reproducible on demand. Observed correlation:

Schedulers running healthy for days (pods started 2026-10-14).
Valkey primary pod restarted (valkey-0 start 2026-10-18T12:50:48Z, ready ~12:52:18Z).
After that, schedulers never enqueued again until (planned) process restart.

Expected behavior

Scheduler either reconnects and resumes ticks after Redis/Valkey failover, or exits so Kubernetes restarts the pod.
Operators can detect a stuck tick loop (health/metrics), not only that PID 1 exists.

Actual behavior

Last schedule.functions.enqueue at 2026-10-18T12:45:01Z, then complete log silence (no exception, no schedule.functions.stopped).
ClickHouse: last trigger = 'schedule' row at 2026-10-18 12:45:01 (484 schedule runs that day, zero after).
Process still Running: wchan=ep_poll, State S, CPU jiffies frozen over multi-second samples.
TCP to Valkey from inside the scheduler pods succeeds again after Valkey recovered.
Same freeze on executions and messages schedulers.
worker-functions kept processing HTTP jobs — only the schedule path died.
No livenessProbe / readinessProbe on these Deployments, so K8s never restarted them.

Evidence

Last enqueue (scheduler logs):

2026-10-18T12:45:01Z  schedule.functions.enqueue
  project.id        688228a50003cf2546fa
  function.id       6883cff4000d4eae152a
  schedule.id       6883e94d6e9b792b466c
  schedule.cron     */5 * * * *
  occurrence.due    2026-10-18T12:45:00+00:00

Valkey:

valkey-0 started 2026-10-18T12:50:48Z
Ready to accept connections ~2026-10-18T12:52:18Z
replicas resynced shortly after

Process while hung:

PID 1  {php} schedule-functions
wchan=ep_poll  State=S
/proc/1 utime+stime unchanged across 3s samples
nc to Valkey: open

Asks / questions

Should the scheduler fail closed (exit or reconnect+resume) when Redis/cache connections drop during failover?
Can you expose a tick heartbeat (HTTP health or a Prometheus counter that advances every tick) so a real livenessProbe is possible? Process-existence probes would not have caught this.
Is this a known issue in 2.5.0 / utopia-php/schedule after Redis reconnect?

Happy to capture more dumps (strace, open FDs, Swoole coroutine dump) before restart if you have a preferred recipe.

Summary

The issue of the scheduler hanging after a Redis restart has been identified, with the scheduler failing to automatically recover after losing connection. A fix is in progress to enable automatic recovery instead of the scheduler getting stuck. Updates will be provided once a solution is implemented.

Harsh

Oct 19, 2026, 13:41

Thanks for the detailed report. I was able to reproduce the scheduler hang after the Redis restart and have been tracing the scheduler flow to see exactly where it stops making progress. So far it looks like the scheduler isnt recovering once that connection is interrupted. Working on a fix so it can recover automatically instead of staying stuck, and will update this thread once I have a solution