Scheduler

The Laravel scheduler is the time-driven entry point to Kraite's workload. Every recurring job — kline fetches, indicator computation, listenKey refreshes, balance snapshots, candle purges, audit-log retention — flows through routes/console.php on ingestion.kraite.com, running on Kraite. The scheduler does not execute trading logic itself; it dispatches into the dispatch daemon and the Horizon queues, which do the work.

This is the subsystem lens view. For the persistent process that replaced one specific scheduled command (steps:dispatch), see Dispatch daemon.


The cron table

CommandCadenceCooldown-gatedPurpose
kraite:cron-flush-dispatcher-saturation1 minnoPersist dispatcher saturation counters
steps:recover-stale (default + trading)1 minnoRecover stale steps, locks, and stalled groups
kraite:cron-sync-orders5 minyesPolling fallback for the user-data WS daemon
kraite:cron-refresh-binance-listen-keys1 minnoKeep Binance listenKeys alive past 60-min auto-expiry
kraite:cron-check-binance-listen-keys-stale5 minnoDetect missing or stale listen-key state
kraite:cron-check-system-health7 minnoUnified health + maintenance sentinel
kraite:cron-check-drifts5 minyesPosition drift, protection, and money guard
kraite:monitor-narrateminutes 7, 27, 47yesDocument an already-open money-guard incident
kraite:cron-create-positions3 minyesOpen new positions
kraite:cron-fetch-klines --only-active-positions5 minyesRefresh klines for tokens with open positions
kraite:cron-fetch-klines --reference-set ... --timeframe=15m15 minyesFeed the market-shock reference basket
kraite:cron-fetch-klines --timeframe=4hevery 4h :05yes4h-bar refresh
kraite:cron-fetch-klines --timeframe=6hevery 6h :05yes6h-bar refresh
kraite:cron-fetch-klines --timeframe=12hevery 12h :05yes12h-bar refresh
kraite:cron-store-accounts-balances5 minyesSnapshot account balances per exchange
kraite:cron-upsert-pnls5 minyesBackfill exchange-reported PnL
kraite:cron-refresh-exchange-symbolshourly :15 except 00/06/12/18yesRefresh catalogues without leverage-bracket work
kraite:cron-refresh-exchange-symbols --with-bracketsevery 6h :15yesRefresh catalogues and all leverage brackets
kraite:cron-conclude-symbols-directionhourly :30yesTAAPI indicator → direction
kraite:cron-renew-subscriptionsdaily 00:00yesProcess monthly renewals
kraite:cron-compute-market-regimehourly :50yesCompute BSCS
kraite:cron-analyse-bscshourly :55yesApply the BSCS cooldown state machine
kraite:cron-detect-market-shock1 minyesFast cascade detector
backup:run --only-dbevery 3h :07noEncrypted database snapshot to B2; two whole-command attempts
backup:monitorevery 6h :15noAlert on stale/unhealthy B2 backups
kraite:purge-candlesdaily 03:00yesRetention sweep on candle data
kraite:cron-purge-position-trailsdaily 03:20yesReclaim expired clean-close breadcrumbs
kraite:cron-purge-failed-backtested-klineshourly :00yesDrop rejected-symbol candles
kraite:purge-old-data ...daily 03:30yesAPI/model log retention
steps:archive (default + trading)daily 04:00 / 04:05yesArchive resolved step trees
steps:purge --only-archive (default + trading)daily 04:30 / 04:35yesKeep five days of archived steps
kraite:cron-optimize-breadcrumb-tablesSundays 03:00–04:36yesStaggered weekly compaction

The hourly direction-conclusion command also owns an application lock, so a manual run and the scheduled run cannot overlap. Its destructive --clean mode is accepted only in local or testing; production refuses the flag before deleting any indicator or direction data.

The split symbol schedule keeps catalogue, token, and availability data fresh every hour without sending a full per-symbol leverage sweep into the cron lane each time. The six-hour run is explicit and manually reproducible; normal hourly runs create zero leverage-bracket steps.


What "cooldown-gated" means

routes/console.php registers step-producing commands only while the Kraite singleton reports is_cooling_down=false. During a release those entries disappear from the schedule, so they cannot create fresh work while queues drain. Recovery, listen-key maintenance, health, and backups stay registered. withoutOverlapping() is a separate Laravel mutex that prevents two instances of one command from running concurrently.

                tick fires


            ┌────────────────────┐
            │ System cooling down?│── yes ─► entry not registered
            └─────────┬──────────┘
                    no

            ┌────────────────────┐
            │ withoutOverlapping │── busy ► skip this tick
            │       mutex        │
            └─────────┬──────────┘
                    ok

              run command


             release mutex

Maintenance mode silences everything — and its sentinel

The scheduler skips every event while the application is in maintenance mode. That is by design during a deploy (cooldown parks each box in maintenance; warmup lifts it) — but it means a box accidentally left in maintenance loses its entire cron table silently: listenKey keepalive, sync fallback, database backups, and every watchdog, all at once, while supervisors and Horizon still look healthy.

Incident — 2026-07-02

An interrupted release warmup left Athena in maintenance mode for 53 hours. The only external symptom was Binance's own listenKeyExpired push every 70 minutes — the keepalive cron that would have prevented it was itself paused, and so was the health watchdog that should have paged. Zero money impact (no open positions), but database backups were silently dead for two days.

Since then, the health watchdog (kraite:cron-check-system-health) is the one scheduled command that runs even in maintenance mode. While the box is down it performs a single check — "has this box been in maintenance longer than the threshold" (default 45 minutes, sized above a full cooldown → deploy → warmup span) — and pages CRITICAL on breach, re-paging every 30 minutes until an operator brings the box up. The full check pass stays skipped during maintenance, so normal deploy windows never produce transient alerts. The release runbooks carry the matching gates: warmup hard-verifies the box answers "UP" after warming, and the fleet health grid renders a Maint column that fails on any leftover maintenance marker.

Recovery after warmup

Warmup starts Horizon and the daemons before it resumes the scheduler on the same host. It then starts a 10-minute recovery grace for the two health signals produced through the default dispatcher: account-balance history and indicators. Those timestamps can still describe the pre-deploy state until the first producer jobs finish.

The grace does not mute mark prices, queues, Redis, the database, daemons, scheduler liveness, or fleet heartbeats. Indicator freshness also checks the exact symbol's producer workflow: a recent query or conclusion in progress is an active repair, while terminal work or an abandoned old step cannot suppress a real alert.

Why the grace is narrow

On 2026-07-16 the balance watchdog ran seconds before the first post-warmup balance write. Later, BNB, ATOM, and GRT paged as stale while their indicator workflows were visibly executing. Both alerts described old timestamps, but neither needed operator action. The recovery rule removes those races without creating a general post-deploy blind spot.

Copied exchange symbols also defer to their fresh Binance source. A delayed copy is not paged while the same token and quote has fresh native analysis; when the Binance source is stale, the source alert remains visible.


Why a daemon replaced steps:dispatch

Architectural decision

The pre-daemon design ran steps:dispatch every second across 10 step-dispatcher groups — 10 forks per second, plus the per-tick scheduler fork itself. At Kraite's growth that became the dominant CPU cost on Athena. The dispatch daemon collapses all 10 group ticks into a single long-lived loop with a 1 s sleep between ticks, dropping load average from 105 to 0.68. The scheduler still owns every other recurring command — only the high-frequency dispatch concern was lifted out.