Reliability & Recovery

Heartbeats, recovery sequencing, and the practices that keep your feed stable

The practices on this page determine whether your feed stays trustworthy under real trading load. Build them in from the start rather than retrofitting them after an incident.

Core recovery process

Principle: Your consumer should be connected essentially 100% of the time. Recovery exists to reconcile snapshot and RMQ data so the system always resumes from the latest known-good state. It isn't a substitute for staying connected.

When a heartbeat timeout, queue shutdown, or other integrity issue triggers recovery, follow this order. The sequence matters as much as the individual steps:

1. Call the Distribution Start API for the affected package/account.
2. As soon as it returns HTTP 200, reconnect the RMQ consumer immediately: start caching incoming messages, but don't apply them to trading state yet.
3. In parallel, call the Snapshot API, filtered by the timestamp of the last healthy heartbeat, to keep the payload manageable.
4. Apply the snapshot to rebuild internal state up to the last healthy point.
5. Reconcile: for each fixture/market, compare the snapshot's last update against the newest cached RMQ message by timestamp, and keep whichever is newer. Snapshot data carries no sequence number, so timestamp is the only comparison available.
6. Only after reconciliation is complete, stop caching and resume applying live RMQ messages directly. Mark the system as recovered.

Don't snapshot first and connect the consumer second. That ordering risks missing real-time changes that occur in the gap. Always connect and start caching before or in parallel with the snapshot call, never after.

Heartbeat monitoring

The feed sends a heartbeat (message type 32) every 5 seconds. Monitor its freshness, not just its presence: a stalled consumer can keep the connection open while data goes stale.

• Run heartbeat monitoring in its own component or thread, independent of your main message handling.
• If no heartbeat arrives for more than 12 seconds, suspend trading immediately and trigger recovery, using the last heartbeat's server timestamp.
• Store recent heartbeats (up to 24 hours) to help troubleshoot after the fact.

Message sequence monitoring

• Sequence numbers aren't global — each is scoped to a message type and entity. A market's sequence covers that fixture/market pair only. Livescore, fixture metadata, and settlement messages each keep their own independent sequence per fixture. See MsgSeq for the field definition.
• Track each of these sequences separately rather than comparing across contexts.
• Gaps of 1–2 are normal within a single context. Watch for upward trends in gaps as a sign of a systemic issue.
• Snapshot data carries no sequence number — reconcile it against cached RMQ messages using the LastUpdate timestamp only, and keep whichever side is newer.

Detecting an uptrend

A single gap rarely means trouble. A trend does. Use a rolling 5-minute window to tell the difference:

1. For each context (fixture/market pair, or fixture for livescore/metadata), track the last sequence number you saw.
2. Each time a gap occurs, log when it happened and how large it was.
3. Keep only the last 5 minutes of gap events; drop anything older.
4. Split the window into five 1-minute slices and count the gaps in each.
5. If the gap count rises for 3 or more consecutive slices, that's an uptrend.

A gap or two in a single slice is normal — ignore it. A steady climb across slices is the signal worth acting on.

An uptrend is a signal to investigate, not a trigger for full recovery on its own. Let heartbeat and queue-health checks decide that.

Timestamps

Field	Source	Use it for
server_timestamp	Message header (ServerTimestamp)	Chronological ordering; filtering snapshots by last healthy heartbeat
timestamp_in_ms	RabbitMQ header	Measuring distribution latency; separating consumption vs. processing performance

Snapshot API: scope and limits

• Use it for recovery and ad-hoc enrichment, not as a substitute for the live RMQ feed in day-to-day operations.
• Keep the RMQ consumer connected and caching while a snapshot call is in flight.
• Filter by the last heartbeat timestamp to keep payload size reasonable.
• Snapshot data is available for up to 24 hours after a fixture completes.

For endpoints, filtering parameters, and request limits, see the Snapshot API reference.

---

Next: Mapping, Testing & Go-Live