> For the complete documentation index, see [llms.txt](https://docs.lsports.eu/u/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lsports.eu/u/trade/integration/reliability-and-recovery.md).

# Reliability & Recovery

The practices on this page determine whether your feed stays trustworthy under real trading load. Build them in from the start rather than retrofitting them after an incident.

## Core recovery process

{% hint style="warning" %}
**Principle:** Your consumer should be connected essentially 100% of the time. Recovery exists to reconcile snapshot and RMQ data so the system always resumes from the latest known-good state. It isn't a substitute for staying connected.
{% endhint %}

When a heartbeat timeout, queue shutdown, or other integrity issue triggers recovery, follow this order. The sequence matters as much as the individual steps:

1. Call the Distribution Start API for the affected package/account.
2. As soon as it returns HTTP 200, reconnect the RMQ consumer immediately: start caching incoming messages, but don't apply them to trading state yet.
3. In parallel, call the Snapshot API, filtered by the timestamp of the last healthy heartbeat, to keep the payload manageable.
4. Apply the snapshot to rebuild internal state up to the last healthy point.
5. Reconcile: for each fixture/market, compare the snapshot's last update against the newest cached RMQ message (by timestamp and sequence), and keep whichever is newer.
6. Only after reconciliation is complete, stop caching and resume applying live RMQ messages directly. Mark the system as recovered.

{% hint style="danger" %}
**Don't** snapshot first and connect the consumer second. That ordering risks missing in-play changes that occur in the gap. Always connect and start caching before or in parallel with the snapshot call, never after.
{% endhint %}

## Heartbeat monitoring

The feed sends a heartbeat (message type 32) every 5 seconds. Monitor its *freshness*, not just its presence: a stalled consumer can keep the connection open while data goes stale.

* Run heartbeat monitoring in its own component or thread, independent of your main message handling.
* If no heartbeat arrives for more than 12 seconds, suspend trading immediately and trigger recovery, using the last heartbeat's server timestamp.
* Store recent heartbeats (up to 24 hours) to help troubleshoot after the fact.

## Message sequence monitoring

* Track sequence per context: separately for each market, live score, and fixture metadata.
* Gaps of 1–2 are normal across the whole feed. Watch for upward trends in gaps as a sign of a systemic issue.
* Use sequence and server timestamp together during recovery to decide which side (snapshot or RMQ) is newest.

## Timestamps

| Field                 | Source                                                                       | Use it for                                                                        |
| --------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| **server\_timestamp** | Message header (`ServerTimestamp`)                                           | Chronological ordering; filtering snapshots by last healthy heartbeat             |
| **timestamp\_in\_ms** | [RabbitMQ header](/u/trade/integration/message-structure/rabbitmq-header.md) | Measuring distribution latency; separating consumption vs. processing performance |

## Snapshot API: scope and limits

* Use it for recovery and ad-hoc enrichment, not as a substitute for the live RMQ feed in day-to-day operations.
* Keep the RMQ consumer connected and caching while a snapshot call is in flight.
* Filter by the last heartbeat timestamp to keep payload size reasonable.
* Snapshot data is available for up to 24 hours after a fixture completes.

For endpoints, filtering parameters, and request limits, see the [Snapshot API reference](/u/trade/integration/apis/snapshot.md).

***

Next: [Mapping, Testing & Go-Live](/u/trade/integration/mapping-testing-and-go-live.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.lsports.eu/u/trade/integration/reliability-and-recovery.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.