> For the complete documentation index, see [llms.txt](https://docs.lsports.eu/u/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.lsports.eu/u/trade/integration/reliability-and-recovery.md).

# Reliability & Recovery

The practices on this page determine whether your feed stays trustworthy under real trading load. Build them in from the start rather than retrofitting them after an incident.

## Core recovery process

{% hint style="warning" %}
**Principle:** Your consumer should be connected essentially 100% of the time. Recovery exists to reconcile snapshot and RMQ data so the system always resumes from the latest known-good state. It isn't a substitute for staying connected.
{% endhint %}

When a heartbeat timeout, queue shutdown, or other integrity issue triggers recovery, follow this order. The sequence matters as much as the individual steps:

1. Call the Distribution Start API for the affected package/account.
2. As soon as it returns HTTP 200, reconnect the RMQ consumer immediately: start caching incoming messages, but don't apply them to trading state yet.
3. In parallel, call the Snapshot API, filtered by the timestamp of the last healthy heartbeat, to keep the payload manageable.
4. Apply the snapshot to rebuild internal state up to the last healthy point.
5. Reconcile: for each fixture/market, compare the snapshot's last update against the newest cached RMQ message by timestamp, and keep whichever is newer. Snapshot data carries no sequence number, so timestamp is the only comparison available.
6. Only after reconciliation is complete, stop caching and resume applying live RMQ messages directly. Mark the system as recovered.

{% hint style="danger" %}
**Don't** snapshot first and connect the consumer second. That ordering risks missing real-time changes that occur in the gap. Always connect and start caching before or in parallel with the snapshot call, never after.
{% endhint %}

## Heartbeat monitoring

The feed sends a heartbeat (message type 32) every 5 seconds. Monitor its *freshness*, not just its presence: a stalled consumer can keep the connection open while data goes stale.

* Run heartbeat monitoring in its own component or thread, independent of your main message handling.
* If no heartbeat arrives for more than 12 seconds, suspend trading immediately and trigger recovery, using the last heartbeat's server timestamp.
* Store recent heartbeats (up to 24 hours) to help troubleshoot after the fact.

## Message sequence monitoring

* Sequence numbers aren't global — each is scoped to a message type and entity. A market's sequence covers that fixture/market pair only. Livescore, fixture metadata, and settlement messages each keep their own independent sequence per fixture. See [MsgSeq](/u/trade/integration/message-structure/trade-message-header.md) for the field definition.
* Track each of these sequences separately rather than comparing across contexts.
* Gaps of 1–2 are normal within a single context. Watch for upward trends in gaps as a sign of a systemic issue.
* Snapshot data carries no sequence number — reconcile it against cached RMQ messages using the `LastUpdate` timestamp only, and keep whichever side is newer.

### Detecting an uptrend

A single gap rarely means trouble. A trend does. Use a rolling 5-minute window to tell the difference:

1. For each context (fixture/market pair, or fixture for livescore/metadata), track the last sequence number you saw.
2. Each time a gap occurs, log when it happened and how large it was.
3. Keep only the last 5 minutes of gap events; drop anything older.
4. Split the window into five 1-minute slices and count the gaps in each.
5. If the gap count rises for 3 or more consecutive slices, that's an uptrend.

A gap or two in a single slice is normal — ignore it. A steady climb across slices is the signal worth acting on.

{% hint style="info" %}
An uptrend is a signal to investigate, not a trigger for full recovery on its own. Let heartbeat and queue-health checks decide that.
{% endhint %}

## Timestamps

| Field                 | Source                                                                       | Use it for                                                                        |
| --------------------- | ---------------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| **server\_timestamp** | Message header (`ServerTimestamp`)                                           | Chronological ordering; filtering snapshots by last healthy heartbeat             |
| **timestamp\_in\_ms** | [RabbitMQ header](/u/trade/integration/message-structure/rabbitmq-header.md) | Measuring distribution latency; separating consumption vs. processing performance |

## Snapshot API: scope and limits

* Use it for recovery and ad-hoc enrichment, not as a substitute for the live RMQ feed in day-to-day operations.
* Keep the RMQ consumer connected and caching while a snapshot call is in flight.
* Filter by the last heartbeat timestamp to keep payload size reasonable.
* Snapshot data is available for up to 24 hours after a fixture completes.

For endpoints, filtering parameters, and request limits, see the [Snapshot API reference](/u/trade/integration/apis/snapshot.md).

***

Next: [Mapping, Testing & Go-Live](/u/trade/integration/mapping-testing-and-go-live.md)


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.lsports.eu/u/trade/integration/reliability-and-recovery.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.