Connecting Two Systems Without Breaking Either: API Integration Patterns

Connecting two systems sounds like a weekend job until you meet the real world. The demo works because everything is online, the data is small, and nothing fails. Then you ship it, and the other API starts returning rate-limit errors, a token expires at 2am, a sync runs while a record is half-written, and both systems quietly decide they own the same field. The gap between a demo and a durable integration is entirely in how you handle the unhappy paths.

This guide is about those paths. It covers the decisions and patterns that separate an integration you can trust from one that drifts out of sync and pages you on a Saturday, with code you can lift into a Vercel or Node service.

Before any code, answer one question: for each piece of data, which system is the authority. This sounds obvious and is the single most common cause of sync bugs. When two systems can both edit the same field and neither is clearly in charge, they will eventually disagree, and you will be left guessing which value is correct.

Make it explicit. Often the cleanest design is one-way: one system owns the data and the other is a read-only mirror, which removes conflict entirely. If you genuinely need two-way sync, decide the conflict rule up front, last-write-wins by timestamp, or field-level ownership where each system is authoritative for specific fields. Write that rule down before you write the sync, because retrofitting it later means untangling data that has already diverged.

How you authenticate shapes how the integration fails. A static API key is simple: store it encrypted, scope it to the least access it needs, and rotate it on a schedule. The risk is leakage, so it should never live in client code or a NEXT_PUBLIC_ variable.

OAuth is more work because access tokens expire, usually within an hour, and you hold a refresh token to mint new ones. The pattern that saves you is to treat a 401 as "refresh and retry once" rather than a hard failure, so a token expiring mid-run is invisible instead of fatal.

async function call(path: string) {
  let res = await fetch(path, { headers: auth() });
  if (res.status === 401) {
    await refreshAccessToken();   // access tokens expire; refresh once
    res = await fetch(path, { headers: auth() });
  }
  return res;
}

Store refresh tokens encrypted, and have a plan for the day a refresh itself fails, because the user revoked access or changed their password. Surfacing a clear "reconnect this integration" prompt beats a silent stream of errors.

Every API has limits, and hitting them is normal, not exceptional. When you get a 429, the provider almost always tells you how long to wait in a Retry-After header, so honor it with a backoff instead of hammering the endpoint and making things worse.

Pagination is the other half. Prefer cursor-based paging over offset paging, because offsets shift when records are added or removed mid-sync and you end up skipping or double-reading rows. And do not pull the entire dataset every time, ask only for what changed since your last run using an updated-since filter or a stored cursor, which keeps each sync fast and well under the rate limit.

async function* pages(url: string) {
  let cursor: string | undefined;
  do {
    const res = await fetch(cursor ? `${url}?cursor=${cursor}` : url, { headers: auth() });
    if (res.status === 429) {
      const wait = Number(res.headers.get("retry-after") ?? 1);
      await sleep(wait * 1000);   // honor the provider's backoff
      continue;
    }
    const body = await res.json();
    yield body.data;
    cursor = body.next_cursor;    // cursor paging is stable while data changes
  } while (cursor);
}

There are two ways data crosses between systems, and good integrations use both. Push, via webhooks, gives you near real-time updates the moment something changes, which is ideal for events like a new order or a payment. Pull, on a schedule, sweeps for changes since the last run, which is ideal for bulk reconciliation and for systems that do not offer webhooks.

The robust design is push for timeliness and pull for completeness: webhooks keep things current, and a periodic incremental pull catches anything the webhooks missed. For the initial load, page through everything once with the rate-limit-aware loop above, then switch to incremental.

A sync is rarely a single action, it is several writes across two systems, and any step can fail after earlier ones succeeded. If you are not careful, you leave half-written state: the order created here but not there, the customer updated but the invoice not.

Two habits prevent this. Make each step idempotent so a retry re-runs the whole sequence safely, deduping on stable ids and using upserts rather than blind inserts. And for sequences that must not be left half-done, record the intent first and process it separately, the transactional outbox pattern, so a crash mid-sync resumes from a durable record instead of vanishing. The aim is that any run can fail at any point and a retry brings both systems back to a consistent state.

Even a careful integration drifts, so build the tools to catch it. Keep a mapping table that links each record's id in one system to its id in the other, which makes lookups exact instead of guesses based on names or emails. On a schedule, run a reconciliation pass that compares the two systems and reports differences, then heals them.

And make the integration legible: log each sync with what it changed, alert when divergence crosses a threshold or a run fails repeatedly, and keep failed items where a human can inspect them. An integration you can see into is one you can trust, the silent ones are the ones that surprise you.

• Decide the source of truth and the conflict rule for every shared field before coding.
• Prefer one-way sync where you can, and make two-way conflict resolution explicit.
• Store keys and refresh tokens encrypted, scope them tightly, and rotate them.
• Treat a 401 as refresh-and-retry, and surface a clear reconnect path when refresh fails.
• Honor Retry-After on 429s, use cursor pagination, and sync incrementally.
• Combine webhooks for timeliness with a scheduled pull for completeness.
• Make every step idempotent, and use an outbox for sequences that must not be left half-done.
• Keep an id mapping table, reconcile on a schedule, and alert on drift.

Should I build a one-way or two-way sync?

Default to one-way whenever you can, because it removes conflicts entirely: one system owns the data and the other mirrors it read-only. Build two-way sync only when both systems genuinely need to edit the same data, and if you do, decide the conflict rule first, either last-write-wins by timestamp or field-level ownership. Most painful sync bugs come from two-way flows where nobody decided who wins a disagreement.

How do I handle API rate limits?

Expect them and back off gracefully. When an API returns a 429, it usually includes a Retry-After header telling you how long to wait, so pause for that duration and resume rather than retrying immediately. Reduce pressure in the first place by syncing only what changed since your last run instead of pulling everything, and by paging through results rather than requesting large batches at once.

What is the source of truth problem?

It is the question of which system is authoritative for each piece of data. When two connected systems can both change the same field and neither is clearly in charge, they drift apart and you cannot tell which value is correct. Deciding the source of truth up front, ideally making one system own the data and the other mirror it, prevents the conflicts that otherwise surface as mysterious sync bugs later.

How do I keep two systems in sync reliably?

Use webhooks for real-time updates and a scheduled incremental pull to catch anything missed, keep a mapping table linking ids across both systems, and make every write idempotent so retries are safe. Then run a periodic reconciliation pass that compares the two systems and heals differences. Real-time sync keeps them current, and reconciliation guarantees they converge even after an outage or a dropped event.

Offset or cursor pagination, which should I use?

Cursor pagination, when the API offers it. Offset paging breaks when records are added or removed while you are iterating, because the offsets shift and you skip or double-count rows, which is common during a live sync. A cursor points at a stable position in the result set, so you keep reading correctly even as the underlying data changes beneath you.

If you are connecting two systems and want them to stay in sync without surprises, tell me what you are integrating and I will give you a straight technical read on the source of truth, the failure modes, and how to make it durable.