CX Alloy API connector lessons when throughput is painfully slow

TL;DR / Key Takeaways

• Slow APIs require connector design that assumes limited throughput from the beginning.
• Nested resources and per-project endpoints can multiply API calls faster than expected.
• You need state, checkpoints, selective sync, and good logging before you scale the connector.
• The right answer is not always "go faster." Sometimes it is "sync smarter."
• A slow connector without observability becomes a black box nobody trusts.

Some APIs are not hard because authentication is impossible. They are hard because every useful answer requires walking through a maze of projects, child resources, nested endpoints, include parameters, and pages.

When an API is slow, optimism is not a strategy. State management is.

This is a practical pattern for designing custom connectors when throughput is limited and the resource model is more complicated than one clean list endpoint.

The hard part is not always authentication

Authentication gets attention because it blocks the first request. But after auth works, the connector may still have harder problems:

• Discovering projects
• Looping through child resources
• Expanding nested data
• Handling pagination
• Avoiding duplicate calls
• Preserving parent-child relationships
• Tracking progress across slow runs

The first 200 response is useful. It is not a sync design.

Why nested APIs destroy naive sync design

A naive connector tries to do this:

1. Get all projects.
2. For every project, get every resource.
3. For every resource, get every child object.
4. Repeat until everything is fetched.

That can work in a tiny account. It breaks down when the number of projects, resources, and child endpoints grows.

Generic examples:

• Projects
• Equipment
• Equipment attributes
• Checklist templates
• Checklist sections
• Checklist lines

Each layer can multiply the number of API calls.

Per-project loops and call explosion

Per-project loops are common in APIs organized around project work. The pattern is understandable, but it can create call explosion.

| Pattern | Connector risk | | --- | --- | | One endpoint per project | Calls grow with project count | | Child endpoints per resource | Calls grow with resource count | | Include parameters | Payloads grow and may slow responses | | Page-based pagination | Long runs need checkpoints |

If a connector has to loop through every project every time, the sync will eventually become too slow or too noisy.

Pagination and checkpointing

Pagination needs checkpoints. A connector should know where it stopped and what it can safely resume.

Useful checkpoint data includes:

• Current project ID or project key
• Resource type
• Page number or cursor
• Last successful timestamp
• Last completed parent object
• Run attempt

Without checkpoints, one failure near the end can force the entire sync to start over.

Parent-child modeling and composite keys

Nested APIs often produce parent-child tables. A checklist line belongs to a checklist section. A checklist section belongs to a template. Equipment attributes belong to equipment.

The destination model needs stable keys.

Composite primary keys are often the practical answer:

project_id + equipment_id + attribute_name
project_id + template_id + section_id
project_id + template_id + section_id + line_id

The exact shape depends on the source API, but the principle is stable: include enough parent context to make the child record unique.

Sync smarter, not harder

The right answer is not always more parallelism. Sometimes more parallel calls only gets you rate-limited faster.

Smarter sync patterns include:

• Sync only active projects.
• Split large resource families into separate runs.
• Use incremental sync where timestamps are available.
• Cache project discovery for the duration of a run.
• Avoid calling child endpoints unless the parent changed.
• Reduce scope when a downstream table is not needed.

A connector should be designed around the data that actually matters.

Observability for slow connectors

Slow connectors need better logs than fast connectors because humans will wonder whether they are stuck.

Log:

• Which project is being processed
• Which resource type is being fetched
• Page or cursor position
• Records returned
• API duration
• Retry count
• Checkpoint updates

Do not log private payloads. Log progress and shape.

Practical checklist

• Map the resource tree before writing code.
• Count expected calls per project and per resource type.
• Decide which resources actually need to sync.
• Use checkpoints for long loops.
• Preserve parent-child relationships in destination keys.
• Add retry and backoff before scaling volume.
• Measure throughput by resource type.
• Log progress in a way an operator can understand.

FAQ

Should a slow API connector use parallel requests?

Maybe, but only after understanding rate limits and failure behavior. Parallelism can make a bad connector fail faster.

What is selective sync?

Selective sync means syncing only the resources, projects, or child tables that matter for the business use case instead of blindly extracting everything.

Why do composite keys matter?

Nested APIs often reuse child IDs across parents. Composite keys keep child rows unique by including the parent context.

When should a connector be split into multiple syncs?

Split syncs when one resource family is slow, optional, or operationally different enough that it should not block the rest of the pipeline.

Related Reading

• Fivetran custom connectors when native connectors are not enough
• Custom Fivetran SDK connectors: a pragmatic guide
• Why AI projects need boring plumbing before they need agents