Custom Fivetran SDK connectors: a pragmatic guide
What is actually involved in writing a production-grade custom connector, including testing, schema drift, retries, and operational support.
TL;DR / Key Takeaways
- A custom connector is production software, not a one-off script.
- Pagination, state, schema drift, and retries need design before coding.
- Tests should cover realistic payloads and failure modes.
- A connector is not done until it has logging, docs, and ownership.
Custom Fivetran SDK connectors are useful when the source system matters and the standard connector does not exist.
They are also real production software. Treating them like a quick script is how data teams inherit brittle pipelines.
The connector is only one part
A working connector needs to extract data, define schema, handle pagination, preserve state, and survive API failures. That is the baseline.
Production work also includes:
- Clear naming for tables and columns
- Tests against realistic API payloads
- Retry behavior for temporary failures
- Logging that explains what happened
- Handling for deleted, changed, or missing fields
- Documentation for the next person who owns it
Schema drift is normal
APIs change. Optional fields appear. Nested objects become inconsistent. A connector should expect that reality instead of crashing on the first surprise.
This does not mean swallowing every error. It means deciding which fields are required, which fields are optional, and what should happen when the source changes.
Operational support matters
The best connector is boring after launch. It syncs, alerts when something is wrong, and leaves enough context for a human to fix the issue.
That means tests, runbooks, version control, and a simple deployment workflow are part of the deliverable. Without those pieces, the connector is not finished.
Practical checklist
- Confirm there is no native connector that already solves the problem.
- Define required tables, fields, and sync state.
- Plan pagination, rate limits, and retry behavior.
- Add fixture-based tests.
- Document how failures should be handled.
Related Reading
Related practical notes
Building a practical alerting system for Fivetran and dbt failures with AWS Lambda, DynamoDB, Teams, and Jira
A useful alerting system does not just say something broke. It knows whether the failure is new, whether it already alerted, whether it recovered, and when to create a ticket.
Read articleCX Alloy API connector lessons when throughput is painfully slow
Some APIs are not hard because authentication is impossible. They are hard because the shape, pagination, nested resources, and throughput force you to design for patience.
Read articleWorkday SOAP APIs, custom connectors, and the joyless art of not getting rate-limited
SOAP APIs can still power serious enterprise integrations, but they demand patience, defensive engineering, and a connector design that respects pagination, retries, payload size, and rate limits.
Read article