Fivetran custom connectors when native connectors are not enough
A practical look at when a custom Fivetran SDK connector makes sense, and what needs to be designed before writing code.
TL;DR / Key Takeaways
- Native connectors are best when they exist and cover the needed data.
- Custom connectors are useful when APIs, pagination, auth, or business rules are specific.
- Schema design, state management, and retry behavior matter more than the first API call.
- Production connectors need logging, tests, and clear ownership.
Fivetran is useful because it removes a lot of common data movement work. When a native connector exists and covers the data you need, use it.
When it does not, a custom SDK connector can be the right answer.
When native connectors are enough
Native connectors are usually the best choice when:
- The source system is supported.
- The required objects and fields are available.
- Sync behavior matches the reporting need.
- The cost and schedule are acceptable.
- The team does not need special business logic during extraction.
Do not write a custom connector just to feel more in control.
When custom SDK connectors make sense
Custom connectors make sense when the source is niche, internal, or unusually shaped.
Common reasons include:
- A vendor API is not supported.
- Authentication requires custom handling.
- Pagination is unusual.
- The API response needs business-specific interpretation.
- The data source is internal and important.
What to define before coding
Before writing code, define the tables, fields, sync state, and failure behavior.
This is the part that saves time later. The first API request is rarely the hard part. The hard part is keeping the sync correct after the API changes, a request fails, or a field disappears.
State, pagination, and schema drift
Every production connector needs a plan for state. The connector should know what it has already synced and how to resume safely.
Pagination needs the same treatment. Some APIs use cursors. Some use timestamps. Some use page numbers. Some return partial data without making the failure obvious.
Schema drift is normal. Decide what should fail fast and what should be treated as optional.
Testing and operational handoff
A useful connector should include:
- Fixture-based tests
- Logging
- Retry handling
- Clear deployment steps
- Runbook notes
- Ownership boundaries
If nobody knows how to debug it, the connector is not done.
Practical checklist
- Confirm native connector coverage first.
- Document the source API limits and auth method.
- Define destination tables and required fields.
- Design sync state before coding extraction.
- Test pagination and failure paths.
- Write a short handoff runbook.
Related Reading
Related practical notes
Building a practical alerting system for Fivetran and dbt failures with AWS Lambda, DynamoDB, Teams, and Jira
A useful alerting system does not just say something broke. It knows whether the failure is new, whether it already alerted, whether it recovered, and when to create a ticket.
Read articleCX Alloy API connector lessons when throughput is painfully slow
Some APIs are not hard because authentication is impossible. They are hard because the shape, pagination, nested resources, and throughput force you to design for patience.
Read articleWorkday SOAP APIs, custom connectors, and the joyless art of not getting rate-limited
SOAP APIs can still power serious enterprise integrations, but they demand patience, defensive engineering, and a connector design that respects pagination, retries, payload size, and rate limits.
Read article