Custom Fivetran SDK connectors: a pragmatic guide
What is actually involved in writing a production-grade custom connector, including testing, schema drift, retries, and operational support.
TL;DR / Key Takeaways
- A custom connector is production software, not a one-off script.
- Pagination, state, schema drift, and retries need design before coding.
- Tests should cover realistic payloads and failure modes.
- A connector is not done until it has logging, docs, and ownership.
Custom Fivetran SDK connectors are useful when the source system matters and the standard connector does not exist.
They are also real production software. Treating them like a quick script is how data teams inherit brittle pipelines.
The connector is only one part
A working connector needs to extract data, define schema, handle pagination, preserve state, and survive API failures. That is the baseline.
Production work also includes:
- Clear naming for tables and columns
- Tests against realistic API payloads
- Retry behavior for temporary failures
- Logging that explains what happened
- Handling for deleted, changed, or missing fields
- Documentation for the next person who owns it
Schema drift is normal
APIs change. Optional fields appear. Nested objects become inconsistent. A connector should expect that reality instead of crashing on the first surprise.
This does not mean swallowing every error. It means deciding which fields are required, which fields are optional, and what should happen when the source changes.
Operational support matters
The best connector is boring after launch. It syncs, alerts when something is wrong, and leaves enough context for a human to fix the issue.
That means tests, runbooks, version control, and a simple deployment workflow are part of the deliverable. Without those pieces, the connector is not finished.
Practical checklist
- Confirm there is no native connector that already solves the problem.
- Define required tables, fields, and sync state.
- Plan pagination, rate limits, and retry behavior.
- Add fixture-based tests.
- Document how failures should be handled.
Related Reading
What we build in this area
Related practical notes
What to Automate First in a Small Business
If you are not sure where to start with automation, start with the task that wastes the most time and carries the least risk.
Read articleWhen Airflow Makes Sense for a Growing Business
Airflow is a workflow scheduler that keeps recurring data jobs reliable, visible, and recoverable when something goes wrong.
Read articleWhat Fivetran Connectors Do and Why Businesses Use Them
Fivetran moves data from your business systems into one place automatically. Here is what that means and when you need a custom connector.
Read article