Workday SOAP APIs, custom connectors, and the joyless art of not getting rate-limited

TL;DR / Key Takeaways

• SOAP is verbose, fragile, and unforgiving, but it is still common in enterprise systems.
• Custom connectors need to treat rate limits, paging, retries, and partial failures as first-class design concerns.
• Large XML payloads should be handled carefully, logged safely, and validated before downstream writes.
• The best connector design is boring: explicit state, predictable retries, clear logs, and safe handoff.
• SOAP does not have to be pretty to be production-worthy.

SOAP is not fun. SOAP is what happens when enterprise software decides XML was not painful enough yet.

That said, SOAP APIs still power serious enterprise systems. Workday is one example of a vendor category where SOAP can remain part of real integration work. If the data matters, the integration matters, even when the protocol feels like it came from a museum with a compliance department.

This is a practical field guide for building custom connectors against large enterprise SOAP APIs without turning the sync into a slow-motion incident.

Why SOAP still shows up in modern data work

Enterprise systems often outlive the API fashion cycle. SOAP remains common because it is formal, contract-driven, and deeply embedded in systems that handle sensitive operational data.

That does not make it pleasant. It does mean engineers still need to support it.

SOAP can show up when a business needs:

• Worker or organization data
• Finance or procurement data
• Configuration-heavy enterprise objects
• Historical records with complex filtering
• Data that is not exposed through a simpler REST endpoint

The job is not to admire the protocol. The job is to move the data safely.

Why SOAP feels painful compared to REST

REST usually feels direct: endpoint, method, headers, JSON body, response.

SOAP often gives you:

• XML envelopes
• Namespaces
• Verbose request bodies
• Large response payloads
• Strict field expectations
• Harder debugging
• Less friendly error messages

The first successful request can feel like a victory. It is not. It is only the point where the actual connector work begins.

The connector problems that matter more than the first successful request

The hard parts usually come after authentication works.

| Connector concern | Why it matters | | --- | --- | | Rate limits | Prevents the connector from getting throttled or blocked | | Pagination | Keeps large extracts from timing out or missing data | | Incremental state | Avoids full reloads when only changed records are needed | | XML parsing | Keeps payload size and memory usage under control | | Validation | Protects downstream tables from malformed records | | Safe logging | Helps debugging without leaking sensitive data |

A production connector is not a request script. It is a repeatable sync process with memory, limits, and failure behavior.

Rate limits and backoff strategy

Rate limits should be designed into the connector from the beginning. Do not wait until the API pushes back.

A practical strategy includes:

• A conservative default request rate
• Exponential backoff for throttle responses
• Jitter so retries do not pile up at the same second
• Clear retry caps
• Safe failure after repeated errors
• Logs that explain which operation was throttled

The goal is not to hammer the API until it gives up. The goal is to sync steadily without becoming a bad citizen.

Pagination, date filters, and incremental sync state

Large enterprise APIs rarely fit into one response. Pagination and filtering are the core of the connector.

Useful state might include:

• Last successful sync timestamp
• Last page or cursor processed
• Entity type being synced
• Date window currently in progress
• Retry count for the current operation

Date filters help reduce payload size, but they need careful handling. If records can be updated late, use a small overlap window and deduplicate downstream.

XML payload size, parsing, and memory pressure

Large XML payloads can create memory pressure faster than expected. Avoid loading more than you need.

Practical options include:

• Request smaller pages
• Parse streams when payloads are large
• Drop unused XML fields early
• Convert records into typed internal objects before writing
• Validate required fields before downstream loading

XML is noisy. The connector should make it boring before the data reaches a table.

Logging without leaking sensitive data

Connector logs should help humans debug sync behavior without exposing private data.

Good logs include:

• Operation name
• Entity type
• Page or cursor position
• Request attempt number
• Status category
• Record counts
• Duration

Bad logs include raw payloads, sensitive values, session tokens, private identifiers, or full XML responses. When in doubt, log counts and fingerprints, not sensitive values.

Validation before loading downstream tables

Before writing records downstream, validate them.

At minimum:

• Required fields exist
• Dates parse correctly
• IDs are present and stable
• Record shape matches the expected field layout
• Unexpected fields are handled intentionally

Validation is where a connector protects the warehouse, dashboard, or automation that depends on it.

How to make SOAP connector work survivable

Survivable connector work is mostly discipline:

• Keep request construction isolated.
• Keep parsing separate from loading.
• Store sync state explicitly.
• Use predictable retries.
• Write small tests around sample payload shapes.
• Document how to restart safely.

The code does not need to be elegant. It needs to be understandable, testable, and safe to operate.

Practical checklist

• Confirm which SOAP service and object types are required.
• Start with the smallest useful data slice.
• Design pagination and date windows before scaling volume.
• Store sync state outside the running process.
• Use exponential backoff with jitter.
• Avoid logging raw XML payloads.
• Validate records before downstream writes.
• Write a restart and failure runbook.

FAQ

Is SOAP still worth integrating with?

Yes, when the source system owns important business data. The protocol may be unpleasant, but the data can still be worth the engineering work.

Should a SOAP connector do full reloads?

Only when the volume is small enough and the API can tolerate it. For larger systems, incremental sync and overlap windows are usually safer.

Can Fivetran custom connectors handle SOAP sources?

Yes, if the connector code handles authentication, request construction, XML parsing, pagination, state, and retries cleanly.

What is the biggest mistake in SOAP connector design?

Treating the first successful request as proof that the connector is almost done. Production behavior is where the real work starts.

Related Reading

• Custom Fivetran SDK connectors: a pragmatic guide
• Fivetran custom connectors when native connectors are not enough
• Why AI projects need boring plumbing before they need agents