Why pipelines have handoffs
Few useful automations are one tool.
A common shape: one tool retrieves something from a place with no proper interface, and hands it to a tool built for processing.
The handoff is where a lot of pipelines go wrong — and it is rarely discussed, because each tool’s documentation covers only its own side.
Send the whole file
The most important decision in the handoff.
Not row by row. The whole file, in one request.
Four reasons:
Fewer executions. A file with five hundred rows sent individually is five hundred calls. Sent whole, it is one. On usage-priced platforms that is a direct cost difference.
Less duplicate risk. A partial send that gets retried can produce duplicates. One file either arrived or it did not.
Clearer failure. “The upload failed” is easier to reason about than “rows 1 to 340 arrived and the rest did not”.
Cleaner separation of responsibility. The retrieving tool’s job is delivering the file. Understanding the contents belongs to the receiving tool.
That last point matters most. If both tools parse the file, both need updating when the format changes.
FIGURE 1: TWO WAYS TO HAND OFF
Whole file, one request
- One execution regardless of size
- Either it arrived or it did not
- Retry is safe
- One tool owns the parsing
Row by row
- Hundreds of executions
- Partial sends possible
- Retries can duplicate
- Both tools parse the format
The mechanics
A file upload over HTTP — a POST with the file attached as form data. The same mechanism a browser uses when you upload a file, done directly.
Three things to get right:
The field name. The receiving end expects the file under a particular name. Agree it and document it.
The content type. Declare what the file is, so the receiver knows how to handle it.
A single request. Not chunked, unless the file is genuinely large enough to need it.
The receiving end
A webhook — a URL that accepts the upload and starts a workflow.
Which requires the receiving system to be reachable from wherever the sender runs. For a self-hosted workflow tool, that means a domain, HTTPS, and a properly configured proxy.
Two things worth doing on the receiving side:
Validate before processing. Was a file received? Is it the expected type? Does it have content?
Return something useful. Not just a status code — a response saying what happened. How many rows were read, whether anything was rejected.
Read the response
The step people skip.
A successful HTTP status does not mean the work succeeded.
An upload can return 200 with a body saying “0 rows processed”. That is a failure, and only the response body reveals it.
On the sending side:
Capture the status code.
Capture the response body.
Log both.
And decide what counts as success. If the receiver reports zero rows processed, should the sending job pass or fail? Usually fail — something is wrong, and you want to know today.
FIGURE 2: WHAT MAKES A HANDOFF RELIABLE
Send the whole file
- One execution, safe to retry, one parser.
Validate before sending
- A bad file caught early is a clear error.
Read the response body
- A 200 with “0 rows processed” is a failure.
Agree the contract
- Field name, content type, what a good response looks like.
Agree the contract
Write down what each side expects. It takes ten minutes and it prevents a category of confusion.
What to record:
The URL the file goes to.
The field name and content type.
What the receiver returns on success and on failure.
What the sender does with each response.
Who owns each side.
Without this, a change on one side breaks the other and nobody knows whose problem it is.
Retries and duplicates
Uploads fail temporarily. Networks drop, services restart.
Retry — with a limit.
Which raises a question: what if the first attempt actually arrived and the response was lost?
Two defences:
Make the receiving side handle duplicates. It should check whether it has already processed this file — by name, by a hash, or by content — and skip it if so.
Include an identifier with the upload, so the receiver can recognise a repeat.
Retries without duplicate handling create duplicate data, and that is much harder to clean up than to prevent.
What to do with the file afterwards
Decide, rather than letting files accumulate.
Keep them for a period. Useful for investigating a problem — you can see exactly what was sent.
Then clean up. A working folder that grows forever eventually fills the disk, which produces a failure nobody expects.
Name them predictably. With a date or a run identifier, so you can find the one from the day something went wrong.
FIGURE 3: THE HANDOFF, END TO END
Validate the file
- Exists, right type, has content
Upload whole
- One request, agreed field name
Read the response
- Status and body, both logged
Clean up
- Keep for a period, then remove
What goes wrong
Row-by-row sending. Hundreds of executions, partial sends, duplicate risk.
Not reading the response. A false success that nobody questions.
No duplicate handling on the receiving side. Retries produce copies.
No agreed contract. A change on one side silently breaks the other.
Files accumulating. Until the disk fills, at which point everything fails at once.
The short version
Send the whole file in one request. Fewer executions, safe retries, and only one tool needs to understand the format.
Validate before sending and read the response after — a successful status with an unsuccessful body is a real and common failure.
Handle duplicates on the receiving side, because retries happen.
And write down the contract. Ten minutes now saves an argument later about whose problem it is.
A pipeline where one tool hands data to another?
Get in touch. The handoff is usually where these break — and it is the cheapest part to get right.
