Few useful automations are one tool.<\/p>\n
A common shape:<\/strong> one tool retrieves something from a place with no proper interface, and hands it to a tool built for processing.<\/p>\n The handoff is where a lot of pipelines go wrong<\/strong> \u2014 and it is rarely discussed, because each tool’s documentation covers only its own side.<\/p>\n The most important decision in the handoff.<\/strong><\/p>\n Not row by row. The whole file, in one request.<\/strong><\/p>\n Four reasons:<\/strong><\/p>\n Fewer executions.<\/strong> A file with five hundred rows sent individually is five hundred calls. Sent whole, it is one. On usage-priced platforms that is a direct cost difference.<\/p>\n Less duplicate risk.<\/strong> A partial send that gets retried can produce duplicates. One file either arrived or it did not.<\/p>\n Clearer failure.<\/strong> “The upload failed” is easier to reason about than “rows 1 to 340 arrived and the rest did not”.<\/p>\n Cleaner separation of responsibility.<\/strong> The retrieving tool’s job is delivering the file. Understanding the contents belongs to the receiving tool.<\/p>\n That last point matters most.<\/strong> If both tools parse the file, both need updating when the format changes.<\/p>\n FIGURE 1: TWO WAYS TO HAND OFF<\/p>\n Whole file, one request<\/p>\n Row by row<\/p>\n A file upload over HTTP<\/strong> \u2014 a POST with the file attached as form data. The same mechanism a browser uses when you upload a file, done directly.<\/p>\n Three things to get right:<\/strong><\/p>\n The field name.<\/strong> The receiving end expects the file under a particular name. Agree it and document it.<\/p>\n The content type.<\/strong> Declare what the file is, so the receiver knows how to handle it.<\/p>\n A single request.<\/strong> Not chunked, unless the file is genuinely large enough to need it.<\/p>\n A webhook<\/strong> \u2014 a URL that accepts the upload and starts a workflow.<\/p>\n Which requires the receiving system to be reachable from wherever the sender runs.<\/strong> For a self-hosted workflow tool, that means a domain, HTTPS, and a properly configured proxy.<\/p>\n Two things worth doing on the receiving side:<\/strong><\/p>\n Validate before processing.<\/strong> Was a file received? Is it the expected type? Does it have content?<\/p>\n Return something useful.<\/strong> Not just a status code \u2014 a response saying what happened. How many rows were read, whether anything was rejected.<\/p>\n The step people skip.<\/strong><\/p>\n A successful HTTP status does not mean the work succeeded.<\/strong><\/p>\n An upload can return 200 with a body saying “0 rows processed”. That is a failure<\/strong>, and only the response body reveals it.<\/p>\n On the sending side:<\/strong><\/p>\n Capture the status code.<\/strong><\/p>\n Capture the response body.<\/strong><\/p>\n Log both.<\/strong><\/p>\n And decide what counts as success.<\/strong> If the receiver reports zero rows processed, should the sending job pass or fail? Usually fail<\/strong> \u2014 something is wrong, and you want to know today.<\/p>\n FIGURE 2: WHAT MAKES A HANDOFF RELIABLE<\/p>\n Send the whole file<\/p>\n Validate before sending<\/p>\n Read the response body<\/p>\n Agree the contract<\/p>\n Write down what each side expects.<\/strong> It takes ten minutes and it prevents a category of confusion.<\/p>\n What to record:<\/strong><\/p>\n The URL<\/strong> the file goes to.<\/p>\n The field name<\/strong> and content type.<\/p>\n What the receiver returns<\/strong> on success and on failure.<\/p>\n What the sender does<\/strong> with each response.<\/p>\n Who owns each side.<\/strong><\/p>\n Without this, a change on one side breaks the other and nobody knows whose problem it is.<\/strong><\/p>\n Uploads fail temporarily.<\/strong> Networks drop, services restart.<\/p>\n Retry<\/strong> \u2014 with a limit.<\/p>\n Which raises a question: what if the first attempt actually arrived and the response was lost?<\/strong><\/p>\n Two defences:<\/strong><\/p>\n Make the receiving side handle duplicates.<\/strong> It should check whether it has already processed this file \u2014 by name, by a hash, or by content \u2014 and skip it if so.<\/p>\n Include an identifier<\/strong> with the upload, so the receiver can recognise a repeat.<\/p>\n Retries without duplicate handling create duplicate data<\/strong>, and that is much harder to clean up than to prevent.<\/p>\n Decide, rather than letting files accumulate.<\/strong><\/p>\n Keep them for a period.<\/strong> Useful for investigating a problem \u2014 you can see exactly what was sent.<\/p>\n Then clean up.<\/strong> A working folder that grows forever eventually fills the disk, which produces a failure nobody expects.<\/p>\n Name them predictably.<\/strong> With a date or a run identifier, so you can find the one from the day something went wrong.<\/p>\n FIGURE 3: THE HANDOFF, END TO END<\/p>\n Validate the file<\/p>\n Upload whole<\/p>\n Read the response<\/p>\n Clean up<\/p>\n Row-by-row sending.<\/strong> Hundreds of executions, partial sends, duplicate risk.<\/p>\n Not reading the response.<\/strong> A false success that nobody questions.<\/p>\n No duplicate handling on the receiving side.<\/strong> Retries produce copies.<\/p>\n No agreed contract.<\/strong> A change on one side silently breaks the other.<\/p>\n Files accumulating.<\/strong> Until the disk fills, at which point everything fails at once.<\/p>\n Send the whole file in one request.<\/strong> Fewer executions, safe retries, and only one tool needs to understand the format.<\/p>\n Validate before sending<\/strong> and read the response after<\/strong> \u2014 a successful status with an unsuccessful body is a real and common failure.<\/p>\n Handle duplicates on the receiving side<\/strong>, because retries happen.<\/p>\n And write down the contract. Ten minutes now saves an argument later about whose problem it is.<\/p>\n A pipeline where one tool hands data to another?<\/p>\n Get in touch. The handoff is usually where these break \u2014 and it is the cheapest part to get right.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":" Why pipelines have handoffs Few useful automations are one tool. A common shape: one tool retrieves something from a place with no proper interface, and hands it to a tool built for processing. The handoff is where a lot of pipelines go wrong \u2014 and it is rarely discussed, because each tool’s documentation covers only […]<\/p>\n","protected":false},"author":3,"featured_media":9377,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23],"tags":[],"class_list":["post-9376","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-robotic-process-automation"],"yoast_head":"\nSend the whole file<\/h2>\n
\n
\n
The mechanics<\/h2>\n
The receiving end<\/h2>\n
Read the response<\/h2>\n
\n
\n
\n
\n
Agree the contract<\/h2>\n
Retries and duplicates<\/h2>\n
What to do with the file afterwards<\/h2>\n
\n
\n
\n
\n
What goes wrong<\/h2>\n
The short version<\/h2>\n