A published government dataset needed to reach a business system.<\/p>\n
The data lives on a public website<\/strong>, released as a ZIP archive containing a CSV.<\/p>\n Doing it by hand:<\/strong> visit the site, download the archive, extract it, find the right file inside, open it, check it looks right, then get it into the target system. Every time the data updates.<\/p>\n Not difficult. Just repetitive, and easy to skip on a busy day<\/strong> \u2014 which means working from stale data without realising.<\/p>\n Two tools, with a clear split.<\/strong><\/p>\n UiPath handles getting the file.<\/strong> Download, extract, find, validate, upload.<\/p>\n A workflow tool handles the contents.<\/strong> Parsing, mapping, checking for existing records, creating them.<\/p>\n Why split it at all?<\/strong><\/p>\n Because the two jobs have different failure modes and different maintenance profiles.<\/p>\n Getting a file from a public website<\/strong> depends on that website. It will change eventually, and when it does, that step needs attention.<\/p>\n Processing the contents<\/strong> depends on your own systems and your own rules. It changes when your business changes.<\/p>\n Keeping them separate means a website redesign affects one small step<\/strong>, not the whole pipeline.<\/p>\n FIGURE 1: THE SPLIT<\/p>\n UiPath<\/p>\n Hand off<\/p>\n Workflow tool<\/p>\n Target system<\/p>\n The archive is fetched by HTTP request, not by driving a browser.<\/strong><\/p>\n This is worth pointing out<\/strong>, because RPA is assumed to mean screen automation. Here it does not.<\/p>\n Where a direct URL exists, requesting it is better in every way<\/strong> \u2014 faster, more reliable, easier to debug, and unaffected by page layout changes.<\/p>\n Navigating a site with a browser is a last resort<\/strong>, for when a login is required or the link is generated per session.<\/p>\n The archive is saved to a known location.<\/p>\n The archive is extracted to a working folder.<\/strong><\/p>\n Then the automation looks for the file it needs<\/strong> \u2014 searching the folder and its subfolders.<\/p>\n And here is a detail worth borrowing:<\/strong> if several matching files are found, take the newest by modification time<\/strong>.<\/p>\n Why that rule:<\/strong> archives sometimes contain more than one file, or a previous extraction has left older files behind. “Newest” is a simple, deterministic rule that does the right thing in both cases \u2014 rather than “the first one found”, which depends on file system ordering and can silently pick the wrong file.<\/p>\n Two checks before anything is sent.<\/strong><\/p>\n Was a file found at all?<\/strong> If not, something went wrong upstream \u2014 an empty archive, a failed download, a changed structure.<\/p>\n Does the file actually exist where the automation expects?<\/strong> Extraction can fail in ways that leave the path but not the file.<\/p>\n Failing here is much better than failing later.<\/strong> A missing file caught before upload is a clear, actionable error. A missing file discovered downstream is a confusing problem in another system.<\/p>\n FIGURE 2: THE CHECKS THAT PREVENT CONFUSING FAILURES<\/p>\n Was a file found?<\/p>\n Does it exist on disk?<\/p>\n Is it the right type?<\/p>\n Fail here, not downstream<\/p>\n The entire CSV is sent in one request<\/strong> to a webhook the workflow tool exposes.<\/p>\n Not row by row. The whole file, once.<\/strong><\/p>\n Three reasons this is the right choice:<\/strong><\/p>\n Fewer executions.<\/strong> A file with hundreds of rows sent individually means hundreds of calls. Sent whole, it is one.<\/p>\n Less duplicate risk.<\/strong> A partial send that gets retried can produce duplicates. One file either arrived or it did not.<\/p>\n Clearer separation.<\/strong> The automation’s job is delivering the file. Understanding what is inside it belongs to the next system.<\/p>\n The upload is a standard multipart form post<\/strong> with the file attached \u2014 the same mechanism a browser uses to upload a file, done directly.<\/p>\n The automation captures the HTTP status and the response.<\/strong><\/p>\n Why this matters:<\/strong> an upload that returns a success status but a response saying nothing was processed is a failure wearing a success badge. Only reading the response body reveals it.<\/strong><\/p>\n A successful upload is logged.<\/p>\n Every stage is wrapped in try-catch<\/strong> \u2014 download, extraction, validation, upload.<\/p>\n When something fails:<\/strong><\/p>\n The error message and stack trace are logged.<\/strong> It runs unattended, so that log is all anyone will have the next morning.<\/p>\n The error is re-thrown<\/strong>, so the job is marked as failed.<\/p>\n That last point is the important one.<\/strong> An automation that catches an error and exits successfully is the worst possible outcome \u2014 the orchestrator records a success, nobody is alerted, and the data silently did not arrive. Somebody discovers it weeks later when a report is short.<\/p>\n A failed job is visible. A quietly-succeeded failure is not.<\/strong><\/p>\n FIGURE 3: WHY THIS DESIGN HOLDS UP<\/p>\n What was done<\/p>\n The alternative<\/p>\n This shape comes up whenever you need data from a source you do not control.<\/strong><\/p>\n The general form:<\/strong><\/p>\n One tool retrieves the file \u2014 using the least fragile method available, and doing nothing else.<\/p>\n It validates that it got something real.<\/p>\n It hands the whole thing to a tool built for processing.<\/p>\n That tool parses, checks against what already exists, and creates records.<\/p>\n The mistake is using one tool for all of it<\/strong> \u2014 usually the RPA tool, because it technically can. It works, and then every change anywhere breaks the whole chain.<\/p>\n Use an HTTP request if a direct URL exists.<\/strong> Do not drive a browser out of habit.<\/p>\n Pick files deterministically.<\/strong> “Newest by modification time” is a real rule. “The first one found” is not.<\/p>\n Validate before handing off.<\/strong> Cheap to check, expensive to discover later.<\/p>\n Send whole files.<\/strong> Fewer executions, less duplicate risk, cleaner failures.<\/p>\n Re-throw errors.<\/strong> A job that silently succeeds after failing is worse than one that fails loudly.<\/p>\n Two tools, one job each. One gets the file. The other understands it.<\/strong><\/p>\n The retrieval step uses an HTTP request rather than screen automation<\/strong>, which makes the fragile part far less fragile than RPA’s reputation suggests.<\/p>\n Validate before handing off, send the whole file at once, and re-throw errors<\/strong> so a failure is visible rather than silent.<\/p>\n That last one is what separates a pipeline you can trust from one that quietly stops working.<\/p>\n Data you need from a source you do not control?<\/p>\n Get in touch. We build pipelines with the fragile part kept small \u2014 and errors that fail loudly rather than silently.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":" The situation A published government dataset needed to reach a business system. The data lives on a public website, released as a ZIP archive containing a CSV. Doing it by hand: visit the site, download the archive, extract it, find the right file inside, open it, check it looks right, then get it into the […]<\/p>\n","protected":false},"author":3,"featured_media":9307,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[23],"tags":[],"class_list":["post-9306","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-robotic-process-automation"],"yoast_head":"\nThe design decision that matters<\/h2>\n
\n
\n
\n
\n
Step 1 \u2014 Download<\/h2>\n
Step 2 \u2014 Extract and find<\/h2>\n
Step 3 \u2014 Validate<\/h2>\n
\n
\n
\n
\n
Step 4 \u2014 Upload the whole file<\/h2>\n
Step 5 \u2014 Check the response<\/h2>\n
Step 6 \u2014 Error handling<\/h2>\n
\n
\n
What the pattern generalises to<\/h2>\n
What we would tell anyone building this<\/h2>\n
The short version<\/h2>\n