No-API publishing · Browser automation · .NET
Partner Publishing Pipeline
A browser-automation pipeline that pushes fund and product data into 30 partner portals that have no API. An analyst approves each push, workers on an autoscaling VM Scale Set log in and publish through the browser, and every push is verified in a fresh session with a screenshot of each step kept as evidence.
What it does
Hourly staff used to hand-key fund data into 30 partner sites. This pipeline does the keying, then proves it got it right.
Analyst-approved pushes
Customers and staff manage product data in the web app. An analyst reviews a customer's products, confirms them, and triggers the push. Nothing goes out unreviewed.
Publishing through the browser
Each profile (a product and its data for a period: AUM, holdings, and the rest) is typed into the partner's own web forms, the way a person would, because none of the 30 sites offered an API.
Verified, with evidence
After submitting, the worker opens a fresh session, reads the published values back and compares them to the source. Every step is logged with screenshots, so a failure shows exactly where it broke.
Architecture
SQL is the record of truth; the queue only carries row IDs. Workers scale with the queue, publish, verify, and write their evidence back.
Technology stack
One C# codebase across the web app and the workers, a real browser for the portals, and plain Azure storage for the queue and the evidence.
Engineering highlights for technical reviewers
The problems that made this hard, and the patterns used to solve them.
SQL row first, queue message second
Approving a push writes a queue row to SQL, then an Azure Queue message carrying only that row's ID. The worker reads the real payload from SQL, so the queue never holds stale data and SQL stays the single place to ask "what happened to this push?"
Scale follows the work
The VM Scale Set autoscales on queue length. A big approval batch brings up more workers, an empty queue lets them drain away, and nobody has to size the fleet by hand.
One codebase, web app and workers
Using PuppeteerSharp instead of Node Puppeteer kept the workers in C#, so they reuse the web app's own data and business logic on the Windows VM image instead of re-implementing it.
Trust nothing until it's read back
A submitted form isn't a published profile. After each push the worker starts a new session, reads the values back from the partner's site and compares them to SQL. Only a match counts as verified.
A screenshot for every step
Old partner portals fail a lot, in new ways. Each step is logged to Azure Blob Storage with a full set of screenshots, so a red push shows exactly which page and which field broke, without re-running it to find out.
A dashboard people want to turn green
The web app's reporting shows every push as green (verified), yellow (unverified) or red (failed), next to the screenshots. Analysts' goal was simple: get their dashboard to all green.
What I'd change, and did
The weak spot was deploying. Shipping a worker change meant pausing automation, updating the VM image, checking it, publishing it, then turning automation back on.
Tried: Azure Functions
I tried moving the workers to Azure Functions to get deploys off the VM image. I never got it working well enough to replace the scale set, so the VM Scale Set stayed as close as this system got.
Carried forward: Linux containers
The next pipeline I built runs its browser workers as Linux containers from a container registry on Azure Container Apps, so a deploy is a new image, not a paused fleet. See the Facility Data Pipeline →
Capabilities demonstrated services & hiring
What building and running this system proves I can deliver.