Data infrastructure · 2025

Cost Data Pipeline

Turn an online cost reference into analysable data.

Looking up one procedure was straightforward; comparing a whole catalogue needed a different approach. This pipeline works through the source application, captures its structured responses and prepares consistent tables of fees, patient costs and geographic breakdowns.

Status
Shipped
Domain
Data acquisition
Role
Pipeline engineering
Year
2025
Cost Data Pipeline · Interactive concept

Explore the workflow with illustrative content.

Stage 03 / 03

Send it to the right next step

A complete sample reaches the output. A sample with a missing field remains in review rather than silently passing through.

Illustrative content · no live data

Follow a record from its source to a usable output

Stack

  • Python
  • Playwright
  • pandas
  • CSV · JSON · XLSX

A browser-driven pipeline that collects procedure-cost information and prepares documented tables for analysis and financial modelling.

01

Use the structure behind the page

Playwright opens the application and captures its structured data responses.

The extraction can then work with named fields and nested records directly, reducing its dependence on how the page happens to display a value.

02

Work through the catalogue reliably

The traversal follows specialties, services and geographic breakdowns while tracking records already collected.

Resumption and partial-failure handling make longer runs manageable without restarting the whole catalogue after an interruption.

03

Deliver data people can interpret

Responses are normalised into service-level tables and exported as CSV, JSON and Excel.

Column documentation explains the measures and their coverage, giving downstream analysts the context needed to use the extracts responsibly.

Pipeline

Extraction

Stage Can reject

  1. 01

    Headless browser

    Application session

  2. 02

    Capture responses

    Structured records

  3. 03

    Traverse

    Collected catalogue

    Progress tracked for resumption

  4. 04

    Flatten

    One row per service

  5. 05

    Emit

    CSV · JSON · XLSX

Behind the work

The skills
that came together.

Data acquisition

  • Browser automation and response capture
  • Catalogue traversal and deduplication
  • Resumable collection

Data preparation

  • Nested-response normalisation
  • Geographic field alignment
  • Documented CSV, JSON and Excel outputs

Project highlights

  • Structured extraction

    Capture the data responses used by the application

  • Repeatable collection

    Tracked progress and partial-failure handling

  • Ready for analysis

    Documented tables in three practical formats