Data infrastructure · 2025
Cost Data Pipeline
Turn an online cost reference into analysable data.
Looking up one procedure was straightforward; comparing a whole catalogue needed a different approach. This pipeline works through the source application, captures its structured responses and prepares consistent tables of fees, patient costs and geographic breakdowns.
- Status
- Shipped
- Domain
- Data acquisition
- Role
- Pipeline engineering
- Year
- 2025
Explore the workflow with illustrative content.
Keep the original input
The first stage represents an incoming record. A separate source copy makes it possible to inspect where an extracted value began.
Check the structured result
Fields are extracted into a consistent shape. Switch to the review example to see a missing field routed for attention.
Send it to the right next step
A complete sample reaches the output. A sample with a missing field remains in review rather than silently passing through.
Illustrative content · no live data
Follow a record from its source to a usable output
Stack
- Python
- Playwright
- pandas
- CSV · JSON · XLSX
A browser-driven pipeline that collects procedure-cost information and prepares documented tables for analysis and financial modelling.
Use the structure behind the page
Playwright opens the application and captures its structured data responses.
The extraction can then work with named fields and nested records directly, reducing its dependence on how the page happens to display a value.
Work through the catalogue reliably
The traversal follows specialties, services and geographic breakdowns while tracking records already collected.
Resumption and partial-failure handling make longer runs manageable without restarting the whole catalogue after an interruption.
Deliver data people can interpret
Responses are normalised into service-level tables and exported as CSV, JSON and Excel.
Column documentation explains the measures and their coverage, giving downstream analysts the context needed to use the extracts responsibly.
Pipeline
Extraction
Stage Can reject
-
01
Headless browser
Application session
-
02
Capture responses
Structured records
-
03
Traverse
Collected catalogue
Progress tracked for resumption
-
04
Flatten
One row per service
-
05
Emit
CSV · JSON · XLSX
Behind the work
The skills
that came together.
Data acquisition
- Browser automation and response capture
- Catalogue traversal and deduplication
- Resumable collection
Data preparation
- Nested-response normalisation
- Geographic field alignment
- Documented CSV, JSON and Excel outputs
Project highlights
-
Structured extraction
Capture the data responses used by the application
-
Repeatable collection
Tracked progress and partial-failure handling
-
Ready for analysis
Documented tables in three practical formats