HOSPITALITY · DATA OPERATIONS · INTEGRATIONS

Making multi-site data pipelines recoverable and easier to operate.

We delivered recurring data workflows across APIs, databases, SFTP and browser-only portals, while improving how the client’s Dagster platform handled partitions, backfills, concurrency, retries and reconciliation.

Dagster run overview showing successful source acquisition, site partition processing and reconciliation — CONFIDENTIAL HOSPITALITY OPERATOR · Data integrations and platform improvement

Client: Confidential
Sector: Multi-site hospitality
Engagement: Data integrations and platform improvement
Status: Delivered into an existing estate

SOURCES

Four source patterns

APIs, ODBC, browser portals and SFTP

PROCESSING

Site-level parallelism

Controlled fan-out and fan-in

RECOVERY

Recoverable processing

Partitions, checkpoints and idempotency

PROJECT SNAPSHOT

The engagement in three parts.

Situation

Operational reporting depended on data from several suppliers, each exposing a different interface. Failures could force large re-runs, and the existing Dagster estate was not consistently using its partition, backfill and concurrency capabilities.

What we delivered

We delivered reusable acquisition and transformation components, site-level parallel processing, bounded backfills, resumable pagination, duplicate-aware finance submission and inspectable failure artefacts.

Our responsibility

We owned acquisition adapters, Dagster patterns, partitioning, recovery behaviour and integration testing inside the existing estate.

What changed

A more capable existing data platform, with reusable integration patterns and narrower recovery across multi-site workloads.

Independent site processing, partition-aware backfills and source-specific concurrency controls were implemented inside the client’s Dagster estate.
Idempotency, atomic writes, checkpoints and failure artefacts improved the safety of replaying incomplete work.

They fitted straight into our weekly standups and helped migrate clients onto our new processing system. Clear, dependable code — and they even improved some of our internal algorithms along the way.

A.C.

Hospitality Software

Eyezilla operator feedback

SOURCE-TO-PLATFORM ARCHITECTURE

Different sources, one recoverable operating model.

Retries and checkpoints form a recovery layer around site-level partitions so incomplete work can resume narrowly.

01 · System layer
Source acquisition
02 · System layer
Input validation
03 · System layer
Site preparation
04 · System layer
Partition processing
05 · System layer
Enrichment
06 · System layer
Reconciliation

SELECTED SYSTEM EVIDENCE

Reusable acquisition and recoverable site-level processing.

REST API, ODBC, portal and SFTP sources moving through acquisition, validation and normalisation into checkpointed partitions

One acquisition model for imperfect source systems

The delivered pipelines handle direct APIs, cursor- and page-based pagination, ODBC extracts, browser-only reporting and SFTP file exchange. Shared browser utilities own startup, cleanup, safe interaction, report dates, downloads and diagnostic evidence.

Why it mattered

New integrations could reuse tested operational behaviour instead of rebuilding it around every supplier.

Parallel site transaction backfill across daily partitions with collection and reconciliation

Parallel site processing with controlled backfills

One high-frequency pipeline discovers configured sites and creates an independent processing unit for each. A custom partition IO manager preserves per-period artefacts even when several partitions are processed in a single backfill.

Why it mattered

Site-scaled workloads no longer had to extend one serial critical path, while individual periods remained rerunnable.

ENGINEERING DECISIONS

Designing for partial failure, replay and source constraints.

Model time as a first-class operating constraint

Daily and weekly partitions, London-aware schedules, UTC conversion and half-open date ranges prevent subtle duplication at day and week boundaries. Repeated intraday refreshes receive unique run keys rather than being mistaken for duplicate work.

Data correctness

Save progress only after successful downstream work

Pagination state is scoped by site and business date and is retained only after the corresponding batch has been indexed. A retry resumes from the last confirmed marker rather than from the last page merely requested.

Recovery

Preserve partition boundaries during efficient backfills

Multi-partition runs can fetch a bounded range once and still persist one atomic artefact per asset partition. Validation checks that every requested partition produced output before the files are replaced.

Orchestration

RESPONSIBILITY