Healthcare data platform lineage documentation is not one artifact but several, and the differences determine what an auditor can reconstruct. Microsoft's healthcare data solutions embed lineage columns directly in silver and gold lakehouse tables, recording creation timestamps, source file paths, and source systems at the record level [1]. An AWS Storage Blog implementation instead attaches FHIR Provenance-aligned annotations to S3 objects as data moves through bronze, silver, and gold buckets [4]. A CareEvolution FHIR-to-OMOP pipeline uses dedicated extension tables to track source lineage, entity relationships, concept standardization decisions, and processing events [5]. Collibra documents lineage as dependency relations between cataloged data objects [6]. Each approach answers a different question: where a record came from, what happened to a file, how a concept was mapped, or which assets depend on each other. Selecting documentation therefore requires matching the mechanism to the audit or research question, not assuming one format covers all needs.

Record-level columns versus object-level annotations

Microsoft's silver lakehouse tables include msftCreatedDatetime, msftModifiedDatetime, msftFilePath, and msftSourceSystem columns to support lineage and traceability at the record and file level [1]. The msftFilePath column stores the full path to the source file in the bronze lakehouse, including shortcuts, while msftSourceSystem corresponds to the Namespace in the unified folder structure [1]. When a field is normalized, flattened, or modified, the original value is preserved in an {columnName}Orig column; examples in the Patient table include meta_lastUpdatedOrig, idOrig, identifierOrig, birthdateOrig, and deceasedDateTimeOrig [1]. Gold lakehouse tables include columns that trace records back to the silver lakehouse [1]. This design embeds lineage inside the analytical tables themselves, so a query against the data can return provenance without a separate metadata lookup.

The AWS S3 annotation approach places lineage outside the data tables but co-located with the objects. Each object receives a lineage annotation using a FHIR Provenance-aligned schema that tracks origin, transformations, and operators, plus a consent annotation tracking which patients' data is in the object [4]. Lambda writes lineage annotations on bronze bucket uploads, and AWS Glue ETL jobs write annotations on transformation to silver and gold [4]. The blog states that co-locating lineage metadata with data avoids metadata drift and makes compliance audits queryable with standard SQL through Athena [4]. The distinction matters: record-level columns support field-by-field tracing within a table, while object annotations support file-level chain traversal across storage tiers. A platform team should not assume one mechanism substitutes for the other.

What transformation lineage captures in FHIR-to-OMOP conversion

The CareEvolution pipeline documents lineage through a four-component framework: data source lineage, entity tracking, concept standardization, and processing events [5]. Source codes are mapped to OMOP standard concepts using the CareEvolution terminology API, which performs exact code and display matching and natural language processing [5]. When standard concepts cannot be identified, source codes are preserved with concept_id = 0 and detailed rationale captured in the PROCESSING_LOG table [5]. This is a different lineage question from storage provenance: it records why a clinical code became a particular standard concept, not merely which file contained the original value.

The dataset behind this documentation came from the PROGRESS study using CareEvolution's MyDataHelps research platform, covering 665 individuals with data from 2,170 unique zip codes across 46 US states and an average of 6.3 source systems per individual [5]. The authors report that lineage tracking revealed transformation patterns, including nearly 522,000 duplicate resources [5]. The pipeline output was run through the OHDSI Data Quality Dashboard and a managed instance of OHDSI's Atlas tool to check adherence to the OMOP CDM standard [5]. This example shows lineage documentation serving research reproducibility rather than regulatory audit, and the extension-table design means the lineage lives in dedicated tables rather than inline columns or object metadata.

A hypothetical validation scenario for a field-level audit

A useful validation exercise, suggested here rather than drawn from any tested deployment, would trace one normalized patient birthdate from a bronze HL7 message to a gold analytics view. The auditor would first query the silver Patient table's birthdateOrig column to recover the original date value and its formatting, then read msftFilePath and msftSourceSystem to identify the bronze source file and originating system [1]. If the pipeline used S3 annotations instead, the auditor would query the lineage annotation on the silver object for source object references and transformation operators, then traverse the chain back to the bronze object [4]. For a FHIR-to-OMOP conversion, the auditor would check the PROCESSING_LOG table for concept mapping rationale where concept_id = 0 [5]. The exercise would reveal which mechanism supports the requested granularity and where documentation ends.

Catalog dependency graphs and the business-technical split

Collibra Data Lineage maps the data lifecycle from source systems to downstream target systems and distinguishes business lineage from technical lineage [6]. Business lineage allows business analysts and stewards to view relations between assets in Data Catalog that represent data objects in external sources, while technical lineage allows data engineers and architects to view the flow of data objects in external sources [6]. Business lineage is accessible via the Diagram tab on asset pages; technical lineage is accessible via the Technical Lineage tab on Column and Table assets and some BI assets [6]. This split means the same underlying pipeline can be documented for two audiences with different granularity.

Collibra represents dependencies as relations of the type "Data Element targets / sources Data Element" [6]. A direct dependency is a data object that is the tail of such a relation; an indirect dependency is the target of a direct or another indirect dependency [6]. The documentation gives a concrete example: if column A targets column B, then column B is the direct dependency of column A; if column B also targets column C, then column C is the indirect dependency of column A [6]. This graph model answers impact-analysis questions, such as which downstream columns change when an upstream source changes. It does not by itself record transformation logic or original values, which is why it complements rather than replaces pipeline-level lineage.

Upsert boundaries and where lineage documentation stops

Microsoft's documentation specifies how bronze records update silver records. Pipelines compare {FHIRResource}.id and {FHIRResource}.meta_lastUpdated values against the id and lastUpdated columns in the ClinicalFhir bronze lakehouse staging table [1]. If a match is found and the bronze record is newer based on lastUpdated, the pipeline updates the existing silver record; if the bronze record is not newer, it is ignored; if no match exists, the bronze record is inserted as a new entry [1]. The documentation also states that some capabilities or specific tables might follow different upsert rules, directing readers to the respective capability documentation [1]. This is a documented boundary: the general rule does not apply universally within the platform.

The AWS blog describes a deployment boundary in its prerequisites and CloudFormation stack. The solution creates encrypted, versioned S3 buckets with public access blocked and access logging enabled, S3 Metadata annotation tables on each data bucket, a federated AWS Glue catalog for Athena, a Lambda function triggered by EventBridge on bronze uploads, two Glue ETL jobs, an Athena workgroup, CloudTrail with S3 data events, and TLS enforcement through DenyInsecureTransport bucket policies [4]. The deployment script generates synthetic HL7 data for verification [4]. This is a sample solution that the blog explicitly frames as adaptable: readers are told to deploy the sample and adapt the annotation schemas to their own pipeline [4]. The documentation does not establish that the sample meets any particular compliance requirement in production.

Governance guidance versus implemented lineage

The Duke-Margolis white paper recommends that data curators establish documented SOPs with recurring validation cycles and traceability for all curation operations, documenting provenance, maintaining auditability, and ensuring traceable metadata [2]. It also recommends extending HL7 FHIR Provenance documentation and ONC transparency expectations, and building on FDA RWE/RWD frameworks [2]. These are policy recommendations from a February 2026 white paper informed by the Duke-Margolis RWE Collaborative's 2025 workstream, not descriptions of a shipped product [2]. The paper's recommendations address what organizations should document, while the vendor and pipeline sources show what specific implementations actually record.

AWS's healthcare analytics reference architecture states that AWS Glue ETL processing workflows transform and normalize data through serverless and horizontally scalable jobs, and that organizations should track data lineage to establish traceability and reproducibility for compliance [3]. It also directs users to verify that AWS services are eligible for the healthcare compliance framework applicable to the workload, such as HIPAA Eligible AWS Services [3]. This is architectural guidance, not a claim that lineage tracking is automatically configured. The reference architecture lists components that can be used as-is or substituted with third-party components [3]. Teams evaluating lineage documentation should distinguish between a vendor's stated capability, a reference pattern, and what their own deployment actually writes.