Your Bronze Layer Is Someone Else's Query

"Bronze is the raw layer." Nearly every medallion-architecture diagram says some version of this, and Databricks — the company that coined the term — is admirably precise about it. Their own documentation says bronze "contains and maintains the raw state of the data source in its original formats," and in their example pipeline, "no data cleanup or validation is performed here" (the same page elsewhere allows for "minimal data validation," worth knowing if you go looking — the rule has an asterisk even at the source). For event logs, clickstreams, sensor feeds, files and API payloads, this is exactly right, and it buys you something real: replay, audit, lineage. Rebuild silver however many times you like — bronze never threw anything away to begin with.

Now try applying that rule to an SAP extract a business unit receives from its IT team. The textbook says: no changes, no cleaning, no filtering. In practice, that extract already arrived full of test rows, client codes, and fields where a date is stored as text and an amount's sign depends on a flag buried three columns over. "No cleanup" isn't a principle at that point — it's an instruction to publish garbage and call it fidelity. For a business unit building a medallion on top of that kind of extract, bronze has to be "raw-plus": mechanically prepared — filtered to the agreed scope, typed, renamed, de-duplicated, lineage added — and never calculating, merging, or exercising business judgment. The test, borrowed straight from the deck this article accompanies: is it a calculation or a merge, or would a business person argue about it? Then it starts in silver, not bronze.

"Raw" is a myth anyway — but that cuts both ways

Lisa Gitelman and Virginia Jackson's introduction to "Raw Data" Is an Oxymoron opens with an epigraph from Geoffrey Bowker: "'Raw data' is both an oxymoron and a bad idea." A few pages later, they make the same point in their own words: data, they write, "are always already 'cooked' and never entirely 'raw.'" It's a fair challenge to the whole idea of an untouched bronze layer — but it cuts in a direction the purists don't always expect. If nothing is ever truly raw, the interesting question stops being "is my bronze pure?" and becomes "how far has this data already travelled before it reached me, and does that distance matter?" For an SAP, PeopleSoft or Archibus extract, the honest answer is: pretty far. Someone else's query, built on someone else's joins and filters, already stands between you and the source tables. Only a homegrown database — an Access file, say — gets you close to anything resembling the tables themselves.

The category error

Here's the claim worth sitting with, because it's a little more pointed than "the textbook is wrong": the textbook isn't wrong. It was just never describing you. Databricks wrote the bronze-is-raw rule with event logs, clickstreams, sensor feeds and API payloads in mind — primary captures, the first digital representation of something happening in the world. Treating that rule as a universal law and bolting it onto a second-hand enterprise extract is a category error, not a difference of opinion. Bowker was right that raw data is an oxymoron — he just wasn't talking about your SAP extract.

So when does "raw-plus" genuinely fit, and when should you leave bronze alone? Six questions do most of the work:

Question Raw-plus bronze fits you Keep bronze genuinely raw
Is “raw” a copy, or the capture itself? A copy — the source system keeps running after you pull the extract. Lose it, ask again. The capture — it happened once and won’t happen the same way twice.
Is the meaning already settled? Yes — a pay code, a GL account, a cost centre were defined by policy before you touched the data. Not necessarily — new methods or classifications can redefine what a reading meant, years later.
Is the next question knowable? Mostly — future questions are variations on operational questions you already answer. No — a future analysis can ask something nobody has thought of yet.
Is the schema a contract or a moving target? A contract — documented, versioned, rarely changing without notice. A moving target — instruments, protocols and methods drift over time.
What gets audited — the number or the method? The number — your gold total reconciles to the source system’s control total. The method — scrutiny can ask for the original capture itself, not a cleaned summary of it.
What does a bad trim cost you? Low — ask for a wider extract next time. The source still exists. Potentially total — there is no re-running an observation that’s already over.

A worked example makes this concrete, and it's a real one. Fisheries and Oceans Canada runs annual groundfish surveys: a vessel works a fixed grid of stations, and at each one the catch is counted, measured and identified by species, while a CTD cast records water temperature, salinity and depth at that exact spot. That capture — this fish, this length, this reading, at this station, on this date — becomes a Research Document and a Science Advisory Report once it clears peer review through the Canadian Science Advisory Secretariat, and rolls up annually into DFO's Sustainability Survey, which categorizes each stock as Healthy, Cautious, Critical or Uncertain. Long-run, the oceanographic readings also land in BioChem, DFO's public biochemical and plankton data repository — browsable through DFO's open data portal if you want to see what genuinely untouched bronze looks like in the wild.

DFO Science Research Vessel

DFO Science Research Vessel

Run that scenario against the six questions and it lands almost entirely on the "keep it raw" side. Nobody can resend last Tuesday's trawl. A temperature reading that looks like sensor drift today might turn out, decades later, to be the first sign of a real warming event — and a species currently treated as one stock might eventually get split into two once genetic methods catch up, which means someone has to go back and re-run the original catch data against boundaries that didn't exist when it was captured. Peer review can ask to see the capture itself, not a cleaned summary of it. And there's no "ask the source for a wider extract" option when the source was a specific afternoon on a specific river.

Compress those six questions into something closer to a flowchart, and you get four gates, each one an AND, not an OR — fail any single one and the honest answer is "keep it raw," or at minimum keep an as-delivered copy rather than trusting your memory of what arrived:

Descision tree for bronze medaillion raw vs copied

Notice this isn't really "business vs. science" — it's "copy of something durable vs. capture of something that happened once." A real-property team doing a one-time environmental baseline survey ahead of a land transfer is a corporate-services group by every other measure, and it would still fail gate one or two. The split runs through what the data actually is, not through which department you sit in.

Two kinds of raw, inside the "copy" world

Even once you've decided you're on the raw-plus side of that test, not everything you receive behaves the same way. Enterprise extracts — SAP, PeopleSoft, Archibus — arrive with a known, fixed shape: rigid, normalized, documented by the system team, even if the meaning is hidden in codes and keys rather than words. That rigidity is exactly what makes trimming safe: you can decide the scope once, filter to it, and move on. Messy files are a different animal entirely — spreadsheets passed through many hands, survey exports, free-text comments. You can't safely trim inside a file before it's even been parsed; the structure only becomes known once silver finds it, and the real cost sits in the parsing, not the volume. For these, bronze stays close to the textbook: parse, type, flag what won't parse, keep the original alongside. You can still drop whole files — duplicates, periods clearly out of scope — just not columns or rows from inside one you haven't understood yet.

This isn't just a DAVHILL house position. On the Databricks Community forum, one practitioner answering exactly this question put it plainly: "If you need a reliable source of truth for reprocessing, keep raw files separate from Bronze." The choice tracks replay, audit and governance needs, not dogma. On the Microsoft Fabric Community, another practitioner working through a similar bronze-layer question described keeping raw files and a straight-copy-to-table version together in the same layer, then transforming into silver — land it, then prepare it, both inside bronze. And even Microsoft's own engineering playbook, written for general pipeline reliability rather than medallion architecture specifically, keeps validation out of bronze on purpose: it describes bronze as holding "a similar copy of the source system data," useful "for pipeline reruns to test validation logic and for data recovery," with validation explicitly placed between bronze and silver, not inside it.

Trimming in bronze makes silver cheap

Once bronze is allowed to trim, the payoff compounds downstream, because every join and calculation in silver scales with what goes in. Business units live inside fixed limits — Fabric capacity, refresh windows, Power BI import sizes — so this isn't academic. Lead with rows: a scope filter (an agreed, documented span of fiscal years, say) shrinks every join that happens after it. Columns matter less if you're in Delta or Parquet, since a query only reads the columns it actually uses — but they matter a great deal in Power Query and import mode, which load everything whether you need it or not. Worth saying honestly, since it's easy to overstate the columns point in either direction.

Keep the as-delivered copy — until someone upstream keeps it for you

None of this means never keeping an exact copy. It means asking first. A generic version of something I've seen play out in practice: a federal finance team's SAP transactions get extracted daily, around 3:00am. IT separately keeps period-end snapshots — including the P13/P14 adjustment periods that eventually become the reconciled year-end P12 — inside SAP or Cognos. Bronze pulls two things: the history file of period-end snapshots for an agreed span, say five years, and the previous day's daily extract. Silver appends them — closed periods come from the history file, the open period comes from the daily extract.

The history file needs no as-delivered copy in bronze, because IT keeps a clean copy you can pull again whenever you need it. The daily extract is different: it's kept only until month-end plus a few days, with that window set by the close calendar (and P13/P14 staying open longer at year end). A couple of things worth watching regardless: silver must not double-count the overlap between the history file and the daily extract, so the timing of the append matters; and most teams only need to trend by period end anyway — if a team genuinely needs movement within a month, raise the snapshot frequency rather than hoarding daily extracts "just in case." And still reconcile the history file in bronze every time — row counts, control totals, the snapshot version that was actually loaded. I've seen IT get a period-end snapshot wrong. It happens, and it's cheaper to catch in bronze than to explain in a board report.

IT usually keeps far more than five years' worth of history, so the five-year span is either a bronze scope filter or something built into IT's own Cognos or BW query. Ask first: if IT can filter for you, bronze doesn't have to.

Where the purists are right

None of this is an argument against purism everywhere. A team that owns ingestion directly — change data capture off a source database, a streaming pipeline — is in a genuinely different situation from a business unit one or two hops downstream of IT's query. Messy, unstructured files earn the same caution, for the reasons above. And anywhere replay is the actual requirement — DFO's groundfish surveys being the clean example — the purist's reasoning isn't excessive caution, it's the entire point of the layer. Give the purists their due; it's what makes the rest of this argument defensible rather than convenient.

Try it yourself

This is one small piece of what gets built live in From Bronze to Gold: Medallion Architecture for High-Performance BI, my workshop at DAMA Days Canada V in Ottawa, December 2–3, 2026. The session includes a short Excel exercise — the same pay-and-branch data used above, minus the fixes — where you find each problem, say which layer it belongs in, and watch gold put itself right. The worksheet is a free download on the session's reference page, whether or not you're in the room.

Where this leaves you

The textbook's test for bronze was always right. It just wasn't written with your SAP extract in mind. The real test, the one worth carrying into your own medallion: is it a calculation or a merge, or would a business person argue about it? Then it starts in silver — and whether bronze gets to trim at all depends on whether what you're holding is a copy of something durable, or the record of something that happened once and is already gone.

— Stephen Davies, DAVHILL Group.

Next
Next

DAMA Days Canada Conference (DDC) V