The Modern Data Stack Hoax

You bought a Ferrari to sit in traffic.

You signed the contract. Databricks or Snowflake on a multi-year commit, dbt layered on top, maybe a slick orchestration tool to round it out. You sent the all-hands email. You stood up in the QBR and told the board you were now a “Modern Data Stack” shop.

Then you went back to your desk, and your team kept writing the exact same bloated, nested, six-levels-deep views they wrote on SQL Server in 2014. Nothing about how the work actually gets done changed. You just changed where the invoice goes.

You didn’t modernize. You re-hosted.

I call this Tool-Shaped Thinking: the belief that the shape of your tools determines the shape of your engineering. It doesn’t. It never has. A premium platform is a permission to do better work. It is not the work. You can run undisciplined, untested, undocumented SQL on the most expensive Lakehouse on earth, and all you have done is buy a Ferrari to sit in traffic.

Here is the law, and you can quote me in your next vendor meeting:

You can’t buy discipline. You install it.

The Hard Truth

Procurement is not strategy. I understand why it feels like it is. Signing a six- or seven-figure platform commitment is a decision. It is a line item. It photographs well in a board deck. It is something you can point to and say “we invested.”

But the operating model is the thing that decides whether that platform saves money or sets it on fire, and the operating model does not come on an invoice. No vendor sells it to you. No SOW installs it. It is the part you have to build, by hand, after the logo is on the contract.

And the platform you just bought changed the rules in a way most leadership never thinks through. On SQL Server, bad SQL was slow. On consumption-based compute, bad SQL is slow and metered. Every nested view, every full-table scan, every unpartitioned join is now a number on a bill that grows while you sleep. I have written before about a “Senior” team that incinerated fifty thousand dollars promoting data from Bronze to Silver in a single run, simply because nobody modeled (see When Did Modeling Die?). That was not a tooling failure. That was old habits running at cloud scale, in real time, with the meter on.

So the tool you bought to save money becomes the tool charging you for your team’s worst habits, faster than ever.

The Crime Scene

Let me show you what this looks like with the lid off.

A few years back, around 2020, I was brought into a client mid-migration to Snowflake. Their source was an RDS Postgres database. One of the jobs feeding their marketing reporting was a single PL/pgSQL script. Five thousand lines. One file.

Hard-coded variables everywhere. Business logic braided directly into the transformation logic, with no separation between what the pipeline did and why it did it. By the time it reached me, the person who handed it over had inherited it from someone else, who had almost certainly inherited it from someone before that. Nobody in the building could fully explain it. I had to interview business stakeholders who were around when it was first written just to reconstruct why certain decisions had been made, and half of those assumptions were years out of date, frozen in code, still running every single night.

It was a load-bearing wall, and no one had the blueprint.

And the plan on the table? Lift and shift. Move the monolith to Snowflake as-is and call it a migration.

Lift-and-shift is just Tool-Shaped Thinking with a migration budget.

Think about what that move actually buys you. You take a script that grinds for hours every night, that nobody understands, that hard-codes business rules from a reality that no longer exists, and you drop it onto a platform that charges you by the second. You have not solved the problem. You have put the problem on a meter.

Who answers for that bill when the CFO comes looking? Not the vendor. Not the script. You do.

Pick the Right Defendant

Here is the part the engineers reading this already know in their gut.

The person who wrote that monolith was not a bad engineer. They were an engineer who was never given an operating model. No version control, no separate environments, no tests, no standard, because no one above them ever required any of it. The Oracle who holds the whole system in their head (yes, the one you privately consider indispensable) is not the villain here. That person is a symptom. I have written a whole piece on how that trap is built, The 10-Year Junior, and the lesson holds at the org level too.

The villain is the organization that confused buying tools with building discipline.

If your team is still writing 2014 SQL on a 2026 platform, that is not your team’s failure. It is a leadership failure. And the good news, the part that should make you exhale, is that it is fixable. Just not by anyone in sales.

The Discipline Audit

Before your team touches another feature of that expensive new platform, you run a Discipline Audit. Four steps. They go in this exact order, because each one is worthless without the one before it.

1. Git or it didn’t happen. Every transformation, every config, every business rule goes into version control before one more thing ships to the new platform. This goes first because nothing above it can be audited, reviewed, or rolled back without it. The translation for the boardroom: if the logic is not versioned, it lives in one person’s head, and you have paid a multi-year commitment to host a single point of failure. When that person leaves, the knowledge leaves with them, and the next hire starts the same interview-the-stakeholders archaeology I had to do. Git is not a developer nicety. It is institutional memory you actually own.

2. Separate the environments. Once the code is versioned, your team needs somewhere to break things that is not production. Dev, test, prod, with real boundaries. This goes second because automating against a single shared environment is just a faster way to set the company’s data on fire. One environment means every deploy is a live bet, and the blast radius is the number you report upstairs. The platform will happily let your team run experiments on the production road at full speed. Discipline is what builds the test track first.

3. Automate the tests. Versioned code plus a safe environment lets you finally gate every change before it reaches a consumer. This goes third because it depends entirely on the first two. On consumption-based compute, an untested transformation is a runaway meter: it does not fail loudly, it fails expensively. A test suite is the circuit breaker between a quiet logic error and a cloud bill that doubled before anyone thought to look. If your only “test” is a Slack alert everyone has muted, you do not have testing. You have a smoke detector with the battery pulled out.

4. Externalize the business logic. Get the hard-coded rules out of the transformation code and into driving tables. This goes last because it is the maturity move, and it only sticks once the first three are in place. It is also the step that pays leadership back directly. When a pricing rule, a regional cutoff, or a revenue definition is buried inside a script, the business pays an engineer every single time that rule changes, and waits in a queue while they do it. When that same rule lives in a table, the business can see its own logic and change it without a deploy. You stop renting access to your own decisions.

Notice what all four have in common. Not one of them came in the box. The platform you bought ships none of them. There is no checkbox in Databricks or Snowflake labeled “discipline.” You install these by hand, in this order, and that is precisely the work no purchase order can do for you.

The Fix

Back to that monolith.

We did not lift and shift it. We deconstructed it. We split the five thousand lines into discrete, named steps. We stripped out the hard-coding and pulled the business logic into driving tables, so the rules could be tracked over time and changed without a code deploy. We rebuilt the heavy lifting as a Python notebook running on Spark, so it processed efficiently instead of grinding for hours.

The results were not abstract.

The job that ran for hours every night moved toward near real-time. Cloud cost for that function dropped by roughly thirty percent. And because the business logic now lived in tables instead of in a script only one departed engineer ever fully understood, the marketing and sales teams could finally see and trial the metrics they cared about, instead of waiting on someone to decode a wall of PL/pgSQL.

Same platform. Same data. One approach prints money. The other one is a Ferrari in traffic with the meter running.

That is the difference between re-hosting and re-architecting.

The Verdict

The Modern Data Stack is not a hoax because the tools are bad. The tools are extraordinary. The hoax is the belief that owning them is the same as deserving them. A logo on an invoice is not an operating model. A migration is not a transformation. And a platform you bought to control costs will, left to your team’s old habits, do the exact opposite, faster and at a higher price than the system you left behind.

You can’t buy discipline. You install it. The only question is whether you install it on purpose, now, or whether you wait for the bill to install it for you.

---

If you have already spent the money and you are quietly watching the bill outrun the value, be clear with yourself about what that gap is and is not. It is not a tooling problem, and your vendor cannot sell you the fix, because the fix is the operating model and that does not come on an invoice. That gap is the work I do: coming in as a fractional architect to install the discipline your platform assumed you already had. If that is the conversation you need to have, book a strategy call.

And if you are the engineer who never signed the contract but got handed the keys anyway, there is a companion piece coming for you: how to install the discipline yourself, on the platform you did not choose, before the bill becomes your name on a postmortem. (And for teams that need to install all four steps at scale, not one engineer at a time, an enterprise track of the coaching program is in the works. More on that soon.)