Why Data Leaders Are Quietly Outpacing the AI Hype
- Gandhinath Swaminathan
- 3 days ago
- 6 min read
I’ve observed this pattern at a client now: Teams deploy an agent for month-end forecast. For the first two weeks, it’s magic. Then someone changes a region code in the ERP and forgets to tell the data team. The agent answers for three days from stale data before anyone notices.
Right now, your team is throwing on every down. An agent here, a text-to-SQL deployment there, another AI pilot next quarter. It feels new. It looks smart. The board notices.
Then something happens.
An agent confidently answers from stale data. A schema changed Friday night, and nobody caught it until Monday morning when the customer saw bad numbers. The system acted on what it believed was true, but wasn't.
That’s when “novelty” stops looking like progress. And that’s when the teams that maintained their fundamentals—harmonization, lineage, semantics, versioning—stop looking old-fashioned. They look bulletproof.
In an air‑raid, the quarterback gets the attention. But the running back keeps the offense on schedule.
Here’s the mapping:
The quarterback is your AI agent—the system that reads the intent, calls audibles, decides what question to ask the warehouse next.
The running back is data engineering. It takes the snap and converts raw events into motion, every play. Clean. Consistent. Repeatable.
The offensive line is your security and governance—it decides who moves through which data gap and who gets stopped before they get there.
The scorekeeper is your semantic layer. It makes sure that “customer” means the same thing in billing, CRM, and product. It's why two teams don't argue about whether the score is 24-21 or 21-24. The score is the score.
The playbook is discipline. It’s the architecture that makes creative play-calling possible without collapsing into chaos on the first defensive blitz.
That last line is where most teams get it wrong.
They stack creativity on chaos and call it innovation. They want the fast plays without the blocking scheme. Then the first defense shows up, and the whole offense collapses.
Creativity doesn’t come before discipline. It comes after—built on top of it. That’s not boring. That’s bulletproof.
Why the air‑raid exposes everything
You already know this. A dashboard with bad data is an argument. Somebody says the numbers are wrong. Somebody else checks. You fix it. Annoying, but containable.
Now put an agent on that same broken stack. The agent doesn’t argue. It acts. It sends an email. It updates a forecast. It changes a customer’s tier. Suddenly “later” isn’t an option.
A silent schema change on Friday? It was a Monday annoyance in the old stack. In the new stack, it’s a Monday morning fire where the agent confidently executed a decision based on false data—and you didn’t catch it until the customer complained.
This is the hard truth: speed exposes everything that discipline keeps hidden.
The old stack moved data. The new stack has to trust the data. That changes everything about how you build it.
The three places your foundation cracks
Most leaders think lineage is the first problem to solve. They’re wrong.
Lineage matters—but it comes second. Harmonization is the first punch that lands.
Harmonization
Making “customer” mean the same thing in billing, CRM, product, and support. It’s unglamorous. It doesn’t ship fast. But it’s where the foundation lives.
An agent can write SQL in seconds. It can’t unify identity across three acquisitions and two legacy ERPs. So it answers your question using the records you didn’t even know existed—billing customer 447 is not CRM contact 812, but your agent won’t know that.
This isn’t a model problem. It’s a data problem. And no amount of prompt engineering fixes it.
Lineage
Lineage. The trail from source to number. When it’s clean, you see what changed upstream, which assets depend on it, which ones are now stale.
Without it? A VP asks why revenue dropped. You dig through job logs. You search table schemas. You call the analyst who built it six months ago and hope he remembers. You spend two hours hunting for a two-minute answer.
Semantics
Semantics. What does “Revenue” actually mean? Without it governed in one place, Revenue becomes whatever the last SQL query computed. Today it means sales gross of returns. Tomorrow it includes annual contracts. Next week it's “whatever Marketing asked for.”
A semantic layer (Cube, dbt, AtScale) defines the metric once. Revenue means Revenue. When an agent queries it, it gets the governed calculation, not a guess. No more arguments about whose numbers are right because there's only one number.
The fix: a semantic layer that exposes metrics as tools the agent can call directly—no guessing, no SQL generation, no ambiguity.
The playbook (discipline first, creativity second)
A modern offense looks chaotic and is disciplined.
Your business wants to ask questions fast: “Why did churn spike?” “How does this segment convert?” “What happens if we raise prices 5%?” You want answers in Slack in seconds, not reports in three days.
The playbook—the architecture, the discipline, the baseline rules—is what makes that possible without turning every answer into a liability.
What the “handoff” is in 2025
Most teams orchestrate like this: Run Task A. Then Task B. Orchestrated by some scheduler. They use Apache Airflow or similar tools, and they work. Right up until they don’t.
The system knows that Task A ran. It doesn’t know what Task A produced. It doesn’t know whether downstream assets are stale. It doesn’t know if the output passed basic sanity checks. It just ran. Task B will run anyway, feeding bad data into what comes next.
This works fine for dashboards. Dashboards are observation. Bad data is a mistake you fix Monday morning.
For agents? The system doesn’t observe. It acts. An agent gets a bad number and moves on it. That’s not a dashboard error. That’s operational risk.
Modern tools like Dagster flip the framing. Instead of tasks, they treat outputs as declared objects with dependencies. The system knows what was produced, not just that a job ran. When something goes stale, it notices and heals automatically instead of waiting for a human to notice.
That’s the handoff. Clean exchange. Eyes up. Ball secured. That’s the running back getting a clean handoff and knowing where the hole is supposed to be.
Checks and versioning
Asset checks are executable tests on data right after materialization. If the data fails, the pipeline stops. Poisoned data doesn’t keep moving downstream into models, into agent answers, into customer decisions.
Versioning is how you replay the exact state that fed a decision when something breaks and nobody can explain why. You don’t guess. You rewind. You see it.
This is how you make creative play-calling survivable. The moment something changes upstream, the whole system knows what broke and why.
Brownfield modernization because (almost) no one starts from zero
Most organizations are Brownfield, meaning they are operating their businesses with legacy ERPs (SAP, Oracle) and a decade of SaaS sprawl. You can't migrate those systems overnight, and “wait until the data is perfect” means waiting forever.
The answer is the Strangler Fig pattern. Don’t rip and replace. Instead, build a modern data plane that gradually pulls data out of the legacy system while the business keeps running.
You tap the ERP’s transaction logs (every insert, update, delete). You stream that data into a modern lakehouse (Snowflake, Databricks). Your agents query the lakehouse—current, clean, fast—instead of hitting the slow, fragile legacy system directly.
You’re not replacing the ERP. You're building a shadow version of it that's modern enough for AI.
This is how you get clean, current data out of old systems without stopping the business. It's not glamorous, but it’s what makes the “modern stack” possible in companies who don't yet have the opportunity to invest in modernization.
The budget question your leaders are really asking
People leaders don’t wake up wanting to fund “data harmonization.” They wake up wanting fewer surprises.
They want month-end close to stop slipping by three days. They want the churn story to stop changing depending on who ran the report. They want the Slack agent to stop causing debates that burn half a day.
So here’s the air‑raid reality. A year from now, nobody will be impressed that you can ask the warehouse a question in plain English. That becomes normal. What will still separate teams is whether those answers are grounded in assets that are fresh, traced, and checked.
If you fund the quarterback and starve the running back, the offense looks exciting until the first real defense shows up. If you fund the running back and never let anyone call plays faster, you end up with clean pipelines that still require a ticket for every question.
The winning posture is layered. Assets with lineage and checks. Harmonization rules. Semantic definitions exposed to agents. Conversational interfaces.
That’s how you get speed without every answer becoming an argument.
If you’re worried that your data foundation can’t actually support the AI vision you’ve been sold, you aren’t alone. Let’s assess your readiness and define the practical steps to fix it.
Comments