dbt Labs acquires Transform - who's next?

Hot takes on data categories

dbt Labs’ acquisition of Transform was not entirely surprising given the state of everything. My sense is that more companies in the data space will either be acquired or shut down in the next few years¹. I asked on Twitter if there was a prediction market for the data ecosystem because my default assumption is that the crowd is smarter than me².

Data Professional (parody) @tayloramurphy

is there a predictions market for which other VC-funded data companies will be acquired? wisdom of the crowds would be useful here to check my own biases 👀

Given that there isn’t and I’m not inclined to make one, here’s my philosophy on the data market broadly³.

DALLE-2 generated photo with the prompt “a photo of a purple toy monster eating a smaller purple toy monster”

2nd or 3rd order

The analytics function, and by extension a data team supporting it, is a 2nd order need of running an organization. A 1st order need is one that is critical to the functioning of the company. I would say most (>90%) companies can function without a data team and the analytics they would produce. They wouldn’t be as successful in this state, but it’s not strictly necessary.

This makes it even more critical for data teams to show their value in an organization. The insights these teams produce are accelerators of an existing business, but if they’re not delivering value then they’ll be cut.

But let’s say we’re in an organization where the need for a data team is recognized and they have executive support. What are the critical elements for the functioning of the data team itself? And what is a 2nd order need of the team that is an accelerator but isn’t critical for it to function?

ELT and BI

There are 3 essential components to any analytics question:

• is the data in the right place (EL + warehouse)

• is the data in the right form (warehouse + T)

• is the data right for the question

And that’s it. To me, this is also reflected in the market. When you think of the big players in the modern data stack, who comes to mind? Likely a combination of a warehouse, a BI tool, an EL tool, and dbt. Who’s absent from this list? A lot of other companies.

What’s hard for me to admit is that many things that we data professionals would deeply care about are actually 2nd order needs of answering data questions. When push comes to shove and you’re asked to cut costs what kinds of work can you do yourself in place of a separate tool?

Basic quality testing via dbt will get you most of what you need. Everything else can be covered with a quick slack message and a hotfix to prod.

Efficient metrics can be covered with some centralized documentation and availability via Slack to answer ad hoc questions.

Questions about which table is best to query is also covered by some documentation and a human in the loop.

Feature stores can be hacked together in your existing warehouse.

Governance can be handled by native features of the data store itself.

Even DataOps is a level of workflow optimization that you can live without for quite a while.

Orchestration is the only category that’s at a 1.5 order level for me. It’s likely the first additional tool you’ll reach for beyond the core set, but even now we’re seeing existing ELT tools do a lot for you.

So what?

To be clear, I’m not saying you don’t need to invest in any of these categories. What I am saying is that ELT+BI are like the food, water, and shelter of data. You’ll be alive with just those, but it’s not a very a complete, fulfilled, or effective life.

When an organization, or individual, needs to scale back expenses, the first ones cut are the non-essential. You’ll cancel Netflix before you stop paying for water. And a data team would stop paying for a data quality tool before their data movement tool.

When data is already a 2nd order need for a company, the 2nd order needs of the data team (i.e. the 3rd order needs of the business) will be the first ones cut.

So what does this mean for the companies that are themselves 3rd order or more? They’ll try to move closer to ELT+BI. Transform attempted to do this by adding more BI features⁴.

And what does this mean for companies that are already in the ELT+BI space? They’ll become larger platforms that make the 3rd order needs of data teams features of their platforms. It’s just easier for users to use a poorer implementation of a feature in one of their current tools than it is to pay for a separate higher quality version of said feature.

Finally, what does that mean for you the data practitioner? It should mean you apply the same rigor and scrutiny to your tools that you do to analytics questions. Be honest about the true needs of your company and where you will get the best value for the money⁵.

It’s still a very fun and exciting time to be working in data. A lot will happen in the next few years and I’m sure it’ll be quite the ride.

1. Not the boldest claim given that companies are acquired or shut down quite regularly!

2. Crowds are smarter than us all.

3. Perhaps it really all just boils down to ZIRP.

4. Maybe if the market was better or if dbt Labs didn’t have such a war chest they would have succeeded. A similar acquisition happened in 2021 when LiveRamp bought Rakam, the makers of the Metriql headless BI/metrics layer.

5. Efficiency is so hot right now

.