Alone, together, in the data

Data is weird, but that's okay because so are we

I started this post around 1 AM in the middle of Coalesce, prior to Benn publishing his amazing summary of Coalesce 2022, “All in one place”. As usual he writes more gooder than most of us. Go read that and then maybe read this.

The Simple Joy of Data Work

I started in data for real in 2013¹. I'd joined a startup in Nashville, TN and learned many of my data skills on the job. SQL, Python, Javascript, regular expressions - that's where the foundation was built. That's where the data love formed.

After a few years I moved to GitLab where I built even more skills as a data engineer by helping a rocket ship head to the moon IPO. I learned an incredible amount about business, politics, and scaling. And I made many mistakes along the way.

Now I work for Meltano - a company that started out as a project called "BizOps" which now has millions in venture funding.

And it all feels weird sometimes.

Data Work is Solo

So much of working in data is working alone. You're staring at your computer, writing some code, creating some charts, generating insights - but often you're alone. You'll have a video chat every once in a while, lots of pings on your favorite chat application, or even a coworker tapping you on the shoulder (if you’re weird and don't work from home).

But the magic of data happens when you're alone. It's you and your computer. Or your pencil and paper if you're old school.

Which makes events and communities like Coalesce both amazing and... weird.

#vendor-content

I attended Coalesce 2020 as a practitioner. I was on the GitLab Data Team and was in a very unique position to be incredibly open about the work I was doing. This was all due to the transparent culture of GitLab. It did wonders for me personally and professionally. I was able to connect with so many people in the dbt Slack and I found a tribe of people who were interested in and working on many of the same problems I was²!

With Coalesce 2020, everyone was on slack and online. It was very unique and is still the very best virtual conference I went to. The energy and chaos was fun and real. Everybody felt more or less equal because we all expressed ourselves through the same somewhat limited channel.

Fast forward to Coalesce 2022 and it's all so much bigger and different. Literally millions billions of dollars in venture capital are represented across all of these companies. All focused on solving data problems. I'm part of that story with Meltano.

And the truth is it changes the game a little bit³. It’s changed because things always change.

It's sometimes feels very hard to connect the day to day reality of data work with these billion dollar ambitions. Especially when you're just working on transforming a single column with SQL, tweaking a single dashboard for your boss, or fixing the indentation of your YAML yet again.

It feels weird to have such a spectacle over something that can be so solitary.

Add to this the complexity of open source. Working on and with open source software rips apart my brain sometimes. I'm torn between wanting this Star Trek post-scarcity reality where there's no need to have proprietary work, and the reality of working in a capitalist system. Some of the joy of dbt is in the fact that so much value is just given away for free⁴.

So many of us found this amazing tool and we're able to make our work lives easier and have a bit of fun doing it. Then we learned that others had the exact same amazing discovery. You learned you were in a community that you didn’t know existed.

Now everybody wants to capture a bit of that magic but likely nobody else can or will. Whether it’s "the funding environment" or "the broader economy" or “our investors breathing down our necks”, companies will feel the pressure to work like proper (older) businesses and actually charge for some of the value they create. And that's not a bad thing - it can just feel discordant with the experience of how so many of discovered and use dbt.

IRL Convos

I had multiple conversations at Coalesce where I talked about my general love for all things data. I find joy in bringing order to chaos. In the fundamentals of building data pipelines that move and transform data. In the challenge of finding bugs and weird corner cases; of seeing how the real world is represented in bits and rendered on my screen⁵.

And when I read Benn’s post I knew I already had answers to his questions:

A far more interesting question we could ask ourselves is, “What would we do if we knew we would fail?” If we knew our company would eventually burn out or fade away, would we still start it? If we knew we’d never be famous, would we still post? If we knew we’d never make partner, would we still join the firm as an associate?1 By separating the work from the reward, we can be more honest with ourselves about why we do the things we do. Are we here because we like the job or believe in its intrinsic value, or because we want the payoff?

Hell yeah I would.

Because I already did.

I did the work of a data professional alone and in the dark. I wrote when nobody read, and posted shit when nobody liked it. I created content that was just for my team but happened to be indexed by Google. I used a rickety-ass tool and joined a slack group with about 300 people in it. I worked on a weird little internal project because I loved the idea.

I feel this joy so deeply that I know that if all of this were to go away I’d still do all the same things. If all of the vendors go bust and we're left with the "basics" of data - I'd still love it and I'd still work "doing data". I'd still be writing my silly little SQL queries and adding nice little features to projects I love. I’d still be alone, working with data, and I’d be happy⁶.

Of course, I hope it doesn’t all go away. If I make some money with Meltano that’d be really cool, but it’s not most of why I do it. I don’t post on Twitter to be a “data influencer”, I just like to post⁷ and connect with awesome people. I work for a data tool vendor because I like data tools. I work in data because I like data.

This joy creates a confidence that means I can have a healthy skepticism of much of what happens in our industry⁸. I can roll my eyes at overly bombastic keynotes. I can poke fun at inflated data jargon. I can acknowledge when valuations are whack. And I can try to not take myself or others too seriously with all of this.

But I can also recognize when moments, and the people that make them up, are truly special. Coalesce 2022 was, for me, one of those special times. It’s a part of the good old days I know I’ll look back on.

And yet… it’s all still a little weird. And that’s ok. I don’t want the weirdness to go away. It’s a hell of a ride and there are some amazing people on it.

1. Is going to grad school and getting a PhD involve real data work? I guess it does, but I’d rather not use Matlab again so let’s just start in 2013.

2. The rush you get when someone asks a question and you're able to answer with a link to the repository where you solved that same problem is real. I loved it. Still do.

3. Ok a lot of bit.

4. Maybe that's just the brilliance of dbt-core generally. So much value had already left the barn prior to the majority of the VCs storming it I do wonder how they'll be able to recoup it all. But maybe it's just a long term play to take Venture Capital and build great things to just give away. dbt, thy name is Robin Hood.

5. Yeah, I even love YAML.

6. I’m reminded of the Chinese proverb about the farmer - “We’ll see”. Very stoic.

7. Correction, shitpost.

8. I hope. Or I’ve just deluded myself. But let’s go with I have a healthy skepticism, k?