Last week, I urged data science teams to focus on delivering value quickly and iteratively (thanks to Ian Blumenfeld for highlighting the importance of keeping an eye on the horizon throughout this process, and not only thinking about your next step). In addition to Ian's comments, I received another question: How do you ship value early when you have to build infrastructure first?
After all, analysis requires data. You can't have data unless you have a way to collect it, and a place to store it. And you can't get anything out of it without a means for querying and analyzing it. Regardless of how agile your team might be, you're limited by how quickly you can provision a warehouse, build ETL pipelines, set up event tracking, and integrate an analytics tool. Doesn't this all take significant time and expertise to build?
The short answer is, not anymore.
All of these systems can be set up in a matter of hours, not weeks and months. As Tristan Handy of Fishtown Analytics explains in this excellent post, modern data tools make it possible for the first iterations of a data infrastructure to be built without the help of specialist data engineers.
Today, analysts themselves can get an entire data infrastructure up and running using off-the-shelf tools. Cloud-first warehouses like Redshift, BigQuery, and Snowflake can be spun up in minutes via web interfaces. Data can be instantly fed into these warehouses without writing a single line of code, thanks to tracking systems like Segment or Snowplow, and ETL tools like Stitch, Fivetran, or ETLeap. And an analytics tool like Mode can be integrated in minutes, immediately giving companies the ability to query, visualize, and share their data.
This new stack represents the most significant development in data engineering of the last five years. They may not seem as exciting as technologies that solve scale and latency on the bleeding-edge of data science, but these “meat-and-potatoes” technologies solve a much broader problem: They make analytics accessible to everyone.
I used to joke with a former coworker, who was notorious for both loving to take bets and being terrible at winning them, that he was always thirty minutes from broke. Today's businesses, even those short on time and people, are now in a similar, albeit far more positive, situation: Data engineer or not, regardless of what analytics infrastructure you have or haven't built, you're only ever thirty minutes from answers.
So don't let the idea of starting keep you from actually starting. The first technical hurdles in analytics have never been lower. And the sooner you get over them, the sooner you can get to work building an analytical decision-making culture.