Why You Need a Long-Tail ETL Connector (do something kind for yourself)

Everyone has finally dialed into the Zoom call. The CEO, the board, the business analysts, and you. The Head of Data.

The goal of the meeting is simple: Create an automated KPI dashboard to track company progress.

You are prepared.

You have designed the most beautiful, comprehensive, Modern Data Stack for analytics.

Data processing. The first thing you did when you joined the company was to put in place a cloud data warehouse. You’re an expert. The separation of storage and compute provides infinite scale, with the speed of processing your team requires.

Data ingestion. You have a real-time pipeline syncing your product data into your analytics environment. You’ve connected Salesforce, Workday, and Zendesk to your data warehouse. Events are being created and relayed from the company website and mobile app into immutable tables.

Transformation. You have a version-controlled repository of SQL queries. Some views. Some materializations (to keep cloud costs low). You have translated the raw data into business semantics that are the foundation of self-service reporting for the company. Everything runs through a staging environment before affecting any production reporting.

Governance. Not only do you have an enterprise data model in place, but you have outlined a whole master data management strategy. You have defined the key entities that matter, and how they relate to each other. Everything is exposed in a data catalog for simple discovery. It doesn’t matter what information people are looking at, everyone is always on the same page about what a metric means.

Data quality. As data pipelines seamlessly crunch data, you’ve added anomaly detection to make sure outputs can be trusted for downstream use cases. Because your team has realized the value of embedded analytics to power client-facing products, data quality is paramount.

Data activation. With all the valuable data sets you’ve created, you make sure to sync the data back into critical business applications so they stay up-to-date with the latest insights. You are Reverse ETLing data back into your CRM system, your advertising platforms, and your email service provider. For your team, this is a great substitute for a customer data platform (CDP) or an integration platform as a service (iPaaS) solution. And it’s all centered around your data warehouse.

Business intelligence. Finally, you have your pipelines set up for data to flow seamlessly into Mode for ad hoc analysis and self-serve reporting. A single platform for business users to find questions to any question that comes to mind. Long story short.

You are ready.

So ready.

The company wants to create an automated KPI dashboard to track company progress?

Nothing will stop you!

Everyone on the Zoom call goes quiet as the CEO pulls up the company’s top 3 KPIs.

1. KPI #1: Increase conversion rates in our hiring pipeline by 20%

2. KPI #2: Increase subscription revenue from the highest value clients by 30%

3. KPI #3: Ensure multi-factor authentication is in place for 80% of business applications (in preparation for the SOC 2 Type 2 audit next quarter) 

You go on mute - hoping no one notices the fear in your eyes.

You ping the head of HR. ‘Hey, what tool do we use to manage our hiring pipeline?’

You urgently flip back through your emails. Where’s the email from the CRO where she mentioned the new subscription billing platform. Was it Baremetrics? ChartMogul? SaaSOptics? Maybe Chargify?

You glance at your bookmarks bar. Open the folder called ‘Frequented Apps’. You have 16 business applications bookmarked. How many applications could there possibly be across the company? Does the team really need to know if multi-factor authentication is enabled for all of these? Really?

The CEO waves her hand on the screen.

‘Hey!’

‘Hey! Can we get a dashboard that shows progress against these KPIs? How long will this take?’

The CEO is asking you a simple question.

‘Can we get a dashboard that shows progress against these KPIs?’

You were prepared for this.

Or at least you thought…

You respond.

‘Um. I’m not sure.’

‘We’ve never accessed this data before. It’s not in our warehouse. And I’m honestly not sure if we can get it. These are pretty niche tools. I’ll need to see if we can find a way to ingest the data. Let me get back to you.’

The CEO rolls her eyes and looks at her Chief of Staff.

‘Can we just create a spreadsheet to track these for the quarter?’

This is a common, and extremely uncomfortable situation for any data team.

Why does this happen?

There are a few reasons:

• Increased Composability - Companies are migrating their tech stacks from large, centralized, monoliths to composable building blocks, so more and more tools are being introduced.

• ETL Has Focused On Big Systems - Most no code ETL tools / ELT tools focus their efforts on the most common data sources. Salesforce, PostgreSQL, Workday, etc. So you can’t find connectors to bespoke applications.

• No Pure-Play Long-Tail ETL Solution - For over a decade, no ETL solution has focused exclusively on the long-tail of business applications.

Luckily, things are changing, and data teams have a few solutions available.

• Data Integration Consultants - There are hundreds of data consultancies that can help in these types of scenarios.

• Open-Source ETL - There are open-source ETL frameworks like Singer which provide a structure for your engineers to build and deploy new, custom connectors faster than starting from scratch.

• Custom ETL Solutions - Portable is a great example of a product-focused exclusively on the long-tail of ETL / ELT connectors. After seeing enough data teams in the situation above, we had to help. With 300+ connectors and counting, we’re just getting started.

In conclusion, the tech stack above is great.

You can have the best data stack in the world, but if you can’t get the data, you have a problem. 

A big problem.

As more and more business applications appear, it’s important to consider how you will get bespoke data sources connected to your analytics environment.