Ad hoc analysis and reporting
Analysts are often asked to answer one-off questions to help someone at their company make a decision. The work that analysts do to answer that question or to make that decision is often referred to as ad hoc analysis.
Because people use ad hoc analysis to answer a huge variety of questions, ad hoc analyses often look very different. A couple quick pivot tables in Excel, a complex report in Mode, and an 80-slide deck could all be considered ad hoc analysis. Moreover, some ad hoc analyses, like developing a recommendation for how to reprice a suite of products, could take months. Other analysis, like figuring out why web traffic dropped to zero, is done in minutes.
The defining feature of ad hoc analysis is that it’s iterative—the answer to one question generates more questions, and those answers generate more questions, and so on. People often have an idea of how they might answer an ad hoc question, but that plan almost always evolves as you learn. It’s more like a journalist doing an investigative report, letting the story’s leads bring them closer to the truth, than it is an architect creating a blueprint and following it exactly until the house is done.
For example, imagine an analyst that works at a software company. The company may already have a BI tools. These tools typically host dashboards that show their company’s KPIs, like the number of daily signups and the company’s current ARR.
Now, suppose signups start to drop. The BI dashboard shows this drop, but it doesn’t explain why signups are dropping. Did something in the signup flow break? Are fewer people finding their website? Are their users inviting fewer people? Without understanding the reason for the drop, the company can’t respond to it.
The analytical work done to figure out the reason for the drop is ad hoc analysis. It’s work that’s motivated by a particular question, opportunity, or challenge, rather than an ongoing need to monitor something for a longer period of time. Because of this, most ad hoc analysis has a shelf life: It does its job, after which people don’t regularly return to it, either to view it or to update it.
There are lots of other potential examples of ad hoc analysis:
An airline company wants to understand the potential impact of a new companion fare promotion they’re considering.
A marketing manager wants to know why a recent blog post performed so much better than others.
An exec team wants to know why app downloads are stagnating on iOS but not on Android.
A product manager wants to know how people are interacting with a new feature.
An analyst wants to figure out why their data pipelines get backed up every Thursday.
A support team wants to decide if they should expand their support hours.
A CSM wants to figure out which use cases they should focus on in an upcoming check-in call with a customer.
The term “ad hoc” undersells ad hoc analysis. The phrase sounds offhand and temporary, like a passing comment of little import, especially when compared to phrases “core metrics” and “key business indicators.”
This couldn’t be less reflective of the value of ad hoc analysis. Ad hoc decisions are, almost by definition, the most important decisions companies make—they’re the ones you only get to make once. Jeff Bezos’ famous one-way doors are the stuff of ad hoc analysis, not a recurring BI report or self-serve dashboard. When your boyfriend proposes to you, at center court of a sold-out Rockets game, your decision is ad hoc—and momentous. Similarly, when a company is deciding if they should shutter a product line, the decision is also ad hoc, despite millions of dollars and hundreds of jobs being at stake.
Even in cases when the decision isn’t new, if the problem is particularly important, we take extra time to consider it—and therefore make it an ad hoc decision.
Consider a CSM team doing quarterly reviews with their customers. For most customers, they may automate the analysis they do to figure out how those customers are using the product. They might have a set of charts they include in their decks, and not adjust them for any one customer. When a CSM prepares to talk to their largest customer, however, they may do some ad hoc work to customize the deck. Even though work is typically predefined, predefined isn’t good enough in the most important situations.
Moreover, we can’t avoid ad hoc analysis in several other ways:
Companies are constantly evolving. Every change—the introduction of a new marketing channel, a new product line, a new business initiative—requires ad hoc analysis to understand. At the start of the pandemic, for example, businesses had to figure out how to operate in an entirely new world. In these moments, predefined reporting goes out the window. Every question was new and most were urgent.
Ad hoc analysis is the foundation of standardized reporting. KPIs and core metrics aren’t preordained; they get developed as a company figures out what’s important to track and what doesn’t matter that much. That development process requires ad hoc analysis.
Standardized reports generate more demand for ad hoc analysis. Reports and dashboards will often surface things that look surprising—in fact, that’s often their very point. Just as a car’s check engine light isn’t there to tell you everything about the car but to tell you when you should pop the hood and investigate, standardized reporting can never show every possible view of the data. When we see anomalies in our reporting, we turn to ad hoc analysis to understand what’s happening.
Ad hoc reporting & analysis is a process where analysts investigate one-off questions that don’t already exist in a dashboard.
In all of these cases, however, creating and learning from ad hoc analysis can feel messy and disorganized. When we look at companies with mature data practices, we only see the final, stable metrics and dashboards. It can feel uncomfortable—and even wrong—to create an exploratory mess that doesn’t fit inside a suite of dashboards or business reports. But, when approaching any analytical problem—from something small like answering a single question to preparing a strategic report for a board meeting—the first steps will always be uneven. Rather than trying to stifle or control this phase, we’re better off having a plan for how to tidy it up later, by working in a sandbox or reserving space for the polished final drafts that we’ll eventually produce. Ad hoc analysis is immensely valuable for businesses—and the best analyses are those that follow the data wherever it takes them.
Related terms:ETL tools , Transformation tools