Eric Mayefsky, head of data science at Quora, has assessed hundreds of job candidates in his half decade in management at various tech companies. But like any manager, he started on the other side of the interview: as an applicant.
What he’s learned from his experiences on both sides of the table can help other data science leaders navigate the chasm of assumptions between interviewee and interviewer, and make more effective hires. His beliefs on data science interviews are defined by five lessons:
- Structure the interview such that expectations are as clear as possible.
- Deliver a clear, honest explanation of the role
- Conduct a challenging interview to hook a great candidate
- Avoid candidates whose interests don’t align well with the role
- Seek out candidates with an appetite for the right type of work
The Early Days
In 2010, Eric was four years into an economics PhD at Stanford University, studying the performance of algorithms in matching markets. The work had obvious real world applications, but Eric was frustrated. The idea of joining the private sector involved a big shift in direction, but after years in academia, Eric felt stifled and dissatisfied by the speed of academia, and the impact of his work. So he applied for an intern role at Facebook, on a team responsible for analyzing performance of its advertising platform. Soon after applying, he was headed to Facebook’s campus for his first private-sector data science job interview.
Eric’s interviewer kicked things off with what sounded like a typical brain teaser:
"How many consecutive zeros are there at the end of the expansion of 1,000 factorial?"
Eric thought he did just okay.
“You know, with some hinting, I got to the right thing. I thought I showed that I wasn't a complete idiot, but I didn't nail it,” he says.
Next, the interviewer wrote some metrics for different hypothetical advertising campaigns on the whiteboard, and asked which of the campaigns performed best. The interviewer asked Eric to do the math, without a calculator.
“The numbers were super awkward, and I didn't expect him to ask me to do the math by hand,” says Eric, “If he wanted me to do the math, I'd have expected him to make the numbers easier, just to move it along.”
The question made Eric think of the GRE, which adjusts questions based on the test taker’s performance. If you answer initial questions accurately, the questions get harder. If you give wrong answers, the questions get easier. Eric worried the interviewer was following similar logic in giving him a basic math question.
“I was thinking I must have completely bombed the first question,” Eric says. He thought the interviewer was just trying to kill time during an already failed interview. But a few days later, he was offered the job.
The assumption that he bombed the question was based on his experience in academia, and it was the wrong assumption. After Facebook hired Eric, he learned that he was asked to do basic math because even experienced analysts don’t always understand the calculations underlying their work. When someone in a quantitative role needs automated assistance every time they see numbers, it slows things down.
Eric remembered that question as he described how he thinks about interviewing data science candidates as a hiring manager. He drills into a candidate’s experiences to identify their assumptions, and avoids time-sucking confusion by giving candidates materials to help them prepare for the interview. At the same time, he keeps candidates on their toes to make sure they have the right skills. Five ideas emerge that explain his thinking.
1: Get on the same page
You don’t want to accidentally overlook a candidate with whatever magic combination of skills your company needs. If you don’t make an effort to understand the person you’re interviewing, you risk misunderstandings that can lead to undesirable outcomes. You might fail to recognize, or lose the interest of, a great candidate. You might accidentally hire someone who seemed great but ends up hating their job. There’s lots of variance in the candidate pool, and lots of variance in the jobs available. To get what you need out of the interview, you need to get on common ground with the candidate.
Eric went into his first interview at Facebook with expectations shaped by years of academia. When he encountered something unexpected in that interview -- an easy question after a more difficult question -- he reacted according to his experience: the GRE. In the hundreds of interviews he’s conducted from the other side of the table, he sees candidates exhibit similar behavior. We contextualize things according to our experiences. Interviewers must understand how a candidate’s background will influence their behavior in order to get an accurate picture of their fit for a role.
There are three types of candidates Eric tends to encounter during the hiring process for data science roles: grad students, wide-eyed undergraduates, and people with some experience in the private sector. Candidates with experience at just one company may have the basic mindset that matches what a company like Quora is seeking, but that perspective can be overwhelmingly shaped by having worked at a single company. Grad student applicants may be stuck in the mindset of writing papers, and still transitioning into thinking about data in terms of practical applications.
“Undergrads are coming from a world where they've often been told what to do,” he adds by way of example. “Candidates who can show that they're very comfortable working on independent projects, that's a big plus.”
The questions you ask in an interview should help illuminate past experiences that have shaped a candidate’s thinking. You can ask candidates what they don’t like about their current situation to gain a sense of their motivation in looking for a role at your company. You can help them meet you halfway by giving them adequate preparation for the interview. Quora sends candidates a “what to expect” memo that outlines the types of problems candidates will encounter and materials they’ll have on hand for solving those problems. The memo offers tips for how to best prepare.
2: Knowing what you don't want doesn't translate to knowing what you do want
Hiring the best candidates is hard work. Many of the best data scientists come out of graduate degree programs, often having entered those programs with the aim of building a career in academia. They have little or no private sector experience and don’t have a detailed understanding of what they’re getting into by looking for a role at a company. Other candidates have only held one job in the private sector, and assume every other job is like the job they are now looking to leave. In either case, they might have a sense of what they don’t want in their next experience, but that doesn’t mean they actually know what they want. In interviews, they might frame what they want as being the opposite of whatever they dislike where they are, which might create a skewed picture.
Job hunters grappling with unknown unknowns can be the most unpredictable. They might approach job applications the same way they did college admissions: apply to a large number of companies, see what response they get, compare, and pick. That’s a rational approach, says Eric, but it can also lead to hasty decisions when applicants don’t have enough information. Hiring managers should be open about the true responsibilities of the job, so that interviewees know what they will be getting into. Eric says he makes a point of explaining Quora’s data science roles as explicitly as he can.
“We are pretty transparent about who we are as a company, and what being a data scientist at Quora entails,” he says. He explains that his team is Quora’s only data team, which means each member has to be ready to work simultaneously on a number of ongoing priorities. That means carving out time for long-term research projects while building tools to help the team work more efficiently, and helping to drive product decisions. Not every company is this upfront about the scope of the role, according to Eric. Some might lead the candidate to think it’s centered entirely on research, when in reality a lot of multitasking is involved.
“After having that very real conversation, if a candidate accepts, I feel that’s good,” he says. But “if I have that kind of conversation and the candidate says no, I’m worried that sometimes it’s because other companies were telling them a rosier story. But it’s a tradeoff I’m happy to make, versus not painting an accurate picture for the candidate.”
3: Keep the interview challenging
The best candidates are looking for jobs that will challenge them, so give them a challenging interview. Eric’s team does this by asking questions that require them to do the sort of work they would actually do day-to-day as a data scientist at Quora. That includes asking candidates how they would go about designing a product, or posing questions that gauge their understanding of users, as well as more quantitative and technical questions. But he also makes a point to go down the rabbit hole with whatever response applicants give, probing and challenging their responses along the way.
He starts, typically, by telling the candidate there’s no particular answer he’s looking for to a given question. A couple examples of retired questions of this type include:
- Say we have a “Suggested Topics” product that will appear in a user's feed. What would be some good metrics of success for this product? How should we determine how much space this component should take up in feed?
- What motivates users to click on email digests?
When the candidate starts responding, he tries to stick with them to the end of their natural train of thought, and ask probing questions about areas where they’ve shown some comfort.
“For example, if I asked them about how they'd evaluate whether a product change was good or bad, there might be a number of metrics and qualitative ways they could propose to do that. Rather than force them into my favorite answer, I would try and say, ‘OK, suppose you look at it the way you suggested and you see [X]. What would you do then, or what would you conclude?’” he says. “If their answer didn't make a ton of sense to me, I would try and ask questions around the pros and cons of their suggestion to understand their thought process and potentially see if they could come to any of the shortcomings of their method on their own.”
He’s careful not to force the candidate down a path totally different from the one they’ve chosen in answering the question. So he doesn’t stop the candidate, or wait for a pause just to ask, "Suppose that doesn't work, what else would you try?" However, after he’s engaged them deeply in one area, he might try to get a sense for the breadth of their thinking by asking if they have other ideas for how to solve the problem.
4: If the dataset is their only interest, be concerned
Data science in the private sector commonly includes skills used in academic research. But if you’re hiring for a hands-on, product-focused data role, you don’t want to hire someone who takes a strictly academic approach to their work. Your dataset may be the only thing about your company that interests some candidates. So even if they have the skills for the role, they won’t be committed to your company’s growth.
Eric does want people to be interested in the data. Quora has more than 200 million monthly active users, who interact through questions ranging from, “Why do bees die after they sting?” to “What are the best tips for raising venture capital funding?”
That data is interesting! But when the first thing that a candidate from a research background says in an interview is that they’re interested in Quora’s dataset, “that can be a red flag. It sounds like you’re still thinking about writing papers.” Quora won’t benefit from hiring a data scientist who is stuck in a pure research mindset to the detriment of discovering insights that can be applied to the company’s product and growth.
This is one of the biggest divisions among candidates. There are those who care about whether their work determines the success of the company where they work, and those who don’t. At a corporation with tens of thousands or hundreds of thousands of employees, employees may not be able to see their impact, and so in some cases it’s okay for them to be hyper-focused without feeling especially responsible for the company’s overall success. But Quora only has about 200 employees. It’s still at a stage where every person’s contribution matters in a direct way.
So when he’s interviewing candidates, Eric is looking for enthusiasm about product work and impactful data analysis. Even when he’s interviewing undergrads, he’s looking for them to be hungry to learn about product development as well as deploying and growing their technical skills. The best candidates are those who demonstrate interest in finding creative ways to increase user engagement, who want to tinker with a/b testing, or want to improve ad targeting in order to better target potential users or customers with the right content and products.
5: Disgruntled is good
Whereas some PhDs can be red-flag candidates due to a single-minded interest in the dataset, there’s another subset of PhDs that Eric says tend to excel in private sector data science jobs. He calls this type the “disgruntled grad student.”
These are graduate students who are sick of how slowly academic research moves, and want something faster and higher energy, where they can see the impact of their work sooner. Eric sees the same drive in undergraduate students who take on projects and jobs outside their studies. He sees it in private sector professionals who want more challenging jobs. What these three types of candidates have in common is that they want their work to make a difference in the success of the company.
The trick to identifying candidates who want to grow their skill sets, and have a zeal for product work, is to give them a chance to show you their best selves. Challenge them, put them under pressure, but also give them a sense of direction. Ultimately you want to make sure you are speaking the same language. Just like the candidates who know what they don’t like, but aren’t sure of what they want, you may not know what the ideal candidate looks like if you don’t give yourself a chance to see all your candidates clearly.
