• Be aware of how your own mindset and mood affects your analysis of a candidate as well as how it impacts how the candidate shows up in the interview. For example, you might be anxious and stressed yourself and that makes the candidate nervous — you may end up experiencing them as nervous, but in fact you are the one that has created that dynamic in the interview.
• Ask the candidate: “If you were hiring someone to fill this role, what criteria would you use?” When someone is particularly good, they are skilled at capturing the essence of what makes someone good at it. This also lets you see how they respond without initial priming and framing.
• The questions a candidate asks can be very high signal: You need to make the candidate feel safe enough to ask their true questions, and you need to answer concisely or you’ll run out of time. Graham writes down each question and sometimes responds with “I’ll answer, but first I’m curious, why did you ask that?” He’s looking for the felt sense of a “hungry mind” based on the way their questions flow.
• The ideal reference check call should take longer than you might think (e.g. 45+ mins). You sometimes need to wear them down over a long period of time before they open up about their real concerns.
• If you aren’t aware of or can’t imagine the downside of working with this person, you haven’t done enough reference checking.
• In each reference you do, it’s useful to ask, “Who else do you know who knows this candidate?”
• One of the art forms inherent in reference checking is controlling for the conditions created by the reference-giver. How much overlap does their map of reality have with mine? If low overlap, they could be right about the candidate but it could not be a fit. This dynamic is also why doing many references helps.
• Ask: “How strong is your endorsement of Jane on a 1-10? (If they answer 7, say actually sorry 7s are not allowed, 6 or 8? If the answer is an 8, ‘What is in that two points?’)” Also, defining the scale by giving an example of what 1 means and what 10 means is key.
• Talent tends to clump — they interact with other people at their level of talent and competence. Your view of the talent level of the list of references is itself signal about the caliber of their peer group.
• Age of the target person is a huge factor. If you’re trying to evaluate someone with fewer years of work experience, being really good at interviewing will be key. If the person is older, reference checking will be more important.
• The more work experience we have, the more our expertise becomes tied to a particular domain and environment. High performance can be extremely context-specific.
• When someone comes from a prestigious company, we often fail to control for the weight that the reputation of the company carries when we form our impression of them.
