Three Tales from an Amazon Bar Raiser
A few examples of situations I went through as a bar raiser, one of the most entertaining positions at Amazon.
Welcome to the Scarlet Ink newsletter. I'm Dave Anderson, an ex-Amazon Tech Director and GM. Each week I write a newsletter article on tech industry careers, and specific leadership advice.
Free members can read some amount of each article, while paid members can read the full article. For some, part of the article is plenty! But if you'd like to read more, I'd love you to consider becoming a paid member!
I recently wrote a couple of articles (one, two) about my time at Amazon. It was fun to write, and people seemed to enjoy them.
One of my best experiences at Amazon was participating in the bar raiser program. If you’re not familiar (and you would rather not read the linked article), I’ll give a brief rundown below.
Today’s article will be a fun exploration of a few experiences I went through in the bar raiser program, and some things I learned along the way.
What’s that about bar raising?
Imagine you started your own company. It was rough going for a bit, but it looks like you’re finally profitable, and you’re hiring many new employees. Good for you.
You’ve been doing it all yourself so far, but you know you need to scale. You need to figure out how to stop being personally involved in every single hire.
But look at all these new managers! How can you trust that they'll make good hiring decisions? You decide to solve it in a relatively simple way.
You identify a few co-workers you really trust, and say that at least one of them must be on every single loop. Those co-workers will have veto power over hiring decisions. You call them “bar raisers” because they ensure that every new hire “raises the bar”. Which colloquially means that they’re good hires. You also tell these bar raisers that they can’t be a bar raiser for their organization’s hires because that’d be totally unfair and biased.
Now you can relax because you know that you’re no longer a bottleneck. Except that a few months later, those trusted co-workers start to also run out of time because your company continues to grow. Too many hires even for this expanded group!
You tell those trusted co-workers that they can identify their most trusted co-workers, and add them to the bar raiser program. And so on.
That’s essentially what Amazon did. Bar raisers are trusted experienced interviewers. They can veto hiring decisions. They run the debrief (decision-making) process, ensuring that there is consistency and quality in the interview process across the company. The general goal is to make sure you don’t hire bad people, and that groups maintain some consistency. What you really want is that a software engineer in one group should be able to join any other group, and not have people say, “What the heck are they doing over there? This employee is horrible!”
Anyway, you can read the linked article if you’d like to know a bit more. Let’s get into the stories.
A time when the hiring team had the wrong bar.
The interview itself was nothing special. Connor was some type of economic scientist candidate with a moderately fine background.
I’m not an economic scientist, and I had read some internal documents to understand what people in that role did. But that’s ok. Part of the bar raiser job is learning how to ask questions to evaluate candidates without experience in the job itself.
Which by itself is a super cool thing to learn how to do. Imagine, for a moment, if you needed to join a group interviewing a software engineer, and you don’t write code.
You wouldn’t ask them to write code, right? Probably not. Because you wouldn’t know if they were writing good code, or bad. But as you’re a smart person, you could likely come up with non-coding questions which could give you a signal about whether this was a good hire or not.
Example: “Tell me about a particularly challenging disagreement you had with a co-worker in the past, and how you resolved it.”
Gives you ideas of what they value, and what they don’t. How much empathy they have, how they interact with their co-workers, and perhaps suggestions of how well they work as a team.
Another example: “Imagine we need to build a Netflix competitor, and I’m your product manager. Let’s figure out our V1 and follow-up version requirements.”
Lets you figure out how well they communicate with product management, and how they negotiate disagreements. You could see if they understand the concepts of MVP (minimal viable product), MLP (minimal loveable product), if they know how to document and clarify expectations, set milestones, and quality and launch processes.
Anyway, the point is that as long as you have skilled software people on the loop, you could fill 1/5th of the interview with valuable questions.
That’s what a bar raiser frequently has to do. So with this economic scientist position, I ensured that the other 4 loop participants covered economic scientist skills, and I took over leadership / communication / project management, and other non-economic scientist-specific skills.
And Connor did what I’d call a C minus job. He answered the questions fine. He didn’t have massive gaps in his skills, but absolutely nothing impressed me. His energy levels were passable, but not exciting.
We don’t hire C minus candidates (the general expectation is that you need a candidate better than the average employee), so I said no in my vote.
Then it came time for the debrief. I pre-read through the feedback from the other 4 people, and it told a pretty clear story. They all read similarly.
“I asked for an analysis of X. He didn’t know Y or Z which was disappointing, but his answer was fine. I guess. I’ll call it good enough. Inclined to hire.”
In other words, a lot of “meh” answers which were barely at or perhaps even below expectations, but they said yes anyway.
When a bar raiser approaches a debrief, I think the good ones have a game plan, or expectations for what is about to happen. I saw 4 people inclined to hire, and their feedback indicated a willingness to have a “meh” new hire. From my experience, that meant I needed to be careful.
If I said, “I don’t think your candidate performed well enough to be hired”, they would insist the candidate did do well, and their notes didn’t reflect the candidate’s great skills accurately. Which is troubling because then you need to decide which version of the feedback is accurate.
Before I suggested a different hiring decision, I needed to get people to verbally confirm what they wrote in their feedback. To be fair, I also had to hear more of their opinions, to ensure I wasn’t incorrect in my assumptions.
We got into the debrief room, and everyone read each other’s feedback (the bar raiser usually reads early so they can prepare). I started the discussion with a fairly standard opener.
I said, “Ok all, let’s get started. To begin with, it looks like everyone was inclined to hire, except for me in their initial feedback. That’s fine. Now that we’ve all read feedback from everyone else, I’d like to hear if you’d like to hire, or not hire, and why. How about we start with you, Elke?”
Per their job titles, it looked like Elke was the most junior person in the room. I started with them for a couple of reasons.
First, as the most junior person, they were less likely to play dumb political games. So I trusted their assessment a bit more.
Second, as a junior person, they were more likely to be influenced by other people’s opinions. So I always tried to have the junior people speak first to avoid peer pressure issues.
Elke looked uncomfortable. “I think we should still hire him?” Elke won’t win any confidence competitions, that’s for sure.
“Ok, sounds good. Why do you say that?” I asked politely.
“He was the best person we’ve interviewed in the last couple of weeks?” he said uncertainly.
I saw a few nods around the table. This was clearly an important point to the room. And yet, it shouldn’t have been said out loud. And perhaps a more senior person wouldn’t have.
One of the jobs of the bar raisers is to maintain a consistent bar for hiring. And one very specific thing we’re supposed to watch for is people lowering their expectations because they’re desperate to hire. We’ve been repeatedly taught that we’d rather hire no one than hire someone who wasn’t an excellent hire. This is also a key reason why bar raisers can never hire for their organization. Because you need a neutral 3rd party to not care about your hiring desperation.
So by saying that he was good because he was the best in the last couple of weeks, he was giving me the exact quote I was supposed to watch for.
“Ok, sounds good. And for my reference, since I’m not an economic scientist, were there any answers which you found particularly great?” I asked.
“Hmm. Not really? They were ok I guess?” Elke said again.
“Thats fine. Thanks! Let’s move on.” I said.
This was a little sneaky of me. I pretended it was a casual question, but it was a critical one for the hiring decision. Because one of the rules for hires is that you’re supposed to hire someone better than the current average employee. One way to gauge if your hiring bar is at the right point is that the person should be exciting to hire. Because you’d say, “Oh yay, they’d really improve my team if they joined.”
A side note — “better than the current average employee” is super difficult to judge, which is why we use proxies like “Are you excited to hire them” or “Did they give some great answers”. And I think the reason we say better than average is that the hiring process is error-prone. The simple explanation is that you’d rather fail high than fail low. So you aim for “great”, and you’ll be ok with “good”.
I continued my questioning similarly with the other 3 people in the room, including the hiring manager Vlad (who went last). Everyone said they still wanted to hire the candidate. Every person agreed that there weren’t any great answers, and some answers were a bit disappointing.
I spoke to the hiring manager next.
I said, “Ok, let’s see if we’re on the same page. The candidate didn’t do a particularly great job on any question, but they didn’t fail any questions badly. Sounds like someone who might meet the bar after they receive some coaching from the team. Does that sound accurate?”
Totally a trick question, because I’m a tricky guy. I said the mentorship thing to suggest that I was willing to go along with his hiring plan, which meant he would be less defensive about the hiring decision. Which meant I might get a more honest assessment.
The hiring manager Vlad nodded. “Yes, I wouldn’t say he did well, but it was better than any other candidates these last couple of weeks. I think with some coaching, he’ll be able to do fine.”
I nodded. “Ok, I think I understand. The concern is that we’re only able to hire candidates who raise the bar. They should be exciting to hire, and should demonstrate some strengths to counter our concerns. I heard that this candidate didn’t show us anything exciting or particularly impressive. I understand that your team would like to hire someone, and you really need this role filled. I sympathize with that. However, I don’t think we can hire this candidate. It seems clear that they don’t raise the bar.”
I know I’d been sneaky, but my intentions were good. I needed everyone to give their honest assessment. Now it was time for the uncomfortable part of the meeting. When people will either agree not to hire, or get really disagreeable.
“What?!” Vlad said with wide eyes. “We’re going to hire him! We decided! The four of us agreed to get him on the team. You don’t need to agree, you just need to get out of our way.”
Ah, he’d chosen disagreeable.
“I understand you four want to hire him, but my job here is to ensure we make the right decision. And I’ve heard repeatedly that this candidate doesn’t meet our expectations.” I said.
Another scientist on the team spoke up. Camilla said, “But we interviewed like 7 people in the last couple of weeks. Connor was the best. We need someone on the team now, we can’t wait for someone who’s really good.”
I wondered if Vlad would pick up on the fact that Camilla had dug the hole deeper. I thought it was possible he’d try to backpedal and claim the candidate was actually great. But he didn’t.
“She’s right!” he said. “He’s not great, but he’s the best in the last couple of weeks, and it’s just taking too long to find someone else. We’d rather have someone here doing the work.”
Which was, conveniently, practically a restatement of exactly why I was here.
“I understand that filling your position is important, but that can’t drive the hiring decision.” I said. “If you’re in a hurry, you might get more resources from the recruiting team to find more candidates. But the hiring decision needs to be based on the candidate’s skills. And this candidate is not at the bar.”
This discussion looped in circles for a bit. A couple of times I had to re-explain the bar raiser position, and I indeed did have the right to veto the hire.
The team members repeatedly explained that they were overworked, and felt that part of the reason was because they were understaffed. The hiring manager repeated that he wanted to make this hire, and was super annoyed I wouldn’t let him.
“I’m going to escalate to get a new bar raiser! This is unacceptable!” Vlad said.
“You can escalate however you’d like Vlad. But you won’t get another bar raiser, and you won’t be making this hire.” I said. Perhaps I could have been more polite, but bah, sometimes you run out of patience.
I ended the meeting. The interviewers stomped out, leaving me with the recruiter who had stayed silent through the debrief. “I’ll reject Connor, and won’t let Vlad work around the process.” the recruiter said. I nodded. It’s a relief when the recruiters are on the ball.
I also wrote to the local bar raiser leadership, letting them know that Vlad might try to work around the system and / or escalate to complain about me. Always good to give people a heads-up.
Vlad did indeed escalate a couple of levels up his hierarchy, and as a result, his manager apparently gave him a nice long lecture on the Amazon hiring process and his role in it. Which felt vindictively pleasant to hear.