Preparing for Calibrations: Manager Guide

Calibration process overview

Manager Pre-Work

In preparation for your Calibration Session with your People Partner, we expect all Managers to complete the following pre-work:

  1. In Lattice, complete draft reviews for your direct reports utilizing the SBI model by Sept 19th
  2. In your Team’s Calibration Roster, fill-in Rating Details (Columns H-I) for every Teammate up for review
  • Note: While we don’t force distribution, we anticipate the following distribution on average:
    • 20% “Exceeding High Bar” [1:1, 1:2 or 2:1]
    • 70% “Meeting High Bar” [2:2s]
    • 10% “Not Meeting High Bar” [3:1, 3:2, 2:3, 1:3 or 3:3]
  • Managers will be required to place each of their direct reports into one of the following categories for both Skill and Values: Exceeding High Bar, Meeting High Bar, Not Meeting High Bar. Additionally, Managers will be required to note who they are considering putting up for promotion.
  • While not required, we highly encourage managers to write notes in the section “M” of the Calibration Roster in preparation for their calibration presentation. You are highly encouraged to copy/paste examples from your Lattice reviews.
  1. On the same roster, mark “Y” - yes or “N” - no (Columns K-L) to indicate if the Teammate is being nominated for a promotion, and if yes, link the completed Promotion Form
  2. Thoroughly prepare to discuss Teammate ratings with your peer calibration participants in a group setting

Calibration Session Agenda

Attendees:

  • Department Managers, Directors, and VPs
  • People Partner
  • (Optional) VP, People and Director, Recruiting
  • (Optional) Cross-functional Leaders

Meetings are facilitated by the People Team in partnership with each department head.

Agenda:

The following is a typical calibration session agenda. Sessions may slightly differ depending on the size of the group being calibrated, as well as the number of calibration meeting participants.

*Note: we expect all Managers to complete their pre-work prior to the calibration meeting.

- Step 1: Overview of the calibration process
- Step 2: The role of calibration participants
- Step 3: Building awareness and mitigating unconscious bias
- Step 4: Performance calibration
- Step 5: Promotion calibration

What to expect during the Calibration Session

Performance calibrations provide an opportunity to review and calibrate performance of all Teammates within a specific department or function, as a collective group. Calibration sessions are interactive discussions where:

  1. Calibrate ‘1’ Ratings: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 1 - Exceeding High Bar category for values and performance, validating the placement with tangible examples. More details here.

    1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations here.
    2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score here.
  2. Calibrate ‘3’ Ratings: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 3 - Not Meeting High Bar category for values and performance, validating the placement with tangible examples. More details here.

    1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations here.
    2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score here.
  3. Calibrate Potential Outliers: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 2 - Meeting High Bar category that may be on the fringe of either Exceeding or Not Meeting

    1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations here.
    2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score here.
  4. Calibrate Promotion Nominations: Managers will explain any “Yes” submissions for promotions and reach final recommendation in partnership with calibrations participants. See details on promotion calibration here.

  5. Validate Final Performance Rating Submissions and Promotions

*Note: The purpose of a performance calibration is not to be adversarial, but to work together to align on a set of standards to be applied to all employees during reviews ensuring the same bar is applied consistently across all teams and individuals. Additionally, the VP within the team will ultimately determine the outcome of promotion nominations in partnership with their People Partner.

Agenda Breakdown

Part 1: Discussion

  • Our calibration discussions are an opportunity to focus on aligning on what Exceeding High Bar means at Sourcegraph. During calibration, Managers will be asked to highlight specific instances where Teammates demonstrate exceptional performance.
  • Of the four required questions on our Manager-to-Direct Report Impact Review, we will focus on two as part of the Manager presentation:
    • How has this Teammate’s performance mapped to the expectations of their role and level over the past 6 months? (Reference the career development framework if applicable). To what extent did they meet their commitments?
    • How does this person’s behavior align with our values? (Provide 2-3 specific examples).
  • Our conversations will focus specifically on the area(s) in which the Teammate was exceeding our high bar: performance, values, or both.
  • Managers will be asked to discuss on individuals who report to them and should come prepared to verbally share the following:
    • Performance and Values rating
  • 2-3 examples of work that supports the rating (the 2 questions bulleted above are a great source for providing these examples!)
  • Explanation for why each example supports the rating
  • Pro Tips:
    1. Utilizing the SBI model when writing reviews, will better prepare Managers to present tangible evidence to support their ratings during the calibration session
    2. When possible, Managers should support their tangible evidence by tying it back to the expectations outlined in the career framework
    3. While not required, we highly encourage Managers to write notes in the Calibration Roster in preparation for their calibration presentation (see example below).
  • Each calibration presentation should be sufficiently detailed that peer Managers can make an informed judgment on the score with no first hand knowledge of the work beforehand.

Example calibration notes (see doc):

[Skills and Values Rating]

Hermione is an IC1 Wizard. I am rating them a 1 - Exceeding High Bar in skill and 2 - Meeting High Bar in values this period.

[Examples of work that supports the rating]

Examples of work that exceeded my expectations this period include:

  1. Polyjuice Potion solution on the Chamber of Secrets Project

(Situation:) Hermione worked on the Chamber of Secrets project with two IC1 Wizards, Ron and Harry. When she joined the project, the team was blocked trying to figure out how to find the chamber of secrets, and the project was at risk. (Behavior:) Hermione joined the project, came up with the idea of brewing polyjuice potion, researched how to do it, and successfully brewed it autonomously. (Impact:) As a result, the team was able to take the potion, learn the location of the chamber of secrets, and destroy Tom Riddle’s diary successfully, saving the school.

[Explanation for why each example supports the rating]

This exceeded my expectations in Proficiency and Execution for an IC1 Wizard because of the technical complexity and precision required to brew the potion. This potion is not taught until IC3, takes 21 days to prep and one month total to brew, and contains 6 complex components to create. Hermione scoped out all of the requirements to this complex, ambiguously defined task, explained the reasoning and trade-offs behind their technical decisions to the other wizards on the team, created all 6 components, and delivered the potion ahead of schedule, which exceeded my expectations for an IC1.

  1. Additional example(as needed)
  2. Additional example (as needed)

Additional resources: Looking for more guidance on crafting a compelling example? Our Handbook page on giving great feedback using the Situation-Behavior-Impact model is a great place to start.

Part 2: Participant expectations

  • When not presenting, Managers are expected to participate in the calibration of every other Teammate presented, not just their direct reports and ICs they are directly familiar with. Participating means:
    • Asking clarifying questions of the presenting Manager until you are able to make an informed determination on if the work presented supports the score given
    • Openly stating if you agree or disagree with the score given, and why, to facilitate the discussion
    • Participate in the discussion until the group is aligned on the score, and the reasoning why

Example participating Manager questions:

Q: “You mentioned that the 6 components were particularly complex - what made these more complex than you would typically expect from an IC1 Wizard?” A: The 6 components were lacewing flies, leeches, fluxweed, knotgrass, powdered horn of a Bicorn and shredded skin of a Boomslang. The lacewing flies had to be brewed for 21 days beforehand, the fluxweed had to be picked at exactly midnight, and for each step she had to calculate the brewing time to the minute based on if the pot was copper or bronze for all ingredients to work together. One minute of variance in that month and the potion would not work. That level of precision in execution exceeds what I expect from IC1 Wizards”

Q: “You mentioned that Hermione led the project, but my impression was that Luna did the customer discovery, wrote the scoping document, and was the one driving the project. Which parts of this was Hermione responsible for?” A: “Luna was involved at the beginning of the project, but was reassigned to another project before Hermione joined. Hermione was the one who came up with using polyjuice potion and the execution from there - Luna did not rejoin the project.”

Q: In Hermione’s self review, she noted that when she took the potion it did not work as intended and she turned into a cat, so there was a problem in the original execution. Does this still exceed your expectations for an IC1? A: Yes, Hermione made the right call to test the potion ahead of time, quickly root caused the issue, and fixed the problem before implementation. This still exceeds my expectations for an IC1 wizard.

Part 3: Aligning on the scores

  • Managers will align on the score. Managers who disagree are expected to state their position and explain why, and participate in the debate to align on the score. Once aligned, we will move to the next Teammate on the calibration roster

Example:

Participating Manager 1: “So far, I have not heard anything that I believe exceeds the high bar for an IC1 Wizard. This potion is well defined in the potion book, and executing on well defined tasks is the expectation of all IC1’s on the team.” Participating Manager 2: “I disagree, the complexity of the execution in making the potion, even if spelled out step by step, exceeds expectations for an IC1 Wizard. In the career development framework, breaking down and scoping complex tasks is an IC2 expectation”

*Note: If Managers are unable to align, the VP of the division is responsible for clarifying the bar and making a decision.

Part 4: Discuss promotion nominations

  • Following calibration discussions, we will review promotion nominations. Managers with promotion nominations will be asked to submit a completed promotion nomination form to their People Partner prior to the meeting, which will be shared with meeting attendees.
  • During the calibration session, Managers will be asked to:
    1. Confirm they would still like to proceed with the promotion conversation, based on earlier performance/values calibration
    2. For individuals whose promotions we will move forward with, we will ask the group to raise concerns about promoting that individual
  • The most senior leader in the meeting is responsible for getting the information they need from the group to make a final promotion decision.

Compensation & Promotion approval

Following calibration sessions, People Partners will work with VP+ level leaders to finalize compensation recommendations. Because we believe in consistently rewarding high performance, our merit increase process will tie increase percentages closely to performance/values ratings.

It’s crucial that we adhere as closely as possible to expected distributions because our merit increase budgets are based on distribution assumptions listed below. The purpose of calibration sessions are to hold a high bar for “Exceeding High Bar” scores, and reward teammates accordingly.

Likewise, final promotion decisions rest with VP+ level leaders. The calibration session is a time for VP+ level leaders to gather information they need to approve, or delay, promotion requests. VP+ level leaders are expected to follow up with Managers by the start of Phase 5 (see the Impact Review Timeline for details on phases) with final decisions, so Managers may communicate to their teams accordingly.


Rating definitions and expected distributions

We rate on two dimensions: performance & values. Each dimension is assigned a score of 1 to 3. The Impact Review Handbook page defines our review philosophy in greater detail.

Ratings

  • 1 - Exceeding High Bar
    • Skill: your contributions consistently raise the high bar set for individuals at your level; your execution, both individually and cross-functionally, is consistently excellent.
    • Values: exceeds in many, exhibits all, deficient in none.
    • Distribution: we expect ~20% of Teammates to fall in the “Top Tier” (high performance) category.
      • Top Tier score = 1:1, 1:2 or 2:1
      • Typically ~15% rated 1 in Skill, ~7% rated 1 in Values, with ~2% rated 1 in both.
    • Merit: Teammates who receive “exceeding high bar” rating can receive an increase twice a year.
      • Recommended merit increase:
        • 1:1 = 7.5-10%
        • 1:2 = 7.50%
        • 2:1 = 7.50%
      • Note: The above merit increase recommendations are based on the expected distribution of 20/70/10 and are subject to change dependent on final distribution.
  • 2 - Meeting High Bar
    • Skill: your contributions are in line with expectations at your current level and sometimes exceed them; you are an essential part of the team.
    • Values: exceeds in some, exhibits most.
    • Distribution: we expect ~70% of Teammates will fall in the “Highly Valued” (strong performance) category
      • Highly-Valued score = 2:2
    • Merit: Teammates who receive “meeting high bar” are eligible to receive a compensation increase once a year, either in the February or August merit cycle. Your People Partner will inform you about a Teammate’s eligibility.
      • Recommended merit increase:
        • 2:2 = 3-5%
      • Note: The above merit increase recommendations are based on the expected distribution of 20/70/10 and are subject to change dependent on final distribution.
  • 3 - Not Meeting High Bar
    • Skill: your contributions are below expectations for your current level and must immediately improve.
    • Values: exhibits some, lacks in some, deficient in some.
    • Distribution: we expect 10% of Teammates will fall in the “Least Effective” (low performance) category.
      • Least Effective score = 3:1, 3:2, 2:3, 1:3 or 3:3
    • Merit: Teammates who receive “not meeting high bar” are not eligible for a compensation increase or a promotion.

Ratings in chart view


Mitigating unconscious bias

  • At Sourcegraph, we strive to consistently reward and develop our team members fairly and equitably. As people leaders, it is our expectation that Managers calibrate teammate performance in a thoughtful manner and with an inclusive lens.
  • These slides illustrate common pitfalls that may arise, consciously or unconsciously, during the performance appraisal process.

5 ways that biases can undermine a performance assessment.

[Based off the SEEDS model from the NeuroLeadership Institute]

Type of Bias Examples (‘How it shows up’) How to Mitigate
Similarity Bias

We prefer what is like us over what is different

“People like me are better than others”

- A manager tends to rate male team members favorably based on future potential, while judging female team members based on past performance

- Members of marginalized groups may be expected to balance opposing expectations (ambition vs likeability) at the cost of being perceived as a leader

- Reflect on whether you’re rating people based on their tangible accomplishments or their future potential

- Reflect on the differing background of your teammates and the narrow/wide bounds of behavior; find value in the various ways of being and doing even if it’s not the typical or what you’re used to / comfortable with

Expedience Bias

We prefer to act quickly rather than take time

“If it feels right, it must be true”

- A sharply dressed, attractive team member might be judged to be more competent than a team member who shows up wearing jeans and a baseball cap

- A teammate’s enthusiasm or positive attitude may overshadow their actual lack of knowledge, skills and abilities

- Reflect on whether you’re just “trusting your gut”; develop a step-by-step approach to ensure you have multiple data points, and apply it to all your directs, before assessing an individual’s performance

Other examples: Halo Effect, Confirmation Bias, Availability Bias

Experience Bias

We take our perception to be the objective truth

“My perceptions are accurate”

- A manager always seeks out second opinions from similar and like-minded peers to validate their rating of a teammate’s performance - When evaluating a team member’s performance or work, actively solicit feedback from peers who tend to challenge your way of doing/thinking

- Reflect on whether you’re obtaining feedback from a variety of peers, and not inadvertently creating an echo chamber to eliminate opposing viewpoints

Distance Bias

We prefer what’s closer over what’s farther away

“Closer is better than distant”

- A manager more closely interacts with teammates in their timezone and tends to provide them with more opportunities to work on highly-visible projects

- A teammate who has been unable to attend a team offsite is regularly tasked with handling office/housekeeping duties versus their peers who’ve been able to attend in-person meetups with their manager

- Ensure you assess performance from a level playing field; teammates can’t be reviewed unfairly for not doing highly-visible, high impact projects if it was never available to them to begin with

- Reflect on whether you are offering business-critical work to teammates who you’ve not met in-person, or whom you tend to work asynchronously with

Safety Bias

We protect against loss more than we seek out gain

“Bad is stronger than good”

- Mistakes of well-represented group members are commonly attributed to external factors and are forgiven over time, while mistakes of marginalized groups are more often attributed to innate ability and become a mark that doesn’t fade over time, often derailing advancement opportunities - Ensure the period of time when performance is evaluated is the same across the board for all teammates

- Look for patterns: whose mistakes do you tend to forgive versus not forgive?