Preparing for Calibrations: Manager Guide

Calibration process overview

The purpose of a calibration session during our impact review process is to ensure that evaluations of teammate performance are fair and consistent across the organization. Calibration sessions typically involve a group of managers or supervisors who meet to review and discuss individual teammate performance ratings, in order to reach a consensus on the accuracy and fairness of those ratings. The goal is to identify any potential biases or discrepancies in the ratings, and to make adjustments as necessary to ensure that all employees are evaluated fairly and objectively.

Calibration sessions can also help identify areas where additional training or development may be needed for teammates, as well as opportunities for recognition and rewards for exceptional performance. Ultimately, the goal of a calibration session is to ensure that the performance review process is transparent, consistent, and fair, and that teammates are evaluated based on their actual job performance, rather than any personal biases or perceptions of their work.

Manager Pre-Work (required before calibration)

In preparation for your Calibration Session with your People Partner, all Managers are required to complete the following pre-work:

Complete all downward reviews for your direct reports utilizing the SBI model. Click here to review best practices for Impact Review writing. Ensure you have reviewed the peer feedback that they received to incorporate into your review.
Review new rating guides along with expected distribution of each new rating.
Log into Lattice and use the tag/note function in the Lattice calibration module. You can find instructions on how to use the tag/notes section in this training video
In Lattice, Managers are expected to place each of their direct reports into one of the following categories for Performance: Distinguised Performance, Superior Performance, Meets Performance Expectations, Partially Successful Performance, Unsuccessful Performance. Additionally, Managers will be required to note who they are considering putting up for promotion.
Review the below resources:

Read this entire handbook page start to finish and educate yourself on “mitigating unconcious bias”
Review our new rating definitions and have a clear understanding of the difference between Distinguished Performance, Superior Performance, Meets Performance Expectations, Partially Successful Performance, and Unsuccessful Performance.
Watch this calibration training video
Review your team’s career framework to ensure you’re well versed on levels… you can also review our company-wide level framework here

Calibration Session Agenda

Attendees:

Department Managers, Directors, and VPs
People Partner
(Optional) VP, People and Director, Recruiting
(Optional, as invited) Cross-functional Leaders

Meetings are facilitated by the People Team in partnership with each department head.

Agenda:

The following is a typical calibration session agenda. Sessions may slightly differ depending on the size of the group being calibrated, as well as the number of calibration meeting participants.

What to expect during the Calibration Session

Performance calibrations provide an opportunity to review and calibrate performance of all Teammates within a specific department or function, as a collective group. Calibration sessions are interactive discussions where:

Calibrate 1 Ratings: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 1 - Distinguished Performance category for performance, validating the placement with tangible examples. More details.
1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations.
2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score.
Calibrate 2 Ratings: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 2 - Superior Performance category for performance, validating the placement with tangible examples. More details.
1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations.
2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score.
Calibrate 4 Ratings: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 4 - Partially Successful Performance category for performance, validating the placement with tangible examples. More details.
1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations.
2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score.
Calibrate 5 Ratings: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 5 - Unsuccessful Performance category for performance, validating the placement with tangible examples. More details.
1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations.
2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score.
Calibrate Potential Outliers: Managers will explain the rating assessment of each Teammate in correspondence to their level and for those Teammates that received a 3 - Meets Performance Expectations category that may be on the fringe of either Superior Performance or Partially Successful Performance
1. Participating Managers will ask clarifying questions, provide feedback/insight, either challenging or confirming the Teammate’s placement. See participant expectations.
2. Managers/Calibration Participants align on scores for Teammates discussed. See details on aligning on score.
Calibrate Promotion Nominations: Managers will explain any “Yes” submissions for promotions and reach final recommendation in partnership with calibrations participants. See details on promotion calibration.
Validate Final Performance Rating Submissions and Promotions

Note: The purpose of a performance calibration is not to be adversarial, but to work together to align on a set of standards to be applied to all employees during reviews ensuring the same bar is applied consistently across all teams and individuals. Additionally, the VP within the team will ultimately determine the outcome of promotion nominations in partnership with their People Partner.

Agenda Breakdown

Part 1: Discussion

Our calibration discussions are an opportunity to focus on aligning on what the rating definitions mean at Sourcegraph. During calibration, Managers will be asked to highlight specific instances where Teammates demonstrate exceptional performance.
Of the four required questions on our Manager-to-Direct Report Impact Review, we will focus on two as part of the Manager presentation:
- How has this Teammate’s performance mapped to the expectations of their role and level over the past 6 months? (Reference the career development framework if applicable). To what extent did they meet their commitments?
Our conversations will focus specifically on the area(s) in which the Teammate had Distinguished Performance, Partially Successful Performance or Unsuccessful Performance.
Managers will be asked to discuss on individuals who report to them and should come prepared to verbally share the following:
- Teammate level/title
- Performance rating
2-3 examples of work that supports the rating (the 2 questions bulleted above are a great source for providing these examples!)
Summary of feedback from peers
Explanation for why each example supports the rating
If putting them up for promotion:
1. Prior performance rating
2. 2-3 examples that demonstrate their performance with the next level from their career ladder
3. If they are not ready right now,how can we set them up for success in the future?
Pro Tips:
1. Utilizing the SBI model when writing reviews, will better prepare Managers to present tangible evidence to support their ratings during the calibration session
2. When possible, Managers should support their tangible evidence by tying it back to the expectations outlined in the career framework
3. While not required, we highly encourage Managers to write notes in the Calibration Roster in preparation for their calibration presentation (see example below).
Each calibration presentation should be sufficiently detailed that peer Managers can make an informed judgment on the score with no first hand knowledge of the work beforehand.

Part 2: Participant expectations

When not presenting, Managers are expected to participate in the calibration of every other Teammate presented, not just their direct reports and ICs they are directly familiar with. Participating means:
- Asking clarifying questions of the presenting Manager until you are able to make an informed determination on if the work presented supports the score given
- Openly stating if you agree or disagree with the score given, and why, to facilitate the discussion
- Participate in the discussion until the group is aligned on the score, and the reasoning why

Example participating Manager questions:

Q: “You mentioned that the 6 components were particularly complex - what made these more complex than you would typically expect from an IC1 Wizard?”

A: The 6 components were lacewing flies, leeches, fluxweed, knotgrass, powdered horn of a Bicorn and shredded skin of a Boomslang. The lacewing flies had to be brewed for 21 days beforehand, the fluxweed had to be picked at exactly midnight, and for each step she had to calculate the brewing time to the minute based on if the pot was copper or bronze for all ingredients to work together. One minute of variance in that month and the potion would not work. That level of precision in execution exceeds what I expect from IC1 Wizards”

Q: “You mentioned that Hermione led the project, but my impression was that Luna did the customer discovery, wrote the scoping document, and was the one driving the project. Which parts of this was Hermione responsible for?”

A: “Luna was involved at the beginning of the project, but was reassigned to another project before Hermione joined. Hermione was the one who came up with using polyjuice potion and the execution from there - Luna did not rejoin the project.”

Q: In Hermione’s self review, she noted that when she took the potion it did not work as intended and she turned into a cat, so there was a problem in the original execution. Does this still exceed your expectations for an IC1?

A: Yes, Hermione made the right call to test the potion ahead of time, quickly root caused the issue, and fixed the problem before implementation. This still exceeds my expectations for an IC1 wizard.

Part 3: Aligning on the scores

Managers will align on the score. Managers who disagree are expected to state their position and explain why, and participate in the debate to align on the score. Once aligned, we will move to the next Teammate on the calibration roster

Example:

Participating Manager 1:

So far, I have not heard anything that I believe exceeds the high bar for an IC1 Wizard. This potion is well defined in the potion book, and executing on well defined tasks is the expectation of all IC1’s on the team.

Participating Manager 2:

I disagree, the complexity of the execution in making the potion, even if spelled out step by step, exceeds expectations for an IC1 Wizard. In the career development framework, breaking down and scoping complex tasks is an IC2 expectation

Note: If Managers are unable to align, the VP of the division is responsible for clarifying the bar and making a decision.

Part 4: Discuss promotion nominations

Following calibration discussions, we will review promotion nominations. Managers with promotion nominations will be asked to submit the promotion recommendation in Lattice. Please ensure the Impact Review packet and notes are submitted and provide detailed examples that support the promotion recommendation, as these will be shared with calibration attendees.
During the calibration session, Managers will be asked to:
1. Confirm they would still like to proceed with the promotion conversation, based on earlier performance/values calibration
2. For individuals whose promotions we will move forward with, we will ask the group to raise concerns about promoting that individual
The most senior leader in the meeting is responsible for getting the information they need from the group to make a final promotion decision.

Compensation & Promotion approval

Following calibration sessions, People Partners will work with VP+ level leaders to finalize compensation recommendations. Because we believe in consistently rewarding high performance, our merit increase process will tie increase percentages closely to performance/values ratings.

It’s crucial that we adhere as closely as possible to expected distributions because our merit increase budgets are based on distribution assumptions listed below. The purpose of calibration sessions are to hold a high bar for “Exceeding High Bar” scores, and reward teammates accordingly.

Likewise, final promotion decisions rest with VP+ level leaders. The calibration session is a time for VP+ level leaders to gather information they need to approve, or delay, promotion requests. VP+ level leaders are expected to follow up with Managers by the start of Phase 5 (see the Impact Review Timeline for details on phases) with final decisions, so Managers may communicate to their teams accordingly.

Mitigating unconscious bias

At Sourcegraph, we strive to consistently reward and develop our team members fairly and equitably. As people leaders, it is our expectation that Managers calibrate teammate performance in a thoughtful manner and with an inclusive lens.
These slides illustrate common pitfalls that may arise, consciously or unconsciously, during the performance appraisal process.

5 ways that biases can undermine a performance assessment.

Based off the SEEDS model from the NeuroLeadership Institute

Type of Bias	Examples (‘How it shows up’)	How to Mitigate
Similarity Bias We prefer what is like us over what is different “People like me are better than others”	A manager tends to rate male team members favorably based on future potential, while judging female team members based on past performance Members of marginalized groups may be expected to balance opposing expectations (ambition vs likeability) at the cost of being perceived as a leader	Reflect on whether you’re rating people based on their tangible accomplishments or their future potential Reflect on the differing background of your teammates and the narrow/wide bounds of behavior; find value in the various ways of being and doing even if it’s not the typical or what you’re used to / comfortable with
Expedience Bias We prefer to act quickly rather than take time “If it feels right, it must be true”	A sharply dressed, attractive team member might be judged to be more competent than a team member who shows up wearing jeans and a baseball cap A teammate’s enthusiasm or positive attitude may overshadow their actual lack of knowledge, skills and abilities	Reflect on whether you’re just “trusting your gut”; develop a step-by-step approach to ensure you have multiple data points, and apply it to all your directs, before assessing an individual’s performance Other examples: Halo Effect, Confirmation Bias, Availability Bias
Experience Bias We take our perception to be the objective truth “My perceptions are accurate”	A manager always seeks out second opinions from similar and like-minded peers to validate their rating of a teammate’s performance	When evaluating a team member’s performance or work, actively solicit feedback from peers who tend to challenge your way of doing/thinking Reflect on whether you’re obtaining feedback from a variety of peers, and not inadvertently creating an echo chamber to eliminate opposing viewpoints
Distance Bias We prefer what’s closer over what’s farther away “Closer is better than distant”	A manager more closely interacts with teammates in their timezone and tends to provide them with more opportunities to work on highly-visible projects A teammate who has been unable to attend a team offsite is regularly tasked with handling office/housekeeping duties versus their peers who’ve been able to attend in-person meetups with their manager	Ensure you assess performance from a level playing field; teammates can’t be reviewed unfairly for not doing highly-visible, high impact projects if it was never available to them to begin with Reflect on whether you are offering business-critical work to teammates who you’ve not met in-person, or whom you tend to work asynchronously with
Safety Bias We protect against loss more than we seek out gain “Bad is stronger than good”	Mistakes of well-represented group members are commonly attributed to external factors and are forgiven over time, while mistakes of marginalized groups are more often attributed to innate ability and become a mark that doesn’t fade over time, often derailing advancement opportunities	Ensure the period of time when performance is evaluated is the same across the board for all teammates Look for patterns: whose mistakes do you tend to forgive versus not forgive?