Incorporate Reviewer Feedback User Guide

Last Updated: April 2020

Purpose: This document provides step-by-step instructions for incorporating Reviewer feedback of a Job Profile in Calibrate.

Audience: This document is meant to be used by Job Profile Editors after Job Profile Reviewers have reviewed a Job Profile and provided feedback.

Assumption: Incorporating Reviewer feedback of a Job Profile consists of three tasks: 1) inviting Reviewers to give you feedback on the Job Profile, 2) collecting feedback from Reviewers and 3) incorporating that feedback into the Job Profile. This User Guide is concerned with task 3 and assumes that tasks 1 and 2 are already complete. If you haven’t performed task 1 yet, see Invite Reviewer User Guide.

Before you begin: Incorporating Reviewer feedback of a Job Profile is an iterative process, which means you'll probably go through the tasks mentioned above and the steps in this User Guide more than once before a Job Profile is ready to be published.

You can find the definition of the capitalized words used in this User Guide in the SkillsEngine dictionary located here.

Instructions:

View Reviewer Feedback

1] After you log in, the Calibrate home page appears. To find the particular Job Profile, click the Profile icon on the left side of the home page.

2] The Job Profiles list appears. There are two ways to locate a particular Job Profile. The first way is to search for it by entering the Job Profile title in the search field. The list will automatically populate with choices.

3] The other way to search for the Job Profile is to use the page control at the bottom of the list. The ">" advances the list ahead one page at a time.

4] You can also jump to a page by clicking on the down arrow and then clicking on that page.

5] Once you've located the Job Profile in the list, click on it. That brings you to the page for that Job Profile.

6] Scroll down the Job Profile to view Reviewer feedback on Work Activities, Knowledge Domains, Soft Skills and Tools & Technologies. Reviewer feedback is in the Reviewer Rating column to the right.

NOTE: The feedback you'll see on the Job Profile page includes: Pending Suggestions: skills suggested by Reviewers not originally in the Job Profile. Reviewer Rating: the degree of critically assigned to Job Profile skills by the Reviewer(s).

 

NOTE: Since each skill may receive multiple different ratings from various Reviewers, the rating you see in the Reviewer Rating column is the median rating of all the ratings from all the Reviewers.

7] To see the breakdown of the median Reviewer Rating for a particular skill, click the "ᴠ" to the right of the skill.

8] The skill breakdown screen appears.

NOTE: There are two very important pieces of information you can gather from the skill breakdown screen: How many Reviewers rated a skill as Critical, Important, Beneficial and Irrelevant. Which types of reviewers (Experts, Contributors, Educators) rated a skill with which criticality. You can use this information to "weight" the ratings to help you determine an overall criticality rating for that skill.

Incorporate Reviewer Feedback

9] When you're ready to begin incorporating Reviewer feedback into your Job Profile, click the Revise button in the upper right corner.

NOTE: If the button in the upper right corner doesn't say Revise, but instead says Edit, that means the Job Profile is already in Draft mode. Just go on to step 10.

10] Click the Edit button in the upper right corner.

Review Work Activities

11] Begin by incorporating Reviewer feedback on Work Activities. Click Work Activities in the Edit Job Profile menu.

12] Start at the top with Pending Suggestions (if there aren't any go to step 15). To see who suggested the Pending Work Activity, what their review type is and how they rated it, click the "ᴠ" to the right of the skill.

13] After evaluating the Pending Work Activity, click Accept or Reject to the left of the skill.

NOTE: If you click Accept, the skill moves into the Unrated Work Activities list below, along with its Reviewer rating. If you click Reject, it goes into the Discarded list on the right. Skills in the Discarded list are not gone forever and can always be retrieved at a later time.

14] Repeat steps 12 & 13 for all Pending Suggestions.

15] Next, proceed through the Unrated Work Activities list one at a time. If necessary, click the "ᴠ" to the right of a Work Activity to view information about it, including where it came from, what the Reviewer types are that rated it (if any) and what rating those Reviewers gave it.

NOTE: When evaluating a Work Activity in the Unrated Work Activity list, you can assign it one of four values: Critical: highly relevant, required and central to the job Important: complementary, and frequently performed within the job, but not required Beneficial: supplementary and helpful, but not necessary for the job Irrelevant: not relevant to the job, candidates gain no benefit from having the skill

16] For each Work Activity in the list, using the Reviewer Rating on the right side as a guide, decide what rating to assign it.

17] Hover your mouse over the three circles or the "x" in the Editor Rating column.

18] If you think the Work Activity is CRITICAL, click the right-most circle. If it's IMPORTANT, click the center circle. If it's BENEFICIAL, click the left-most circle. The circles will fill in to indicate your choice. If it's IRRELEVANT click the "x" to the left.  

NOTE: If you fill in the circle(s), the Work Activity moves down to the Rated Work Activities List. If you click the "x", the Work Activity moves to the Discarded list.

19] If you want to change your rating, navigate down to the Rated Work Activities list, locate the Work Activity, and click the circle(s) that reflect your new rating. 

20] If you marked it IRRELEVANT and change your mind, locate it in the Discarded list on the right and add it back into the Unrated list by clicking "+". If you're unsure how to use the lists on the right, see Create Job Profile User Guide.

21] If you want to make a Work Activity in the Rated list IRRELEVANT, you'll first have to click the undo arrow to the left of the skill. That moves it up to the Unrated list and then you can click "x".

NOTE: To the right of each Work Activity in the Rated Work Activities list you may see a color status in the information “i” column. The legend for the color status is right under the Edit Job Profile menu. If you don’t see any color code there, that means there’s agreement between your value and the Reviewer median rating. Red means there’s a disagreement.

 

NOTE: When you get to the end of the Work Activities list, if you think there is a Work Activity that is still missing from the list, you can add it in now. If you do, you’ll probably want to send the Job Profile out for review again

22] If you want to add a Work Activity at this point, you can add one from the lists on the right (Related, Library, Discarded) or you can create a new one from scratch by clicking the Create a New Work Activity button at the bottom of the list.

NOTE: If you're unsure how to use the lists on the right, see Create Job Profile User Guide. If you're unsure how to create a Work Activity from scratch, see Create Work Activity User Guide.

Review Knowledge Domains, Soft Skills and Tools & Technologies

23] To Validate Knowledge Domains, click KNOWLEDGE DOMAINS in the Edit Job Profile menu and repeat steps 11 - 22.

24] To Validate Soft Skills, click SOFT SKILLS in the Edit Job Profile menu and repeat steps 11 - 22.

25] To Validate Tools & Technologies, click TOOLS & TECHNOLOGIES in the Edit Job Profile menu and repeat steps 11 - 22.

26] When you have incorporated the Reviewer feedback and reviewed the skills, click Finish to return to the Job Profile dashboard.

NOTE: The updates that you made to this Job Profile are not visible to your Reviewers yet. For them to see your changes and provide additional feedback, you must send the Job Profile update for Validation.

27] At this point you have three options. Option 1: you can go back and make more changes to the Job Profile by clicking the Edit button in the upper right corner.

28] Option 2: if you're done with the Job Profile and don't want any more feedback from Reviewers and just want to go straight to publication, on the right side menu click Publish.

29] Option 3: you can send the update for Validation and invite Reviewers to give you more feedback by clicking the Validate button.

30] The Validation pop up appears. Click Okay.

31] The Notify Reviewers pop up appears. To immediately send out emails to Reviewers notifying them that there is a new revision of of the Job Profile, click the Send button. Otherwise click the Don't Send button.

NOTE: Finishing the Validation does not mean it’s closed. You still have to close the validation to be done with it. CAUTION: Once you close a Validation it cannot be reopened.

32] To close a Validation, click the Close Validation button in the upper right corner.

33] When the Warning pop up appears that asks you if you want to close the Validation, click Close Validation. The Validation status changes to Closed.