Incorporate Reviewer Feedback User Guide

Purpose: This document provides step-by-step instructions for incorporating Reviewer feedback of a Job Profile in Calibrate.

Audience: This document is meant to be used by Job Profile Editors after Job Profile Reviewers have reviewed a Job Profile and provided feedback.

Assumption: Incorporating Reviewer feedback of a Job Profile consists of three tasks: 1) inviting Reviewers to give you feedback on the Job Profile, 2) collecting feedback from Reviewers and 3) incorporating that feedback into the Job Profile. This User Guide is concerned with task 3 and assumes that tasks 1 and 2 are already complete. If you haven’t performed task 1 yet, see Invite Reviewer User Guide.

Before you begin: Incorporating Reviewer feedback of a Job Profile is an iterative process, which means you'll probably go through the tasks mentioned above and the steps in this User Guide more than once before a Job Profile is ready to be published.

You can find the definition of the capitalized words used in this User Guide in the SkillsEngine dictionary located here.

Instructions:

View Reviewer Feedback

1] After you log in, the Calibrate Dashboard appears. To find the particular Job Profile, click the Profile icon on the left side of the home page.

2] The list of Job Profiles and Cluster Profiles appears.

NOTE: The are three ways to locate the Job Profile whose Reviewer feedback you want to incorporate: Use the search box Filter by Tags Use the page controls

3]  To use the search box to locate the Job Profile, enter the Job Profile title in the search box (Job Profiles appear automatically). 

NOTE: When you use the search box to find a Job Profile, the list returned may be so long that you will have to use the page controls to locate it within the search results. If so, see steps 6 and 7 for more details.

4] To filter by Tags, start typing a Tag in the Filter by textbox.

5] A list of Tags appears. Click on the desired Tag.

6] To use the page controls to locate the Job Profile, you can either navigate through the list one page at a time using the page control at the bottom of the list by clicking the ">".

7] Or you can jump directly to a specific page in the Job Profile list by clicking the down arrow next to the page number and then clicking on the page you want to jump to from the list.

8] When you locate the Job Profile, click on it. The Job Profile Dashboard appears.

9] Scroll down the Job Profile to view Reviewer feedback on Work Activities, Knowledge Domains, Soft Skills and Tools & Technologies. Reviewer feedback is in the Reviewer Rating column to the right.

NOTE: The feedback you'll see on the Job Profile page includes: Pending Suggestions: skills suggested by Reviewers not originally in the Job Profile. Reviewer Rating: the degree of critically assigned to Job Profile skills by the Reviewer(s).

 

NOTE: Since each skill may receive multiple different ratings from various Reviewers, the rating you see in the Reviewer Rating column is the median rating of all the ratings from all the Reviewers.

10] To see the breakdown of the median Reviewer Rating for a particular skill, click the "ᴠ" to the right of the skill.

11] The Reviewer detail screen appears.

NOTE: There are two very important pieces of information you can gather from the skill breakdown screen: How many Reviewers rated a skill as Critical, Important, Beneficial and Irrelevant. Which types of reviewers (Experts, Contributors, Educators) rated a skill with which criticality. You can use this information to "weight" the ratings to help you determine an overall criticality rating for that skill.

 

Incorporate Reviewer Feedback

12] When you're ready to begin incorporating Reviewer feedback into your Job Profile, click the Revise button in the upper right corner.

NOTE: If the button in the upper right corner doesn't say Revise, but instead says Edit, that means the Job Profile is already in Draft mode. Just go on to step 13.

13] Click the three dot menu in the upper right corner.

14] In the pop up menu, click Edit Details.

Review Skills

15] Across the top menu, click the skill type you want to review such as Work Activities.

16] Start at the top with Pending Suggestions (if there aren't any go to step 19). To see who suggested the pending skill, what their review type is and how they rated it, click the "ᴠ" to the right of the skill.

17] After evaluating the pending skill, click Accept or Reject to the left of the skill.

NOTE: If you click Accept, the skill moves into the unrated skills list below, along with its Reviewer rating. If you click Reject, it goes into the Discarded list on the right. Skills in the Discarded list are not gone forever and can always be retrieved at a later time.

18] Repeat steps 16 & 17 for all Pending Suggestions.

19] Scroll down and proceed through the Unrated skills list one at a time. If necessary, click the "ᴠ" to the right of a skill to view information about it, including where it came from, what the Reviewer types are and what rating (if any) those Reviewers gave it.

NOTE: When evaluating a skill, you can assign it one of four values: Critical: highly relevant, required, and central to this job. Candidates are disqualified if these skills are missing or deficient. Important: important, complementary, and frequently performed within this job, but not required. Candidates can acquire or become proficient in these skills while on the job. Beneficial: supplementary and helpful, but not necessary for the job. Candidates will benefit from having these skills but are qualified without them. Irrelevant: not relevant to the job. Candidates gain no benefit from having the skill.

20] For each skill in the Unrated list, using the Reviewer Rating on the right side as a guide (if available), decide what rating to assign it.

21] Hover your mouse over the three circles or the "x" in the Editor Rating column.

22] If you think the skill is Critical, click the right-most circle. If it's Important, click the center circle. If it's Beneficial, click the left-most circle. The circles will fill in to indicate your choice. If it's Irrelevant, click the "x" to the left.  

NOTE: If you fill in the circle(s), the skill moves down to the Rated skills list. If you click the "x", the skill moves to the Discarded list.

23] After you've completed the Unrated skills list, scroll down to the Rated skills list and proceed through them one at a time.

24] If you want to change the rating of a skill, click the circle(s) that reflect your new rating. 

25] If you want to make a skill in the Rated list Irrelevant, you'll first have to click the undo arrow to the left of the skill. That moves it up to the Unrated list and then you can click "x".

26] If a skill was marked as Irrelevant and you change your mind, locate it in the Discarded list on the right and add it back into the Unrated list by clicking "+". If you're unsure how to use the lists on the right, see Create a Job Profile User Guide.

NOTE: To the right of each Work Activity in the Rated Work Activities list you may see a color status in the information “i” column. The legend for the color status is right under the Edit Job Profile menu. If you don’t see any color code there, that means there’s agreement between your value and the Reviewer median rating. Red means there’s a disagreement.

 

NOTE: When you get to the end of a skill list, if you think there is a skill that is still missing from the list, you can add it in now. If you do, you’ll probably want to send the Job Profile out for review again.

27] If after you get through the Rated skills list, you think a skill is missing from the Job Profile and would like to add a it, you can from the lists on the right (Related, Library, Discarded) or you can create a new one from scratch by clicking a button like the Create a New Work Activity button at the bottom.

NOTE: If you're unsure how to use the lists on the right, see Create a Job Profile User Guide. NOTE: If you're unsure how to create a skill from scratch, see the following: Create Work Activity User Guide Create Knowledge Domains User Guide Create Soft Skills User Guide Create Tools & Technologies User Guide

 

28] Repeat steps 15 - 27 for other skill types if necessary.

29] When you have incorporated the Reviewer feedback and reviewed the skills, click Finish to return to the Job Profile dashboard.

NOTE: The updates that you made to this Job Profile are not visible to your Reviewers yet. For them to see your changes and provide additional feedback, you must send the Job Profile out for Validation again.

30] At this point you have three options. Option 1: you can go back and make more changes to the Job Profile by clicking the three dot menu in the upper right and then clicking Edit Details .

31] Option 2: if you're done with the Job Profile and don't want any more feedback from Reviewers and just want to go straight to publication, click the three dot menu and then click Publish.

32] Option 3: you can send the updated Job Profile out for Validation again and invite Reviewers to give you more feedback by clicking the Validate button.

33] The Validation pop up appears. Click Okay.

34] The Notify Reviewers pop up appears. To immediately send out emails to Reviewers notifying them that there is a new revision of of the Job Profile to review, click the Send button. Otherwise click the Don't Send button.

NOTE: Finishing the Validation does not mean it’s closed. You still have to close the validation to be done with it. CAUTION: Once you close a Validation it cannot be reopened.

35] To close a Validation, click the Close Validation button in the upper right corner.

36] When the Warning pop up appears that asks you if you want to close the Validation, click Close Validation. The Validation status changes to Closed.