Calibrate Release Highlights Q1 2019

This quarter we focused on improving and enhancing existing and beta features and, in one case, removing a feature. The beta version of the Job Profile Validations that landed in Q4 2018 reached public release (version 1) and is now enabled for all Calibrate accounts! An overview of Job Profile Validations is available in Release Highlights Q4 2018 and detailed documentation can be found in the Job Profile Validation User Guide.

 Job Profile Validations v1.0

The feedback on the beta version of Job Profile Validations resulted in a number of simplifications and refinements to the work flows and user interfaces. 

The Job Profile section received the following updates:

• Pending Skill Suggestions now display the skill and the ranking. Previously, when a Reviewer recommended a new skill, all that was shown was the skill.

• Reviewer median rankings are now displayed in the Job Profile Report. Originally, when a Job Profile Report was created, only the Editor ranking was displayed.

• Deactivating a Job Profile with a Pending or Active Validation now closes the Validation.

In the Validation section, the following updates were made:

• Added the ability to filter, sort, and paginate the list of Active and Pending Job Profile Validations.
• A new Closed Job Profile Validation section was added allowing users to review previous Closed Job Profile Validations.

• The list of Reviewers on a Job Profile Validation can now be filtered and paginated.

• Active Validations are now automatically updated and Reviewers are notified when the associated Job Profile is updated. Previously, Editors were required to take extra steps to activate an update. This also allowed us to remove the Pending Update state for Job Profile Validations.
• Editors can now manage Reviewers regardless of the current state of a Validation (Pending or Active) and directly manage Reviewers from the main screen of a Job Profile Validation instead of a separate screen.

• Editors can now select multiple Review Types, providing more flexibility when describing the type of review a Reviewer is giving.

• Improved the handling of existing Reviewers including displaying an error when an Editor tries to re-invite an existing Reviewer and ensuring that all Reviewers in the account are displayed in the Existing Reviewer drawer.

The Reviewer section also received some attention, including: 

• Allowing Reviewers to select multiple Review Types on the Review Welcome screen.

• Combining the Work Activity Category steps into a single step.
• Updating the layout of the steps in the Review to include separate lists for Unranked Skills, Ranked Skills and Pending Reviewer Skill Suggestions.
• Allowing Reviewers to mark a Review as Complete indicating more explicitly to the Job Profile Editors that they are done with their review. 

Awards Dashboard

There were several efforts aimed at improving the usability of the Award Dashboard:

• Previously, the Award Dashboard displayed just the Courses but not the Course Outcomes. Now it displays the Course Outcomes in the Summary panel for each Course. This way users have more information when deciding what Courses a skill goes in.

• For each skill in an Award, Calibrate now tries to suggest Courses that this skill may be taught in. This is based on an analysis of the Course and it's information.

• In an Award, users can target up to five Job Profiles (or more by using Job Profile Clusters). Initially, users had to align each Job Profile individually, which didn't make sense since there are typically a lot of the same skills across the Target Job Profiles. Now Calibrate consolidates and de-duplicates the skills for multiple Job Profiles into a single list which results in a lot less work for users.

Security

Security was enhanced by requiring stronger passwords and establishing better user session timeouts. Previously there were very few restrictions on passwords and user sessions expired after 30 days.

Other Improvements and Enhancements

Performance

The following pages were optimized to improve response time:

• Job Profile Create, View and Edit
• Job Profile Validations List 
• Award Dashboard
• Select Job Profile Targets

Work was also done to remove outdated and unused code and improve our deployment infrastructure to reduce initial load times for Calibrate in general.

We improved Calibrate's handling of "bounced" emails. The application can now identify the reason why an email was bounced and whether it was a hard bounce or a soft bounce. This improves performance by ensuring we are not re-sending emails repeatedly that have no chance of reaching their destination.

Better Loading and Empty States

Pages that require loading a lot of data would often appear  empty for users on slower internet connections. Calibrate gave no visual clue to the users that anything is happening. Now, when a page or section of a page is loading additional data, a loading graphic appears to let users know the page is loading. 

Search queries that return no results also produced blank pages. Now these queries explicitly inform the user when no results were found.

Bounced Email Handling

Calibrate Work Activities have typically been grouped by Category. However, after gathering feedback from users, we have ungrouped them in the Job Profile and Award sections to make them easier to work with and reduce the amount of clicking required by users. 

Removed Features

The beta version of the in-app tutorials feature has been removed. The tutorials were not widely used, caused performance issues and required a lot of engineering effort to maintain. We learned a lot from implementing them and gathering feedback from users. We will revisit in-app tutorials as a feature of Calibrate in the future.