Review a Course

Penn State’s Course Review Process​

Penn State’s QM course review process is all volunteer-based. A call for volunteers is posted three times a year seeking courses to be reviewed and course reviewers. A team of three reviewers works to review each course submitted. To participate in the process, reviewers must have completed the Applying the QM Rubric (APPQMR) Workshop. There is no prerequisite for submitting a course for review.

Review Cycle Schedule

An open call for courses to be reviewed and volunteers to serve on review teams will occur three times a year on July 1, November 1, and March 1.

Open Call for Courses and Review Volunteers
 Course Review Periods
July 1st to September 15th October 1st to December 15th
Nov 1st to January 15th February 1st to April 30th
March 1st to May 15th June 1st to August 31st

Note: Requests for formal course reviews and QM self-reviews may be conducted at any time.

Why volunteer?​

Faculty or designers who volunteer courses for review receive detailed feedback and recommendations for course improvement from reviewers trained in applying the QM Rubric. Note the QM Rubric contains known elements of quality for course design only. It is not a measure of faculty teaching or performance.

Faculty and designers who volunteer to review courses have an opportunity to see online courses from other units or campuses. Reviewers may glean design strategies or ideas from other courses. The review also makes reviewers more aware of QM standards and hones the reviewer’s ability to recognize and meet standards in their own courses.

 

Submit a Course for Review

Faculty or designers may have their fully online courses reviewed. There are four steps in the course review process. Time commitment may vary by course based on review findings, however, estimates for the time required for each step in the process are included below. The time required for the entire course review is 3 to 6 weeks.

Detailed Process, Roles, and Tasks

Step 1: Submit a Course for Review
Faculty Time: ~10 minutes

To volunteer a course for review, the faculty or course instructional designer must complete and submit a short online form with summary information about the course.

Tips

  • Only one person may have access to complete and submit this form. The default is to assign faculty as the designated contact person for the course review. However, in some cases, it may be more appropriate for the designer to complete forms and serve as the contact person.
  • Be sure to designate the contact person (either faculty or designer) when you nominate your course for review.

Step 2: Complete the Course Information Worksheet
Faculty Time: ~1-2 hours

Once the course has been accepted for the QM review, faculty or an instructional designer must complete the detailed course information form to start the review process. This form is online. It takes an estimated 1-2 hours to complete.

Tips

  • A course cannot move forward through the review process without learning objectives. Please check that the course has learning objectives at the course and lesson levels prior to submitting it for review.
  • If you have detailed activities within the course, be sure to explain the purpose, connection to learning objectives, and materials used to support the activity on the form. Without an explanation of the context and use of the activity, reviewers may miss the connection to course learning objectives.

Next Steps

After the Course Information Worksheet is complete, the Penn State QM Coordinator is able to assign a review team to the course and move the review to the next step. Note a course may not move forward in the course review process until this form is completed and submitted online.

Step 3: Meet the Review Team
Faculty Time: ~30 minutes

This meeting is usually brief and introduces faculty to the review process, the reviewers, their roles, and what faculty can expect from the review. The review team may also use this time to ask clarifying questions about the course. This usually requires less than one hour of time.

Tips

  • The review team will need access to the course prior to this meeting.
  • Most review participants prefer to create a copy of the master course in Canvas that is used solely for the course review.
  • External reviewers such as those reviewing courses through a formal QM review are required to get Friends of Penn State accounts to be enrolled in a course.

Next Steps

After this initial meeting, the review team will meet as a group as needed to discuss the review process and findings. The review team members will also spend individual time reviewing the course based on the QM rubric items. Reviewers mark each item in the rubric as “met” or “not met.” When the review is submitted, a notification that the findings are ready for review will be sent.

Step 4: Respond to Reviewer Findings

The time required for this step in the process varies. At this point, faculty review the final report from reviewers and decide how they would like to proceed.

Faculty Time: ~5 minutes
Faculty intent to respond to the review (or not) is submitted through a short form online.

Faculty Time: varies based on the extent of work to be completed
Faculty have an opportunity to collaborate with their assigned designers to make additions or revisions to the course based on the feedback received. If changes are made, the course can be resubmitted to the review team.

Next Steps

If the review team agrees standards previously not met are now met, the review will be updated to reflect this and an amended final report will be issued.

Resources

The QM rubric and sample worksheets are available online at https://www.qmprogram.org/review/index.cfm. Note: A QM password is required to view this resource material.

 

Be A Reviewer: Guide for Volunteer Course Reviewers

Faculty and instructional designers who have received the Applying the QM Rubric (APPQMR) training may volunteer to review Penn State online courses. There are five steps in the course review process. Reviewers use the Course Review Management System (CRMS) to record their findings and submit reports. Time commitment may vary by course based on review findings. However, estimates for time required are included below. The entire review process usually takes 3 to 6 weeks.

Step 1: Review Team Meeting

This is the first chance for the team of three course reviewers to meet, discuss roles and responsibilities, review the overall process, and set a timeline. One member of the review team is designated as the lead reviewer and is responsible for submitting the final report. This meeting also serves as an opportunity to review the functions of the CRMS and be sure that all review members can find the documents needed to perform the review.

Step 2: Kick-off Meeting with Faculty

This meeting introduces the review team to faculty and familiarizes faculty with the review process. All review team members should have access to the course to be reviewed and also read the Course Worksheet completed by faculty prior to the kick-off meeting. The review team can then raise any questions with the course faculty prior to starting the review.

Step 3: Individual Review

After the initial meetings and planning, the review team members work to review the course individually and record their ratings and comments in the online Course Review Management System. Ideally, individual reviews are completed within 2-3 weeks.

Step 4: Review Team Debrief

After individual reviews are complete, the review team reconvenes to discuss the Final Report and any inconsistencies in review findings. The Final Report is then submitted through the CRMS.

Step 5: Faculty Debrief (Optional)

Some faculty and/or designers may have questions about the review findings and recommendations in the final report. In this case, faculty may request to meet with the lead reviewer and/or the QM Coordinator to answer questions about the findings.

 

Create Your Own Quality Assurance Process

Customized Review Process

Not all courses may be ready for a course review. You may want to review a course on your own prior to submitting it for review by others. The Self-Review tool is available for you to use individually to examine the different design components of an online course and begin to understand the best practices that create quality courses. You may also request an internal review team composed of 2-3 members of your own unit or department to have space to collaboratively review a course.

Customized Rubric

You may find the QM Rubric in its default form does not meet your specific needs for course review. In this case, you may want to use a subset of the QM Rubric standards or add customized items to the rubric.

Design with the Rubric in Mind

You may also choose to leverage QM as part of your program planning or course development process. The QM Rubric can inform the design of new online programs or courses – or be used to define areas for improvement prior to revision.

Consulting Available

There are many options for customizing and implementing QM based on your specific quality assurance needs. Penn State’s QM Coordinator is available to answer your questions about QM and make recommendations. Contact Penny Ralston-Berg through Contact form.