Some courses on the edX platform have maintained a commitment to evaluating students using methods that mirror, as closely as possible, the assessments used in residential course settings. JusticeX is not one of those courses. In interviews with Professor Sandel and his course team, they described the goals of their course as more focused on sharing their perspectives and sparking student interest and reflection than in evaluating, sorting, and certifying students.

In the spirit of creating a “course” rather than a lecture series, Professor Sandel did include assessments in JusticeX, and in an interview he described multiple-­‐choice questions as the “least bad” of the available option for assessing student competence at scale in the humanities. In evaluating grades and certificates in JusticeX, it is important to keep in focus the limits of the assessments used and the course team’s belief that the learning opportunities provided to “auditors” and others with no interest with the problems and certifications were as important as the experience of students who chose to engage with the for-credit problems.

Figure 8. Distribution of grades across all JusticeX participants who answer at least one for-credit question correctly (n= 12,330).

Figure 8

Of the 76,079 students who registered for the course on or before the final due of August 2, 2013, 5,442 people earned a certificate. (Of interest: two of the 133 people who registered for the course on August 2 earned a certificate.)

Many analyzing large-scale online courses are interested in the percentage of students who earn a certificate, and the corresponding inverse. As a set of purely descriptive statistics, we can take the number of students who earned a certificate (5,442) and divide that by the number of students who registered (79,787, yielding 6.8%); by the number of students who registered before the final due date (76,079, yielding 7.2%); by the number of participants who viewed the course and registered by the final due date (47,469, yielding 11.5%); or by the number of participants who viewed more than half the course and registered by the final due date (8,285, yielding 65.7%). However, no data that we have for JusticeX can allow us to identify the number of students who intended to complete the course and the proportion who actually did so.

Interestingly, there is little variation in certification rate based on when students signed up for the course. Figure 9 shows an average trend line (loess approximation) of pass rate by daily registration cohort. About 7% of the students who register on any given day pass the course, slightly lower before the launch and slightly higher after.

Figure 9. Average trend line (loess approximation) of pass rate (certified students over all registered students) by daily registration cohort (n=76,079).

Figure 9