JusticeX: Spring 2013 Course Report



Add Comment

This course is an introduction to moral and political philosophy. It explores classical and contemporary theories of justice, and applies these theories to contemporary legal and political controversies. Topics include affirmative action, income distribution, same-sex marriage, the role of markets, debates about rights (human rights and property rights), arguments for and against equality, dilemmas of loyalty in public and private life. The course invites students to subject their own views on these controversies to critical examination.
       —from the JusticeX syllabus

Justice is an introduction to moral and political philosophy, including discussion of contemporary dilemmas and controversies.

ER22x was offered as a HarvardX open online course in Spring 2013 on edX, a platform for massive open online courses (MOOCs). It was taught by Professor Michael Sandel.

Reich, J., Nesterko, S., Seaton, D.T., Mullaney, T., Waldo, J., Chuang, I., & Ho, A.D. (2014) JusticeX: Spring 2013 Course Report (HarvardX Working Paper No. 4).

Content Type: 


Introduction to the JusticeX Course Report

This report describes one of the first courses, JusticeX, offered by HarvardX on the edX platform. The report was prepared by researchers external to the course team, based on an examination of the courseware, analyses of data collected by the edX platform, and interviews with the course faculty and team members.

The report proceeds in four parts. We begin by describing the goals and structure of JusticeX, with the belief that any learning environment should be evaluated in the context of its intent, values, and vision. We then provide descriptive statistics about the students who registered for JusticeX and compare them to other HarvardX students. With an understanding of what the course team created and the learners who took an interest in the course, we then turn to examining how participants interacted with the resources, including their patterns of assessment-­‐taking, persistence, and overall activity. We end by examining the limits of our understanding of student learning in JusticeX and discuss future research directions for online humanities courses.

Our hope is that this report and its companion reports—including a synthetic multiple-course report and other reports from the first HarvardX courses—will inspire new avenues of research and provide insights to future course designers in designing the next generation of open online courses.

The Structure of JusticeX

Professor Michael Sandel’s Justice class has been a fixture of Harvard’s College of Arts and Sciences for years, serving over 15,000 students on campus. Professor Sandel’s teaching style is distinguished by the Socratic dialogues he facilitates with hundreds of students in Sanders Theatre, and he was featured in Ken Bain’s research study of excellent teachers, What the Best College Teachers Do. Like several of the first HarvardX classes, Justice had an online life before HarvardX. In 2009, WGBH partnered with Professor Sandel and Harvard University to broadcast the Justice course on television, which was complemented in 2011 by justiceharvard.org, a site combining online videos and discussion forums. The first of the 12 episodes of WGBH’s Justice production has been viewed by millions on television and on YouTube. This is a course with a long, highly regarded history.

JusticeX extends the Justice learning experience onto the edX platform. The course officially launched on March 3, 2013; content was released starting March 12. All course content, including the final exam, was released by May 31, and students wishing to earn a certificate needed to submit all graded work by August 2. Participants could still register and use the courseware beyond this submission date, and we collected data on activity through September 8, 2013.

Table 1. Important Dates in JusticeX

December 19, 2012 Registration Opens
March 3, 2013 Official Course Start Date
March 12, 2013 First Content Released
May 31, 2013 Final Exam and All Content Released
July 31, 2013 All Graded Material Due for Certificates
September 8, 2013 Date of Report Data Collection

The courseware consisted of 24 content chapters, each organized around a video lecture of approximate 20–30 minutes. Each lecture, mainly recut from PBS video, was packaged and accompanied by a selected reading or two; a three-question self-test (un-graded); a lecture-specific discussion prompt; and a poll question. The poll questions asked people to respond to a specific issue of moral reasoning with a “yes” or “no” answer. When poll respondents selected yes or no, they were shown the results of the poll and taken to a discussion forum specific to their response. In this discussion forum, the course team offered a counter argument and solicited a response.

Suggestions for approaching the JusticeX course material from the course FAQ:
1. Do the reading(s) with an eye to the topic of the lecture (as you go, write down questions that you have about the reading(s) and try to see how the reading helps thinking about the topic of the lecture);
2. Watch the lecture video (the lecture addresses the central parts of the reading and explains how it may help thinking about the ethical issues under consideration);
3. Respond to the poll question (here you get a chance to articulate your own view about a controversial ethical issue and to clarify it in a discussion with your classmates);
4. Do the self-test (this will give you a good idea of your understanding of the central ideas explained in the lecture);
5. Respond to the discussion prompt and continue the discussion with friends, family, and others.

To illustrate how these pieces fit together, it is useful to explore the components of the Lecture 1 chapter. In the first video lecture, Professor Sandel introduces the subject of JusticeX by posing a hypothetical scenario in which a train trolley is headed towards a collision that will kill five people, but this accident can be avoided if a person throws a switch to divert the trolley, killing a single person. Following the lecture, the poll question asks if it is morally permissible to divert a train trolley in such a way that it will kill one person, but save five lives. If a person responds yes, they are asked if it would be permissible to kill a healthy person and distribute their organs to save five others. Students are also given the option to change their responses to the poll questions. The self-test questions in the Lecture 1 chapter probe students’ understanding of the definitions of “consequentialist” and “categorical” moral reasoning. The additional discussion prompt asks people to reflect on a hypothetical about using torture to extract information about a bomb planted in New York City. The accompanying reading comes from Professor Sandel’s book Justice: What’s the Right Thing to Do?, and it further explores the trolley dilemma, illustrated by an account of actual events from Afghanistan where soldiers who released two teenage goatherds found themselves soon after attacked by a large group of Taliban fighters. Together, these resources provide an introduction to an approach to moral philosophy that attempts to navigate the complexities of universal principles and specific contexts and situations.

Two chapters were released per week, and every three to six chapters, a five-question multiple-­choice quiz was released, for a total of five quizzes in all. At the end of the course, a 25-question multiple choice exam was released as well. Students needed to get 60% of all the questions correct (30 of 50) in order to pass the course.

The JusticeX course had several additional pieces of content. After every two weeks, the course staff released a “weekly forum digest” summarizing issues that arose in the forums and making suggestions for discussion norms and strategies. Professor Sandel hosted two live Q&A sessions that were recorded and made part of the courseware. In the 2013 residential course, Professor Sandel had several satellite sections participating from other universities around the world, and he engaged students in these sections in dialogue with his Harvard students in Sanders Theatre. Three of these videos were included in the courseware for JusticeX. Finally, teaching fellows hosted “office hours” using the forums three times a week, choosing dedicated times to respond rapidly in the forums.

The Students of JusticeX

For JusticeX, edX has records of 79,787 students who registered for the course. In most ways, these registrants are similar in demographic characteristics to students in the other four early HarvardX large-scale courses. The registrants were about 60% men; nearly half were in their 20s, and as a group they were highly educated. In terms of their highest completed degrees, over 70% of students had earned at least a four-year college degree. Of all registrants, 37% had a bachelor’s degree, an additional 28% had a master’s or professional degree, and an additional 5% of registrants had earned a doctorate.

Figure 1. Gender distribution of JusticeX registrants (n=72,980; 6,807 missing) and registrants from first four other 2012–2013 HarvardX large-scale courses (n= 311,080; 28,447 missing).

Figure 1

Figure 2. Educational attainment distribution of JusticeX registrants (n=72,599; 7,188 missing) and registrants from other HarvardX large-scale courses (n=309,617; 29,910 missing).

Figure 2

Figure 3. Age distribution of JusticeX registrants (n= 72,089; 7,698 missing) and registrants from other HarvardX large-­scale courses (n=308,934; 30,593 missing).

Figure 3

As with other HarvardX courses, JusticeX students came from all over the world. Nearly a third of registrants with identifiable locations came from the United States, but many students also came from India, Brazil, China, and over 150 other countries around the world.

Table 2. Counts and percentages of students country of residence among 72,990 with identifiable locations. (6,797 missing)

  Count Percent
United States 23,616 32%
India 5,182 7%
U.K. 3,560 5%
Brazil 3,348 5%
Canada 2,569 4%
Germany 1,717 2%
Australia 1,407 2%
Japan 1,374 2%
China 1,336 2%
Spain 1,320 2%

Participants began enrolling in the course as soon as registration opened, and they continued doing so even beyond the end of the course. From the opening of registration until course launch, registrations averaged 518 people per day. After the course launched, an average of 220 people registered per day.

Figure 4. Cumulative enrollment for JusticeX (n=79,787).

Figure 4

Of those who signed up for JusticeX, the degree and kind of participation in the course varied considerably. In other studies of large-scale online courses, researchers have clustered students based on assessment patterns (students who complete the first assignment, students who complete all assignments, etc.). In JusticeX, however, the emphasis of the course was much more on participation and access than on assessment or certification. Indeed, the course team considered their multiple choice assessments to be the “least bad” of the assessment options available to humanities instructors, and they placed little emphasis on quiz completion. Students were encouraged to watch videos, participate in discussions, and complete practice and for-­credit questions as suited their own interests. There are multiple meaningful ways to participate in JusticeX.

To illustrate these diverse pathways through the course, Figure 5 shows a scatterplot of participant grades versus a simple metric of course activity: the number of the 24 “chapters” or courseware sections the students opened and viewed.

Figure 5. Scatterplot of grades versus chapters viewed for JusticeX registrants (n=79,787).

Figure 5

This scatterplot of activity illustrates the diverse ways that students used the JusticeX courseware. In the top section of the figure, we see that most participants who earn a certificate viewed most or all of the course content, but some certificate earners viewed only part of the course, and a few passed the course by only viewing the required quiz and exam chapters. Some of these users may already have been proficient with the course material and only wanted expeditious certification of their proficiency. We describe this as “optimizing” behavior for certification. However, most certified students are in the top right, exhibiting patterns akin to those of a “completionist,” students who get a perfect score and view all of the content.

In the bottom right section of the figure, we show those who did not earn a certificate, but viewed more than half of the course chapters. We selected this cutoff to identify students who viewed a substantial proportion of the chapters, and we describe these students as having “explored” the curriculum. In the bottom right of the figure, we circle students who exhibit behavior consistent with “listeners”—borrowing the MIT term for a course auditor—who viewed every chapter and answered zero for-credit questions correctly. In the bottom left, we show students who viewed fewer than half of the chapters, and we describe these participants as having “viewed” the course. In the bottom left, we circle those students who never opened the courseware at all after registering for the course. The coverage of data across this scatterplot reflects the very diverse ways in which people engaged with the JusticeX courseware.

Motivated by this variation, we defined four subsamples of participants to explore in this series of HarvardX course reports. In JusticeX, Registrants consist of all 79,787 students recorded by edX as having registered for the course. Viewers are the 50,044 registrants who opened up the courseware. The courseware is the section of the course site containing the content of the course: the lecture units, videos, poll questions, and problems. Explorers are the 8,415 registrants who viewed at least half of the chapters in the courseware. Certificate Earners are the 5,442 registrants who earned a certificate in the course. As the example of the optimizer shows, not all certificate earners are explorers. To clearly distinguish among these nested groups, we identify (in Figure 6) four mutually exclusive and exhaustive sets of students who “only registered, “only viewed,” “only explored,” and were “certified.”

Figure 6. JusticeX participants clustered into four disjoint sets (not to scale) (n=79,787).

Figure 6

While these groups are unevenly distributed in numbers, they are similar in demographic composition. Compared to the overall gender distribution that was 60% male, certified students were 62% male, and “only registered” students were 58% male. “Only explored” students were slightly more educated than other groups, with 72% having a Bachelor’s degree or higher, and they were slightly older, with a median age of 31 compared to the overall median age of 28. Broadly, however, demographic characteristics did not strongly correlate with particular patterns of activity.

Figure 7. Distribution of females and students with Bachelors degree in four disjoint groups of JusticeX students (n=79,787).

Figure 7

Participation and Activity in JusticeX

Professor Sandel has a demonstrated commitment to sharing his teaching widely with the world. He kept enrollment open in his on-campus course as registration neared 1,000 students; he produced a television version of the course with WGBH; he has lectured widely on moral reasoning to popular audiences around the world; and JusticeHarvard.org provided an online version of this course years before the emergence of edX. JusticeX was designed to expand access to Justice’s ideas about the importance of moral reasoning in society and in participant’s lives. This intent is important to keep in focus in reviewing how students choose to participate in the various learning opportunities in JusticeX.

Grades and Certification

Some courses on the edX platform have maintained a commitment to evaluating students using methods that mirror, as closely as possible, the assessments used in residential course settings. JusticeX is not one of those courses. In interviews with Professor Sandel and his course team, they described the goals of their course as more focused on sharing their perspectives and sparking student interest and reflection than in evaluating, sorting, and certifying students.

In the spirit of creating a “course” rather than a lecture series, Professor Sandel did include assessments in JusticeX, and in an interview he described multiple-­‐choice questions as the “least bad” of the available option for assessing student competence at scale in the humanities. In evaluating grades and certificates in JusticeX, it is important to keep in focus the limits of the assessments used and the course team’s belief that the learning opportunities provided to “auditors” and others with no interest with the problems and certifications were as important as the experience of students who chose to engage with the for-credit problems.

Figure 8. Distribution of grades across all JusticeX participants who answer at least one for-credit question correctly (n= 12,330).

Figure 8

Of the 76,079 students who registered for the course on or before the final due of August 2, 2013, 5,442 people earned a certificate. (Of interest: two of the 133 people who registered for the course on August 2 earned a certificate.)

Many analyzing large-scale online courses are interested in the percentage of students who earn a certificate, and the corresponding inverse. As a set of purely descriptive statistics, we can take the number of students who earned a certificate (5,442) and divide that by the number of students who registered (79,787, yielding 6.8%); by the number of students who registered before the final due date (76,079, yielding 7.2%); by the number of participants who viewed the course and registered by the final due date (47,469, yielding 11.5%); or by the number of participants who viewed more than half the course and registered by the final due date (8,285, yielding 65.7%). However, no data that we have for JusticeX can allow us to identify the number of students who intended to complete the course and the proportion who actually did so.

Interestingly, there is little variation in certification rate based on when students signed up for the course. Figure 9 shows an average trend line (loess approximation) of pass rate by daily registration cohort. About 7% of the students who register on any given day pass the course, slightly lower before the launch and slightly higher after.

Figure 9. Average trend line (loess approximation) of pass rate (certified students over all registered students) by daily registration cohort (n=76,079).

Figure 9


A number of metrics can illustrate the nature and diversity of student activity in JusticeX. In Figure 10, we show the distribution of chapters viewed by JusticeX participants. Chapters represent the highest-­‐level unit in the edX courseware, including lecture units, quizzes, the final exam, and several logistical chapters like the introduction to the course and supplementary materials. The number of chapters viewed by a participant, therefore, is a rough summary of what fraction of course content they had a chance to learn from. The distribution of chapters viewed has a bi-modal distribution that is a characteristic of many distributions of usage data on the edX platform—a large number of people do very little, and then there is a second mode for students who seem interested in viewing the entire the course. Here, we see that most people only open a few chapters, but many people also open nearly all of them. (Together, Figures 8 and 10 represent the marginal distributions of the variables comprising Figure 5.)

Figure 10. Distribution of chapters viewed for JusticeX participants who viewed courseware (n=50,044).

Figure 10

We can gain another perspective on student activity in JusticeX by looking at students’ use of assessments. Justice had 24 sets of lecture review questions, five quizzes, and a final exam. Figure 11 shows that both views and attempts peak early in the course and then decline at a steady rate after the first quiz, two weeks into the 12-week course. The peaks of activity in the quizzes shows that more people viewed and attempted the quizzes than the lecture review questions.

Figure 11. Number of students viewing and attempting each problem set in JusticeX of all students who view courseware (n=50,044). Numbered sets are practice questions, Q1–Q5 are for-credit quizzes, and Final is the final exam.

Figure 11

This decline of activity over time in massive open online courses has been widely reported in early studies of the form, and it is useful to also compare these patterns of activity to other online media. For instance, Figure 12 displays the number of views, as of mid-November 2013, of the PBS Justice videos that were posted to YouTube in October of 2009. We see here similar patterns of “attrition” in watching this lecture series, where the initial video has been viewed over four million times, and then, by the fifth video in the series, views range from 200,000 to 300,000.

Figure 12. Number of views of PBS Justice videos hosted on YouTube since September 2009, as of November 2013.

Figure 12

An additional metric of student activity is a simple measure of the number of “clicks” or actions taken within the course site. Figure 13 shows the distribution of student clicks on a logarithmic scale for both certificate earners and all other participants.

Figure 13. Number of participant clicks (i.e. recorded actions) plotted on a log scale for JusticeX certificate earners (n=5,442) and non-certificate earners (n=73,345).

Figure 13

One immediate insight is that, in coarse terms, activity predicts achievement: those who do a lot in the course tend to be those who pass the course. Similar to Figure 5, this indicates that there are some certificate earners who can pass the course with only a few hundred actions, while there are also listeners who do not pass the course but take thousands of actions. In general, however, those who do many actions in the course are more likely to earn a certificate than those who do little, a simplistic insight that echoes findings from other early studies of lecture-based large-­scale online courses.


We also examined student persistence through the course. Unlike courses that begin and end on set dates, students could join JusticeX at any point. Moreover, there were no incremental due dates throughout the course—only a final due date—so late registrants could complete all of the work of the course, even those who registered on the final day of the course. The official course launch was March 2, 2013; the first content was released on March 12; the final exam was made available on May 28; and all assignments were due by August 2. Some students engaged in the course at the pace that the content was released, completing two lecture units every week from March through May. Students could also sign up for Justice in July, and, to borrow a phrase from contemporary television viewing habits, “binge-watch” the course and complete all assignments over a weekend.

Therefore, to examine student persistence, we focused more on each student’s relative timescale—the time from their enrollment or the course content launch, whichever was later—rather than on the absolute time in the course.

In Figure 14, we show an average hazard function derived by empirically calculating the hazard proportions of each weekly registration cohort and then averaging across the hazard proportion in each relative week. Thus we evaluated the questions: “What proportion of students stop participating during their first week? Their second week? (and so on…).” We can then infer the proportion of students who would remain from week to week in each cohort. This is an “implied survivor function” calculated directly from the average hazard function.

Figure 14. Hazard function comprised of the average of all registration cohort hazard functions plotted on relative course week (where week 0 is the course launch or initial registration week, whichever is later) and truncated at week 21 when the final exam was due. Implied survival function is calculated directly from the average hazard function.

Figure 14

The key insight from this figure is that, within each weekly registration cohort, students are very likely to cease activity during the week that they register and right afterwards. After that, however, hazard proportions drop to below .2 and level out below .1 after the fifth week. Colloquially, if a student gets hooked on a course within the first two weeks—regardless of when s/he starts—s/he is likely to stay, or at least the risk of dropping out in any subsequent week is constant.

This survival model gives some sense of how students persist throughout the whole course, examining the span of time from first action to last action. However, a student could log in only twice, on the first day and the last day, and be counted as “surviving” through the length of the course. This motivates an alternative metric that counts how many discrete days students view course material. Among all registrants, we find that median number of days of activity is two, and 75% of registrants have seven or fewer days of activity.

For a more granular view, therefore, in Figure 15 we examine the daily activity of those who have viewed over half the course (those who “explored”) as well as certificate earners. We see that among these active students, most had between 10 and 40 days of activity, with a long tail of students who spent over 100 days within the period from March 3 to our date of data collection on September 8.

Figure 15. Days with activity, the number of discrete days (demarked in UTC time) during the observational period where participants had at least one action, for explorers and certificate earners (n=8,415).

Figure 15

These contrasting figures illustrate the diversity of ways of participating substantially in the Justice course. Most people took an action one or two days a week over the 26 week period that we examined; however, some students participated over the course of only a few days, and still others engaged much more frequently.

Learning in JusticeX

A natural question to ask at the end of any course is: What have students learned as a result of this course? We end this report with a cautionary suggestion to avoid interpreting any of these data as answers to this question, and we call for future research that can provide more satisfying and informative answers to this crucial question.

JusticeX certification is a limited proxy for JusticeX learning, not only because Professor Sandel and his course team did not emphasize assessment and certification, but also because of our limited understanding of the incoming proficiency of any given student. A registrant who has taken the course before, or who is indeed an expert in the subject, could enter the courseware, take all of the quizzes, and leave, not having learned, but having gained a certification of existing mastery. Although this is also possible in an on-campus, residential course, the monetary and opportunity costs of enrolling in a residential program, registering in a course, and completing assignments is far higher and disincentivizes using a residential course for certification rather than learning.

With the data we have now, we have a baseline from which to track improvements in HarvardX courses, by examining ways to have more students register, to have students persist longer, or to have students engage more often in a greater percentage of the course. While these dimensions may be important, they are at best imperfect proxies for student learning.

If we want to work towards improving large-scale online learning environments in a data-­informed way, we need to continue to develop innovative ways to assess what students are learning. Deeper examinations of the discussion forums could launch a productive line of inquiry. HarvardX has begun a program of surveying registrants in courses to better understand their level of familiarity with the course material and previous experience in similar courses. In some cases, surveys include pre-tests that provide information about students’ pre-course levels of competence. Research into computational and peer assessment of student work might open new avenues of insight into student learning, and new tools—such as new systems for the online annotation of texts—could provide new data sources for students, instructors, and researchers to better gauge learning.

Despite our challenges in precisely characterizing the learning in JusticeX, we have no doubts that many students in JusticeX enjoyed rich, meaningful learning experiences. Michael Sandel has shared his insights on moral reasoning to millions through books, videos, residential courses, lectures, online courses, and now through HarvardX. His efforts provide a compelling example of how faculty members can use multiple media channels to both share ideas widely and build communities of learners engaged collaboratively in deep inquiry. HarvardX provides a unique opportunity to study one of these communities.