A natural question to ask at the end of any course is: What have students learned as a result of this course? We end this report with a cautionary suggestion to avoid interpreting any of these data as answers to this question, and we call for future research that can provide more satisfying and informative answers to this crucial question.

JusticeX certification is a limited proxy for JusticeX learning, not only because Professor Sandel and his course team did not emphasize assessment and certification, but also because of our limited understanding of the incoming proficiency of any given student. A registrant who has taken the course before, or who is indeed an expert in the subject, could enter the courseware, take all of the quizzes, and leave, not having learned, but having gained a certification of existing mastery. Although this is also possible in an on-campus, residential course, the monetary and opportunity costs of enrolling in a residential program, registering in a course, and completing assignments is far higher and disincentivizes using a residential course for certification rather than learning.

With the data we have now, we have a baseline from which to track improvements in HarvardX courses, by examining ways to have more students register, to have students persist longer, or to have students engage more often in a greater percentage of the course. While these dimensions may be important, they are at best imperfect proxies for student learning.

If we want to work towards improving large-scale online learning environments in a data-­informed way, we need to continue to develop innovative ways to assess what students are learning. Deeper examinations of the discussion forums could launch a productive line of inquiry. HarvardX has begun a program of surveying registrants in courses to better understand their level of familiarity with the course material and previous experience in similar courses. In some cases, surveys include pre-tests that provide information about students’ pre-course levels of competence. Research into computational and peer assessment of student work might open new avenues of insight into student learning, and new tools—such as new systems for the online annotation of texts—could provide new data sources for students, instructors, and researchers to better gauge learning.

Despite our challenges in precisely characterizing the learning in JusticeX, we have no doubts that many students in JusticeX enjoyed rich, meaningful learning experiences. Michael Sandel has shared his insights on moral reasoning to millions through books, videos, residential courses, lectures, online courses, and now through HarvardX. His efforts provide a compelling example of how faculty members can use multiple media channels to both share ideas widely and build communities of learners engaged collaboratively in deep inquiry. HarvardX provides a unique opportunity to study one of these communities.