HarvardX and MITx: The First Year of Open Online Courses—Fall 2012–Summer 2013

2014
mm/dd/yyyy

Content

Add Comment

This report is the result of a collaboration between the HarvardX Research Committee at Harvard University and the Office of Digital Learning at MIT.

Ho, A. D., Reich, J., Nesterko, S., Seaton, D. T., Mullaney, T., Waldo, J., & Chuang, I. (2014). HarvardX and MITx: The first year of open online courses (HarvardX and MITx Working Paper No. 1).
33

Content Type: 

Format: 

Executive Summary

  • In the year from the fall of 2012 to the summer of 2013, the first 17 HarvardX and MITx courses launched on the edX platform. In that year, 43,196 registrants earned certificates of completion. Another 35,937 registrants explored half or more of course content without certification. An additional 469,702 registrants viewed less than half of the content. And 292,852 registrants never engaged with the online content. In total, there were 841,687 registrations from 597,692 unique users across the first year of HarvardX and MITx courses. (See Table 2.)
  • The most typical course registrant is a male with a bachelor’s degree who is 26 or older; however, this profile describes fewer than one in three registrants (222,847, 31%). A total of 213,672 (29%) registrants report their gender as female; 234,463 (33%) report a high school education or lower; 45,884 (6.3%) report that they are 50 or older; and 20,745 (2.7%) have IP or mailing addresses from countries on the United Nations list of Least Developed Countries. Small percentages are not small numbers. The diversity of registrants resists singular profiles; registrants are notable for their differences. (See Table 3 and Table 4.)
  • Course certification rates are misleading and counterproductive indicators of the impact and potential of open online courses. (See Figure 1.)
    • Large numbers of non-certified registrants access substantial amounts of course content.
    • Open online registration is not equivalent to enrollment in conventional courses, where traditional enrollment generally entails monetary costs, opportunity costs, and accountability.
    • Certification rates can be useful indicators when enrollments are limited. For a fixed number of registrants, higher certification rates accurately reflect larger numbers of certified registrants. For open online courses that support large-scale enrollment, there is no forced tradeoff between numbers of certified and non-certified registrants—both numbers can increase freely by design. In these circumstances, focusing on certification rates alone penalizes desirable activities like browsing and exploring courses, which open online courses are generally designed to support.
    • Pressure to increase certification rates may decrease the impact of open online courses, by encouraging instructors and administrators to suppress or restrict registration, lower certification standards, deemphasize recruitment of target subpopulations, or disregard interventions that may disproportionately increase numbers of non-certified registrants over certified registrants.
  • There are considerable differences in average demographics across courses, in terms of gender (13%–49% female), college degree attainment (54%–85%), median age (23–30), and percentage from the US (16%–36%). These differences are best appreciated in the context of the diversity of course offerings, the intentions of the instructor teams, and the outreach and dissemination efforts of course teams. In spite of average differences, all large-scale courses had hundreds of registrants with only high school degrees or who are under 15, and also had hundreds of registrants with postdoctoral degrees or who are over 50. (See Figures 3–8, and Tables 2–6.)
  • Unlike conventional courses, open online enrollment occurs continuously throughout courses, with enrollment rates rising as course launch dates approach and then declining more quickly after launch dates pass. Exploration and certification is more likely among registrants who enroll near the launch dates, but viewing likelihood is stable through the run of the courses. Course exploration and certification may benefit from synchronous course schedules and the cohorts that they build. Managing asynchronicity to maintain registrant involvement regardless of enrollment date is an ongoing challenge for instructors and a fertile area for future research. (See Table 5, Figure 8, and Figure 9.)
  • New metrics, far beyond grades and course certification, are necessary to capture the diverse usage patterns in the data. A simple comparison of grades and viewed content shows thousands of users who fit a range of profiles. Of particular interest may be those students who accessed substantial course content but did not participate in assessments. Metrics include course chapters accessed, forum usage, total numbers of “clicks,” and numbers of active days in the course. (See Table 6 and Figure 13.)
  • The average percentage of registrants who cease activity in these open online courses is highest in the first week at around 50%. The average percentage of registrants who cease activity in the second week declines sharply to 16% for registrants who persist to that point, and these percentages continue to decline over subsequent weeks. This indicates that registrants who are active after the first week have a relatively high chance of visiting again in subsequent weeks. (See Figure 12.)
  • Over four thousand registrants earned more than one certificate across HarvardX and MITx, including 1,912 who earned at least one certificate from both institutions. A total of 76 registrants earned 5 or more certificates from the first 17 courses.

Introduction

HarvardX and MITx are collaborative institutional efforts between Harvard University and MIT to enhance campus-based education, advance educational research, and increase access to online learning opportunities worldwide. Over the year from the fall of 2013 to the summer of 2014, HarvardX and MITx launched 17 courses on edX, a jointly founded platform for delivering online courses. This report is a joint effort by institutional units at Harvard and MIT to describe the registrant and course data provided by edX in the context of the diverse efforts and intentions of HarvardX and MITx instructor teams.

With the second full academic year of HarvardX and MITx well underway, and with more than double the number of course offerings in the pipeline, this report is an opportunity to guide course development and set baseline expectations for Year 2 results. It is also an opportunity to deepen public understanding of the considerable diversity between and within university efforts to support open online learning. Companion reports from HarvardX and MITx describe individual courses in greater depth and include, among other details, the differing learning goals that different instructors had for their students. We strongly encourage readers to review these reports as a package to understand the full story of the HarvardX and MITx initiatives in their first year.

These first 17 courses were released on the edX platform over a period of rapid development, expansion, and change. Systematic data collection protocols and integrated tools for randomized controlled experiments were initially not in place. Although research capabilities have advanced considerably since this first year, data from these first courses cannot provide definitive answers about what works in open online education. Nonetheless, there are plentiful indications that this is an unconventional data source of considerable potential, and herein descriptive results will be important to guide the design of future experiments and the interpretation of future results.

We begin the next section by elaborating on this new data context and providing a framework for interpreting the figures and findings to come. We then reflect on differences and similarities among the first HarvardX and MITx courses, before turning to statistics describing meaningful subpopulations of participants. We present registration, certification, demographic, enrollment, geographic, and activity data across all the first HarvardX and MITx courses, and then conclude with perspectives gained from the study.

I. Interpreting Findings from a New Enterprise

In accordance with the research missions of HarvardX and MITx, both Harvard and MIT have convened groups charged with advancing research. Harvard University Provost Alan Garber convened the HarvardX Research Committee, a body comprised of 15 faculty members from around the university. MIT President Rafael Reif announced the Office of Digital Learning, whose mission includes a commitment to research on digital learning. Together, these two bodies have been working to understand, organize, analyze, interpret, and disseminate the data that edX delivers to its founding institutions.

With hundreds of thousands of registrants, open registration, asynchronous use of course resources, and a rapidly evolving platform, familiar educational variables like “enrollment” and “achievement” were challenging to operationalize. Registration into HarvardX or MITx courses hosted on the edX platform requires nothing more than a few keystrokes and a few clicks. Course registrants are accountable to no one and use course resources whenever and however they wish. Terms like “student,” “grade,” and “course” nonetheless bring to mind conventional analogs in higher education, and related terms like “enrollment” and “completion” similarly trigger specific interpretations. We emphasize and demonstrate that this educational data context differs substantially from that of any course where course registration costs more time and money than a few seconds and zero dollars. This perspective is consistent with some of the growing literature on MOOCs. [1]

With these challenges in mind, we offer caution in the form of four common fallacies that we perceive as particular threats to the interpretation of data from large open online courses.

1) We have all the data we could want.

The edX platform collects a large amount of data, approximately 20 GB of data per course. However, many variables that may interest researchers were not collected systematically in this first year. Examples include socioeconomic status, prior knowledge, motivations for enrolling in particular courses, detailed video interaction behaviors, and externally validated assessments of student learning. These variables are also rare in on-campus college courses, and one of the promises of HarvardX and MITx is the potential for more rigorous research on learning for on-campus courses as well. Online systems may make it easier to collect relevant data for research; however, the ability to log detailed online interactions does not necessarily confer upon the data any educational or policy relevance.

2) A small percentage is a small number.

In a new data context, interpreting the magnitude of numbers is challenging and subject to “framing”: the tendency for interpretations to differ depending upon an initial frame of reference. If the number of certificate earners in an open online course is 1,000, is that a large or small number? From an on-campus frame of reference, a professor may take years or decades to teach 1,000 students. From an online frame of reference, 1,000 is vanishingly small compared to the sizes of many online populations.

Percentages seem to address this problem by providing a frame of reference for comparison. If 100,000 students register, then one might expect that 100,000 have the opportunity to become certified. If 1,000 are ultimately certified as completers, then 1,000/100,000=1%, and this seems small. We argue that this is misleading. There is no doubt that course certification numbers are important indicators of the impact of an open online course offering, however the diversity of possible uses of open online courses make certification percentages problematic.

As one of many anecdotes that illustrates the problem with certification percentages, consider the evening of July 24, 2013, when Anant Agarwal, the president of edX, appeared on the Colbert Report, a satirical news show hosted by the comedian Stephen Colbert on the Comedy Central television network. Figure 1 plots day-to-day registration cohorts as a solid thick line and shows that enrollment in HarvardX courses [2] more than tripled after the broadcast, with 406 registrations on Wednesday, July 24 (UTC) to 1356 registrations on Thursday, after the Colbert Report broadcast. The numbers of these registrants who ultimately become certified in a course are shown as a thin solid line. The five-day average before the broadcast was 12 certified registrants per day, and the five-day average after the broadcast was 24 certified registrants per day, a doubling of certification numbers.

Of course, if certification doubles but registration triples, certification rates will drop. The bottom half of Figure 1 illustrates this slight drop, from 3.2% to 2.5% in the five-day average. Clearly the courses did not suddenly change in quality, rather, the audience changed in average composition. Yet we do not think that any instructor, researcher, or policymaker should begrudge Stephen Colbert for tripling registration and doubling certification. An increase in the number of registrants who are not ultimately certified can decrease certification rates, but if it is accompanied by an increase in the absolute numbers of registrants who learn, we argue that it should be regarded positively.

Figure 1. Daily number of registrations in HarvardX courses from June 24 to August 23, with the broadcast date of Anant Agarwal’s appearance on The Colbert Report shown (July 24, 2013). The number and percentage of these registrants who become certified are also shown.

Figure 1

3) Certification indicates learning.

While certificates are easy to count, certification is a poor proxy for the amount of learning that happens in a given course. Many registrants engage in courseware without choosing to complete the assessments for credit. And certification is difficult or impossible for registrants who register late or after the course closes. This is part of the explanation for the low certification rates shown in Figure 1, particularly in the August timeframe, when certification for most courses was no longer possible. That so many registrants register and participate in courses without hope of earning a certificate illustrates how limited certification and certification rates are at describing learning.

Noncertified registrants may have learned a great deal from a course, and certified registrants may have learned little. Some registrants may already be experts and may merely wish certification, a situation that is rare in residential education because of the larger monetary and opportunity cost students bear when registering for a course. More generally, instructors have limited assessment capabilities and grading options compared to residential courses. And protections against academic dishonesty are still limited. These challenges do not render the assessment of learning an impossible challenge, but they should limit interpretations of certification or certification rates as a proxy for registrant or course-wide learning.

4) A course is a course is a course.

This report reviews courses that differ dramatically on multiple dimensions. Beyond the most obvious difference of course content, there are structural differences in the design and duration of courses. There are essential contrasts in the philosophy of the instructors and the expectations of the registrants. Instructors took dramatically different approaches to video design and distribution. Approaches to assessments and criteria for certification differed widely. Although MITx courses have more in common among them, structurally, than HarvardX courses, we emphasize that the diversity among HarvardX and MITx courses reflects the diversity of the curriculum of their parent institutions and is considerable.

We intend comparisons of certification rates, gender ratios, grade distributions, and relative activity to reflect the variation in course content and design, as well as variation in registrant background and intention. These metrics should not be misinterpreted to indicate that a course, its instructors, and its registrants are somehow “better” than others on any dimension. Such comparisons are at best unsupported by the data and at worst obviously incorrect.

 

II. Differences Among the First HarvardX and MITx Courses

Table 1 lists courses and key dates for the 17 courses from the first year of HarvardX and MITx. We abbreviate courses for convenience and exposition. Some abbreviations are a matter of course convention, like HarvardX’s CS50X, HeroesX, and JusticeX, and others are a shorthand used only in these reports, like SSChem-2, referring to the second offering of 3.091x: Solid State Chemistry, at MITx. The time periods between registration openings and course launches, and course launches and course wrap dates, differ considerably across courses. As registration windows and the length of the course are likely to influence initial registration numbers and other course statistics, these are important contextual features to appreciate. Some courses also remain open long after the final due date of all materials, and enrollments continue to rise even though certification is no longer possible. This asynchronicity is a key feature of open online course design and usage. This report uses data up through September 8, 2013, a date after the close of all 17 courses.

Table 1. The first 6 HarvardX and 11 MITX courses from the 2012–2013 academic year, their instructors, and key dates.

Table 1

These courses are notable for their differences, from the humanities to the sciences, from small courses (CopyrightX instructor William Fisher capped enrollment at 500) to the most registered course thus far on all of edX, CS50X: Introduction to Computer Science. The MITx courses generally had a similar structure, running 12–15 weeks and with relatively similar numbers of problems, videos, and e-text pages. HarvardX courses differed more from each other on a number of dimensions, including course length, enrollment, and the relative emphasis on course components like video, assessments, and forums. The individual course reports for HarvardX and MITx provide more information about the differences among courses.

Because CopyrightX was a limited-enrollment online course, we do not include it in subsequent comparisons and refer interested readers to the HarvardX report (Fisher, 2014). Tables and figures that concern online activity also exclude the HarvardX computer science course, CS50X, because the instructor, David Malan, ran a substantial proportion of his course on a platform (cs50x.org) that provided alternative activity statistics. We refer interested readers to his slide deck reviewing the course (Malan, 2013).

III. Descriptive Statistics

We identify four subpopulations of interest within each course, to illustrate the considerable variability in registrants by their actions. Figure 2 provides a screenshot of the edX interface to illustrate these distinctions. Among registrants, we define those who “viewed” the course as those who accessed the “courseware” tab on edX, the leftmost tab in Figure 2, where the core content of the course, including video and assessments, generally reside. Note that it is possible to view course information, the syllabus, and discussion forums, and still not “view” the course, by this definition. Among those who “viewed” the courseware, we define those who “explored” as those who accessed half or more of the “chapters” in the courseware. Figure 2 also illustrates chapters, which are the highest organizational units in the courseware. To become a “viewer,” a registrant must merely “click” on the courseware. To become an “explorer,” the viewer must click on content within half or more of the chapters.

Figure 2. Screenshot of the edX interface from HeroesX, with the “courseware” shown on the left. Registrants must click on the “courseware” to meet the definition of “viewed,” and they must click on half or more of the “chapters” (the primary tabs on the left, in this case, CB22X: Coming Soon, Hour 0, Hour 1…) to meet the definition of “explored.”

Figure 2

The cutoff that distinguishes viewers from explorers is arbitrary, but it is one way to identify registrants who appear to access substantial amounts of the courseware. The sidebar to the right further identifies mutually exclusive and exhaustive categories of course registrants. Of particular interest may be those who “only explore,” that is, those who access substantial amounts of the courseware but appear to be uninterested or unsuccessful when it comes to certification. Note the difference between “explored” and “only explored,” where the former overlaps with certified registrants and the latter explicitly excludes them. The last category describes certified registrants, those whose weighted average score, expressed as a percentage, is above the instructor-selected minimum cutoff percentage. This cutoff ranges from 50% to 60% across MITx courses and from 50% to 80% across HarvardX courses.

sidebar image

IV. Registration and Certification

Table 2 displays total course enrollments as measured by the number of registrants in the course. Across these first HarvardX and MITx courses, 43196 registrants were certified, and another 35937 registrants explored more than half of the courseware chapters but were not certified. A total of 469702 registrants viewed less than half of the courseware chapters, and 292852 registrants never entered the courseware at all. In total, there were 841687 registrations across the first year of HarvardX and MITx courses.

Table 2. Registrants, membership of registrants in mutually exclusive registration categories, and exploration/certification rates.

Table 2

Average course certification rates are 62% among registrants who explored the course, 9% among registrants who viewed the course, and 6% among all registrants in the course. Note that these unweighted, average percentages are different from rates that weight by the number of registrants in the course, which can generally be calculated from the raw numbers in Table 2. Unweighted averages, as presented, prevent large courses from proportionately influencing summary statistics and better describe the typical course than the typical registrant.

As we argued earlier, course certification rates are difficult to interpret without understanding whether and which registrants are making an informed commitment to become certified. Further, certification is a poor proxy for learning without the benefit of pretest data or other baseline measures. Information about informed commitment was not available when these courses were administered, and pretest information was not collected systematically across courses. HarvardX now has a common survey administered across courses, and MITx has common instruments available to course instructors, so future annual reports can better disaggregate results by stated initial intentions.

Figure 3 displays results from Table 2 and illustrates numbers of registrants in particular categories. The CS50X course has more than double the registration of other courses, although it has also been open for registration the longest, since July 24, 2012, and it remains open now. Again, this reflects the asynchronicity that is a hallmark of many open online courses. MITx courses had fewer registrants than HarvardX courses on average. This may be explainable in part by the specialized audience for some of the more MITx courses, as indicated by their prerequisites.

Figure 3. Numbers of total registrants in HarvardX and MITX courses, by registrant category. (CS50X numbers of only viewed and only registered are 102K and 67K, respectively.)

Figure 3

V. Demographics

Registrant variation on common demographic variables far exceeds variation in residential universities, both within and across courses. Table 3 lists summary statistics about education, age, and gender, and Table 4 lists counts and percentages of registrants from subpopulations that illustrate the variation in registrant backgrounds.

Table 3. Demographic information for registrants and certificate earners. Percentages exclude missing data (3–5% for gender, 3–6% for education, 4–8% for age).

Table 3

Table 4. Selected subpopulation counts and percentages of nonmissing data by course and for all HarvardX and MITx.

Table 4

Education

Figure 4 shows the percentage of registrants and certificate earners, respectively, who have completed a bachelor’s degree, a master’s degree, or a doctorate. Certified registrants generally had a higher average educational level than noncertified registrants. These percentages exclude from the denominator both missing data and completed educational levels reported as “other.” Figure 5 shows the full distribution of registrants’ completed educational levels, ranging from “none” to “doctorate.” Percentages of registrants who choose not to report their education range from 3–6% across courses.

Figure 4. Percentage of registrants who reported completion of a bachelor’s degree or higher, among all registrants (light gray) and certificate-earners (dark gray) who report their education level in each course. (Nonresponse rates from 3–6% across courses.)

Figure 4

Figure 5. Distribution of completed educational levels reported by all registrants, by course. (Nonresponse rates from 3–6% across courses.)

Figure 5

The median completed educational level across all courses was “bachelor’s” with one exception: HealthStat certificate earners had a median completed educational level of “master’s.” The two courses with the highest percentages of registrants with earned doctorates were HealthStat and Biology:7.00x, with percentages near 10%. Among those with lower levels of completed education, Table 4 and Figure 6 also show that the percentages of registrants who have, at most, a high school degree or less range from 15% (HealthStat) to 43% (MechRev:8.MReV). In many technical courses, around 5% of registrants have only completed junior high/middle school or less. This reinforces our earlier point that seemingly small percentages of registrants with low educational attainment can be a large number (over 8000 registrants without high school degrees for CS50X, over 3000 for CS-1:6.00x and CS-2:6.00x).

Age

Figure 6 shows the age distribution in 2013 as estimated by the birth year provided at edX registration. The median age for all registrants is below 30 in every course, but the median age for certificate earners is higher than that of all registrants in all courses. The median age of certificate earners is particularly high in HeroesX, at 36. As might be expected from the course topics, as well as the relationship between age and educational attainment, the average HarvardX registrant is both older and has a higher completed educational level than the average MITx registrant. HeroesX and Biology:7.00x had particularly broad age distributions. All age distributions show considerable positive skew, in particular in HeroesX and JusticeX, with more than 10% of registrants in their 50s and above, and around a quarter of registrants who are in their 30s. Table 4 lists numbers and percentages of registrants with ages 15 and younger and ages 50 and older.

Figure 6. Distributions of the reported ages of registrants in each course as “box and whisker” plots, with the median reported age of all registrants (horizontal line) and certificate earners (diamond) shown. Box borders are the 25th and 75th percentiles. Whiskers terminate at the 5th and 95th percentiles. (Nonresponse rates from 4–8% across courses.)

Figure 6

Gender

Figure 7 shows the gender distribution across courses. On average, science, technology, engineering, and mathematics (STEM) courses have lower percentages of female registrants than courses that are not conventionally described as STEM courses (including HealthEnv, HeroesX, GlobPov:14.73x, and JusticeX). In many STEM courses, certificate earners are less likely to be female than registrants overall, although the magnitudes of distributional gender differences between certificate earners and registrants are small.

Figure 7. Percentage of registrants reporting a gender of female in each course, for all registrants and certificate earners. (Nonresponse rates from 3-5% across courses.)

Figure 7

VI. Enrollment

Figure 8 displays one of the many ways in which open online course registration differs from conventional models. Although most registrants enroll before the launch of the course, there is considerable variability across courses, driven in large part by variation in preregistration windows. Table 5 shows the percentages of registrants who register before course launch, during the course, and after the course closes and certification is generally not a possibility. Many of these latter percentages are small only because the course closing date was close to our data collection horizon of September 8, 2013. Many of these courses remain open to registrants but closed for certification, which means that course certification percentages will continue to decline over time. This is another demonstration of the short-sighted, misleading nature of certification percentages as evaluation metrics.

Figure 8. Distribution of enrollment dates in days relative to course launch, for all registrants by course. Median registration day for certificate earners is shown as a hollow diamond. Box borders are the 25th and 75th percentiles. Whiskers terminate at the 5th and 95th percentiles.

Figure 8

Table 5. Enrollment distributions and related statistics showing wide variation in enrollment times relative to course windows.

Table 5

Figure 8 also illustrates that there are small differences in the median registration times between certificate earners and registrants overall. Figure 9 shows overall enrollment, average view rates, explore rates, and certification rates, all referenced to the launch date of courses, across all HarvardX and MITx offerings. The likelihood of certification as well as exploration is highest for registrants enrolling near the launch date. The enrollment pattern also shows overall acceleration in registration in the weeks closest to launch dates, although these vary across courses. In contrast, viewing likelihood is stable after course launch dates. This suggests that exploration and certification benefits from synchronous course schedules and the cohorts that they build. Managing asynchronicity to maintain registrant involvement regardless of enrollment date is an ongoing challenge for instructors and a fertile area for future research.

Figure 9. Enrollment by week relative to launch (above), with the percentage of registrants who ultimately view, explore, and are certified, across all HarvardX and MITx courses (below).

Figure 9

VII. Geography

Columns 4 through 6 of Table 6 begin to describe the geographical distribution of registrants and certificate earners. The country was located by IP address or, if the IP address is missing, the country was located by the parsed mailing address submitted at initial edX registration, if possible. Column 4 lists percentages of registrants who access course content primarily from the United States, and Column 6 lists corresponding percentages of certificate earners. Column 5 lists percentages from India, the #2 country by enrollment in almost all courses. Figure 10 ranks the top 25 countries by the numbers of registrants in HarvardX and MITx and shows the numbers and percentages of registrants accounted for by these countries. More recent registration data by country and course are available online (Nesterko, 2014).

Table 6. Percentage of registrants with modal IP addresses or mailing addresses from the USA and India (10% unidentifiable), activity statistics for forum posts, and median total numbers of active days, disaggregated by viewed vs. explored or certified registrants.

Table 6

Figure 10. The top 25 countries, by numbers of registrants, for all HarvardX and MITx registrants. The country was located by modal IP address or, if the IP address is missing, the country was located by the parsed mailing address submitted at initial edX registration. This process estimates countries for 90% of registrations.

Figure 10

Table 7 lists the top 30 countries by the percentage of registrants certified, among the 77 countries with greater than 1000 registrations. The United States is not shown and ranks 55th. Again, certification rates must be interpreted with caution. The relatively low US certification rate indicates more browsing and less certification among US registrants on average. However, registrants are clearly non-representative of countries, and certification rates, as we have argued, are distinct from achievement under conventional conditions. As the Colbert Report example demonstrates in Section I, cross-country differences are due at least in part to the background and interests of the registrants who happen to find their way to the registration page. Nonetheless, these data establish a baseline from which instructors and administrators may set future recruitment and certification goals.

Table 7. The top 30 countries, by certification rate, for countries with registrations > 1000.* Numbers of registrations by country and percentages of total HarvardX- and MITx-wide registrations also shown.

Table 7

VIII. Activity

Table 6 also shows selected activity statistics for each course. CS50X is excluded, as CS50X course activity was logged on a separate platform. The number of “clicks” is operationalized as the number of events (e.g. video plays, e-text page accesses, problem attempts, forum posts) in the server log files. This serves as a rough indicator of the total discrete actions that a user takes in a course. This number is unsurprisingly much higher for registrants who explored or were certified. The “active days” metric is a similar effort at describing a user’s activity in a particular course. It is simply the number of discrete days, demarcated in UTC time, that a user engages in some type of course activity.

For all registrants who viewed a course, the typical registrant accessed course content over two to five separate days, depending on the course. For registrants who explored or were certified, the typical registrant accessed course content over 24 to 63 separate days, depending on the course. Variation across courses is related to the length of the course and the amount of content, among other factors.

Discussion forum usage rates range across courses. Table 6 shows numbers of registrants who post one or more times in the discussion forums. Figure 11 displays the percentages of course “viewers” who post at least once in the forum, which range from 6.5% of viewers in Circuits- 3:6.002x to 25.7% in JusticeX and 33.3% in Biology:7.00x. Among certified registrants, discussion forum usage is more prevalent, with percentages ranging from 37% in Circuits- 3:6.002x to 62% in HeroesX and 70% in Biology:7.00x. Distributions of forum usage show extreme positive skew, with small but substantial numbers of registrants posting frequently. Excluding instructors and staff, dozens of registrants had post counts in the hundreds, and a few had post counts in the thousands.

Figure 11. Forum participation: percentages of “viewed” registrants and certified registrants with 1 or more forum post, by course.

Figure 11

Figure 12 shows percentages of registrants whose last actions in a course are in a given week, referenced by the beginning of the course or the registration enrollment date, whichever is sooner. This is known in some fields as a plot of “hazard probabilities” that describe the rate of attrition in a given time period, for individuals who have persisted to that time period. In this context, the plot describes the week-to-week percentages of registrants whose last action in a course is in that particular week. The plot shows largely similar patterns across courses, where, on average, half of registrants in the first week have their last activity in that week. In subsequent weeks, the percentage of students who remain are unlikely to have their last activity in any remaining week. This pattern continues until the ends of courses (not shown due to variability in course durations), where percentages rise again as courses end, and viewers and explorers are active for the last time. The plot indicates that registrants who are active after the first week have a fairly low chance of leaving in subsequent weeks.

Figure 12. Average percentage of active registrants whose last action in a course is in a particular week. For example, among registrants still active by Week 5 of a course, around 10% will have their last action in that week. Only includes registrants enrolling before the third week of the course. CS50X is not included. Selected individual course estimates are shown in gray to illustrate course-to-course variation. The course-level average percentage is shown in black.

Figure 12

Finally, Figure 13 begins to demonstrate how course activity metrics can tell a more complete story about opportunities to learn than assessments can alone. Figure 13 plots distributions of the percent of course chapters accessed by students along the horizontal axis. On the vertical axis is the course grade, adjusted in a piecewise linear fashion so that 0% and 100% retain their meaning but 60% is the passing cutoff for all courses. Grades of 0% and 1% are excluded in the histogram to avoid distorting the scale but are included in the scatterplot. Every registrant is represented as a dot on this plot, with “only registered” registrants at the origin, “only viewed” registrants in the lower left quadrant, certified students above the horizontal line, and “explorers” to the right of the vertical line. The registrants in the lower right quadrant are of particular interest and generally overlooked, we argue, by short-sighted certification rates. However, the overall story concerns the vast numbers of students who exist in all regions of this scatterplot, representing the immense diversity of learning approaches that registrants take in these open online courses.

Figure 13. Distributions of course activity (in terms of the percentage of chapters accessed) and course grades (for grades above 1%, linearly adjusted across courses to a common certification cutoff of 60%). CS50X is not included.

Figure 13

IX. Conclusion

This report offers summaries of descriptive statistics about the courses, registrants, and activity in HarvardX and MITx over their first year. We have already placed considerable emphasis on the limitations of these data. Here, we conclude with general recommendations, reflecting in part on our experiences not only with these quantitative data but with course teams designing courses. These recommendations may help to guide both research and the design of technology-mediated learning environments.

HarvardX and MITx registrants are not “students” in a conventional sense, and they and their behavior differ from traditional students in K-12 and post-secondary institutions. Registration requires no cost or commitment, thus traditional metrics, like certification rates and enrollment rates, miss many new facets of course engagement, such as skilled learners dropping in to learn one specific aspect of a course. This report and its companion reports are an effort to broaden the discussion and perspective on open online learning, and the data encourage more nuanced consideration of broadly used terms like “students” and “learning.”

Registrant activity differs considerably within and across courses. Registrants are engaging with courses in diverse ways, and many instructors are deliberately building courses that honor diverse forms of participation. Certificate earning is one possible learning pathway. Others include simply watching videos or reading text. Some registrants sample a couple of chapters and then take their interests elsewhere, only to register in other courses and sign up for the second instance of courses. Some registrants focus on assessments to test themselves. Nearly any way that one can imagine a registrant using a course to learn is actually revealed in the data. Certification rates are a misleading representation of this diversity.

There will be no grand unifying theory of MOOCs. A national discussion has emerged over the past two years concerning Massive Open Online Courses (MOOCs). Our results suggest that describing MOOCs as though they are a monolithic collection of courses misses the differential usefulness and impact MOOCs may have from sector to sector. Courses from professional schools like the Harvard School of Public Health exemplify how strategies are likely to differ. That these registrants are more highly educated and that higher percentages of registrants from outside the US should hardly be surprising, and the public policy implications of these efforts should be evaluated in a context differently than an introductory computer science course. The implications of courses in each sector are different and need to be considered in context.

Given how different some of these courses and sectors are, their commonalities are surprising. In every course, people use resources in diverse ways. In every course, we see registrants who are active but not assessed, assessed but hardly active, and those who do both to extremes. Regardless of course and enrollment times, most registrants leave within a week or two of their entering the course, but remaining registrants are far less likely to leave in subsequent weeks. Certification rates are similar on average and in their variability across HarvardX and MITx, in spite of the substantial differences between HarvardX and MITx courses.

Asynchronicity is a defining feature of open online learning, with implications for how we study it. Open enrollment periods and unrestricted use of course resources raise important questions for analysis and design. Registrant trajectories through the course depend upon at least three timeframes, 1) a registrant-oriented timeframe that references each registrant’s enrollment date, 2) a course-oriented timeframe that references curricular milestones in the course, and 3) a calendar-oriented timeframe that acknowledges days of the week, holidays, and weeks in the year. Longitudinal research in these courses requires specification of the time or times relevant for analysis, and results are likely to depend on the choice.

Measuring learning requires a greater investment in assessment and research. This problem is inherited from parent institutions. Some fields have well established large-scale assessments, but most areas of higher education do not. Online courses can offer rich, real-time data to understand and improve student learning, but current data describes activity more often than learning gains or desired future outcomes. We need to invest more in high-quality, scalable assessments, as well as research designs, including pretesting and experiments, to understand what and how registrants are learning.

Open online courses are neither useless nor the salvation of higher-education. Large-scale, “low-touch” learning platforms will have sectors and niches where they are very useful and others where they are less so. Our understanding of tradeoffs and our ability to identify new opportunities will improve with continued research. Thoughtful instructors and administrators in schools and universities will take advantage of resources that can be saved by using these technologies and redeploy those resources to places where “high touch” matters. The results we present here and in companion reports can begin to frame this discussion, as well as set a baseline for evaluating the expansion of our efforts that is already well underway.

Notes

[1] An overview of relevant literature follows. The pilot edX course has been studied by Breslow, et al. (2013), and DeBoer, et al. (2014) use data from the same course to argue for the reconceptualization of educational variables. Kizilcec, et al.(2013) motivate and implement a nuanced student classification approach, and Koller, et al. (2013) suggest that certification should be considered in the context of learner intent. Emanuel, et al. (2013) show that 83% of survey responders taking University of Pennsylvania open online courses have 2- or 4-year college degrees, and Perna, et al. (2013) find persistence and achievement rates below 15%. Good reviews of press coverage and selected research findings are found at Nature and by Watters (2013).

[2] MITx courses reveal a similar pattern but are excluded because some MITx courses were opening and closing enrollment during the time period of interest.