Data

Our data sources for this analysis were student posts retrieved from a database that saved every student post along with any subsequent edits on the discussion forum. Access to the discussion forum was prominently displayed as a tab on the course site, and a section entitled ‘How to use the question/answer forum’ was available on the course information page. After clicking on the discussion forum tab, students were directed to a page on which initial posts, known as ‘threads,’ were visible. This page could be sorted by date so that more recent threads appeared first; by activity so that threads with the most recent responses appeared first; by answers so that threads with the greatest number of responses appeared first; or by votes so that threads with the greatest number of votes appeared first. When sorting the threads by votes, students could choose to sort by number of ‘up-votes,’ which indicated student appreciation for the post, or ‘down-votes,’ which indicated their dislike for the post. There was also a search function enabling students to peruse the forum for topics of interest.

When we retrieved the data in November, a total of 96,696 posts were made by students during and subsequent to the course. For this study, we used only the 90,441 posts that were made during the 14 weeks of the course from March 4 through June 15, 2012. Of the 108,008 students who clicked at least once on the course site, 8,894, or slightly over eight percent of them posted at least one time on the discussion forum. The distribution of posts per student was positively skewed (Mean = 10.23, SD = 44.66, Median = 2, Mode = 1), with the total posts per student ranging from 1 to 1442 for those who posted at all. Each student post was time-stamped and identified so that complete conversations could be re-constructed. The data were de-identified and the time stamp was removed prior to initiation of our analysis.

Phase 1 – Development of Coding Framework

Our research question, ‘What is the topic of students’ posts and the role assumed by posters on an open discussion forum?’ provided the conceptual framework for this study. Our approach entailed classifying the topics into meaningful categories so that they could be quickly located and clustered in ways that would facilitate future research. To accomplish this, we applied codes1 that described the primary themes of the posts.

Our first task was to gain perspective regarding the appropriate number and types of categories that should be used to describe the posts. This inductive methodology, referred to as open coding, is commonly used at this stage to identify, describe, or categorize phenomena found in qualitative data (Corbin & Strauss, 1990). A committee of five researchers performed open coding on a series of randomly selected threads to gain a general awareness of the type of communication that occurred in this space. Our findings from this examination were wide-ranging; we noted student exchanges containing advice for studying course material, questions and responses related to understanding course content or accessing course materials, commiseration with fellow students about difficulties experienced in solving homework problems or in using the course technology, requests for group formation in other online sites, e.g., Facebook, and even comments related to students’ physiological states, indicating hunger or sleep deprivation.

Following this first review, we developed a tentative list of descriptive codes that would encompass the topics and roles we had noted. Creation of codes that possess conceptual and structural order is important to the viability of a coding schema. Miles and Huberman (1994) emphasize that codes should relate to one another in coherent, study-important ways. We chose codes that we posited to be associated with our student outcomes of interest—persistence and achievement. We began with four codes for topic—content, user interface/course structure, community-building/interpersonal, and nonspecific affective—and two codes for role—help-seeker and help-giver.

After defining or operationalizing our codes, we trialed them on randomly selected posts. It immediately became apparent that we needed to expand this framework and refine our definitions. We added the code ‘tangential topic’ to describe posts in which students appeared to be transferring their understanding of circuits and electronics to other contexts. We also combined the two codes—community-building/interpersonal and nonspecific affective—to create one code—social/affective—because we found these posts to be closely related. We developed separate codes for user interface/course structure, which we renamed course website/technology and course structure/policies because there appeared to be two distinct types of posts and we hypothesized that each type may show different association with our outcomes of interest. We developed the codes ‘missing data,’ for posts had been removed by course staff, and ‘non-English,’ for posts that appeared in other languages. For role of the poster, we added ‘other’ as a code to be applied to posts in which students did not ask for or receive help or information, but instead expressed an opinion.

We elaborated upon our former and newly added definitions, and then trialed the revised framework on an additional 500 posts. This trial was conducted so that each of the five researchers coded a total of 200 posts, with 100 of those overlapping posts coded by another researcher, and the remaining 100 overlapping those coded by a different researcher. This type of procedure, known as check-coding, allows researchers to identify code definitions that need elaboration, or even codes that need to be re-defined (Miles & Huberman, 1994). It also allowed us to evaluate inter-coder reliability or degree of agreement between coders. For this round of coding, the inter-coder reliability, calculated via percent agreement, ranged from 75% to 91% between the five possible combinations of researchers for topic of the post, and from 76% to 86% for role of the poster.

Following this trial, we each met with the two other researchers who coded the same posts to discuss discrepancies in our coding. We continued to refine our code definitions until we reached agreement on a framework that contained well-defined codes for all of the types of posts we encountered. During this process, we changed the ‘tangential topics’ code to ‘other coursework’ to categorize posts in which students made references to courses other than 6.002x. We found that identifying posts that discussed topics tangential to course content was difficult; we reasoned that identifying posts in which students referred to other courses may also allow us to determine some transfer of course understanding to other contexts. To further clarify the meaning of our codes, we found specific examples of posts related to each one and incorporated all of the information into a printed document that we referred to as our ‘codebook.’

The product of our efforts was a two-dimensional coding framework that would classify posts according to topic of the post and role that the poster assumed. We defined eight subcategories for topic and three subcategories for role of poster as shown in Tables 1 and 2 (See Appendix A for further detail).

Table 1. Topic of Post

Code Definition
1. Content Posts specifically addressing circuits and electronics material
2. Other coursework Posts discussing courses other than circuits and electronics
3. Social/affective Posts addressing social, emotional, or community-building aspects of the class
4. Course website/technology Posts that addressed the online interface
5. Course structure/policies Posts regarding the course organization, guidelines, or requirements
6. Other Posts conveying anything not related to class content, other courses, social aspects of the class, course website or technology, or course requirements
7. Missing data Posts in which data had been censored by course staff (in these cases, particular thread numbers could not be located)
8. Non-English Posts written in other languages

 

Table 2. Role of Poster

Code Definition
1. Help-seeker (or Information-seeker) Posts in which the poster asked for help, information, pointers, etc.
2. Help-giver (or Information-giver) Posts in which the poster gave help, insight, or provided information
3. Other Posts in which the poster was not explicitly seeking, declaring, or providing information, such as an opinion

As we trialed each iteration of the coding framework, we had to make several important decisions about our coding protocol. First, we found it necessary to allow dual codes to encompass the complexities of the topic and role dimensions. This meant that two codes could be applied to categorize the topic of one post, and two additional codes could be applied to describe multiple roles assumed by the poster. In a single post, it was not uncommon to find a response to another student’s question about course content, an expression of emotion about understanding or not understanding that content, as well as a general question to other students regarding whether they would give a different answer to the question. In that illustration, the student’s response to another student’s question about content would be coded as ‘help-giver’ for role and ‘content’ for topic; their expression of emotion would be coded as ‘social/affective’ as a second topic; and their question about alternate answers would be coded as ‘help-seeker’ as a second role.

Our second important decision was that we would code posts independently of the thread in which they appeared. When students began a conversation about a particular topic, they started a new ‘thread’ on the forum. Each thread contained a series of posts, in which other students commented on or responded to the original question or statement. Our preliminary work involved coding only the first post of each thread because we believed this would identify the topic of that particular conversation. However, we found that one thread often contained multiple topics and that the topics were not always accurately identified in the first post. Often students’ original statement of the problem was a misdiagnosis. The most common examples of this were threads in which students attributed difficulties with homework to technology errors that were actually their own errors in calculation or in understanding course content. In order to gain a more accurate sense of the number of posts in each of our coding categories, we elected to code at the individual post level rather than at the thread level. This decision also allowed us to gain more information regarding help- or information-givers because the first posts in threads were typically initiated by students asking questions, and the help- or information-giving role became more evident in subsequent posts within the same thread.

Phase 2 – Coding of Posts by Independent Coders

A further check of our coding framework was conducted by handing it off to independent coders to determine if our categories were comprehensive and our category descriptions provided adequate guidance for coding to those who were unfamiliar with the data. Inherent in this phase was selecting the sample of posts to be used for coding. Realizing that it would be impossible for a human to code each of the 96,696 posts in any timely fashion, we chose to code slightly under five percent, or 4,500 posts. In our preliminary work, we noted that the central focus of students’ posts changed as the course progressed. To capture this within the data, we divided the course into four time periods (quarters) and randomly selected 1125 posts from each quarter. We did not select posts that occurred after the end of the course, even though students continued to communicate with each other on the site well over a year after the course officially ended. We also noted that students tended to post more frequently at the beginning and end of the course than in the middle. To gain a more accurate estimate of posts described by each of our coding categories, we will weight the number of codes per category in each time period by the proportion of the total posts that occurred within that time period.

Two graduate students who were not on our original committee were recruited to code the 4500 posts. After their initial training and check-coding, these individuals would conduct the final coding of the data. The training consisted of orienting them to the codebook and discussing examples of each code. Following the training session, each coder was given 2500 posts, with 500 of the posts overlapping so that we could check for inter-coder reliability. The coders were instructed to code 32 of the overlapping posts from each of the four time periods (128 total for each coder), using codes from what we considered to be our final coding framework. Following completion of this first round of coding, we checked for inter-coder reliability by calculating percent agreement and found it to be only 59% for topic and role combined. We met with the coders to discuss difficulties in utilizing the codebook and to resolve disagreements. We made clarifications to the code definitions, and the coders then applied codes to another 128 posts. For this second round, they reached 81% agreement for topic, and 73% agreement for role of the poster. The coders then returned to the first 128 posts and re-coded, this time reaching 73% agreement for topic and 72% for role of the poster. Although there is no agreed upon threshold for appropriate inter-coder reliability (Campbell, Quincy, Osserman, & Pedersen, 2013), these percentages fall within a range considered acceptable for this type of data (Fahy, Crawford, Ally, Cookson, & Keller, 2000). Following the third round of coding and check-coding, the coders were instructed to code the remainder of their 2500 posts.

Lessons Learned

Utilization of the coding framework presented some particular challenges when implemented with the 6.002x discussion forum data. One significant challenge—that of interpretation based on the researcher’s own bias—is inherent in all qualitative data analysis (Miles & Huberman, 1994) and was also present as we resolved differences between coders after they coded the same posts. In our research context, where posts were dissociated from the thread in which they occurred, it became very easy to infer the motive of the poster based on personal interpretations. For example, the role of the poster in the comment, ‘I got 1.25 and it still isn’t right,’ could be viewed as information-giver, but it could also be viewed as an information- or help-seeker because the student obviously hadn’t arrived at the correct answer. It was very tempting to speculate the circumstances under which this post was made. Yet another perspective might be to see this post as social/affective, voicing frustration or confusion over repeated attempts to correctly solve the problem. We soon realized that from the latter perspective, many posts could be classified as social/affective. The fact that students were posting at all could be interpreted to mean that they were attempting to make a connection with other students.

Our resolution to these dilemmas was to clarify what our codes could or could not distinguish and what this meant for our future research questions. Some posts were simply difficult to put in a particular ‘bin.’ During our check-coding discussions, we began to approach these situations by thinking about what type of information would be needed to answer our future research questions and what was not relevant to code for. For example, when considering our future research question regarding factors that may be related to students’ persistence or achievement in the course, we would not be able to accurately assess the relationship of social/affective communication to those outcomes if we had liberally assigned that code to any post in which we inferred that students were reaching out to make social connections. Similarly, attributing emotion or an affective component to a post due to our own interpretation of the poster’s tone would also add error to our future analysis. We agreed to rely only on information overtly displayed in the post itself, which for the above examples, meant that the poster stated an emotion-laden phrase such as “I really love this course,” or made an overt social gesture such as “Would anyone like to join a group that meets in person?” rather than to guess or infer anything about the context or motivation behind the post. Although we may have lost some information by making this choice, we felt that it strengthened the reliability of our coding in addition to reducing potential error from simply being wrong about what the poster intended.

Additionally, our experience reinforced the importance of orienting coders to the coding framework with actual data, not just for training purposes, but as a chance to get to know the character of the posts and understand how the abstract notions on which the codes are based are realized in the forum (i.e., how people ask for help, norms regarding off-topic postings, etc.).