Utilization of the coding framework presented some particular challenges when implemented with the 6.002x discussion forum data. One significant challenge—that of interpretation based on the researcher’s own bias—is inherent in all qualitative data analysis (Miles & Huberman, 1994) and was also present as we resolved differences between coders after they coded the same posts. In our research context, where posts were dissociated from the thread in which they occurred, it became very easy to infer the motive of the poster based on personal interpretations. For example, the role of the poster in the comment, ‘I got 1.25 and it still isn’t right,’ could be viewed as information-giver, but it could also be viewed as an information- or help-seeker because the student obviously hadn’t arrived at the correct answer. It was very tempting to speculate the circumstances under which this post was made. Yet another perspective might be to see this post as social/affective, voicing frustration or confusion over repeated attempts to correctly solve the problem. We soon realized that from the latter perspective, many posts could be classified as social/affective. The fact that students were posting at all could be interpreted to mean that they were attempting to make a connection with other students.
Our resolution to these dilemmas was to clarify what our codes could or could not distinguish and what this meant for our future research questions. Some posts were simply difficult to put in a particular ‘bin.’ During our check-coding discussions, we began to approach these situations by thinking about what type of information would be needed to answer our future research questions and what was not relevant to code for. For example, when considering our future research question regarding factors that may be related to students’ persistence or achievement in the course, we would not be able to accurately assess the relationship of social/affective communication to those outcomes if we had liberally assigned that code to any post in which we inferred that students were reaching out to make social connections. Similarly, attributing emotion or an affective component to a post due to our own interpretation of the poster’s tone would also add error to our future analysis. We agreed to rely only on information overtly displayed in the post itself, which for the above examples, meant that the poster stated an emotion-laden phrase such as “I really love this course,” or made an overt social gesture such as “Would anyone like to join a group that meets in person?” rather than to guess or infer anything about the context or motivation behind the post. Although we may have lost some information by making this choice, we felt that it strengthened the reliability of our coding in addition to reducing potential error from simply being wrong about what the poster intended.
Additionally, our experience reinforced the importance of orienting coders to the coding framework with actual data, not just for training purposes, but as a chance to get to know the character of the posts and understand how the abstract notions on which the codes are based are realized in the forum (i.e., how people ask for help, norms regarding off-topic postings, etc.).