The First Design Cycle
Initial design of the observation protocol began by watching videos of high school mathematics classrooms that had high amount of in-class student work. As we watched the video, we began to record student behaviors that we found relevant to the question of measuring students’ use of resources to resolve a problem. This list was then supplemented with behaviors that we had observed in our own experience. This list of behaviors became our open codes:
Teacher identifies resources
Teacher provides instruction on use of resources
Teacher does not create opportunities for other resources (lecture/quiz/etc)
Teacher answers question that could have be redirected to class/student
Student asks for more student-student interaction
Student asks for more work time
Student requests information on resources
Student requests instruction on use of resources
Student asks question of teacher
Teacher creates opportunities for students to ask questions of each other
Teacher asks open-ended question
Students ask questions of each other
Students share observations with other students
Students observe and learn from each others’ approaches to problems
Student builds on another student’s comment
Student challenges another student’s comment
Student expresses uncertainty to another student
Student waits for teacher to come along to solve the problem
Internet access restricted
Student passively uses assigned resource (read text/watch video)
Internet access made available
Teacher provides students with control of reference (textbook/video/etc).
Teacher assigns creation of reference (journal/wiki/etc).
Student reviews textbook
Student reviews video tutorial
Student flips back in notes/handout
Student reviews student created reference
Students create connections in student created reference
Student uses website
Student suggests or brings in resource
Student asks metacognition/reflection questions
Student connects problem to out of classroom experience.
Student demonstrates intrinsic motivation to learn material
Student creates a sub-problem for themselves
Student reorganizes or re-represents the problem (ex: draws a diagram)
Student categorizes problem by underlying mathematical principle
Student makes their own observations beyond what is planned by the teacher
Student makes their own problems beyond what is planned by the teacher
Student proposes an experiment
Initial axial coding and first trial coding
Our initial organization of the data involved classifying the open codes into major categories based on who the actors were, and sub-categories based on the type of interaction:
Student-Teacher
Teacher identifies resources
Student requests instruction
Student-Student
Teacher opens collaboration
Students share observations
Students ask questions of each other
Students observe each other
Students (build on/challenge) each other’s comments
Student expresses uncertainty to another
Student-Reference
Student uses external reference
Student-Internal
This initial coding scheme was used to code classroom video from the Trends in International Mathematics and Science Study (TIMSS) website. The videos were coded using a form designed in the iCoda iPad app. iCoda is software tool for field observation in which a form consists of “buttons.” Whenever an event corresponding to a code is observed to start, a button is pressed to activate it, and then pressed again to deactivate the button. iCode records the start time, end time, and title for all buttons pressed during an observation. Early versions of our form no longer exist, but a sample later version can be seen in Figure 6. The resulting data was studied using analysis software written in-house by the authors.
Figure 1: Example code histogram of code frequencies (counts)
Figure 2: Sample timeline showing the time of occurrence of each code during the class video. Vertical axis is code (in the same order as the histogram in figure 1). Horizontal axis is time.
Second axial coding and first trial coding
During this design cycle and discussion, a second way of organizing the codes emerged, which led to a second axial coding of the codes. We decided that scoring a class based on student actions wasn’t reasonable in situations where the instructor didn’t give the students an opportunity to act. For example, in the timeline above (Figure 1), you can see a long period of time at the beginning of the class when the teacher is introducing the problem (The leftmost black bar representing “student-teacher” interaction).
In this second organizing scheme, we placed the observed behaviors in two dimensions. The dimension was our original top-level category of the actors, while the second dimension was based what kinds of opportunities the students had to use resources, and whether or not they took those opportunities. The new axial coding scheme looked something like (Figure 3).
Figure 3: Table of results for two-dimensional axial coding
Figure 4: Count histogram under two-dimensional axial coding scheme
Figure 5: Example timeline under two-dimensional axial coding scheme.
The Second Design Cycle
After the initial design cycle, we found that there was a dissonance between focusing on pairs of actors in one dimension, and opportunities in the second dimension because opportunities involved only a single actor. We redesigned the codes to combine these two categories into a single resources dimension.
Teacher is resource
Teacher points to resource
Student uses teacher
Student uses student
Student uses object
However, we still wanted to make a commentary on quality of content, so we added a new second dimension based on Depth of Knowledge (DoK) levels in an attempt to track when students were relying on the teacher for simple recall situations or when their need required more analysis or deeper thinking. The resulting iCoda form can be seen in Figure 6.
Figure 6: iCoda form for cycle 2. Each button corresponds to a code. The observer presses a button to activate the code, and again to release the button, and iCoda records start and end times.
Figure 7. Comparison of count and duration histograms for the same class, showing very different pictures of the classroom.
We began to observe a large number of overlapping event instances; this meant that our initial approach of counting code instances was not viable. Because each button only had an on-off state, overlapping instances would be recorded only as a single event. For example students asking for help often overlapped with each other, leading to a misleadingly low count of student’s asking for help. We compensated for this by making a move to tracking the duration of a code rather than counting instances (Figure 7). This showed us a very different picture of the classroom, where the students spend a lot of time waiting for the teacher to come help them.
A second problem was the interface of the redesigned form. Decisions identifying actors could be made very quickly, (as soon as the actor began speaking), while decisions about depth of knowledge required more insight into the content. This led to a time lag where the observer would often know which code to choose in the resources dimension long before they knew which DoK level was appropriate to code for. After this iteration and the challenges presented by the additional coding of DoK, the decision was made to remove this feature from the coding.
Figure 8: Redesigned iCoda form separating the two dimensions to account for time lag.
The Third Design Cycle
At this point, our codes became relatively fixed. The third design cycle was primarily about negotiating the meanings of individual codes. Due to the limited number of TIMSS videos, we moved to coding classroom video from two high school algebra classes in Arizona.
Figure 9: Sample inter-rater reliability analysis for code “Student uses Student.” Blue bars are the codes by the two raters. Black represents times where raters agree there is no code. Green represents times when raters agree there is a code. Orange and magenta represent times of disagreement when only one rater coded. This particular example shows 49.082% agreement between coders.
Initial inter-rater reliability tests with the form had low ratings (Figure 9). We returned to the video to explore times when the coders disagreed and began negotiating the meanings of codes.
Teacher is resource:
The primary disagreement here was whether teacher is resource was only lecture, or whether a teacher responding to a question from a student counted as teacher is resource. Essentially the disagreement was about whether teacher as resource must always be teacher initiated. We decided for the later option: That any time a teacher acted as a source of information, the teacher was acting as a resource, regardless of who initiated it.
Teacher points to resource:
The primary disagreement here was whether assigning an activity that required a tool (such as a calculator) counted as teacher points to resource, because students were expected to use (and be introduced to) the tool. Unlike the case above, we decided that teacher points to resource needs to be defined as the teacher suggesting the use of a tool that was not part of a lesson. For example, if a lesson is set up in such as way that the student is expected to use a graphing calculator as a means to solving a problem, we would not code this as “teacher points to resource.” We would, however, code the use of a graphing calculator as “teacher points to resource” if the calculator was not part of the lesson plan.
Fourth Design Cycle
In our third trial, we tested both the electronic form and a new paper form (Figure 10). The paper form was used as a checklist. During observations, the observer kept a timer, and every minute would check the boxes corresponding to any code events during that minute. The paper form also had a section for the observer to write notes on their observations.
Figure 10. Sample paper form, showing timeline and accompanying notes.
The team universally preferred the paper form for three reasons. First, coding minute by minute instead of second by second gave the team more time to think and analyze while coding. Second, the paper form allowed for the creation of a timeline without specialized software; and third, the team liked the opportunity to add notes to the timelines. These notes served as both explanations for an observer’s’ particular choice of code, and reminders to the observer of what happened for future discussion. The paper form became the sole tool used to gather date during the final inter-rater reliability run.
