Thank you for presenting your paper.  It's always nice to see quantitative studies around CATs as they are rare.  Have you considered adding a section to the end of your paper discussing potential limitations of the study.  For these types of studies there can be other variables that can produce corrleation between the studied variables.

For example, is it possible that "more commited" students who care about their grades would volunteer to participate in the CATs exercise?   As participating in the CAT could be viewed as "extra work".  Those not willing to do more or extra work might then fall into the non-cat group and their work ethic could potentially affect their final score.

I don't have a lot of experience with this type of study, but that might seem worth considering and perhaps even exploring in future research. 

Just a thought.  Thanks for sharing your research.  This is worthy topic and particulary important in non-traditional higher education.

Jason

Hi Jason. You bring up an excellent point. I beleive that this is a very valid limitation for this study. We were originally going to compare this to courses that did not use CATs at all, but we changed it to a caomparison within courses that use CATs. More research definitely needs to be done, but we have found, at least unofficially, that students have been performaing better in courses since we began implementing CATs less than a year ago. We have further studies planned. Thank you for pointing out this limitation. I will definitely be adding this to the discussion.

Rick

Good work on this CATs essay! I think that doing periodic spot checks in the classroom is essential to assess student understanding of the course objectives. The only challenge I gather from doing this in the online classroom, is that if it is not required, then the students may not do it. In my experiences, the majority of the students do not participate in any CATs because they only get possible participation credit for it, and they can get participation credit elsewhere in the discussion forum. In the traditional setting, an instructor can actually ask the students and get them to respond verbally. It is more difficult as an instructor to get online students more involved in challenging themselves as far as participating in CATs. Meaning, do we offer additional incentives so more students partake in CATs? For instance, giving an extension on a weekly submission if replying to a CAT could be an incentive. I am not saying this is a good idea, I am just suggesting an example of a possible incentive that online students may find beneficial.

Hi Kristen. In our format, it is not a requirement to use the CATs, however many of us do see a substantial portion of the class participate. I think it is more about how interesting or meningful the CAT is. We need to think about their purpose each time we use them. I do not see us getting to 100% participation, but I have seen many CATs get about 50-75% of the class involved.

Jason and Kristen,

   Thank you for your feedback. It is good to hear we are on the right track with this study. We found the same concerns as you did, Jason. There are just not very many quantitative studies regarding classroom assessment techniques. Formative assessment can be a difficult element to quantify; however, we feel that this was a good place to start. Initially, we are working on incorporating these formative assessment techniques to take a proactive approach to our lessons in the classroom. We want to identify any areas of need or possible remedial instruction ahead of time prior to students submitting their final assignment or summative assessment.  I feel that providing additional incentives may take away from the goal of these techniques. However, this might be an interesting study in itself. If we had a  control class and that of an "incentives based" class, what might the difference in student success be?

-Emily

This was a very interesting paper about introducing an assessment technique designed to determine if student's have a complete understanding of what is expected of them in a given assignment.  As an online instructor, this review is from the perspective of an online environment. 

It has been a frequent occurrence in online courses for students to rush through assignment instructions and not attend to the minutia detailing what precisely is expected of them.  The CAT provides a chance to for students to pause and attend to details they may otherwise miss.  Could a CAT be implemented as a required checklist to complete prior to activating the ability to submit an assignment?  In essence, the could the CAT unlock the drop box in an online environment?   It would seem that just the process of making the student stop and think prior to submission could be very beneficial, particularly based on the research results demonstrating there was a significant increase in grades for students who completed the CAT. 

Note that there was a minor grammatical error noted on p. 6 in the first line of the paragraph entitled "Data Collection".  Otherwise, this was a very compelling presentation of research related to coming up with creative ways to assess what students are really grasping in terms of what is expected of them.

Jennifer

This is an interesting article and provides direction in use of and the results of use of Classroom Assessment Techniques (CATs). This study provides an opportunity for the audience to review an example of a particular CAT and the impact that a specific CAT could have on student achievement. This will benefit future students and online instructors two-fold; first in providing an example for CATs use and second in indicating the benefits of CATs.

I would have also been interested in knowing how the CAT was explained before administering and further recommendations for practice resulting from the study. One final editing for mechanical and APA issues could also be a plus.

Rick, Emily, and Shelia this was obviously a study that was needed.  I commend you for taking on this study. 

As I read it, I found one area where it could be stronger.  There is a basic assumption that the group that declined the CAT and those that used it were equal at the beginning of the study.  Do you have any additional information indicating that is the case?  What would have happened had you randomly assigned about half of the 39 that agreed to participate to a control group that did not use the CAT's?  Would the difference in gpa between the two groups of volunteers be as large as it is?  Or, did the act of volunteering to participate separate higher performing learners from lower performing learners?

This study certainly raises some important questions that demand further study.  Thank you for your work in shedding some light on this topic for all of us.

John

Jennifer, thank you for your comments. By using CATs in the online classroom, we give the opportunity to add support for student understanding. We also give ourselves the opportunity to check for understanding before the summative assessment. We have seen a growing amount of participation in our CAT postings and we are continually adding to our base of strategies.

Sherion, the CATs are placed in the discussion forums as an extra discussion question and opportunity to earn participation points. One idea of CATs is that they should be added in a way that students do not really know that they ae being assessed, necessarily. We do not do much explaining before implementation because of this.

John, thank you for your thoughts. I like your idea of a control group. We have identified a limitation as the possibility that lower performing students do not participate as fully in the CATs. This will be added to our final draft. This has been something that we are trying to ontrol as well, since we need to find a way to get the low performing students to take part in the formative assessments. Further research is continuing in this area as well. Thank you again for your thoughtful comments.

Rick

Rick,

In a campus classroom we often try to engage marginal students with a personal conversation after class in the office or hallway.  What do you think would be the online equivalent?  I have tried sending messages through the Individual Forum and weekly feedback in the gradebook.  However, they often generate no response.  I expect they would be part of the group that chose not to participate.  I appreciate your efforts to learn more about this common problem that all of us face.

John

Hi John. Great question. As Online Full Time Faculty, we reach out on the telephone quite often. We also use the Individual Forum and Gradebook text boxes, as you have mentioned. One other opportunity that we have as OFTF is to contact the students' counselors. We have found that some students are intimidated by instructors at the beginning of a course. But, persistence is the key and once they realize we are trying to help them become successful, this usually gets them to communicate more freely.

Emily, Rick & Sheila, I appreciate this study on CATs and how a specific CAT, namely the Misconception/Preconception Check, can help students better understand the requirements of a summative assignment. Below are some suggestions/thoughts on the study. I have placed them in what I perceive to be priotity order. 

1. Since you are using a One-Way ANOVA (between subjects), I would suggest changing your dependent variable from a 4 point scale (i.e., GPA) to a 100 point scale. This type of statistical test is meant to be used with continuous data. Therefore, changing the students' grades into percentages that range from 0 to 100 would help better justify using a One-Way ANOVA here.
2. I agree with several of the other comments above. There seems to be a major confounding variable that potentially exists here. Is the CAT actually helping the students or is the CAT an added element in the course that differentiates the better students from the weaker ones? Numeorus studies have shown that students who participate more online tend to do better; therefore, this "CAT" may just act as a filter that preselected the good students from the weaker students. An additional benefit to this study might be to examine the "CAT participants" grades on previous assignments with the "non-CAT participants" to see if there is a strong correlation. If there is not then your CAT may have made a statistical difference. If there is then it is probably just acting as a filter. 
3. Under Data Collection you mention you used two data sets. Perhaps it would be helpful to identify them as dependent and independent variables instead. 
4. It might be better to use "the findings suggest" rather than "the findings show" in the last sentence of your abstract.       

It is great to see an updated version of a CATs.  Is it your intent to present this to other on-line universities? Although I know I used the same techniques at my previous colleges, it is alarming that the research is over a decade behind so to speak. I am concerned as such techniques can often be copied or reworded by a few members of the same class. (Personal experience with my own students) I wonder if you have considered taking CATs to the next level and incorporate student conducted CATs? Each week have a few different students research, conduct, and apply? Maybe I am am reaching too far? Again, I am very pleased to see current CAT trends in the on-line classroom being addressed. 

Thanks for presenting this research study as it will help both students and instructors understand each other's perspectives and expectations. In addition to reviewers' comments above-students' performance in the classroom can easily be viewed by their instructors to determine their levels of participation. Therefore, could that be the reason why only 40 percent or 36 out of 96 (as stated in your study) students participated in CAT study, because students who were not doing well in their assignments or participations might not view CAT beneficial, thereby, not motivated to participate? Second, formative assessment could help potential struggling students prepared more than summative assessment techniques -that way such students will have the opportunity to decide their course of approach in bringing their grades up to speed.

Thanks,

J.Ibeh Agbanyim

This research is impressive and reflects work completed in the arena of distance learning during the last decade.  Most academics agree that replacing the majority of quantitative assessments with qualitative measures enhances student learning abilities, while encouraging the learner to achieve higher levels of student learning outcomes. 

Hello all. It is wonderful to see so many insightful recommendations and thoughts. This is certainly providing assistance in fine-tuning this article.

Scott-There are some great suggestions here. Just for clarification, would the GPA scale from 0.0-4.0 not be continuous as well? In addition, I do think that there are so many opportunities with this concept in terms of research as there has not been much exploration of this topic in online modalities. We may find that CATs can be an identifier of a specific group of students; however, I do think that there must be further data collected.

Leisha-Thank you for your comments. I am to be a part of this attempt to explore more current trends of this topic. I do think that in the online modality, we have a different set of challenges to face like your example. Students will certainly see others' answers when CATs are posted in the open forums. I love the thought of student generated CATs. There is actually a CAT that is called Student Generated Test Questions (CAT # 25). This is quite similar to what you are suggesting. There is great power in having students identify what they feel is important from your lessons. This will give you an idea of whether you were teaching what you thought you were teaching. Great idea.

-Emily