Preparing data for Cohen's Kappa in SPSS

July 14, 2011 6:48 PM

Statistics Filter: help me set up my coding data to calculate Cohen's Kappa as a measure of interrater reliability. Taking data from CAT to SPSS.

For reasons of convenience, I had coders on my project use the CAT tool-- which worked great, was free, enabled remote coding, and is is fairly simple. The bad news is, I had assumed the Kappa that was available as a standard comparison within CAT was Cohen's Kappa. Turns out it wasn't; it was Fleiss's Kappa, which is only appropriate for more than two coders.

Since I only had two coders, Cohen's Kappa is the statistic I need. Sadly, there's no easy way to export my data from CAT in an SPSS-ready format, so I'll have to do so manually. My question is, what's the best way to format the data as I do this?

My scenario is this: I have two coders who coded posts in an online discussion as having/not having 12 attributes.

The easiest way I've come up with to format my data is in 2 columns, one per coder, with codes stacked down the side, like so (showing only 3 codes for each message, not 12 like I really have):

Message # Code #     Coder 1     Coder 2
Message 1 Code 1              1      0
Message 1 Code 2              0      0
Message 1 Code 3              1      0
Message 2 Code 1              0      1
Message 2 Code 2              1      0
Message 2 Code 3              1      1
Message 3 Code 1              1      0
Message 3 Code 2              1      1
Message 3 Code 3              1      0

However, this demo on running Cohen's Kappa in SPSS suggests data be formatted differently. Of course, the data in that example's a bit different from mine, and I'm a little confused as to the origin of the summarized count variable in that example.

Can anyone advise or point me to steps on prepping and executing Cohen's Kappa on binary data like mine? Or tips or other methods for calculating this?

posted by activitystory to Education (3 answers total)

I would not advise starting with the summary table format, since it is much harder to check errors or do anything else with the data. In long format, it is trivial to switch it into the table with the crosstab menu. You're going to get a kappa per-attribute since it is only for mutually exclusive categories.

As an aside, CAT claims to be able to export to standard formats like CSV. It seems plausible that a little tinkering should allow you to do what you need. I've never used it.
posted by a robot made out of meat at 7:07 PM on July 14, 2011

Export from CAT to CSV and import into Excel then import into SPSS.
posted by k8t at 7:20 PM on July 14, 2011

Response by poster: Thanks for the responses-- I had already tried to import the CAT-exported CSVs into SPSS, but was having trouble since they seem so strangely formatted. If you've done this before: was there an easy way to transform the data to something workable?
posted by activitystory at 8:06 AM on July 15, 2011