Statistics Filter: help me set up my coding data to calculate Cohen's Kappa as a measure of interrater reliability. Taking data from
For reasons of convenience, I had coders on my project use the CAT tool
-- which worked great, was free, enabled remote coding, and is is fairly simple. The bad news is, I had assumed the Kappa that was available as a standard comparison within CAT was Cohen's Kappa. Turns out it wasn't; it was Fleiss's Kappa, which is only appropriate for more
than two coders.
Since I only had two coders, Cohen's Kappa is the statistic I need. Sadly, there's no easy way to export my data from CAT in an SPSS-ready format, so I'll have to do so manually. My question is, what's the best way to format the data as I do this?
My scenario is this: I have two coders who coded posts in an online discussion as having/not having 12 attributes.
The easiest way I've come up with to format my data is in 2 columns, one per coder, with codes stacked down the side, like so (showing only 3 codes for each message, not 12 like I really have):
Message # Code # Coder 1 Coder 2
Message 1 Code 1 1 0
Message 1 Code 2 0 0
Message 1 Code 3 1 0
Message 2 Code 1 0 1
Message 2 Code 2 1 0
Message 2 Code 3 1 1
Message 3 Code 1 1 0
Message 3 Code 2 1 1
Message 3 Code 3 1 0
However, this demo on running Cohen's Kappa in SPSS
suggests data be formatted differently. Of course, the data in that example's a bit different from mine, and I'm a little confused as to the origin of the summarized count variable in that example.
Can anyone advise or point me to steps on prepping and executing Cohen's Kappa on binary data like mine? Or tips or other methods for calculating this?