---
title: "Inter-rater reliability: gesture form similarity coding"
author: "Marlou Rasenberg"
date: '(this version: `r format(Sys.Date())`)'
output:
  github_document:
    toc: true
  html_document:
    toc: true
    toc_float: true
editor_options:
  chunk_output_type: console
---

```{r global_options, include=FALSE}
knitr::opts_chunk$set(echo=TRUE, warning=FALSE, message=FALSE, fig.path='3_out/')
```

```{r packages, results='hide',include=F}
# Packages
list.of.packages <- c("dplyr", "irr")
new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
if(length(new.packages)>0) install.packages(new.packages)
lapply(list.of.packages, require, character.only=T)
rm(list.of.packages,new.packages)

```

This script supports the paper 'The primacy of multimodal alignment in converging on shared symbols for novel referents' by Marlou Rasenberg, Asli Özyürek, Sara Bögels and Mark Dingemanse (for the CABB team). To increase readability, not all code chunks present in the .Rmd source are shown in the output.

## Aim
This script is used to compute inter-rater reliability for gesture form similarity coding. We compare coder's judgments for:  
* handshape  
* movement  
* orientation  
* position  
Note that handedness of gestures was already coded in earlier stages; inter-rater reliability for handedness is computed in the script 'reliability_gesturecoding'.

## Data
As input we use 'reliability_gestureformsimilarity.csv', which contains form similarity coding of two coders for N=108 gesture comparisons (=25% of all gesture comparisons for which form similarity was coded).  
The key columns for the current analysis are 'gesture_similarity_coding_V' and 'gesture_similarity_coding_J' (one for each coder, both were student assistants). The annotations in those columns specify whether the gestures of the gesture pair were considered similar (1) or different (0) with respect to each form feature. The code '9' is used for missing values, i.e., when it was not possible to code similarity.  
E.g., "H1P1S0M0R1" means there was overlap in handedness (H), position (P) and orientation (R), but not in handshape (S) and movement (M). 

```{r read in data}
df <- read.csv2("1_data/referential_task/inter_rater_reliability/gesture_form_similarity/reliability_gestureformsimilarity.csv")
```

## Subsetting data
The column 'include' signifies whether gesture comparisons were included in the analysis (see supplementary_analysis_gestureformsimilarity). 
Gesture comparisons where both gestures had the same Fribble subpart are included (1), those who had 'main' or '-WF' (N=5) are not (0). The latter should thus be filtered out here:

```{r subset data}
df_subset <- subset(df, include==1)
nrow(df_subset)==nrow(df)-5
```

## Analysis

### Handshape
```{r handshape}
table(df_subset$handshape_V, df_subset$handshape_J)
agree(df_subset[, c("handshape_V", "handshape_J")]) #88.3
kappa2(df_subset[, c("handshape_V", "handshape_J")]) #0.712
```

### Movement
```{r movement}
table(df_subset$movement_V, df_subset$movement_J)
agree(df_subset[, c("movement_V", "movement_J")]) #85.4
kappa2(df_subset[, c("movement_V", "movement_J")]) #0.632
```

### Orientation
```{r orientation}
table(df_subset$orientation_V, df_subset$orientation_J)
agree(df_subset[, c("orientation_V", "orientation_J")]) #75.7
kappa2(df_subset[, c("orientation_V", "orientation_J")]) #0.539
```

### Position
```{r position}
table(df_subset$position_V, df_subset$position_J)
agree(df_subset[, c("position_V", "position_J")]) #77.7
kappa2(df_subset[, c("position_V", "position_J")]) #0.468
```

```{r session info, include=F}
sessionInfo()

# R version 4.0.2 (2020-06-22)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows 10 x64 (build 19043)
# 
# Matrix products: default
# 
# locale:
# [1] LC_COLLATE=Dutch_Netherlands.1252  LC_CTYPE=Dutch_Netherlands.1252    LC_MONETARY=Dutch_Netherlands.1252 LC_NUMERIC=C                      
# [5] LC_TIME=Dutch_Netherlands.1252    
# 
# attached base packages:
# [1] stats     graphics  grDevices utils     datasets  methods   base     
# 
# other attached packages:
# [1] irr_0.84.1     lpSolve_5.6.15 dplyr_1.0.0   
# 
# loaded via a namespace (and not attached):
#  [1] Rcpp_1.0.7       crayon_1.3.4     digest_0.6.25    R6_2.4.1         lifecycle_0.2.0  magrittr_1.5     evaluate_0.14    pillar_1.4.4     rlang_0.4.11    
# [10] rstudioapi_0.11  ellipsis_0.3.1   vctrs_0.3.0      generics_0.0.2   rmarkdown_2.2    tools_4.0.2      glue_1.4.1       purrr_0.3.4      xfun_0.14       
# [19] yaml_2.2.1       compiler_4.0.2   pkgconfig_2.0.3  htmltools_0.4.0  tidyselect_1.1.0 knitr_1.28       tibble_3.0.1  
```


