An investigation of inter-operator reliability tests for real-time analysis system.

Sports performance data has been measured and collected by a variety of methods including different measurement tools or systems in the field of performance analysis of sport. Lapse-time analysis (LTA) has often been used rather than real-time analysis (RTA) because of the difficulties of data gathering, presentation of the results, non-feasibility of feedback during match time and the selection of valid performance indicators (PIs). RTA data has often been used to enhance performances and, therefore, it is necessary to investigate the reliability tests used on such data. Pearson’s r, Chi-square, % error and Kappa tests statistics were used to evaluate the reliability between independent observers. A peer review process of modelling different levels of reliability with synthetic data demonstrated that kappa was the only one of the four statistics to have construct validity for the purpose of reliability assessment. With the Cybersport system for basketball, kappa values for the system as a whole of 0.8 or above are interpreted as good while values of 0.6 to 0.8 are acceptable. When using kappa for post hoc reliability analysis of individual event types, values of 0.5 or above can be considered as acceptable.