A new digital watermarking evaluation and benchmarking methodology using an external group of evaluators and multi‐criteria analysis based on ‘large‐scale data’

Digital watermarking evaluation and benchmarking are challenging tasks because of multiple evaluation and conflicting criteria. A few approaches have been presented to implement digital watermarking evaluation and benchmarking frameworks. However, these approaches still possess a number of limitations, such as fixing several attributes on the account of other attributes. Well‐known benchmarking approaches are limited to robust watermarking. Therefore, this paper presents a new methodology for digital watermarking evaluation and benchmarking based on large‐scale data by using external evaluators and a group decision making context. Two experiments are performed. In the first experiment, a noise gate‐based digital watermarking approach is developed, and the scheme for the noise gate digital watermarking approach is enhanced. Sixty audio samples from different audio styles are tested with two algorithms. A total of 120 samples were evaluated according to three different metrics, namely, quality, payload, and complexity, to generate a set of digital watermarking samples. In the second experiment, the situation in which digital watermarking evaluators have different preferences is discussed. Weight measurement with a decision making solution is required to solve this issue. The analytic hierarchy process is used to measure evaluator preference. In the decision making solution, the technique for order of preference by similarity to the ideal solution with different contexts (e.g., individual and group) is utilized. Therefore, selecting the proper context with different aggregation operators to benchmark the results of experiment 1 (i.e., digital watermarking approaches) is recommended. The findings of this research are as follows: (1) group and individual decision making provide the same result in this case study. However, in the case of selection where the priority weights are generated from the evaluators, group decision making is the recommended solution to solve the trade‐off reflected in the benchmarking process for digital watermarking approaches. (2) Internal and external aggregations show that the enhanced watermarking approach demonstrates better performance than the original watermarking approach. © 2016 The Authors. Software: Practice and Experience published by John Wiley & Sons Ltd.