What is the Impact of Bad Layout in the Understandability of Social Goal Models?

The i* community has published guidelines, including model layout guidelines, for the construction of models. Our goal is to evaluate the effect of the layout guidelines on the i* novice stakeholders' ability to understand and review i* models. We performed a quasi-experiment where participants were given two understanding and two reviewing tasks. Both tasks involved a model with a bad layout and another model following the i* layout guidelines. We evaluated the impact of layouts by combining the success level in those tasks and the required effort to accomplish them. Effort was assessed using time, perceived complexity (with NASA TLX), and eye-tracking data. Participants were more successful in understanding than in reviewing tasks. However, we found no statistically significant difference in the success, time taken, or perceived complexity, between tasks conducted with models with a bad layout and models with a good layout. Most participants had little to no prior knowledge in i*, making them more representative of stakeholders with no requirements engineering expertise. They were able to understand the models fairly well after a short tutorial, but struggled when reviewing models. In the end, adherence to the existing i* layout guidelines did not significantly impact i* model understanding and reviewing performance.