TESTAR is an open source tool for automated software testing that generates test sequences on the fly based only on information derived from the Graphical User Interface (GUI). At the core of TESTAR is the way to automatically select which actions to test; finding the right algorithm to carry out this task can make significant differences to the testing outcome. In this work we evaluate Q-learning as a metaheuristic for action selection and carry out experiments with a range of paramenters, using random selection as a baseline for the comparison. Two applications are used as Software Undder Test (SUT) in the experiments, namely MS Powerpoint (a proprietary desktop application) and the Odoo enterprise management system (an open source web-based application). We introduce metrics to evaluate the performance of the testing with TESTAR, which are valid even under the assumption that access to the source code is not available and testing is only possible via the GUI. These metrics are used to perform statistical analysis, showing that the superiority of action selection by Q-learning can only be achieved through an adequate choice of parameters. Mots-Clefs. Automated GUI Testing,Testing Metrics, Testing Web Applications, Q-learning