Homes are becoming an ecosystem of digital devices and appliances, which can be interconnected and controlled. This interconnection can be facilitated by a central smart hub on which home automation software is deployed. Commercially available hubs, while easy to install and use, often support a limited set of devices and protocols, and have a high total cost of ownership. Open-source home automation systems provide an affordable and open alternative, bringing support for devices and services that are unsupported by commercial alternatives. In recent years, the number of available open-source home automation systems has increased drastically. Each system comes with its own set of functionalities and limitations, making choosing a specific solution challenging, as a wrong decision may be costly. In this work, we overview 20 of the prominent open-source home automation systems, from which we select the four most promising ones. To evaluate and compare these systems, we identify key features from a set of use cases and extract specific features for home automation. This results in a two phase study. In the first phase, we perform a use case based analysis based on the extracted features. In the second phase, we perform a criteria-based analysis with 34 criteria that covers aspects such as setup time, quality of documentation, pricing, and hardware requirements. We also identify the commonalities in the architecture that emerge from the systems. The results help to identify the strengths and weaknesses of the various systems and can help the developer and the practitioner make an informed choice when selecting an open-source home automation solution.