With the prevalence of web services, a large number of similar web services are provided by different providers. To select the optimal service among these service candidates, Quality of Service (QoS), representing the non-functional characteristics, plays an important role. To obtain the QoS values of web services, a number of web service QoS prediction methods have been proposed. Collaborative web service QoS prediction is one of the most popular approaches. Based on the historical QoS data, collaborative QoS prediction methods employ memory-based collaborative filtering (CF), model-based CF, or their hybrids to predict QoS values. However, these methods usually only consider the QoS information of similar users and services, neglecting the correlation between them. To enhance the prediction accuracy, we propose a novel method to predict QoS values based on factorization machine, which leverages not only QoS information of users and services but also the user and service neighbor's information. To evaluate our approach, we conduct experiments on a large-scale real-world dataset with 1,974,675 web service invocations. The experiment results show that our approach achieves higher prediction accuracy than other QoS prediction methods.