Interactions of users and items can be naturally modeled as a user-item bipartite graph in recommender systems, and emerging research is devoted to exploring user-item graphs for collaborative filtering methods. In reality, user-item interaction usually stems from more complex underlying factors, such as the users’ specific preferences. A user-item bipartite graph could be used to understand the differences in motivation. However, existing research has not clearly proposed and modeled the factors that affect the differences, ignoring the similarities between user pairs and item pairs, preventing them from capturing fine-grained user preferences more effectively. This paper has developed a novel recommendation model MI-CF, which explicitly proposes and models multi-attribute and implicit relationship factors for collaborative filtering recommendation. MI-CF aggregates multi-attribute spaces through the user-item bipartite graph and additionally establishes user-user and item-item graphs to model the similar relationship information of neighbor pairs through a memory model. In addition, the sparse regularizer is utilized to alleviate the overfitting problem. Extensive experimental results on three public datasets not only show the significant performance gain of the proposed model but also prove the effectiveness and interpretability of fine-grained implicit factors modeling.