Abstract Visual place recognition (VPR) in environments subject to extreme appearance variation due to changing weather, illumination or seasons is a challenging task. Recent works have shown that features learned from CNNs can achieve promising performance. However, most of the existing methods concentrate so much on the image itself that they neglect the architecture of the network, especially different filters that may carry more meaningful information. In this paper, we develop a learnable feature map filtering (FMF) module constrained by triplet loss to re-calibrate the weight of the individual feature map. In this way, specific feature maps that encode invariant characteristics of location are extracted. Moreover, to make full use of the rich global mutual information that resides in the sample set, we propose an influence-based graph attention network (IB-GAT) with a verification subnet to better incorporate the relations among samples during the training process. Different from conventional GAT approaches, IB-GAT enables feature nodes to attend over the influence of other nodes instead of the original feature. Thus refined features with more discriminative power could be generated. Extensive experiments have been conducted on six public VPR datasets with varying appearances. Ablation analysis verifies the potential efficacy of the FMF module and the IB-GAT components. The experimental results also demonstrate that the proposed methods can achieve better performance than the current state of the art.