Similarity matrices for colored graphs