The nature of the visual companions of AP and AM stars