Interactions of the nonbenzodiazepine anxiolytic, buspirone, with serotonin (5-HT) were studied using behavioral and neurochemical procedures. Punished responding was studied in pigeons as this behavior is a generally acknowledged preclinical predictor of anxiolytic activity and because buspirone increases punished responding of pigeons with greater potency and efficacy than in other species. Keypeck responses were maintained under either fixed-interval or fixed-ratio schedules of food presentation; every 30th response produced a brief electric shock and suppressed responding (punishment). Buspirone (0.1-5.6 mg/kg i.m.) produced dose-related increases in punished responding which reached a maximum at 1 mg/kg. A serotonin agonist, MK-212 (0.01 mg/kg), antagonized whereas the 5-HT antagonist, cyproheptadine (0.01 mg/kg), potentiated the effects of buspirone without having behavioral effects of their own. The characteristics of [3H]-5-HT binding in pigeon brain membranes were similar to results reported in mammalian brain. Neither buspirone, MJ-13805 (gepirone, a related analog), nor MJ-13653 (a buspirone metabolite), significantly affected [3H]-5-HT binding and none of the compounds appreciably inhibited uptake of [3H]-5-HT into pigeon cerebral synaptosomes. Hill coefficients significantly less than unity for all drugs except 5-HT suggested multiple serotonergic binding sites for buspirone and analogs. Buspirone and MJ-13805 (1 nM) inhibited [3H]ketanserin binding (a measure of 5-HT2 binding sites) in pigeon cerebrum with Ki values above 10(-6) M. The number of [3H]ketanserin binding sites was estimated to be 109 fmol/mg of protein in pigeon cerebrum compared to 400 fmol/mg of protein in rat cerebrum.(ABSTRACT TRUNCATED AT 250 WORDS)