During early cardiac development, progenitors of the valves and septa of the heart are formed by an epithelial-mesenchymal cell transformation of endothelial cells of the atrioventricular (AV) canal. We have previously shown that this event is due to an interaction between the endothelium and products of the myocardium found within the extracellular matrix. The present study examines signal transduction mechanisms governing this differentiation of AV canal endothelium. Activators of protein kinase C (PKC), phorbol myristate acetate (PMA) and mezerein, both produced an incomplete phenotypic transformation of endothelial cells in an in vitro bioassay for transformation. On the other hand, inhibitors of PKC (H-7 and staurosporine) and tyrosine kinase (genistein) blocked cellular transformation in response to the native myocardium or a myocardially-conditioned medium. Intracellular free calcium concentration ([Ca2+]i) was measured in single endothelial cells by microscopic digital analysis of fura 2 fluorescence. Addition of a myocardial conditioned medium containing the transforming stimulus produced a specific increase in [Ca2+]i in "competent" AV canal, but not ventricular, endothelial cells. Epithelial-mesenchymal cell transformation was inhibited by pertussis toxin but not cholera toxin. These data lead to the hypothesis that signal transduction of this tissue interaction is mediated by a G protein and one or more kinase activities. In response to receptor activation, competent AV canal endothelial cells demonstrate an increase in [Ca2+]i. Together, the data provide direct evidence for a regional and temporal regulation of signal transduction processes which mediate a specific extracellular matrix-mediated tissue interaction in the embryo.