Machine learning algorithms have gained widespread usage across a variety of domains, both in providing predictions to expert users and recommending decisions to everyday users. However, these AI systems are often black boxes, and end-users are rarely provided with an explanation of the algorithmic output, which could lead to a significant loss of trust and willingness to use. The critical need for explanation and justification by AI systems has led to calls for algorithmic transparency, including the "right to explanation" in the EU General Data Protection Regulation (GDPR), which requires many companies to provide a meaningful explanation to involved parties. These initiatives all presuppose that we know what constitutes a meaningful or good explanation, but there has been limited research on this question in the context of AI systems. In this paper, we (1) develop a generalizable framework grounded in philosophy, psychology, and interpretable machine learning to investigate and define characteristics of good explanation, and (2) conduct a large-scale lab experiment to measure the impact of different factors on perceptions of understanding, fairness, and trust within a loan application context. The framework and study together form a concrete guide for managers to present algorithmic prediction rationales to end-users to foster trust and adoption. They also highlight elements of explanation to be considered by AI researchers and engineers in designing, developing, and deploying explainable machine learning algorithms.