Modern lifestyle and technology development naturally comes with a greater use of software packages and solutions in everyday life, as well as in education. This often results with numerous new software solutions on the market, usually similar between each other. These similarities lead to dilemma what is the software solution that can be recognized as most suitable for use. This paper presents the way to select the most suitable mathematical software package that could be used in teaching, with respect of technical and mathematical study courses in higher education. The selection is made by applying the mathematical method analytical hierarchical process (AHP). AHP is multi – criteria decision making method which is widely used in resolving the conflict situations for decision making processes. In order to achieve more realistic and relevant assessment, the criteria used for the decision making and selection performance are given by the international standard for software product quality
ISO 9126. This combination of AHP method and ISO 9126 standard provides a good basis for optimization problem in a specific selection issue - selection of mathematical educational software.