Evaluating a Rubric for Assessing Constraint-Based Solid Models

A study was conducted during the Fall 2013 semester to examine the effectiveness of a rubric for evaluating constraint-based solid models. The rubric was created after studying conceptual frameworks and other research related to evaluating constraint-based CAD models. Since only one researcher evaluated the models in the 2013 study, it was recommended that a study be conducted where multiple experts evaluated the same models using this original rubric. During the Fall 2015 semester, three faculty experts in constraint-based modeling used the same rubric to evaluate a representative sample of models created by students in an introductory engineering graphics course. This paper presents literature related to evaluating constraint-based solid models and inter-rater reliability, describes the methodology and results of the study, and provides recommendations for further research related to evaluating constraint-based solid models. Introduction / Review of Literature As the tools for creating virtual models have evolved, engineering graphics educators have continued to adjust their methods for accurately and consistently evaluating students’ modeling strategies. Some of these methods include using concise rubrics for evaluating models 1-6, developing activities where students can evaluate their own models 7, and using automated electronic evaluation tools 8-9. One of the main challenges has been developing a method that clearly informs students about how their models will be evaluated, is a valid and reliable tool for assessing design intent, and allows faculty to evaluate models in a timely and consistent manner. Rubrics have been shown to provide reliable scoring of performance and have the potential to promote learning and/or improve instruction 10. The main purpose of the rubric used for evaluating the models in this study was to create a valid and consistent method for scoring constraint-based models used in engineering graphics courses. It was created based on a review of literature of several key topics in engineering graphics, graphicacy and modeling, and constraint-based CAD 4. These topics include CAD modeling strategies 11-13, conceptual framework research of CAD expertise 14-18, studies related to evaluating CAD models 5-9, 19-20, and engineering graphics literacy 1-4. Figure 1 displays the main categories of the rubric with an explanation of each. The rubric described here was used in a previous study to evaluate three models created by 23 students in a second level engineering graphics course 4. The purpose of that study was to compare this rubric to a more elaborate rubric used to assess engineering graphics literacy 1-3. Conclusions from this study revealed that scores were significantly higher when evaluated with the new rubric than when evaluated with the older rubric. There were also concerns that the older rubric required a great deal of time to evaluate models. Since only one person evaluated all of the models, it was recommended that a study be conducted to evaluate the inter-rater reliability of multiple raters using the same rubric. Category Points Base/Core Feature correctly identified For some objects this is clear. For other objects there is some flexibility. The base feature should create a good foundation for modeling the rest of the object in an efficient manner. 10 Orientation of initial sketch plane The initial sketch plane is important for establishing the viewing direction of the model and also how the model will be oriented in the assembly. It is also critical for establishing the main symmetry plane for models. 10 Best model origin As with the base/core feature, the location of the origin is flexible. It should, however, reflect the design intent of the model. For example, if an object has obvious symmetry, it is a good idea to have the origin in the center of the symmetry. This allows one to take advantage of the defaults planes in the part for establishing symmetry. 10 Sketches are simple, fully constrained, and reflect appropriate design intent Are the dimensions given in the problem the ones used in the sketches? If symmetry is needed, is this built into the sketch – omitting dimensions in place of appropriate constraints? 20 Appropriate feature end conditions For example, Through-All and Next 10 Correct application of symmetry/duplication When key dimensions are modified, do features stay centered per design intent? Is this correctly built into sketches and features? 10 Accuracy/Complete model Is the total volume accurate? Are all features taken into consideration? 20 Modeling strategy efficiency Does the modeling strategy reflect an economy of features, only using necessary sketches and features? Are features based on the given problem? 10 Figure 1. The Rubric. Inter-judge / Inter-rater Reliability Inter-judge or inter-rater reliability “refers to the degree of agreement between two or more observers/judges with respect to their categorization of n subjects/objects” (p. 669) 21. When examining the relationships between two judges or raters with interval or ratio data, Spearman’s rank order correlation coefficient can be used. When there are more than two judges or raters and the data is interval or ratio, Kendall’s coefficient of concordance with the intraclass correlation coefficient can be employed (p. 1053) 22.