Applying Machine Learning to the Choice of Size Modifiers

People use different size modifiers to refer to different object sizes; “the long table” is likely to be a different table from “the small table”. However, the details influencing the selection of size modifier have not yet been uncovered. When is something “long”, and when is something “small”? We introduce a connection between the visible dimensions of objects and the kinds of language people use to refer to them. First, we conduct an experiment to elicit size-denoting modifiers from images of real world objects. We find that we are able to effectively model the relationship between dimensional features and modifier choice using decision trees. The images are then used as input to an object segmentation algorithm, and we compare how well we can predict speakers’ behavior using the real world measurements of the pictured objects and the image pixel-based measurements. We find that real world measurements are the best predictors of modifier choice, suggesting that people infer real world size features from images. However, automatically extracted pixel measurements do perform relatively well at predicting modifier choice, offering a potential connection between computer vision and natural language. When speaker identity is taken into account, modifier choice can be predicted with even greater accuracy (around 75%), and the difference between automatically extracted and real world measurements is no longer significant.