A psychophysical approach to modeling image semantics

Current image retrieval systems compare images based on low-level visual primitives, such as color, color layout, texture and shape. However, recent psychophysical experiments show that human observers primarily use high-level semantic descriptors and categories to judge image similarity. To model these high-level descriptors in terms of low-level primitives we use hierarchical clustering to segment the psychophysically determined image similarity space into semantically meaningful categories. We then conduct a series of psychophysical experiments to evaluate the perceptual salience of these categories. For each category we investigate the correlation with low-level pictorial features to identify semantically relevant features, their organization and distribution. Our findings suggest a new semantically based image similarity model.