Putting the Content Into Context: Features and Gaps in Image Retrieval

Digital management of medical images is becoming increasingly important as the number of images being created in medical settings everyday is growing rapidly. Content-based image retrieval or techniques based on the query-by-example paradigm have been studied extensively in computer vision. However, the global, low level visual features automatically extracted by these algorithms do not always correspond to high level concepts that a user has in his mind for searching. The role of image retrieval in diagnostic medicine can be quite complex, making it difficult for the user to express his/her information needs appropriately. Image retrieval in medicine needs to evolve from purely visual retrieval to a more holistic, case-based approach that incorporates various multimedia data sources. These include multiple images, free text, structured data, as well as external knowledge sources and ontologies.