From medical image to automatic medical report generation

Abstract

We present a novel method for automatic breast radiology report generation from image data. We formalize this problem as learning to map a set of diverse image measurements to a set of discrete semantic descriptor values that represent the standard radiology lexicon. We use a structured learning framework to model individual semantic descriptors and their relationships. The parameters of the learned model are efficiently learned based on a training set of images using the structured support vector machine (SVM). The output report for a new image is generated in the form of a set of radiological lexicon descriptors. If the proposed method is used in a computer aided diagnosis (CAD) system, radiologists should be able to easily understand the diagnosis decision of the system since the system output is the standard radiological lexicon used to make a diagnosis. We applied the method to breast imaging modalities, sonography, and mammography. Our experiments indicate that our method generalizes better than competing approaches. Although the proposed method is tested for breast imaging report generation, it should be useful in general doctors' practice, wherein there is a predefined set of medical descriptors to be acquired by a doctor during image investigation.