perceive@dieei.unict.it   @PeRCeiVeLab   +39 095 738 7902

Knowledge Based Image Classification

"Learning never exhausts the mind"

Fruit Ontology


Our Fruit Ontology has been designed with the assistance of three expert agronomists to visually describe the fruit domain. Moreover, it extends a generic Annotation Ontology, which provides the structure necessary to embed a domain-specific ontology in the annotation tool in order to guide and constrain the annotation process.
See the following VOWL (Visual OWL); high resolution image: zoom in to see classes and properties.
Click here to download the owl file containing the Fruit Ontology definition and its 27 instances (one for each fruit variety in our dataset).

image/svg+xml Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of leaf has length leaf has length fruit has species fruit has species tree has crown density tree has crown density peduncle has relative insertion peduncle has relative insertion tree has habit tree has habit peduncle has length peduncle has length fruit russet has type fruit russet has type fruit has russet fruit has russet Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of peduncle has thickness peduncle has thickness fruit has size fruit has size fruit has colour fruit has colour fruit has overcolour description fruit has colour description sample has BB sample has BB fruit has variety fruit has variety Subclass of Subclass of fruit has lenticels fruit has lenticels fruit has shape fruit has shape has sample has sample is sample of is sample of is annotation sample of physical object is annotation sample of physical object physical object has annotation sample physical object has annotation sample physical object has sample is example of physical object is sample of physical object physical object has example physical object has sample is sample of physical property is sample of physical object physical property has sample fruit is in shrub fruit is in shrub shrub has fruit shrub has fruit Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of leaf has colour leaf has colour corolla is in flower corolla is in flower flower has corolla flower has corolla shrub has leaf shrub has leaf leaf is in shrub leaf is in shrub leaf has petiole leaf has petiole petiole is in leaf petiole is in leaf petal has shape petal has shape Subclass of Subclass of petiole has colour petiole has colour Subclass of Subclass of fruit has peduncle fruit has peduncle peduncle is in fruit peduncle is in fruit Subclass of Subclass of corolla has petals arrangement corolla has petals arrangment Subclass of Subclass of Subclass of Subclass of petal edge has type petal edgehas type petal has edge petal has edge corolla has size corolla has size physical object has size physical object has size corolla has shape corolla has shape corolla petals arrangement has type corolla petal arrangment has type petal has colour petal has colour leaf has edge leaf has edge physical object has edge physical object has edge fruit has overcolour fruit has overcolour leaf has width leaf has width leaf has shape leaf has shape leaf edge has type leaf edge type edge has type edge has type flower is in tree flower is in tree tree has flower tree has flower. tree has leaf tree has leaf leaf is in tree leaf is in tree shrub has flower shrub has flower flower is in shrub flower is in shrub Subclass of Subclass of peduncle relative insertion has type peduncle relative insertion has type peduncle thickness has type peduncle thickness has type fruit is in tree fruit is in tree tree has fruit tree has fruit leaf shape has type leaf shape has type petal shape has type petal shape has type Subclass of Subclass of Subclass of Subclass of fruit has colour description fruit has colour description physical object has colour physical object has colour fruit lenticels has type fruit lenticels has type fruit shape has type fruit shape has type tree has vigour tree has vigour petiole has length petiole has length physical object has length physical object has length Subclass of Subclass of Subclass of Subclass of petal is in corolla petal is in corolla corolla has petal corolla has petal physical object is part of physical object has part physical object has part physical object is part of physical object is part of physical object has physical property physical object is part of physical object has shape Subclass of Subclass of corolla shape has type corolla shape has type shape has type shape has shape has type Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of tree habit has type tree habit has type physical property has type physical property has type Subclass of Subclass of Subclass of Subclass of plant has habitat plant has habitat topDataProperty topDataProperty annotation sample is visible annotation sample is visible fruit has stripes fruit has stripes fruit lenticels has colour fruit lenticels hasc olour fruit lenticels has diameter fruit lenticels has diameter fruit russet has distribution fruit russet has distribution physical object has width physical object has width has narrower has narrower is top concept in scheme is top concept ins scheme has broader has broader is in scheme is in scheme sample has image samplei s in image Fruit Russet Fruit Russet 11 Literal Literal sizeRange sizeRange Literal Literal AnnotationSample widthRange widthRange Physical object Physical Object Sample Sample Phisycal property Physical Property Peduncle Relative Insertion Peduncle Relative Insertion. 2 Edge Edge Data One Of Data One Of Petal Petal 27 hexBinary hexBinary Flower Flower 27 Corolla Petals Arrangement Corolla Petals Arrangment. 4 hexBinary hexBinary Literal Literal Petal edge Petal edge 1 Leaf Leaf 27 Plant Plant Plant part Plant Part Peduncle Thickness Peduncle Thickness 3 Peduncle Peduncle 27 Concept Concept (external) Fruit Fruit 27 Fruit Lenticels Fruit Lenticels 19 Species Species 3 Shrub Shrub Leaf shape Leaf shape 5 standardRange standardRange Petiole Petiole 27 Leaf edge Leaf edge 6 Corolla Corolla 27 Corolla Shape Corolla Shape 1 Literal Literal Tree Habit Tree Habit 11 Variety Variety 72 Habitat Habitat 9 Shape Shape Literal Literal Fruit shape Fruit shape 14 Petal shape Petal shape 3 Annotation 110831 lengthRange lengthRange Literal Literal Thing Thing Thing Thing string string string string Tree Tree 27 Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of lengthRange lengthRange lengthRange lengthRange Data One Of Data One Of widthRange widthRange sizeRange sizeRange Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of lengthRange lengthRange sizeRange sizeRange standardRange standardRange standardRange standardRange hexBinary hexBinary

Annotation Tool


The developed Annotation Tool exploits the knowledge base encoded in the ontology to ensure the annotations correctness and to reduce the experts working time.
Left: experts had only to annotate one target object per image.
Right: bounding box annotations provided by non-experts. All the bounding box labels were inferred automatically from expert annotation through ontology.






Fruit Image Dataset


We used the tool to build the Fruit Image Dataset, a collection of 5,547 images of 3 common fruit species, namely, malus domestica (apple), prunus avium (cherry) and pyrus communis (pear), and 27 fruit varieties in total. We also generated over 110,000 bounding boxes (depicting the different varieties of fruits, leaves, peduncles, etc.) and over 1,000,000 OWL triples (representing high-level knowledge on context and attributes).





Malus Domestica Pyrus Communis Prunus Avium
Variety #img #bb Variety #img #bb Variety #img #bb
Ambrosia 224 7,310 Abate 459 3,768 Bing 111 3,702
Braeburn 149 4,561 Colombia Red Anjou 250 1,897 Black Tartarian 96 3,683
Cameo 75 2,447 Conference 159 2,044 Burlat 133 4,128
Fuji 179 5,801 Coscia 351 3,702 Ferrovia 106 3,640
Golden Delicious 289 5,813 Doyenne du Comice 125 1,556 Lapins 118 3,847
Granny Smith 342 4,012 Flordahome 192 2,143 Rainer 341 9,233
Pink Lady 260 6,612 Green Anjou 116 1,841 Stella 74 2,698
Reinette 281 7,277 Kaiser 124 2,658


Royal Gala 113 3,520 Gebhard Red Anjou 122 2,347


Stark Delicious 450 6,881 Williams 308 3,270


Total 2,362 54,234 Total 2,206 25,676 Total 979 30,921
Total Dataset Total Images Total Bounding Boxes

5,547 110,831



Classification Performance


We developed a simple semantic CNN-based object classifer, which is able to exploit the real-world semantics available in the Fruit Image Dataset. The underlying idea of our classifier consisted of building an ontology instance for test image I and comparing such instance with the C ground-truth instances (in our case, C = 27, corresponding to the considered varieties as defined in the Fruit ontology), provided by the domain experts, to find the best match. In particular, we treated the classification problem as a graph matching one; in practice, we grounded ontology instances to weighted graphs and computed the similarity between graphs.
In the following table we report the Mean Classification Accuracy achieved a) when using only low and middle-level visual descriptors (first 4 columns) and b) when integrating high- level knowledge. In the latter case we had a performance increase of about 11%.






Learning Visual Descriptors Exploiting High-Level Knowledge
Species VLFeat KDES OverFeat GoogLeNet Belief propagation
All Species 4.21% 6.11% 16.49% 20.11% 33.42%
Malus domestica 15.95% 19.21% 24.74% 30.34% 40.76
Pyrus communis 23.54% 24.07% 32.44% 34.77% 44.76%
Prunus avium 12.21% 17.41% 21.94% 24.27% 36.23%

"Nature is the source of all true knowledge. She has her own logic, her own laws,
she has no effect without cause nor invention without necessity." L. Da Vinci."