perceive@dieei.unict.it   @PeRCeiVeLab   +39 095 738 7902

Knowledge Based Image Classification

"Learning never exhausts the mind"

Fruit Ontology


Our Fruit Ontology has been designed with the assistance of three expert agronomists to visually describe the fruit domain. Moreover, it extends a generic Annotation Ontology, which provides the structure necessary to embed a domain-specific ontology in the annotation tool in order to guide and constrain the annotation process.
See the following VOWL (Visual OWL); high resolution image: zoom in to see classes and properties.
Visit the page to download the owl file containing the Fruit Ontology definition and its 24 instances (one for each fruit variety) and the annotated dataset.

image/svg+xml Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of leaf has length leaf has length fruit has species fruit has species tree has crown density tree has crown density peduncle has relative insertion peduncle has relative insertion tree has habit tree has habit peduncle has length peduncle has length fruit russet has type fruit russet has type fruit has russet fruit has russet Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of peduncle has thickness peduncle has thickness fruit has size fruit has size fruit has colour fruit has colour fruit has overcolour description fruit has colour description sample has BB sample has BB fruit has variety fruit has variety Subclass of Subclass of fruit has lenticels fruit has lenticels fruit has shape fruit has shape has sample has sample is sample of is sample of is annotation sample of physical object is annotation sample of physical object physical object has annotation sample physical object has annotation sample physical object has sample is example of physical object is sample of physical object physical object has example physical object has sample is sample of physical property is sample of physical object physical property has sample fruit is in shrub fruit is in shrub shrub has fruit shrub has fruit Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of leaf has colour leaf has colour corolla is in flower corolla is in flower flower has corolla flower has corolla shrub has leaf shrub has leaf leaf is in shrub leaf is in shrub leaf has petiole leaf has petiole petiole is in leaf petiole is in leaf petal has shape petal has shape Subclass of Subclass of petiole has colour petiole has colour Subclass of Subclass of fruit has peduncle fruit has peduncle peduncle is in fruit peduncle is in fruit Subclass of Subclass of corolla has petals arrangement corolla has petals arrangment Subclass of Subclass of Subclass of Subclass of petal edge has type petal edgehas type petal has edge petal has edge corolla has size corolla has size physical object has size physical object has size corolla has shape corolla has shape corolla petals arrangement has type corolla petal arrangment has type petal has colour petal has colour leaf has edge leaf has edge physical object has edge physical object has edge fruit has overcolour fruit has overcolour leaf has width leaf has width leaf has shape leaf has shape leaf edge has type leaf edge type edge has type edge has type flower is in tree flower is in tree tree has flower tree has flower. tree has leaf tree has leaf leaf is in tree leaf is in tree shrub has flower shrub has flower flower is in shrub flower is in shrub Subclass of Subclass of peduncle relative insertion has type peduncle relative insertion has type peduncle thickness has type peduncle thickness has type fruit is in tree fruit is in tree tree has fruit tree has fruit leaf shape has type leaf shape has type petal shape has type petal shape has type Subclass of Subclass of Subclass of Subclass of fruit has colour description fruit has colour description physical object has colour physical object has colour fruit lenticels has type fruit lenticels has type fruit shape has type fruit shape has type tree has vigour tree has vigour petiole has length petiole has length physical object has length physical object has length Subclass of Subclass of Subclass of Subclass of petal is in corolla petal is in corolla corolla has petal corolla has petal physical object is part of physical object has part physical object has part physical object is part of physical object is part of physical object has physical property physical object is part of physical object has shape Subclass of Subclass of corolla shape has type corolla shape has type shape has type shape has shape has type Subclass of Subclass of Subclass of Subclass of Subclass of Subclass of tree habit has type tree habit has type physical property has type physical property has type Subclass of Subclass of Subclass of Subclass of plant has habitat plant has habitat topDataProperty topDataProperty annotation sample is visible annotation sample is visible fruit has stripes fruit has stripes fruit lenticels has colour fruit lenticels hasc olour fruit lenticels has diameter fruit lenticels has diameter fruit russet has distribution fruit russet has distribution physical object has width physical object has width has narrower has narrower is top concept in scheme is top concept ins scheme has broader has broader is in scheme is in scheme sample has image samplei s in image Fruit Russet Fruit Russet 11 Literal Literal sizeRange sizeRange Literal Literal AnnotationSample widthRange widthRange Physical object Physical Object Sample Sample Phisycal property Physical Property Peduncle Relative Insertion Peduncle Relative Insertion. 2 Edge Edge Data One Of Data One Of Petal Petal 27 hexBinary hexBinary Flower Flower 27 Corolla Petals Arrangement Corolla Petals Arrangment. 4 hexBinary hexBinary Literal Literal Petal edge Petal edge 1 Leaf Leaf 27 Plant Plant Plant part Plant Part Peduncle Thickness Peduncle Thickness 3 Peduncle Peduncle 27 Concept Concept (external) Fruit Fruit 27 Fruit Lenticels Fruit Lenticels 19 Species Species 3 Shrub Shrub Leaf shape Leaf shape 5 standardRange standardRange Petiole Petiole 27 Leaf edge Leaf edge 6 Corolla Corolla 27 Corolla Shape Corolla Shape 1 Literal Literal Tree Habit Tree Habit 11 Variety Variety 72 Habitat Habitat 9 Shape Shape Literal Literal Fruit shape Fruit shape 14 Petal shape Petal shape 3 Annotation 110831 lengthRange lengthRange Literal Literal Thing Thing Thing Thing string string string string Tree Tree 27 Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of lengthRange lengthRange lengthRange lengthRange Data One Of Data One Of widthRange widthRange sizeRange sizeRange Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of Data One Of lengthRange lengthRange sizeRange sizeRange standardRange standardRange standardRange standardRange hexBinary hexBinary

Annotation Tool


The developed Annotation Tool exploits the knowledge base encoded in the ontology to ensure the annotations correctness and to reduce the experts working time.
Left: experts had only to annotate one target object per image.
Right: bounding box annotations provided by non-experts. All the bounding box labels were inferred automatically from expert annotation through ontology.






Fruit Image Dataset


We used the tool to build the Fruit Image Dataset, a collection of 3,872 images of 3 common fruit species, namely, malus domestica (apple), prunus avium (cherry) and pyrus communis (pear), and 24 fruit varieties in total. We also generated over 60,000 bounding boxes (depicting the different varieties of fruits, leaves, peduncles, etc.) and over 1,000,000 OWL triples (representing high-level knowledge on context and attributes).





Malus Domestica Pyrus Communis Prunus Avium
Variety #img #bb Variety #img #bb Variety #img #bb
Ambrosia 117 3,837 Abate 286 4,293 Bing 67 2,677
Braeburn 83 1,486 Anjou 190 3,008 Black Tartarian 45 1,258
Cameo 70 874 Conference 181 2,586 Burlat 71 2,568
Fuji 181 3,710 Coscia 167 2,963 Ferrovia 77 5,307
Golden Delicious 324 5,554 Doyenne du Comice 77 1,186 Lapins 87 3,037
Granny Smith 360 1,901 Kaiser 111 1,170 Rainer 140 3,849
Pink Lady 239 2,737 Williams 165 4,871 Stella 31 1,173
Reinette 242 2,233





Royal Gala 150 2,070





Stark Delicious 411 4431





Total 2,177 28,833 Total 1,177 20,023 Total 518 19,869
Total Dataset Total Images Total Bounding Boxes

3,872 68,547



Classification Performance


We developed a simple semantic CNN-based object classifer, which is able to exploit the real-world semantics available in the Fruit Image Dataset. The underlying idea of our classifier consisted of building an ontology instance for test image I and comparing such instance with the C ground-truth instances (in our case, C = 24, corresponding to the considered varieties as defined in the Fruit ontology), provided by the domain experts, to find the best match. In particular, we treated the classification problem as a graph matching one; in practice, we grounded ontology instances to weighted graphs and computed the similarity between graphs.
In the following table we report the Mean Classification Accuracy achieved a) when using only low and middle-level visual descriptors (first 4 columns) and b) when integrating high- level knowledge. In the latter case we had a performance increase of about 11%.






Learning Visual Descriptors Exploiting High-Level Knowledge
Species VLFeat KDES OverFeat GoogLeNet Belief propagation
All Species 4.21% 6.11% 16.49% 20.11% 33.42%
Malus domestica 15.95% 19.21% 24.74% 30.34% 40.76
Pyrus communis 23.54% 24.07% 32.44% 34.77% 44.76%
Prunus avium 12.21% 17.41% 21.94% 24.27% 36.23%

"Nature is the source of all true knowledge. She has her own logic, her own laws,
she has no effect without cause nor invention without necessity." L. Da Vinci."