Special Section

Tissue Level Based Deep Learning Framework for Early Detection of Dysplasia in Oral Squamous Epithelium

Rachit Kumar Gupta1,*, Mandeep Kaur2, Jatinder Manhas3
Author Information & Copyright
1Department of Computer Science and IT, University of Jammu, Jammu, J&K, India, gupta.rachit1990@gmail.com.
2Department of Oral Pathology, Indira Gandhi Govt. Dental College, Jammu, J&K, India, dr_mandeep_kaur@yahoo.com
3Department of Computer Science & IT, Bhaderwah Campus, University of Jammu, Jammu, J&K, India, manhas.jatinder@gmail.com
*Corresponding Author : Rachit Kumar Gupta, Department of Computer Science and IT, University of Jammu, Jammu, J&K, India, gupta.rachit1990@gmail.com.

© Copyright 2019 Korea Multimedia Society. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Apr 30, 2019; Revised: Jun 03, 2019; Accepted: Jun 05, 2019

Published Online: Jun 30, 2019


Deep learning is emerging as one of the best tool in processing data related to medical imaging. In our research work, we have proposed a deep learning based framework CNN (Convolutional Neural Network) for the classification of dysplastic tissue images. The CNN has classified the given images into 4 different classes namely normal tissue, mild dysplastic tissue, moderate dysplastic tissue and severe dysplastic tissue. The dataset under taken for the study consists of 672 tissue images of epithelial squamous layer of oral cavity captured out of the biopsy samples of 52 patients. After applying the data pre-processing and augmentation on the given dataset, 2688 images were created. Further, these 2688 images were classified into 4 categories with the help of expert Oral Pathologist. The classified data was supplied to the convolutional neural network for training and testing of the proposed framework. It has been observed that training data shows 91.65% accuracy whereas the testing data achieves 89.3% accuracy. The results produced by our proposed framework are also tested and validated by comparing the manual results produced by the medical experts working in this area.

Keywords: oral cancer; oral epithelial tissue; oral dysplasia; deep learning


India has one of the highest rates in oral cancer in the world partly attributed to high prevalence of tobacco chewing. In parallel to the increase in Oral Cancer, border-line malignant lesions which range from epithelial dysplasia to intra epithelial carcinoma have also increased in numbers [1]. In the past few years deep learning has proven to be efficient in many classification problems especially in medical imaging data. In the last few years a large number of machine learning based frameworks especially deep learning were employed by researchers to detect and classify the oral cancer. These deep learning based systems have proved to be very efficient and has achieved accuracy almost equal to the specialist pathologist that does its work manually and has ample of experience in his field [2].

The term dysplasia was introduced by Reagon in 1958 in a study where he described the features of dysplasia [3]. In medical terms dysplasia means an abnormal development in cells while histomorphologically any cellular or structural change in epithelium is dysplasia. Major pathological microscopic changes at cellular level as well as tissue level in dysplastic tissue are given in Table 1: [4].

Table 1. Pathological microscopic changes at the cellular level and tissue level in dysplasia
Cellular level changes Tissue level changes
• Abnormal variation in cell size (Anisocytosis) • Loss of polarity
• Increased nuclear/cytoplasmic ratio • Disordered maturation from basal to squamous cells
• Abnormal mitotic figures (abnormal in shape or location) • Increased cellular density
• Enlarged nuclei and cells • Dyskeratosis (premature keratinization and keratin pearls deep in epithelium)
• Abnormal variation in nuclear size (anisonucleosis) • Secondary extensions (nodules) on rete tips
• Increased number and size of nucleoli
Download Excel Table

Dysplasia is a part of the pathway to malignancy. Dysplastic cells cannot be called malignant until and unless it invades the connective tissue or metastasize. The dysplastic cells can be classified into mild, moderate and severe dysplasia.

The progressive change in dysplastic cells from mild to moderate and then from moderate to severe can be seen in the epithelium. The invasive activity of dysplastic cells starts from the lower layer, then progress to the middle layer and finally full thickness of epithelium. Next, it will invade the basement membrane [5].

Fig. 1. Showing the progression of dysplasia from normal epithelium to in-citu carcinoma.
Download Original Figure

As the dysplasia arises in the basal layer of the epithelium and extends, with progression, to the upper epithelial layers, the scheme classifies mild dysplasia as involvement of the lower third of the epithelium only, moderate dysplasia as extension to the middle third, and severe dysplasia as extension to the superficial third of the epithelium [6].

Early detection plays a key role in cancer diagnosis and can improve long-term survival rates. Medical imaging is a very important technique for early cancer detection and diagnosis. As is well known, medical imaging has been widely employed for early cancer detection, monitoring, and follow-up after the treatments [7].

This paper aims to classify oral dysplasia by using popular deep learning technique namely Convolutional Neural Network. Since our data is in image format that’s why CNN (Convolutional Neural Network) is preferable over other deep learning techniques. Recent studies show that CNN achieve promise performance in cancer detection and diagnosis. [8, 9].


This section is intended to provide a brief review of recent studies on applying deep learning for early cancer detection, cancer diagnosis and prognosis.

In recent years, a bunch of papers has been published about the application of deep learning in cancer detection and diagnosis. In [10], Gustav Forslid et al. have used CNN on oral cancer dataset. Two different CNN models were used namely ResNet and VGG. Results were very good as compared to other image diagnosis techniques. In [11], Geert Litjens et al. have presented Prostate cancer identification in biopsy specimens and breast cancer metastasis detection in sentinel lymph nodes using deep learning and concluded that deep learning improves the accuracy of prostate cancer diagnosis and breast cancer staging. Authors employed CNN. In [12], Albayrak et al. developed a deep learning based feature extraction algorithm to detect mitosis in breast histopathological images. In the proposed algorithm, the CNN model was used to extract features which were used to train a support vector machine (SVM) for the detection of mitosis. In [13], Krizhevsky et al. used AlexNet to construct a CNN model to classify benign or malignant tumors from the breast histopathological images. In [14], Akshay Iyer et al. have used the pre-trained model, VGG19 to extract information from the pathological images specific for lung cancer. A model consisting of deep convolution network-based image classification has been proposed for predictions on mutations in genes signature information of lung cancer among Indian populations. In [15], Bassma El-Sherbiny et al. propose - Brain/Lung/Breast (BLB) automated detection system. It precisely predicts the occurrence of cancer and segments the expected region of tumor/cancer in MRI/CT scan/Mammography images. This system proposes different classification techniques including Support Vector Machine (SVM), ExtraTrees and convolutional neural network (CNN). CNN performed exceptionally well in the detection of lung cancer. In [16], Chen et al. proposed a deep cascade network for mitosis detection in breast histology slides. They first trained a fully connected network model to extract mitosis candidates from the whole histology slides and then fine-tuned a CaffeNet model for the classification of mitosis. Three networks with different configurations of fully connected layers were trained and the scores were averaged to generate the final output. In [17], Ida Arvidsson et al. compared two different techniques; by training the networks using color augmentation and by using digital stain separation using an auto encoder. The author achieved accuracies of 95% for classification of benign versus malignant tissue and 81% for Gleason grading for data from the same site as the training data. In [18], Albarqouni et al. explored deep CNN in a biomedical context, a multiscale CNN architecture was developed with an aggregation layer was introduced after the softmax layer to aggregate the prediction results with the annotation results from multiple participation. In [19], Sawon Pratiheret al. have deployed deep learning framework to classify elastic scattering spectra of biological tissues into normal and cancerous ones. Authors experimented to show the superiority of the convolutional neural network extracted deep features over classical handcrafted biomarkers. The proposed method employs elastic scattering spectra of the tissues as input to CNN. In [20], Wichakam et al. proposed a combined system consisting of CNN and SVM for mass detection on digital mammograms. CNN was used on mammographic patches to get the high-level feature representation of the image. This high-level feature set was used as input to SVM for classification of mammograms. In [21], Xu et al. proposed a stacked sparse auto-encoder based algorithm to classify nuclei in breast cancer histopathology. In [22], Tian Xia et al. proposed a system of tumor classification by pre-training a CNN from samples of different tissue types in histopathological images, then fine-tuning the obtained pre-trained on a particular tissue type. CNN showed improvement over training from scratch with limited data. In [23], Mina Khoshdeli et al. applied convolutional neural networks for grading of the tumor and decomposing tumor architecture from H&E stained histology sections of the kidney. In [24], Xu Y et al. proposed deep convolutional neural network activation features to perform classification, segmentation and visualization in large-scale tissue histopathology images. Authors used a pre-trained ImageNet network on features extracted by CNN.


In this study we have taken data of 52 patients suffering from oral dysplasia. 2 H&E stained histopathological samples are taken from each patient. The data is taken from Indira Gandhi Govt. Dental College and Hospital, Jammu, India. Based on CIN (Cervical intraepithelial neoplasia) classification, these data samples were classified into 4 different classes namely normal, mild dysplasia, moderate dysplasia and severe dysplasia.

There are a total of 2688 images in our dataset. 663 images belonging to a normal class, 681 belonging to mild dysplasia, 675 belonging to moderate dysplasia and 669 belonging to severe dysplasia.

CNN was created and trained from scratch. Python language along with the Tensor flow and Keras deep learning libraries were used to build the CNN model.

Fig. 2. Showing Basic structure of a CNN.
Download Original Figure

Dataset was divided into training set and testing set in a 70:30 ratio respectively. The training set contained 1882 images and testing set contained 806 images. The research methodology diagram is given in figure 4:

Fig. 3. CNN Model Layer wise.
Download Original Figure
Fig. 4. Research methodology diagram.
Download Original Figure


The classification of the dysplastic tissue was carried out by CNN into 4 different classes.

When CNN was trained on training data it gave an accuracy of 91.65% and in testing, it gave an accuracy of 89.3%. Our CNN was trained for a total of 75 epochs. The Accuracy and loss of model is given in figure 5(a) and figure 5(b) respectively.

Fig. 5. Training results.
Download Original Figure

Our proposed CNN produced the following output of classified tissue images along with labels:

Fig. 6. Predicted Tissue images along with labels.
Download Original Figure

The confusion matrix obtained from the experiment is given in figure 3:

Fig. 7. Confusion matrix for testing of 806 oral epithelial tissue images.
Download Original Figure

Precision, recall, f1-score for each class and average of all classes are given in Table 2.

Table 1. Precision, Recall and f1-score of testing.
Class Precision Recall F1-score
Normal 0.93 0.89 0.91
Mild 0.83 0.89 0.86
Moderate 0.86 0.87 0.87
Severe 0.95 0.92 0.93
Average 0.89 0.89 0.89
Download Excel Table


Our proposed deep learning system although not in real time but produced promising results. Our proposed deep learning system has achieved higher accuracies almost as the experienced specialist concerned oral pathologist.


In the future, we will increase the dataset size to input more and more training data so that our proposed deep learning model will learn more efficiently. Also, we will fine-tune our deep learning model so that it can achieve more accuracy in predicting the true label.


I would like to thank the faculty of the Department of Oral Pathology and Department of Oral Medicine and Radiology from the Institute of Indira Gandhi Govt. Dental College and Hospital for their timely support in undertaking this research work.



Yoshiko Ariji, Motoki Fukuda, Yoshitaka Kise, Michihito Nozawa, Yudai Yanashita, Hiroshi Fujita, Akitoshi Katsumata, Eiichiro Ariji, “Contrast-enhanced computed tomography image assessment of cervical lymph node metastasis in patients with oral cancer by using a deep learning system of artificial intelligence”, Oral Surgery, Oral Medicine, Oral Pathology and Oral Radiology, 2018


Gnananandar G, Masthan K, Babu A. N, Jacobina J.,” Histomorphometric Analysis of oral Epithelial Dysplasia and oral Squamous Cell Carcinoma”. Biomed Pharmacology Journal, 2017


Dena J. Fischer, Joel B. Epstein, Thomas H. Morton, Stephen M. Schwartz,” Interobserver reliability in the histopathologic diagnosis of oral pre-malignant and malignant lesions“, Journal of Oral Pathology Medicine, Vol. 33, No. 2, pp. 65–70, 2004.


Barnes L, Eveson JW, Reichart P, Sidransky D, editors. “WHO classification of tumours: pathology and genetics of head and neck tumours”. Lyon: ARC Press; 2005.


S. Warnakulasuriya, “Histological grading of oral epithelial dysplasia: revisited”. Journal of Pathology. pp. 294–297, 2001.


Hong D. Xiao, Carlo B. Bifulco, “Cellular and Molecular Pathology, Oral, Head and Neck Oncology and Reconstructive Surgery”, Elsevier, pp. 57-78, 2018.


Zilong Hu, Jinshan Tang, Ziming Wang, Kai Zhang, Lin Zhang, Qingling Sun, “Deep Learning for Image-based Cancer Detection and Diagnosis A Survey”, Pattern Recognition 2018.


Y. LeCun, Y. Bengio, G. Hinton, “Deep learning”, Nature, Vol. 521, pp. 436-444, 2015.


G. Forslid; Håkan Wieslander; Ewert Bengtsson; Carolina Wählby; Jan-Michael Hirsch; Christina Runow Stark; Sajith Kecheril Sadanandan, “Deep Convolutional Neural Networks For Detecting Cellular Changes Due To Malignancy”. IEEE International Conference on Computer Vision Workshops (ICCVW), Venice, pp. 82-89, 2017.


G. Litjens, Clara I. Sánchez, Nadya Timofeeva, Meyke Hermsen, Iris Nagtegaal, Iringo Kovacs, Christina Hulsbergen - van de Kaa, Peter Bult, Bram van Ginneken & Jeroen van der Laak, "Deep learning as a tool for increased accuracy and efficiency of Histopathological diagnosis", Scientific Reports, Vol. 6, pp. 262-86, 2016.


A. Albayrak, G. Bilgin, “Mitosis detection using convolutional neural network based features”, IEEE 17th International Symposium on Computational Intelligence and Informatics (CINTI), pp. 335-340, 2017.


A. Krizhevsky, I. Sutskever, G.E. Hinton, “Imagenet classification with deep convolutional neural networks”, Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.


A. Iyer, A. Vyshnavi, M. Hima, P. K. Namboori, Krishnan, “Deep Convolution Network-Based Prediction Model For Medical Diagnosis Of Lung Cancer - A Deep Pharmacogenomic Approach: deep diagnosis for lung cancer,” Second International Conference on Advances in Electronics, Computers and Communications, pp.1-4, 2018.


Bassma El-Sherbiny, Nardeen Nabil, Youssef Emad, Nada Ayman, Taraggy Mohiy, Ashraf Abdel Raouf, “BLB (Brain/Lung cancer detection and segmentation and Breast Dense calculation),” First International Workshop on Deep and Representation Learning (IWDRL), Cairo, pp. 41-47, 2018.


H. Chen, Q. Dou, X. Wang, J. Qin, P.-A. Heng, “Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks”, AAAI, pp. 1160-1166, 2016.


Ida Arvidsson, Niels Christian Overgaard, Felicia-Elena Marginean, Agnieszka Krzyzanowska, Anders Bjartell, Kalle Åström, Anders Heyden, “Generalization of prostate cancer classification for multiple sites using deep learning”, IEEE 15th International Symposium on Biomedical Imaging, Washington, DC, pp. 191-194, 2018.


S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, N. Navab, “Aggnet: deep learning from crowds for mitosis detection in breast cancer histology images”, IEEE transactions on Medical Imaging, Vol. 35, pp. 1313-1321, 2016.


Sawon Pratiher, Shubhobrata Bhattacharya, Sabyasachi Mukhopadhyay, Nirmalya Ghosh, Gautham Pasupuleti, Prasanta K. Panigrahi, “Deep features using a convolutional neural network for early-stage cancer detection” Proceedings Volume 10679, Optics, Photonics, and Digital Technologies for Imaging Applications Vol. 1067902, 2018.


I. Wichakam, P. Vateekul, “Combining deep convolutional networks and SVMs for mass detection on digital mammograms”, 8th International Conference on Knowledge and Smart Technology (KST), IEEE, pp.239-244, (2016).


J. Xu, L. Xiang, R. Hang, J. Wu, “Stacked Sparse Autoencoder (SSAE) based framework for nuclei patch classification on breast cancer histopathology”, 2014 IEEE 11th International Symposium on Biomedical Imaging, IEEE, pp. 999-1002, 2014.


T. Xia, A. Kumar, D. Feng and J. Kim, "Patch-level Tumor Classification in Digital Histopathology Images with Domain Adapted Deep Learning," 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, pp. 644-647, 2018.


M. Khoshdeli, A. Borowsky and B. Parvin, "Deep Learning Models Differentiate Tumor Grades from H&E Stained Histology Sections," 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, pp. 620-623, 2018.


Y. Xu, Z. Jia, L. B. Wang, Y. Ai, F. Zhang, M. Lai, E. I. Chang, “Large scale tissue histopathology image classification, segmentation, and visualization via deep convolutional activation features,” BMC Bioinformatics, Vol. 18, No. 1, p. 281, 2016.


Rachit Kumar Gupta


Rachit Kumar Gupta has received his Master in Computer Application degree in 2014 from university of Jammu. He has worked as lecturer in the Department of computer science and IT, university of Jammu for 2 years. Currently he is pursuing his Ph.D from Department of computer science and IT, university of Jammu. His area of interest include machine learning, deep learning, medical image analysis,

Jatinder Manhas


Jatinder Manhas has received his MCA and Ph.D. in Computer Science from Department of Computer Science & IT, University of Jammu. He has a vast experience of over 13 years in the field of networking, databases, and website design issues. Currently he is working as a Senior Assistant Professor in the Department of Computer Science & IT at Bhaderwah Campus of University of Jammu. His area of interest includes Artificial intelligence, machine learning, IoT, deep learning, medical image analysis, website design issues, etc.

Dr. Mandeep Kaur


Dr. Mandeep Kaur has received her MDS. She is working as Asstt. Prof in Department of Oral pathology and Microbiology at Indira Gandhi Govt dental College and Hospital, Jammu. She has a teaching experience of more than 10 years. Her expertise includes oral lesions, oral cancer, dysplasia.