Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. St...

Full description

Autores:: Gustavo Martinez Villalobos
Gil-González, Julian
Fernandez-Gallego, Jose A.
Álvarez-Meza, Andrés Marino
Castellanos-Dominguez, Cesar German

Tipo de recurso:: Article of investigation

Fecha de publicación:: 2023

Institución:: Universidad de Ibagué

Repositorio:: Repositorio Universidad de Ibagué

Idioma:: eng

id	UNIBAGUE2_7c2070815e5b682483b000ede61c83f6
oai_identifier_str	oai:repositorio.unibague.edu.co:20.500.12313/5585
network_acronym_str	UNIBAGUE2
network_name_str	Repositorio Universidad de Ibagué
repository_id_str
dc.title.eng.fl_str_mv	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
title	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
spellingShingle	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification Entropía cruzada generalizada Múltiples anotadores Chained approach Classification Deep learning Generalized cross-entropy Multiple annotators
title_short	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
title_full	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
title_fullStr	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
title_full_unstemmed	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
title_sort	Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification
dc.creator.fl_str_mv	Gustavo Martinez Villalobos Gil-González, Julian Fernandez-Gallego, Jose A. Álvarez-Meza, Andrés Marino Castellanos-Dominguez, Cesar German
dc.contributor.author.none.fl_str_mv	Gustavo Martinez Villalobos Gil-González, Julian Fernandez-Gallego, Jose A. Álvarez-Meza, Andrés Marino Castellanos-Dominguez, Cesar German
dc.subject.armarc.none.fl_str_mv	Entropía cruzada generalizada Múltiples anotadores
topic	Entropía cruzada generalizada Múltiples anotadores Chained approach Classification Deep learning Generalized cross-entropy Multiple annotators
dc.subject.proposal.eng.fl_str_mv	Chained approach Classification Deep learning Generalized cross-entropy Multiple annotators
description	Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. Still, traditional multiple-annotator methods must account for the varying levels of expertise and the noise introduced by unreliable outputs, resulting in decreased performance. In addition, they assume a homogeneous behavior of the labelers across the input feature space, and independence constraints are imposed on outputs. We propose a Generalized Cross-Entropy-based framework using Chained Deep Learning (GCECDL) to code each annotator’s non-stationary patterns regarding the input space while preserving the inter-dependencies among experts through a chained deep learning approach. Experimental results devoted to multiple-annotator classification tasks on several well-known datasets demonstrate that our GCECDL can achieve robust predictive properties, outperforming state-of-the-art algorithms by combining the power of deep learning with a noise-robust loss function to deal with noisy labels. Moreover, network self-regularization is achieved by estimating each labeler’s reliability within the chained approach. Lastly, visual inspection and relevance analysis experiments are conducted to reveal the non-stationary coding of our method. In a nutshell, GCEDL weights reliable labelers as a function of each input sample and achieves suitable discrimination performance with preserved interpretability regarding each annotator’s trustworthiness estimation.
publishDate	2023
dc.date.issued.none.fl_str_mv	2023-04
dc.date.accessioned.none.fl_str_mv	2025-09-01T21:26:39Z
dc.date.available.none.fl_str_mv	2025-09-01T21:26:39Z
dc.type.none.fl_str_mv	Artículo de revista
dc.type.coar.none.fl_str_mv	http://purl.org/coar/resource_type/c_2df8fbb1
dc.type.coarversion.none.fl_str_mv	http://purl.org/coar/version/c_970fb48d4fbd8a85
dc.type.content.none.fl_str_mv	Text
dc.type.driver.none.fl_str_mv	info:eu-repo/semantics/article
dc.type.version.none.fl_str_mv	info:eu-repo/semantics/publishedVersion
format	http://purl.org/coar/resource_type/c_2df8fbb1
status_str	publishedVersion
dc.identifier.citation.none.fl_str_mv	Triana-Martinez, J., Gil-González, J., Fernandez-Gallego, J., Álvarez-Meza, A. y Castellanos-Dominguez, C. (2023). Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification. Sensors, 23(7), 3518. DOI: 10.3390/s23073518
dc.identifier.doi.none.fl_str_mv	10.3390/s23073518
dc.identifier.issn.none.fl_str_mv	14248220
dc.identifier.uri.none.fl_str_mv	https://hdl.handle.net/20.500.12313/5585
identifier_str_mv	Triana-Martinez, J., Gil-González, J., Fernandez-Gallego, J., Álvarez-Meza, A. y Castellanos-Dominguez, C. (2023). Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification. Sensors, 23(7), 3518. DOI: 10.3390/s23073518 10.3390/s23073518 14248220
url	https://hdl.handle.net/20.500.12313/5585
dc.language.iso.none.fl_str_mv	eng
language	eng
dc.relation.citationissue.none.fl_str_mv	7
dc.relation.citationstartpage.none.fl_str_mv	3518
dc.relation.citationvolume.none.fl_str_mv	23
dc.relation.ispartofjournal.none.fl_str_mv	Sensors
dc.relation.references.none.fl_str_mv	Zhang, J.; Sheng, V.S.; Wu, J. Crowdsourced label aggregation using bilayer collaborative clustering. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 3172–3185. Parvat, A.; Chavan, J.; Kadam, S.; Dev, S.; Pathak, V. A survey of deep-learning frameworks. In Proceedings of the 2017 International Conference on Inventive Systems and Control (ICISC), Coimbatore, India, 19–20 January 2017; pp. 1–7. Liu, Y.; Zhang, W.; Yu, Y. Truth inference with a deep clustering-based aggregation model. IEEE Access 2020, 8, 16662–16675. Gil-Gonzalez, J.; Orozco-Gutierrez, A.; Alvarez-Meza, A. Learning from multiple inconsistent and dependent annotators to support classification tasks. Neurocomputing 2021, 423, 236–247. Sung, H.E.; Chen, C.K.; Xiao, H.; Lin, S.D. A Classification Model for Diverse and Noisy Labelers. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining; Springer: Berlin/Heidelberg, Germany, 2017; pp. 58–69. Yan, Y.; Rosales, R.; Fung, G.; Subramanian, R.; Dy, J. Learning from multiple annotators with varying expertise. Mach. Learn. 2014, 95, 291–327. Xu, G.; Ding, W.; Tang, J.; Yang, S.; Huang, G.Y.; Liu, Z. Learning effective embeddings from crowdsourced labels: An educational case study. In Proceedings of the 2019 IEEE 35th International Conference on Data Engineering (ICDE), Macao, China, 8–11 April 2019; pp. 1922–1927. Tanno, R.; Saeedi, A.; Sankaranarayanan, S.; Alexander, D.C.; Silberman, N. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 11244–11253. Davani, A.M.; Díaz, M.; Prabhakaran, V. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguist. 2022, 10, 92–110. Kara, Y.E.; Genc, G.; Aran, O.; Akarun, L. Modeling annotator behaviors for crowd labeling. Neurocomputing 2015, 160, 141–156. Cao, P.; Xu, Y.; Kong, Y.; Wang, Y. Max-mig: An information theoretic approach for joint learning from crowds. arXiv 2019, arXiv:1905.13436. Chen, Z.; Wang, H.; Sun, H.; Chen, P.; Han, T.; Liu, X.; Yang, J. Structured Probabilistic End-to-End Learning from Crowds. In Proceedings of the IJCAI, Yokohama, Japan, 7–21 January 2021; pp. 1512–1518. Ruiz, P.; Morales-Álvarez, P.; Molina, R.; Katsaggelos, A.K. Learning from crowds with variational Gaussian processes. Pattern Recognit. 2019, 88, 298–311. G. Rodrigo, E.; Aledo, J.A.; Gámez, J.A. Machine learning from crowds: A systematic review of its applications. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2019, 9, e1288. Zhang, P.; Obradovic, Z. Learning from inconsistent and unreliable annotators by a gaussian mixture model and bayesian information criterion. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Athens, Greece, 5–9 September 2011; pp. 553–568. Zhang, J. Knowledge learning with crowdsourcing: A brief review and systematic perspective. IEEE/CAA J. Autom. Sin. 2022, 9, 749–762. Zhu, T.; Pimentel, M.A.; Clifford, G.D.; Clifton, D.A. Unsupervised Bayesian inference to fuse biosignal sensory estimates for personalizing care. IEEE J. Biomed. Health Inform. 2018, 23, 47–58. Song, H.; Kim, M.; Park, D.; Shin, Y.; Lee, J.G. Learning from noisy labels with deep neural networks: A survey. IEEE Trans. Neural Netw. Learn. Syst. 2022. Cheng, L.; Zhou, X.; Zhao, L.; Li, D.; Shang, H.; Zheng, Y.; Pan, P.; Xu, Y. Weakly supervised learning with side information for noisy labeled images. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020, Proceedings, Part XXX 16; Springer: Berlin/Heidelberg, Germany, 2020; pp. 306–321. Lee, K.; Yun, S.; Lee, K.; Lee, H.; Li, B.; Shin, J. Robust inference via generative classifiers for handling noisy labels. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 3763–3772. Chen, P.; Liao, B.B.; Chen, G.; Zhang, S. Understanding and utilizing deep neural networks trained with noisy labels. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 1062–1070. Yu, X.; Han, B.; Yao, J.; Niu, G.; Tsang, I.; Sugiyama, M. How does disagreement help generalization against label corruption? In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 7164–7173. Lyu, X.; Wang, J.; Zeng, T.; Li, X.; Chen, J.; Wang, X.; Xu, Z. TSS-Net: Two-stage with sample selection and semi-supervised net for deep learning with noisy labels. In Proceedings of the Third International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI 2022), SPIE, Guangzhou, China, 12–14 August 2022; Volume 12509, pp. 575–584. Shen, Y.; Sanghavi, S. Learning with bad training data via iterative trimmed loss minimization. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 5739–5748. Ghosh, A.; Manwani, N.; Sastry, P. Making risk minimization tolerant to label noise. Neurocomputing 2015, 160, 93–107. Ghosh, A.; Kumar, H.; Sastry, P.S. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017; Volume 31. Albarqouni, S.; Baur, C.; Achilles, F.; Belagiannis, V.; Demirci, S.; Navab, N. Aggnet: Deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 2016, 35, 1313–1321. Rodrigues, F.; Pereira, F. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 32. Chattopadhay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 839–847. Rizos, G.; Schuller, B.W. Average jane, where art thou?–recent avenues in efficient machine learning under subjectivity uncertainty. In Proceedings of the International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, Lisbon, Portugal, 15–19 June 2020; pp. 42–55 Zhang, J.; Wu, X.; Sheng, V.S. Imbalanced multiple noisy labeling. IEEE Trans. Knowl. Data Eng. 2014, 27, 489–503. Dawid, A.P.; Skene, A.M. Maximum likelihood estimation of observer error-rates using the EM algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.) 1979, 28, 20–28. Raykar, V.C.; Yu, S.; Zhao, L.H.; Valadez, G.H.; Florin, C.; Bogoni, L.; Moy, L. Learning from crowds. J. Mach. Learn. Res. 2010, 11, 1297–1322. Groot, P.; Birlutiu, A.; Heskes, T. Learning from multiple annotators with Gaussian processes. In Proceedings of the International Conference on Artificial Neural Networks, Espoo, Finland, 14–17 June 2011; pp. 159–164. Xiao, H.; Xiao, H.; Eckert, C. Learning from multiple observers with unknown expertise. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Gold Coast, Australia, 14–17 April 2013; pp. 595–606. Morales-Alvarez, P.; Ruiz, P.; Coughlin, S.; Molina, R.; Katsaggelos, A.K. Scalable variational Gaussian processes for crowdsourcing: Glitch detection in LIGO. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 44, 1534–1551. Gil-Gonzalez, J.; Alvarez-Meza, A.; Orozco-Gutierrez, A. Learning from multiple annotators using kernel alignment. Pattern Recognit. Lett. 2018, 116, 150–156. Morales-Álvarez, P.; Ruiz, P.; Santos-Rodríguez, R.; Molina, R.; Katsaggelos, A.K. Scalable and efficient learning from crowds with Gaussian processes. Inf. Fusion 2019, 52, 110–127 Rodrigues, F.; Pereira, F.; Ribeiro, B. Sequence labeling with multiple annotators. Mach. Learn. 2014, 95, 165–181. Wang, X.; Bi, J. Bi-convex optimization to learn classifiers from multiple biomedical annotations. IEEE/ACM Trans. Comput. Biol. Bioinform. 2016, 14, 564–575. Zhang, Z.; Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. Adv. Neural Inf. Process. Syst. 2018, 31, 1–11. Gil-González, J.; Valencia-Duque, A.; Álvarez-Meza, A.; Orozco-Gutiérrez, Á.; García-Moreno, A. Regularized chained deep neural network classifier for multiple annotators. Appl. Sci. 2021, 11, 5409. Zhao, X.; Li, X.; Bi, D.; Wang, H.; Xie, Y.; Alhudhaif, A.; Alenezi, F. L1-norm constraint kernel adaptive filtering framework for precise and robust indoor localization under the internet of things. Inf. Sci. 2022, 587, 206–225. Box, G.E.; Cox, D.R. An analysis of transformations. J. R. Stat. Soc. Ser. B (Methodol.) 1964, 26, 211–243. Saul, A.; Hensman, J.; Vehtari, A.; Lawrence, N. Chained Gaussian processes. In Proceedings of the Artificial Intelligence and Statistics, Cadiz, Spain, 9–11 May 2016; pp. 1431–1440 Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems; O’Reilly Media: Sebastopol, CA, USA, 2019. Xiao, H.; Rasul, K.; Vollgraf, R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv 2017, arXiv:1708.07747. Hernández-Muriel, J.A.; Bermeo-Ulloa, J.B.; Holguin-Londoño, M.; Álvarez-Meza, A.M.; Orozco-Gutiérrez, Á.A. Bearing health monitoring using relief-F-based feature relevance analysis and HMM. Appl. Sci. 2020, 10, 5170. LeCun, Y.; Boser, B.; Denker, J.; Henderson, D.; Howard, R.; Hubbard, W.; Jackel, L. Handwritten digit recognition with a back-propagation network. Adv. Neural Inf. Process. Syst. 1989, 2, 396–404. Dogs vs. Cats—Kaggle.com. Available online: https://www.kaggle.com/c/dogs-vs-cats (accessed on 6 January 2023). Peterson, J.C.; Battleday, R.M.; Griffiths, T.L.; Russakovsky, O. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27–28 October 2019; pp. 9617–9626. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556. Tzanetakis, G.; Cook, P. Musical genre classification of audio signals. IEEE Trans. Speech Audio Process. 2002, 10, 293–302. Rodrigues, F.; Pereira, F.; Ribeiro, B. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recognit. Lett. 2013, 34, 1428–1436. Gil-Gonzalez, J.; Giraldo, J.J.; Alvarez-Meza, A.; Orozco-Gutierrez, A.; Alvarez, M. Correlated Chained Gaussian Processes for Datasets with Multiple Annotators. IEEE Trans. Neural Netw. Learn. Syst. 2021. MacKay, D.J.; Mac Kay, D.J. Information Theory, Inference and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003. Demšar, J. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 2006, 7, 1–30. Li, Z.L.; Zhang, G.W.; Yu, J.; Xu, L.Y. Dynamic Graph Structure Learning for Multivariate Time Series Forecasting. Pattern Recognit. 2023, 138, 109423. Leroux, L.; Castets, M.; Baron, C.; Escorihuela, M.J.; Bégué, A.; Seen, D.L. Maize yield estimation in West Africa from crop process-induced combinations of multi-domain remote sensing indices. Eur. J. Agron. 2019, 108, 11–26. Montavon, G.; Binder, A.; Lapuschkin, S.; Samek, W.; Müller, K.R. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning; Springer Nature: Cham, Switzerland, 2019; pp. 193–209. Holzinger, A.; Saranti, A.; Molnar, C.; Biecek, P.; Samek, W. Explainable AI methods-a brief overview. In Proceedings of the xxAI-Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, Vienna, Austria, 18 July 2020, Revised and Extended Papers; Springer: Berlin/Heidelberg, Germany, 2022; pp. 13–38. Bennetot, A.; Donadello, I.; Qadi, A.E.; Dragoni, M.; Frossard, T.; Wagner, B.; Saranti, A.; Tulli, S.; Trocan, M.; Chatila, R.; et al. A practical tutorial on explainable ai techniques. arXiv 2021, arXiv:2111.14260. Saranti, A.; Hudec, M.; Mináriková, E.; Takáč, Z.; Großschedl, U.; Koch, C.; Pfeifer, B.; Angerschmid, A.; Holzinger, A. Actionable Explainable AI (AxAI): A Practical Example with Aggregation Functions for Adaptive Classification and Textual Explanations for Interpretable Machine Learning. Mach. Learn. Knowl. Extr. 2022, 4, 924–953.
dc.rights.none.fl_str_mv	© 2023 by the authors.
dc.rights.accessrights.none.fl_str_mv	info:eu-repo/semantics/openAccess
dc.rights.coar.none.fl_str_mv	http://purl.org/coar/access_right/c_abf2
dc.rights.license.none.fl_str_mv	Atribución-NoComercial 4.0 Internacional (CC BY-NC 4.0)
dc.rights.uri.none.fl_str_mv	https://creativecommons.org/licenses/by-nc/4.0/
rights_invalid_str_mv	© 2023 by the authors. http://purl.org/coar/access_right/c_abf2 Atribución-NoComercial 4.0 Internacional (CC BY-NC 4.0) https://creativecommons.org/licenses/by-nc/4.0/
eu_rights_str_mv	openAccess
dc.format.mimetype.none.fl_str_mv	application/pdf
dc.publisher.none.fl_str_mv	MDPI
dc.publisher.place.none.fl_str_mv	Suiza
publisher.none.fl_str_mv	MDPI
dc.source.none.fl_str_mv	https://www.mdpi.com/1424-8220/23/7/3518
institution	Universidad de Ibagué
bitstream.url.fl_str_mv	https://repositorio.unibague.edu.co/bitstreams/59defeca-895f-45bf-8e60-ddb262ff92d7/download https://repositorio.unibague.edu.co/bitstreams/f6d6b9e4-3e8f-4cdb-be28-eb3c81ad62af/download https://repositorio.unibague.edu.co/bitstreams/ec31ef08-6d78-4e58-bbde-3952e7d6baf3/download https://repositorio.unibague.edu.co/bitstreams/17c88058-ba23-418d-baec-8a1acc94812d/download
bitstream.checksum.fl_str_mv	3cca7b8dbd04f114dbb34fbd3eae3fc7 5020a86d3cefa1f3debe2590e2101c2e a11501b33891eb787fafc15634f336c2 2fa3e590786b9c0f3ceba1b9656b7ac3
bitstream.checksumAlgorithm.fl_str_mv	MD5 MD5 MD5 MD5
repository.name.fl_str_mv	Repositorio Institucional Universidad de Ibagué
repository.mail.fl_str_mv	bdigital@metabiblioteca.com
_version_	1851059988508704768
spelling	Gustavo Martinez Villalobos35a65c11-a18c-4c18-9fbe-0b91d60e3b62600Gil-González, Juliand4dc3d5d-c43a-46c7-813c-26d2fb86299b-1Fernandez-Gallego, Jose A.5c39dc3b-5876-4b8c-bceb-1e5bce9dc255-1Álvarez-Meza, Andrés Marinoe7baa84c-f992-4731-8f15-4cca0d36381c-1Castellanos-Dominguez, Cesar Germance4c7baa-c275-4476-84e4-cbbb2f4a961f-12025-09-01T21:26:39Z2025-09-01T21:26:39Z2023-04Supervised learning requires the accurate labeling of instances, usually provided by an expert. Crowdsourcing platforms offer a practical and cost-effective alternative for large datasets when individual annotation is impractical. In addition, these platforms gather labels from multiple labelers. Still, traditional multiple-annotator methods must account for the varying levels of expertise and the noise introduced by unreliable outputs, resulting in decreased performance. In addition, they assume a homogeneous behavior of the labelers across the input feature space, and independence constraints are imposed on outputs. We propose a Generalized Cross-Entropy-based framework using Chained Deep Learning (GCECDL) to code each annotator’s non-stationary patterns regarding the input space while preserving the inter-dependencies among experts through a chained deep learning approach. Experimental results devoted to multiple-annotator classification tasks on several well-known datasets demonstrate that our GCECDL can achieve robust predictive properties, outperforming state-of-the-art algorithms by combining the power of deep learning with a noise-robust loss function to deal with noisy labels. Moreover, network self-regularization is achieved by estimating each labeler’s reliability within the chained approach. Lastly, visual inspection and relevance analysis experiments are conducted to reveal the non-stationary coding of our method. In a nutshell, GCEDL weights reliable labelers as a function of each input sample and achieves suitable discrimination performance with preserved interpretability regarding each annotator’s trustworthiness estimation.application/pdfTriana-Martinez, J., Gil-González, J., Fernandez-Gallego, J., Álvarez-Meza, A. y Castellanos-Dominguez, C. (2023). Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification. Sensors, 23(7), 3518. DOI: 10.3390/s2307351810.3390/s2307351814248220https://hdl.handle.net/20.500.12313/5585engMDPISuiza7351823SensorsZhang, J.; Sheng, V.S.; Wu, J. Crowdsourced label aggregation using bilayer collaborative clustering. IEEE Trans. Neural Netw. Learn. Syst. 2019, 30, 3172–3185.Parvat, A.; Chavan, J.; Kadam, S.; Dev, S.; Pathak, V. A survey of deep-learning frameworks. In Proceedings of the 2017 International Conference on Inventive Systems and Control (ICISC), Coimbatore, India, 19–20 January 2017; pp. 1–7.Liu, Y.; Zhang, W.; Yu, Y. Truth inference with a deep clustering-based aggregation model. IEEE Access 2020, 8, 16662–16675.Gil-Gonzalez, J.; Orozco-Gutierrez, A.; Alvarez-Meza, A. Learning from multiple inconsistent and dependent annotators to support classification tasks. Neurocomputing 2021, 423, 236–247.Sung, H.E.; Chen, C.K.; Xiao, H.; Lin, S.D. A Classification Model for Diverse and Noisy Labelers. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining; Springer: Berlin/Heidelberg, Germany, 2017; pp. 58–69.Yan, Y.; Rosales, R.; Fung, G.; Subramanian, R.; Dy, J. Learning from multiple annotators with varying expertise. Mach. Learn. 2014, 95, 291–327.Xu, G.; Ding, W.; Tang, J.; Yang, S.; Huang, G.Y.; Liu, Z. Learning effective embeddings from crowdsourced labels: An educational case study. In Proceedings of the 2019 IEEE 35th International Conference on Data Engineering (ICDE), Macao, China, 8–11 April 2019; pp. 1922–1927.Tanno, R.; Saeedi, A.; Sankaranarayanan, S.; Alexander, D.C.; Silberman, N. Learning from noisy labels by regularized estimation of annotator confusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 11244–11253.Davani, A.M.; Díaz, M.; Prabhakaran, V. Dealing with disagreements: Looking beyond the majority vote in subjective annotations. Trans. Assoc. Comput. Linguist. 2022, 10, 92–110.Kara, Y.E.; Genc, G.; Aran, O.; Akarun, L. Modeling annotator behaviors for crowd labeling. Neurocomputing 2015, 160, 141–156.Cao, P.; Xu, Y.; Kong, Y.; Wang, Y. Max-mig: An information theoretic approach for joint learning from crowds. arXiv 2019, arXiv:1905.13436.Chen, Z.; Wang, H.; Sun, H.; Chen, P.; Han, T.; Liu, X.; Yang, J. Structured Probabilistic End-to-End Learning from Crowds. In Proceedings of the IJCAI, Yokohama, Japan, 7–21 January 2021; pp. 1512–1518.Ruiz, P.; Morales-Álvarez, P.; Molina, R.; Katsaggelos, A.K. Learning from crowds with variational Gaussian processes. Pattern Recognit. 2019, 88, 298–311.G. Rodrigo, E.; Aledo, J.A.; Gámez, J.A. Machine learning from crowds: A systematic review of its applications. Wiley Interdiscip. Rev. Data Min. Knowl. Discov. 2019, 9, e1288.Zhang, P.; Obradovic, Z. Learning from inconsistent and unreliable annotators by a gaussian mixture model and bayesian information criterion. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Athens, Greece, 5–9 September 2011; pp. 553–568.Zhang, J. Knowledge learning with crowdsourcing: A brief review and systematic perspective. IEEE/CAA J. Autom. Sin. 2022, 9, 749–762.Zhu, T.; Pimentel, M.A.; Clifford, G.D.; Clifton, D.A. Unsupervised Bayesian inference to fuse biosignal sensory estimates for personalizing care. IEEE J. Biomed. Health Inform. 2018, 23, 47–58.Song, H.; Kim, M.; Park, D.; Shin, Y.; Lee, J.G. Learning from noisy labels with deep neural networks: A survey. IEEE Trans. Neural Netw. Learn. Syst. 2022.Cheng, L.; Zhou, X.; Zhao, L.; Li, D.; Shang, H.; Zheng, Y.; Pan, P.; Xu, Y. Weakly supervised learning with side information for noisy labeled images. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, 23–28 August 2020, Proceedings, Part XXX 16; Springer: Berlin/Heidelberg, Germany, 2020; pp. 306–321.Lee, K.; Yun, S.; Lee, K.; Lee, H.; Li, B.; Shin, J. Robust inference via generative classifiers for handling noisy labels. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 3763–3772.Chen, P.; Liao, B.B.; Chen, G.; Zhang, S. Understanding and utilizing deep neural networks trained with noisy labels. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 1062–1070.Yu, X.; Han, B.; Yao, J.; Niu, G.; Tsang, I.; Sugiyama, M. How does disagreement help generalization against label corruption? In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 7164–7173.Lyu, X.; Wang, J.; Zeng, T.; Li, X.; Chen, J.; Wang, X.; Xu, Z. TSS-Net: Two-stage with sample selection and semi-supervised net for deep learning with noisy labels. In Proceedings of the Third International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI 2022), SPIE, Guangzhou, China, 12–14 August 2022; Volume 12509, pp. 575–584.Shen, Y.; Sanghavi, S. Learning with bad training data via iterative trimmed loss minimization. In Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA, 9–15 June 2019; pp. 5739–5748.Ghosh, A.; Manwani, N.; Sastry, P. Making risk minimization tolerant to label noise. Neurocomputing 2015, 160, 93–107.Ghosh, A.; Kumar, H.; Sastry, P.S. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017; Volume 31.Albarqouni, S.; Baur, C.; Achilles, F.; Belagiannis, V.; Demirci, S.; Navab, N. Aggnet: Deep learning from crowds for mitosis detection in breast cancer histology images. IEEE Trans. Med. Imaging 2016, 35, 1313–1321.Rodrigues, F.; Pereira, F. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, LA, USA, 2–7 February 2018; Volume 32.Chattopadhay, A.; Sarkar, A.; Howlader, P.; Balasubramanian, V.N. Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks. In Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Tahoe, NV, USA, 12–15 March 2018; pp. 839–847.Rizos, G.; Schuller, B.W. Average jane, where art thou?–recent avenues in efficient machine learning under subjectivity uncertainty. In Proceedings of the International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems, Lisbon, Portugal, 15–19 June 2020; pp. 42–55Zhang, J.; Wu, X.; Sheng, V.S. Imbalanced multiple noisy labeling. IEEE Trans. Knowl. Data Eng. 2014, 27, 489–503.Dawid, A.P.; Skene, A.M. Maximum likelihood estimation of observer error-rates using the EM algorithm. J. R. Stat. Soc. Ser. C (Appl. Stat.) 1979, 28, 20–28.Raykar, V.C.; Yu, S.; Zhao, L.H.; Valadez, G.H.; Florin, C.; Bogoni, L.; Moy, L. Learning from crowds. J. Mach. Learn. Res. 2010, 11, 1297–1322.Groot, P.; Birlutiu, A.; Heskes, T. Learning from multiple annotators with Gaussian processes. In Proceedings of the International Conference on Artificial Neural Networks, Espoo, Finland, 14–17 June 2011; pp. 159–164.Xiao, H.; Xiao, H.; Eckert, C. Learning from multiple observers with unknown expertise. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Gold Coast, Australia, 14–17 April 2013; pp. 595–606.Morales-Alvarez, P.; Ruiz, P.; Coughlin, S.; Molina, R.; Katsaggelos, A.K. Scalable variational Gaussian processes for crowdsourcing: Glitch detection in LIGO. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 44, 1534–1551.Gil-Gonzalez, J.; Alvarez-Meza, A.; Orozco-Gutierrez, A. Learning from multiple annotators using kernel alignment. Pattern Recognit. Lett. 2018, 116, 150–156.Morales-Álvarez, P.; Ruiz, P.; Santos-Rodríguez, R.; Molina, R.; Katsaggelos, A.K. Scalable and efficient learning from crowds with Gaussian processes. Inf. Fusion 2019, 52, 110–127Rodrigues, F.; Pereira, F.; Ribeiro, B. Sequence labeling with multiple annotators. Mach. Learn. 2014, 95, 165–181.Wang, X.; Bi, J. Bi-convex optimization to learn classifiers from multiple biomedical annotations. IEEE/ACM Trans. Comput. Biol. Bioinform. 2016, 14, 564–575.Zhang, Z.; Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. Adv. Neural Inf. Process. Syst. 2018, 31, 1–11.Gil-González, J.; Valencia-Duque, A.; Álvarez-Meza, A.; Orozco-Gutiérrez, Á.; García-Moreno, A. Regularized chained deep neural network classifier for multiple annotators. Appl. Sci. 2021, 11, 5409.Zhao, X.; Li, X.; Bi, D.; Wang, H.; Xie, Y.; Alhudhaif, A.; Alenezi, F. L1-norm constraint kernel adaptive filtering framework for precise and robust indoor localization under the internet of things. Inf. Sci. 2022, 587, 206–225.Box, G.E.; Cox, D.R. An analysis of transformations. J. R. Stat. Soc. Ser. B (Methodol.) 1964, 26, 211–243.Saul, A.; Hensman, J.; Vehtari, A.; Lawrence, N. Chained Gaussian processes. In Proceedings of the Artificial Intelligence and Statistics, Cadiz, Spain, 9–11 May 2016; pp. 1431–1440Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems; O’Reilly Media: Sebastopol, CA, USA, 2019.Xiao, H.; Rasul, K.; Vollgraf, R. Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. arXiv 2017, arXiv:1708.07747.Hernández-Muriel, J.A.; Bermeo-Ulloa, J.B.; Holguin-Londoño, M.; Álvarez-Meza, A.M.; Orozco-Gutiérrez, Á.A. Bearing health monitoring using relief-F-based feature relevance analysis and HMM. Appl. Sci. 2020, 10, 5170.LeCun, Y.; Boser, B.; Denker, J.; Henderson, D.; Howard, R.; Hubbard, W.; Jackel, L. Handwritten digit recognition with a back-propagation network. Adv. Neural Inf. Process. Syst. 1989, 2, 396–404.Dogs vs. Cats—Kaggle.com. Available online: https://www.kaggle.com/c/dogs-vs-cats (accessed on 6 January 2023).Peterson, J.C.; Battleday, R.M.; Griffiths, T.L.; Russakovsky, O. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27–28 October 2019; pp. 9617–9626.Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv 2014, arXiv:1409.1556.Tzanetakis, G.; Cook, P. Musical genre classification of audio signals. IEEE Trans. Speech Audio Process. 2002, 10, 293–302.Rodrigues, F.; Pereira, F.; Ribeiro, B. Learning from multiple annotators: Distinguishing good from random labelers. Pattern Recognit. Lett. 2013, 34, 1428–1436.Gil-Gonzalez, J.; Giraldo, J.J.; Alvarez-Meza, A.; Orozco-Gutierrez, A.; Alvarez, M. Correlated Chained Gaussian Processes for Datasets with Multiple Annotators. IEEE Trans. Neural Netw. Learn. Syst. 2021.MacKay, D.J.; Mac Kay, D.J. Information Theory, Inference and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003.Demšar, J. Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 2006, 7, 1–30.Li, Z.L.; Zhang, G.W.; Yu, J.; Xu, L.Y. Dynamic Graph Structure Learning for Multivariate Time Series Forecasting. Pattern Recognit. 2023, 138, 109423.Leroux, L.; Castets, M.; Baron, C.; Escorihuela, M.J.; Bégué, A.; Seen, D.L. Maize yield estimation in West Africa from crop process-induced combinations of multi-domain remote sensing indices. Eur. J. Agron. 2019, 108, 11–26.Montavon, G.; Binder, A.; Lapuschkin, S.; Samek, W.; Müller, K.R. Layer-wise relevance propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning; Springer Nature: Cham, Switzerland, 2019; pp. 193–209.Holzinger, A.; Saranti, A.; Molnar, C.; Biecek, P.; Samek, W. Explainable AI methods-a brief overview. In Proceedings of the xxAI-Beyond Explainable AI: International Workshop, Held in Conjunction with ICML 2020, Vienna, Austria, 18 July 2020, Revised and Extended Papers; Springer: Berlin/Heidelberg, Germany, 2022; pp. 13–38.Bennetot, A.; Donadello, I.; Qadi, A.E.; Dragoni, M.; Frossard, T.; Wagner, B.; Saranti, A.; Tulli, S.; Trocan, M.; Chatila, R.; et al. A practical tutorial on explainable ai techniques. arXiv 2021, arXiv:2111.14260.Saranti, A.; Hudec, M.; Mináriková, E.; Takáč, Z.; Großschedl, U.; Koch, C.; Pfeifer, B.; Angerschmid, A.; Holzinger, A. Actionable Explainable AI (AxAI): A Practical Example with Aggregation Functions for Adaptive Classification and Textual Explanations for Interpretable Machine Learning. Mach. Learn. Knowl. Extr. 2022, 4, 924–953.© 2023 by the authors.info:eu-repo/semantics/openAccesshttp://purl.org/coar/access_right/c_abf2Atribución-NoComercial 4.0 Internacional (CC BY-NC 4.0)https://creativecommons.org/licenses/by-nc/4.0/https://www.mdpi.com/1424-8220/23/7/3518Entropía cruzada generalizadaMúltiples anotadoresChained approachClassificationDeep learningGeneralized cross-entropyMultiple annotatorsChained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators ClassificationArtículo de revistahttp://purl.org/coar/resource_type/c_2df8fbb1http://purl.org/coar/version/c_970fb48d4fbd8a85Textinfo:eu-repo/semantics/articleinfo:eu-repo/semantics/publishedVersionPublicationTEXTArtículo.pdf.txtArtículo.pdf.txtExtracted texttext/plain7513https://repositorio.unibague.edu.co/bitstreams/59defeca-895f-45bf-8e60-ddb262ff92d7/download3cca7b8dbd04f114dbb34fbd3eae3fc7MD53THUMBNAILArtículo.pdf.jpgArtículo.pdf.jpgIM Thumbnailimage/jpeg30496https://repositorio.unibague.edu.co/bitstreams/f6d6b9e4-3e8f-4cdb-be28-eb3c81ad62af/download5020a86d3cefa1f3debe2590e2101c2eMD54ORIGINALArtículo.pdfArtículo.pdfapplication/pdf182196https://repositorio.unibague.edu.co/bitstreams/ec31ef08-6d78-4e58-bbde-3952e7d6baf3/downloada11501b33891eb787fafc15634f336c2MD51LICENSElicense.txtlicense.txttext/plain; charset=utf-8134https://repositorio.unibague.edu.co/bitstreams/17c88058-ba23-418d-baec-8a1acc94812d/download2fa3e590786b9c0f3ceba1b9656b7ac3MD5220.500.12313/5585oai:repositorio.unibague.edu.co:20.500.12313/55852025-09-02 03:02:55.741https://creativecommons.org/licenses/by-nc/4.0/© 2023 by the authors.https://repositorio.unibague.edu.coRepositorio Institucional Universidad de Ibaguébdigital@metabiblioteca.comQ3JlYXRpdmUgQ29tbW9ucyBBdHRyaWJ1dGlvbi1Ob25Db21tZXJjaWFsLU5vRGVyaXZhdGl2ZXMgNC4wIEludGVybmF0aW9uYWwgTGljZW5zZQ0KaHR0cHM6Ly9jcmVhdGl2ZWNvbW1vbnMub3JnL2xpY2Vuc2VzL2J5LW5jLW5kLzQuMC8=

Chained Deep Learning Using Generalized Cross-Entropy for Multiple Annotators Classification

Publicaciones similares