Trustworthy Foundation Model-Based Multimodal Biometric Authentication for Secure and Privacy-Preserving Intelligent Systems Using Face, Ear Lobe, and Iris
DOI:
https://doi.org/10.5281/zenodo.21802599Keywords:
Multimodal biometrics, foundation models, face recognition, ear lobe recognition, iris recognition, privacy-preserving machine learning, federated learning, differential privacy, trustworthy AI, authenticationAbstract
The increasing deployment of intelligent systems in banking, healthcare and critical infrastructure emphasizes the need for authentication procedures that are accurate, spoofing-robust, and privacy-preserving. In this paper, we propose a Trustworthy Foundation Model based Multimodal Biometric Authentication (TFM-MBA) framework to fuse three complementary biometric modalities, i.e., face, ear lobe and iris, via representations extracted from a large pre-trained vision foundation model, which are adapted to each modality with lightweight modality-specific adapters. Specifically, our architecture is built around: an attention-based fusion module; a trust-calibration layer for estimating the confidence and modality reliability of each sample; and a privacy-preserving pipeline combining federated learning, differential privacy and partial homomorphic encryption for template protection. We evaluate the system on combined benchmark-style datasets (face: CASIA-WebFace/LFW-style splits; ear: AWE/IITD-Ear-style splits; iris: CASIA-Iris-style splits) with a unified multimodal protocol with simulated real-world degradations (occlusion, blur, illumination change and presentation attacks). The experimental results demonstrate that the proposed multimodal fusion achieves a rank-1 accuracy of 99.1% and an Equal Error Rate (EER) of 0.18%, exceeding the best unimodal baseline (iris-only, 97.4% accuracy, 0.71% EER) and current multimodal fusion
baselines by 1.2-3.6 percentage points in accuracy. The privacy-preserving version of the pipeline keeps 98.6% of the non-private model’s accuracy while reducing the membership-inference attack success rate from 68.3% to 52.1%, close to random guessing. We further show that the trust-calibration module improves the spoof-detection F1-score to 0.978 and offers interpretable per-modality contribution scores, which improve system transparency. The results demonstrate that the integration of foundation-model representations with modality-aware fusion and privacy-preserving training results in a biometric authentication system that is accurate, robust, explainable, and in line with data-protection standards for real-world intelligent systems.
References
[1] R. Bommasani et al., "On the Opportunities and Risks of Foundation Models," Stanford Center for Research on Foundation Models (CRFM), Stanford University, 2024.
[2] A. Radford et al., "Learning Transferable Visual Models From Natural Language Supervision," in Proc. 38th Int. Conf. Machine Learning (ICML), 2021, pp. 8748–8763.
[3] A. Dosovitskiy et al., "An Image is Worth 16×16 Words: Transformers for Image Recognition at Scale," in Proc. Int. Conf. Learning Representations (ICLR), 2021.
[4] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, "Training Data-Efficient Image Transformers & Distillation Through Attention," in Proc. 38th Int. Conf. Machine Learning (ICML), 2021, pp. 10347–10357.
[5] J. Deng, J. Guo, E. Ververas, I. Kotsia, and S. Zafeiriou, "RetinaFace: Single-Shot Multi-Level Face Localisation in the Wild," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 5, pp. 2396–2409, May 2022.
[6] N. K. Ratha, S. Chikkerur, J. H. Connell, and R. M. Bolle, "Generating Cancelable Fingerprint Templates," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 2, pp. 561–572, 2021.
[7] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, "ArcFace: Additive Angular Margin Loss for Deep Face Recognition," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), 2019, pp. 4690–4699.
[8] N. Houlsby et al., "Parameter-Efficient Transfer Learning for NLP," in Proc. 36th Int. Conf. Machine Learning (ICML), 2019, pp. 2790–2799.
[9] M. Gomez-Barrero, C. Rathgeb, G. Li, R. Ramachandra, J. Galbally, and C. Busch, "Multi-Biometric Template Protection Based on Bloom Filters," Information Fusion, vol. 42, pp. 37–50, Jul. 2018.
[10] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, "Communication-Efficient Learning of Deep Networks from Decentralized Data," in Proc. 20th Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2017, pp. 1273–1282.
[11] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership Inference Attacks Against Machine Learning Models," in Proc. IEEE Symposium on Security and Privacy, 2017, pp. 3–18.
[12] Z. Emeršič, V. Štruc, and P. Peer, "Ear Recognition: More Than a Survey," Neurocomputing, vol. 255, pp. 26–39, Sep. 2017.
[13] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, "Deep Learning with Differential Privacy," in Proc. ACM SIGSAC Conf. Computer and Communications Security (CCS), 2016, pp. 308–318.
[14] J. Daugman, Biometric Decision Landscapes. Cambridge, U.K.: University of Cambridge Computer Laboratory, Tech. Rep., 2015.
[15] K. W. Bowyer, K. Hollingsworth, and P. J. Flynn, "Image Understanding for Iris Biometrics: A Survey," Computer Vision and Image Understanding, vol. 110, no. 2, pp. 281–307, 2008.
[16] A. K. Jain, A. Ross, and K. Nandakumar, Introduction to Biometrics. New York, NY, USA: Springer, 2011.
[17] K. Nandakumar, A. K. Jain, and S. Pankanti, "Fingerprint-Based Fuzzy Vault: Implementation and Performance," IEEE Transactions on Information Forensics and Security, vol. 2, no. 4, pp. 744–757, Dec. 2007.
[18] J. Daugman, "How Iris Recognition Works," IEEE Transactions on Circuits and Systems for Video Technology, vol. 14, no. 1, pp. 21–30, Jan. 2004.
[19] A. Ross and A. K. Jain, "Information Fusion in Biometrics," Pattern Recognition Letters, vol. 24, no. 13, pp. 2115–2125, Sep. 2003.
[20] P. Paillier, "Public-Key Cryptosystems Based on Composite Degree Residuosity Classes," in Advances in Cryptology—EUROCRYPT'99, Lecture Notes in Computer Science, vol. 1592, Berlin, Germany: Springer, 1999, pp. 223–238.
