A Hybrid Stacking Model for Legal Document Classification Using TF-IDF Features
DOI:
https://doi.org/10.5281/zenodo.21817471Keywords:
Legal NLP, Document Classification, Hybrid Stacking Model, TF-IDF, Support Vector Machine, Extra Trees, Multi-Layer Perceptron, Logistic RegressionAbstract
In Legal Natural Language Processing (Legal NLP), legal document classification is a fundamental task that can help legal professionals to automatically analyze case documents and determine the treatment of previous judicial decisions in new cases. Traditional machine learning models are good at efficiency but don't have a semantic understanding while deep learning models are more flexible in extracting contextual information, but are prone to over-fitting and have high computational costs. In this paper, a hybrid stacking model is proposed, which integrates the classical machine learning, ensemble learning, and neural learning, so that it can achieve a good balance between accuracy, robustness and efficiency. The proposed architecture is divided into two texts and then uses the TF-IDF feature extraction for the legal document. The extracted features are fed to the base classifiers as the linear support vector machine, extra trees classifier and multi-layer perceptron and the outputs of the base classifiers are passed to the logistic regression meta-classifier for stacking. The model was tested on a synthetic legal corpus that was created using a set of 660 documents, split 80:20 train:test and five-fold crossvalidation. The hybrid model performed best with a test accuracy of 94.7%, a cross-validation accuracy of 92.1% and a macro F1-score of 94.7%, which is similar to the best individual classifiers, and has better robustness due to stacking.
References
1. Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D., & Lampos, V. (2016). Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective. PeerJ Computer Science, 2, e93. https://doi.org/10.7717/peerj-cs.93
2. Al-Qurishi, M., AlQaseemi, S., & Souissi, R. (2022). AraLegal-BERT: A pretrained language model for Arabic legal text. In Proceedings of the Natural Legal Language Processing Workshop 2022 (pp. 338–344). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.nllp-1.31
3. Ariai, F., & Demartini, G. (2025). Natural language processing for the legal domain: A survey of tasks, datasets, models, and challenges. ACM Computing Surveys. Advance online publication. https://doi.org/10.1145/3777009
4. Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., & Androutsopoulos, I. (2020). LEGAL-BERT: The muppets straight out of law school. In Findings of the Association for Computational Linguistics: EMNLP 2020 (pp. 2898–2904). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.findings-emnlp.261
5. Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D. M., & Aletras, N. (2022). LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1, pp. 4310–4330). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.297
6. Chalkidis, I., Pasini, T., Zhang, S., Tomada, L., Schwemer, S., & Søgaard, A. (2022). FairLex: A multilingual benchmark for evaluating fairness in legal text processing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol. 1, pp. 4389–4406). Association for Computational Linguistics.
7. Cui, J., Shen, X., Nie, F., Wang, Z., Wang, J., & Chen, Y. (2023). A survey on legal judgment prediction: Datasets, metrics, models and challenges. IEEE Access, 11, 102050–102071. https://doi.org/10.1109/ACCESS.2023.3317083
8. Elaraby, M., & Litman, D. (2022). ArgLegalSumm: Improving abstractive summarization of legal documents with argument mining. In Proceedings of the 29th International Conference on Computational Linguistics (pp. 6187–6194). International Committee on Computational Linguistics.
9. Elnaggar, A., Otto, R., & Matthes, F. (2018). Named-entity linking using deep learning for legal documents: A transfer learning approach. arXiv. https://arxiv.org/abs/1810.06673
10. Hendrycks, D., Burns, C., Chen, A., & Ball, S. (2021). CUAD: An expert-annotated NLP dataset for legal contract review. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS 2021). https://arxiv.org/abs/2103.06268
11. Katz, D. M., Bommarito, M. J., II, & Blackman, J. (2017). A general approach for predicting the behavior of the Supreme Court of the United States. PLOS ONE, 12(4), Article e0174698. https://doi.org/10.1371/journal.pone.0174698
12. Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2023). GPT-4 passes the bar exam [Working paper]. SSRN. https://doi.org/10.2139/ssrn.4389233
13. Malik, V., Sanjay, R., Nigam, S. K., Ghosh, K., Guha, S. K., Bhattacharya, A., & Modi, A. (2021). ILDC for CJPE: Indian legal documents corpus for court judgment prediction and explanation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol. 1, pp. 4046–4062). Association for Computational Linguistics.
14. Nigam, S. K., & Goel, N. (2022). nigam@COLIEE-22: Legal case retrieval and entailment using cascading of lexical and semantic-based models. arXiv. https://arxiv.org/abs/2204.07853
15. Pakhale, K. (2023). Comprehensive overview of named entity recognition: Models, domain-specific applications and challenges. arXiv. https://arxiv.org/abs/2309.14084
16. Shukla, A., Bhattacharya, P., Poddar, S., Mukherjee, R., Ghosh, K., Goyal, P., & Ghosh, S. (2022). Legal case document summarization: Extractive and abstractive methods and their evaluation. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Vol. 1, pp. 1048–1064). Association for Computational Linguistics.
17. Wu, Y., Zhou, S., Liu, Y., Lu, W., Liu, X., Zhang, Y., Sun, C., Wu, F., & Kuang, K. (2023). Precedent-enhanced legal judgment prediction with LLM and domain-model collaboration. arXiv. https://arxiv.org/abs/2310.09241
18. Yu, W., & Ti, P. (2024). A study of legal judgment prediction based on deep learning multi-fusion models—Data from China. SAGE Open, 14(3). https://doi.org/10.1177/21582440241257682
19. Zheng, L., Guha, N., Anderson, B. R., Henderson, P., & Ho, D. E. (2021). When does pretraining help? Assessing self-supervised learning for law and the CaseHOLD dataset. In Proceedings of the 18th International Conference on Artificial Intelligence and Law (pp. 159–168). Association for Computing Machinery. https://doi.org/10.1145/3462757.3466088
20. Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020). How does NLP benefit legal system: A summary of legal artificial intelligence. arXiv. https://arxiv.org/abs/2004.12158
