Random Forest vs Elastic-Net Penalized Logistic Regression for Patient Discharge Classification in BPJS Primary Care

David Kevin Hutabarat; Sawaluddin Sawaluddin; Syahriol Sitorus; Sutarman Sutarman

doi:10.20956/j.v22i2.48609

Authors

David Kevin Hutabarat Department of Mathematics, Universitas Sumatera Utara
Sawaluddin Sawaluddin Department of Mathematics, Universitas Sumatera Utara
Syahriol Sitorus Department of Mathematics, Universitas Sumatera Utara
Sutarman Sutarman Department of Mathematics, Universitas Sumatera Utara

DOI:

https://doi.org/10.20956/j.v22i2.48609

Keywords:

BPJS Kesehatan, Elastic Net, Hyperparameter Tuning, Imbalanced Classification, Random Forest

Abstract

This study analyzes and compares Random Forest and Penalized Logistic Regression (Elastic Net, SAGA solver) for classifying patient discharge status at BPJS Kesehatan primary care facilities (FKTP). The large-scale dataset consists entirely of nominal predictors with class imbalance (~64.9% majority). The experimental design applies an 80/20 train–test split, one-hot encoding, and class_weight = balanced for both models. Hyperparameters are tuned via a staged coarse→fine randomized search with a local-optimum convergence rule (improvement threshold ε = 1e−6, patience = 10), followed by 10-fold cross-validation for internal validation and final testing on the hold-out set. We evaluate three primary metrics: F1-Score, Precision–Recall AUC (PR-AUC), and Brier Score. On the test set, Random Forest attains F1 = 0.996679, PR-AUC = 0.999933, and Brier = 0.002646; Penalized Logistic Regression attains F1 = 0.996676, PR-AUC = 0.999928, and Brier = 0.002017. The near-identical F1 and PR-AUC indicate comparable discrimination between methods, while the lower Brier Score for Penalized Logistic Regression demonstrates superior probability calibration. Overall, both approaches lie on the same performance plateau for discrimination, with a consistent calibration advantage for Penalized Logistic Regression; method choice can thus be guided by whether operational needs prioritize calibrated probabilities or flexible non-linear decision boundaries.

References

[1] Austin, A. M., Ramkumar, N., Gladders, B., Barnes, J. A., Eid, M. A., Moore, K. O., Feinberg, M. W., Creager, M. A., Bonaca, M., & Goodney, P. P., 2022. Using a cohort study of diabetes and peripheral artery disease to compare logistic regression and machine learning via random forest modeling, BMC Medical Research Methodology, Vol. 22, No. 300, 1–10, doi: 10.1186/s12874-022-01774-8.

[2] Bouthillier, X., Laurent, C., & Vincent, P., 2019. Unreproducible Research is Reproducible, in Proceedings of the 36th International Conference on Machine Learning, 2019, 725–734. [Online]. Available: https://proceedings.mlr.press/v97/bouthillier19a

[3] Boyd, K., Eng, K. H., & Page, C. D., 2013. Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals, in Joint European conference on machine learning and knowledge discovery in databases, Berlin: Springer Berlin Heidelberg, 2013, 451–466. doi: 10.1007/978-3-642-40994-3_29.

[4] Breiman, L., 2001. Random Forests, Machine Learning, Vol. 45, 5–32, doi: 10.1023/A:1010933404324.

[5] Breiman, L., Jerome Friedman, R.A. Olshen, & Charles J. Stone, 1984. Classification and Regression Trees, 1st ed. New York, 1984. doi: 10.1201/9781315139470.

[6] Brier, G. W., 1950. Verification of forecasts expressed in terms of probability, Monthly Weather Review, Vol. 78, No. 1, 1–3, doi: 10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2.

[7] Bröcker, J., 2009. Reliability , Sufficiency , and the Decomposition of, The Quarterly Journal of the Royal Meteorological Society, Vol. 135, No. 643, 1512–1519, doi: 10.1002/qj.456.

[8] Calster, B. Van, Mclernon, D. J., Smeden, M. Van, Wynants, L., & Steyerberg, E. W., 2019. Calibration: the Achilles heel of predictive analytics, BMC Medicine, Vol. 17, No. 230, doi: 10.1186/s12916-019-1466-7.

[9] Chen, C., Liaw, A., & Breiman, L., 2004. Using Random Forest to Learn Imbalanced Data. [Online]. Available: https://statistics.berkeley.edu/tech-reports/666

[10] Chowdhury, M. Z. I., Leung, A. A., Walker, R. L., Sikdar, K. C., Beirne, M. O., Quan, H., & Turin, T. C., 2023. A comparison of machine learning algorithms and traditional regression ‑ based statistical modeling for predicting hypertension incidence in a Canadian population, Scientific Reports, Vol. 13, No. 13, doi: 10.1038/s41598-022-27264-x.

[11] Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, Y., & Calster, B. Van, 2019. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models, Journal of Clinical Epidemiology, Vol. 110, 12–22, doi: 10.1016/j.jclinepi.2019.02.004.

[12] Davis, J., & Goadrich, M., 2006. The Relationship Between Precision-Recall and ROC Curves, in Proceedings of the 23rd International Conference on Machine Learning, 2006, 233–240. doi: 10.1145/1143844.1143874.

[13] Davis, S. E., Lasko, T. A., Chen, G., Siew, E. D., & Matheny, M. E., 2017. Calibration drift in regression and machine learning models for acute kidney injury, Journal of the American Medical Informatics Association, Vol. 24, No. 6, 1052–1061, doi: 10.1093/jamia/ocx030.

[14] Defazio, A., Bach, F., & Lacoste-Julien, S., 2014. SAGA: A Fast Incremental Gradient Method with Support for Non-Strongly Convex Composite Objectives, Advances in Neural Information Processing Systems, Vol. 2, No. January, 1646–1654, doi: 10.48550/arXiv.1407.0202.

[15] Eriksson, D., Pearce, M., Gardner, J. R., Turner, R., & Poloczek, M., 2019. Scalable Global Optimization via Local Bayesian Optimization, in Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019, 5496–5507. doi: 10.48550/arXiv.1910.01739.

[16] Falkner, S., Klein, A., & Hutter, F., 2018. BOHB: Robust and Efficient Hyperparameter Optimization at Scale, in Proceedings of the 35th International Conference on Machine Learning, 2018, 1437–1446. [Online]. Available: https://proceedings.mlr.press/v80/falkner18a

[17] Farhan, M., Santosa, B., & Sholihah, M., 2024. Identification of Referral Pattern in Indonesian Primary Healthcare Facilities Using Data Mining Techniques, in IEEE Technology & Engineering Management Conference - Asia Pacific (TEMSCON-ASPAC), 2024, 1–6. doi: 10.1109/TEMSCON-ASPAC62480.2024.11024874.

[18] Feurer, M., Eggensperger, K., Falkner, S., Lindauer, M., & Hutter, F., 2022. Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning, Journal of Machine Learning Research, Vol. 23, 1–61, doi: 10.48550/arXiv.2007.04074.

[19] Flach, P. A., 2015. Precision-Recall-Gain Curves: PR Analysis Done Right, in Proceedings of the 29th International Conference on Neural Information Processing Systems, 2015, 838–846. [Online]. Available: https://dl.acm.org/doi/abs/10.5555/2969239.2969333

[20] Galiën, O. P. Van Der, Hoekstra, R. C., Gürgöze, M. T., Manintveld, O. C., Bunt, M. R. Van Den, Veenman, C. J., & Boersma, E., 2021. Prediction of long‑term hospitalisation and all‑cause mortality in patients with chronic heart failure on Dutch claims data: a machine learning approach, BMC Medical Informatics and Decision Making, Vol. 21, No. 303, doi: 10.1186/s12911-021-01657-w.

[21] Genuer, R., Poggi, J.-M., & Tuleau-Malot, C., 2010. Variable selection using random forests, Pattern Recognition Letters, Vol. 31, No. 14, 2225–2236, doi: 10.1016/j.patrec.2010.03.014.

[22] Gneiting, T., & Raftery, A. E., 2007. Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association, Vol. 102, No. 477, 359–378, doi: 10.1198/016214506000001437.

[23] Handayani, P. W., Dartanto, T., Moeis, F. R., Pinem, A. A., Azzahro, F., Hidayanto, A. N., Denny, & Ayuningtyas, D., 2021. The regional and referral compliance of online healthcare systems by Indonesia National Health Insurance agency and health-seeking behavior in Indonesia, Heliyon, Vol. 7, No. 9, doi: 10.1016/j.heliyon.2021.e08068.

[24] Hendrawan, D., Ariawan, I., Sartono, B., Wahyuningsih, W., Negara, S. I., Mawardi, J., Fatah, C. J., Sutara, F. A., & Nugraha, N. S., 2021. Data Sampel BPJS Kesehatan 2015-2020, Jakarta. [Online]. Available: https://data.bpjs-kesehatan.go.id/bpjs-portal/action/blog-detail.cbi?id=79f03774-6397-11ec-bd5e-bb284b79c3ff

[25] Ho, T. K., 1995. Random Decision Forests, in Proceedings of 3rd International Conference on Document Analysis and Recognition, 1995, 8–12. doi: 10.1109/ICDAR.1995.598994.

[26] Hoerl, A. E., & Kennard, R. W., 1970. Ridge Regression: Biased Estimation for Nonorthogonal Problems, Technometrics, Vol. 12, No. 1, 55–67, doi: 10.2307/1267351.

[27] Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X., 2013. Applied Logistic Regression, 3rd ed. New Jersey: John Wiley & Sons, Inc., 2013.

[28] Kadra, A., Hutter, F., Lindauer, M., & Grabocka, J., 2021. Well-tuned Simple Nets Excel on Tabular Datasets, in 35th Conference on Neural Information Processing Systems, 2021, 1–24. doi: 10.48550/arXiv.2106.11189.

[29] Kull, M., Filho, T. de M. e S., & Flach, P., 2017. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers, in Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017. [Online]. Available: https://proceedings.mlr.press/v54/kull17a

[30] Motz, M., Krauß, J., & Schmitt, R. H., 2022. Benchmarking of hyperparameter optimization techniques for machine learning applications in production, Advances in Industrial and Manufacturing Engineering, Vol. 5, doi: 10.1016/j.aime.2022.100099.

[31] Murphy, A. H., 1973. A New Vector Partition of the Probability Score, Journal of Applied Meteorology, Vol. 12, No. 4, 595–600, doi: 10.1175/1520-0450(1973)012<0595:ANVPOT>2.0.CO;2.

[32] Nasional, D. J. S., 2025. Monthly Report Monitoring JKN. [Online]. Available: https://kesehatan.djsn.go.id/kesehatan/doc/laporan-bulanan/Monthly_Report_JKN_06_2025.pdf

[33] Nembrini, S., Konig, I. R., & Wright, M. N., 2018. The revival of the Gini importance?, Bioinformatics, Vol. 34, No. 21, 3711–3718, doi: 10.1093/bioinformatics/bty373.

[34] Park, Y., & Ho, J. C., 2020. CaliForest: Calibrated Random Forest for Health Data, in CHIL ’20: Proceedings of the ACM Conference on Health, Inference, and Learning, Toronto, 2020, 40–50. doi: 10.1145/3368555.3384461.

[35] Powers, D. M. W., 2011. Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness & Correlation, Journal of Machine Learning Technologies, Vol. 2, No. 1, 37–63, doi: https://doi.org/10.48550/arXiv.2010.16061.

[36] Probst, P., Wright, M. N., & Boulesteix, A. L., 2019. Hyperparameters and tuning strategies for random forest, May 01, 2019, Wiley-Blackwell. doi: 10.1002/widm.1301.

[37] Rachael, M., 2020. Comparison of Elastic Net and Random Forest in identifying risk factors of stunting in children under five years of age in Kenya, University of Nairobi, 2020. [Online]. Available: https://erepository.uonbi.ac.ke/handle/11295/154085

[38] Saito, T., & Rehmsmeier, M., 2015. The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets, PLOS ONE, Vol. 10, No. 3, 1–21, doi: 10.1371/journal.pone.0118432.

[39] Sanchez-Pinto, L. N., Venable, L. R., Fahrenbach, J., & Churpek, M. M., 2018. Comparison of Variable Selection Methods for Clinical Predictive Modeling, International Journal of Medical Informatics, Vol. 116, 10–17, doi: 10.1016/j.ijmedinf.2018.05.006.

[40] Schmidt, M., Le Roux, N., & Bach, F., 2017. Minimizing Finite Sums with the Stochastic Average Gradient, Mathematical Programming, Vol. 162, No. 1–2, 83–112, doi: 10.1007/s10107-016-1030-6.

[41] Seyam, E. A., 2025. Predicting High-Cost Healthcare Utilization Using Machine Learning : A Multi-Service Risk Stratification Analysis in EU-Based Private Group Health Insurance, Risks, Vol. 13, No. 7, 133, doi: 10.3390/risks13070133.

[42] Shrestha, A., Bergquist, S., Montz, E., & Rose, S., 2018. Mental Health Risk Adjustment with Clinical Categories and Machine Learning, Health Services Research, Vol. 53, 3189–3206, doi: 10.1111/1475-6773.12818.

[43] Si, Y., Sun, L., Chen, S., Fan, J., Pishgar, E., Alaei, K., Placencia, G., & Pishgar, M., 2025. Retrospective Machine Learning Approach for Forecasting In-Hospital Death in ICU Patients After Cardiac Arrest, medRxiv, 1–22, doi: 10.1101/2025.05.05.25327009.

[44] Steele, A. J., Denaxas, S. C., Shah, A. D., Hemingway, H., & Luscombe, N. M., 2018. Machine learning models in electronic health records can outperform conventional survival models for predicting patient mortality in coronary artery disease, PLoS ONE, Vol. 13, No. 8, 1–20, doi: 10.1371/journal.pone.0202344.

[45] Strobl, C., Boulesteix, A., Kneib, T., Augustin, T., & Zeileis, A., 2008. Conditional variable importance for random forests, BMC Bioinformatics, Vol. 9, No. 307, 1–11, doi: 10.1186/1471-2105-9-307.

[46] Su, Y., Buist, D. S. M., Lee, J. M., Ichikawa, L., Miglioretti, D. L., Erin, J., Bowles, A., Wernli, K. J., Kerlikowske, K., Tosteson, A., Lowry, K. P., Henderson, L. M., Sprague, B. L., & Hubbard, R. A., 2023. Performance of statistical and machine learning risk prediction models for surveillance benefits and failures in breast cancer survivors, Cancer Epidemiol Biomarkers Prev, Vol. 32, No. 4, 561–571, doi: 10.1158/1055-9965.EPI-22-0677.

[47] Tay, J. K., Narasimhan, B., & Hastie, T., 2023. Elastic Net Regularization Paths for All Generalized Linear Models, Journal of Statistical Software, Vol. 106, No. 1, doi: 10.18637/jss.v106.i01.

[48] Thongpeth, W., Lim, A., Wongpairin, A., Thongpeth, T., & Chaimontree, S., 2021. Comparison of linear, penalized linear and machine learning models predicting hospital visit costs from chronic disease in Thailand, Informatics in Medicine Unlocked, Vol. 26, doi: 10.1016/j.imu.2021.100769.

[49] Tibshirani, R., 1996. Regression Shrinkage and Selection Via the Lasso, Journal of the Royal Statistical Society. Series B: Methodological, Vol. 58, No. 1, 267–288, doi: 10.1111/j.2517-6161.1996.tb02080.x.

[50] Timofeev, R., 2004. Classification and Regression Trees (CART) Theory and Applications, Humboldt University, Berlin, 2004.

[51] Varma, S., & Simon, R., 2006. Bias in error estimation when using cross-validation for model selection, BMC Bioinformatics, Vol. 7, No. 91, doi: 10.1186/1471-2105-7-91.

[52] Xiao, Y., Xing, E. P., & Neiswanger, W., 2022. Amortized Auto-Tuning: Cost-Efficient Bayesian Transfer Optimization for Hyperparameter Recommendation, arXiv, 1–23, doi: 10.48550/arXiv.2106.09179.

[53] Zela, A., Siems, J., Zimmer, L., Lukasik, J., Keuper, M., & Hutter, F., 2022. Surrogate NAS Benchmarks: Going Beyond the Limited Search Spaces of Tabular NAS Benchmarks, in International Conference on Learning Representations, 2022. doi: https://doi.org/10.48550/arXiv.2008.09777.

[54] Zhou, K., Hong, L., Hu, S., Zhou, F., Ru, B., Feng, J., & Li, Z., 2022. DHA: End-to-End Joint Optimization of Data Augmentation, Transactions on Machine Learning Research, 1–19, doi: 10.48550/arXiv.2109.05765.

[55] Zou, H., & Hastie, T., 2005. Regularization and variable selection via the elastic net, Journal of the Royal Statistical Society. Series B (Statistical Methodology), Vol. 67, No. 2, 301–320, [Online]. Available: http://www.jstor.org/stable/3647580