Multiclass Benchmarks

Dataset Overview

Dataset Instances Features Classes Numeric Categorical Target
Cmc 1473 9 3 2 7 Contraceptive_method_used
Cnae-9 1080 856 9 856 0 Class

Quick Check

Dataset Type Family / Variant Test accuracy Test F1
Cmc Best classical Trees / CatBoost 0.56380 ± 0.01882 0.53572 ± 0.02054
Best transformed ViT / BarGraph 0.56290 ± 0.01452 0.56045 ± 0.01604
Cnae-9 Best classical MLP / MLP 0.97037 ± 0.00805 0.96992 ± 0.00846
Best transformed ViT+MLP / REFINED 0.96173 ± 0.00517 0.96194 ± 0.00487

Cmc

Source: OpenML · Original

Leaderboard

Family Best variant Test Accuracy (↑) Test F1 (↑) Train time (s) #Params FLOPs
Trees CatBoost 0.56380 ± 0.01882 0.53572 ± 0.02054 6.64661 — —
MLP MLP 0.52670 ± 0.01711 0.52548 ± 0.01566 20.21081 899,000 53,760
ViT BarGraph 0.56290 ± 0.01452 0.56045 ± 0.01604 70.66815 3,222,435 31,831,688
ViT+MLP Combination 0.55837 ± 0.01855 0.55904 ± 0.01354 54.89044 1,167,587 257,758,240
CNN Combination 0.55475 ± 0.02707 0.55731 ± 0.02529 37.43239 26,275 8,797,568
CNN+MLP Combination 0.54751 ± 0.01535 0.54588 ± 0.00203 38.03557 33,571 96,932,704
Architecture Results
Tree Baselines
Method Test Acc Test F1 Test Precision Test Recall Train time (s)
XGBoost 0.55385 ± 0.00405 0.51932 ± 0.00482 0.53787 ± 0.00435 0.51698 ± 0.00474 0.71723 ± 0.01057
CatBoost 0.56380 ± 0.01882 0.53572 ± 0.02054 0.55398 ± 0.01577 0.53372 ± 0.01914 6.64661 ± 0.10867
LightGBM 0.55928 ± 0.01380 0.53068 ± 0.01289 0.54305 ± 0.01377 0.52804 ± 0.01305 0.49725 ± 0.09115
MLP
Train loss Val loss Test loss Test accuracy Test precision Test recall Test F1 Test ROC AUC Test LogLoss Test MCC Total params FLOPs MACs
0.77463 ± 0.01142 0.90521 ± 0.00770 0.97395 ± 0.00960 0.52670 ± 0.01711 0.53118 ± 0.01556 0.52670 ± 0.01711 0.52548 ± 0.01566 0.70562 ± 0.00858 0.97326 ± 0.01015 0.26985 ± 0.02506 899,000 53,760 26,624
ViT
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 116.61239 ± 2.29721 0.51403 ± 0.01304 0.51244 ± 0.01460 0.50566 ± 0.01671 0.51403 ± 0.01304 0.68201 ± 0.00601 0.99348 ± 0.00872 18,557,531 185,223,012
IGTD 148.18030 ± 1.54484 0.42986 ± 0.00000 0.18478 ± 0.00000 0.25846 ± 0.00000 0.42986 ± 0.00000 0.50000 ± 0.00000 1.06630 ± 0.00002 18,756,425 961,520,359
REFINED 131.15941 ± 5.83637 0.55747 ± 0.01675 0.57576 ± 0.01607 0.55971 ± 0.01710 0.55747 ± 0.01675 0.73846 ± 0.01551 0.91167 ± 0.01975 8,034,345 405,902,263
DistanceMatrix 128.09730 ± 3.34499 0.52941 ± 0.01535 0.53164 ± 0.01732 0.52638 ± 0.01697 0.52941 ± 0.01535 0.70803 ± 0.00756 0.99914 ± 0.03235 4,389,851 154,472,600
BarGraph 70.66815 ± 2.09838 0.56290 ± 0.01452 0.56920 ± 0.01562 0.56045 ± 0.01604 0.56290 ± 0.01452 0.72685 ± 0.00615 0.94493 ± 0.01276 3,222,435 31,831,688
Combination 50.64266 ± 3.07094 0.54480 ± 0.02650 0.55690 ± 0.01973 0.53989 ± 0.02733 0.54480 ± 0.02650 0.73836 ± 0.00875 0.92103 ± 0.01228 1,152,003 23,116,164
SuperTML 123.16419 ± 3.17526 0.52851 ± 0.00671 0.52976 ± 0.02017 0.52090 ± 0.01065 0.52851 ± 0.00671 0.71625 ± 0.00255 0.95196 ± 0.02299 14,585,091 480,755,835
FeatureWrap 134.36641 ± 1.40692 0.51131 ± 0.01154 0.50469 ± 0.01292 0.49867 ± 0.02234 0.51131 ± 0.01154 0.66903 ± 0.00290 1.00951 ± 0.00536 9,119,321 89,289,244
BIE 124.84714 ± 6.21710 0.55294 ± 0.01821 0.56052 ± 0.01137 0.55140 ± 0.01781 0.55294 ± 0.01821 0.71498 ± 0.00598 0.94962 ± 0.00646 21,438,339 730,066,308
ViT + MLP
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 117.50083 ± 2.54110 0.49683 ± 0.01409 0.48894 ± 0.01589 0.48781 ± 0.01639 0.49683 ± 0.01409 0.68982 ± 0.00179 0.99148 ± 0.00483 18,560,475 3,658,451,148 ± 525,865,378.91535
IGTD 149.76682 ± 7.78152 0.52489 ± 0.02950 0.52757 ± 0.02946 0.51519 ± 0.03628 0.52489 ± 0.02950 0.69828 ± 0.01713 0.97191 ± 0.01441 18,356,809 23,306,968,083 ± 1,754,835,821.31258
REFINED 125.47250 ± 13.94367 0.55113 ± 0.01480 0.57143 ± 0.02202 0.55077 ± 0.01840 0.55113 ± 0.01480 0.73435 ± 0.00985 0.92641 ± 0.01140 7,760,777 3,748,450,744 ± 740,852,628.00000
DistanceMatrix 150.77362 ± 4.50453 0.54389 ± 0.01930 0.55826 ± 0.01901 0.54059 ± 0.02547 0.54389 ± 0.01930 0.71359 ± 0.01572 0.98598 ± 0.01211 4,368,251 5,390,830,368 ± 304,416,115.04435
BarGraph 66.23054 ± 0.78443 0.54027 ± 0.03543 0.55883 ± 0.03626 0.54345 ± 0.03433 0.54027 ± 0.03543 0.72027 ± 0.02308 0.94456 ± 0.02947 3,262,851 1,024,690,320 ± 90,009,869.65352
Combination 54.89044 ± 0.72794 0.55837 ± 0.01855 0.57543 ± 0.01271 0.55904 ± 0.01354 0.55837 ± 0.01855 0.74016 ± 0.00807 0.91011 ± 0.02151 1,167,587 257,758,240 ± 50,943,945.25477
SuperTML 143.16534 ± 2.41321 0.53122 ± 0.01711 0.53484 ± 0.01654 0.52844 ± 0.01931 0.53122 ± 0.01711 0.71292 ± 0.00740 0.94419 ± 0.01945 14,344,355 10,194,143,094 ± 895,464,193.63100
FeatureWrap 139.72991 ± 11.27500 0.53122 ± 0.00757 0.52775 ± 0.00806 0.52487 ± 0.00862 0.53122 ± 0.00757 0.67457 ± 0.00162 1.01469 ± 0.00180 8,971,289 4,481,517,712 ± 253,067,916.86262
BIE 139.94504 ± 2.70236 0.54299 ± 0.02619 0.54634 ± 0.02360 0.54168 ± 0.02300 0.54299 ± 0.02619 0.70445 ± 0.01312 0.95276 ± 0.01201 21,571,715 7,198,581,280 ± 1,422,744,547.91564
CNN
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 99.38911 ± 0.98989 0.47602 ± 0.02904 0.47440 ± 0.02399 0.46387 ± 0.02599 0.47602 ± 0.02904 0.65798 ± 0.02125 1.03012 ± 0.02895 13,356,803 477,365,760
IGTD 59.34190 ± 1.03583 0.51131 ± 0.01319 0.50829 ± 0.01456 0.50027 ± 0.01458 0.51131 ± 0.01319 0.68163 ± 0.00613 1.01187 ± 0.04742 11,252,291 201,483,328
REFINED 86.79629 ± 6.35222 0.52398 ± 0.01480 0.53632 ± 0.02372 0.52086 ± 0.01808 0.52398 ± 0.01480 0.69901 ± 0.01071 1.01439 ± 0.05581 1,171,875 21,680,800
DistanceMatrix 41.15781 ± 0.98381 0.55475 ± 0.02707 0.56809 ± 0.02266 0.55731 ± 0.02529 0.55475 ± 0.02707 0.72989 ± 0.01112 0.92750 ± 0.01318 764,595 124,157,184
BarGraph 67.33546 ± 4.62728 0.53394 ± 0.02437 0.54430 ± 0.02776 0.53428 ± 0.02707 0.53394 ± 0.02437 0.71769 ± 0.00987 0.95512 ± 0.02738 1,114,931 191,019,072
Combination 37.43239 ± 2.38739 0.54751 ± 0.01781 0.56023 ± 0.01100 0.54867 ± 0.01771 0.54751 ± 0.01781 0.72817 ± 0.00822 0.93757 ± 0.02026 26,275 8,797,568
SuperTML 129.37107 ± 0.30095 0.53937 ± 0.02424 0.55776 ± 0.02714 0.52630 ± 0.01799 0.53937 ± 0.02424 0.71672 ± 0.00635 0.95154 ± 0.01303 301,923 591,587,968
FeatureWrap 37.97508 ± 3.29608 0.50769 ± 0.01613 0.50451 ± 0.01334 0.49074 ± 0.02660 0.50769 ± 0.01613 0.68416 ± 0.01095 0.98980 ± 0.02289 47,035 1,170,576
BIE 75.70332 ± 0.15330 0.55294 ± 0.01256 0.55550 ± 0.00999 0.54402 ± 0.01736 0.55294 ± 0.01256 0.73295 ± 0.01282 0.93518 ± 0.01830 1,531,619 1,648,116,864
CNN + MLP
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 58.12462 ± 1.90736 0.48688 ± 0.01909 0.48561 ± 0.02730 0.47571 ± 0.02089 0.48688 ± 0.01909 0.66685 ± 0.01691 1.06728 ± 0.05824 11,121,187 5,201,861,280
IGTD 67.84435 ± 0.69047 0.50045 ± 0.01342 0.48193 ± 0.06481 0.47949 ± 0.03770 0.50045 ± 0.01342 0.67919 ± 0.00750 1.03052 ± 0.03764 10,765,475 2,205,616,160
REFINED 104.82458 ± 3.45941 0.52127 ± 0.01127 0.54837 ± 0.01201 0.52341 ± 0.00904 0.52127 ± 0.01127 0.70467 ± 0.01246 0.99529 ± 0.03033 1,184,771 238,771,104
DistanceMatrix 33.44982 ± 2.31255 0.54208 ± 0.02180 0.56618 ± 0.01260 0.54219 ± 0.02130 0.54208 ± 0.02180 0.71733 ± 0.01292 0.94284 ± 0.01894 699,059 1,364,290,400
BarGraph 56.81205 ± 2.08185 0.53575 ± 0.00248 0.55262 ± 0.02107 0.53573 ± 0.00406 0.53575 ± 0.00248 0.71719 ± 0.00791 0.95926 ± 0.03626 1,105,427 2,101,001,056
Combination 38.03557 ± 2.37867 0.54480 ± 0.02650 0.54956 ± 0.03032 0.54352 ± 0.02811 0.54480 ± 0.02650 0.71769 ± 0.01576 0.94702 ± 0.02667 33,571 96,932,704
SuperTML 136.04745 ± 2.61454 0.54751 ± 0.01535 0.55968 ± 0.01551 0.54463 ± 0.01651 0.54751 ± 0.01535 0.72676 ± 0.00391 0.92772 ± 0.00556 353,731 6,508,601,440
FeatureWrap 57.12546 ± 1.01150 0.51765 ± 0.01304 0.53846 ± 0.00605 0.51622 ± 0.01403 0.51765 ± 0.01304 0.68590 ± 0.00847 1.00077 ± 0.02192 109,771 14,251,248
BIE 75.70332 ± 0.15330 0.55294 ± 0.01256 0.55550 ± 0.00999 0.54402 ± 0.01736 0.55294 ± 0.01256 0.73295 ± 0.01282 0.93518 ± 0.01830 1,531,619 1,648,116,864

Cnae-9

Source: OpenML · Original

Leaderboard

Family Best variant Test Accuracy (↑) Test F1 (↑) Train time (s) #Params FLOPs
Trees XGBoost 0.95432 ± 0.00338 0.95461 ± 0.00345 8.44052 — —
MLP MLP 0.97037 ± 0.00805 0.96992 ± 0.00846 10.34689 706,057 1,411,072
ViT IGTD 0.93827 ± 0.01310 0.93838 ± 0.01313 26.78283 1,269,455 12,377,946
ViT+MLP REFINED 0.96173 ± 0.00517 0.96194 ± 0.00487 75.39355 1,989,455 10,130,914,304
CNN IGTD 0.93704 ± 0.01537 0.93723 ± 0.01538 23.99849 190,897 63,914,592
CNN+MLP IGTD 0.95556 ± 0.00805 0.96572 ± 0.00999 157.23238 2,804,841 18,386,727,744
Architecture Results
Tree Baselines
Method Test Acc Test F1 Test Precision Test Recall Train time (s)
XGBoost 0.95432 ± 0.00338 0.95461 ± 0.00345 0.95848 ± 0.00428 0.95432 ± 0.00338 8.44052 ± 0.11371
CatBoost 0.93951 ± 0.00805 0.94003 ± 0.00780 0.94732 ± 0.00626 0.93951 ± 0.00805 6.54664 ± 0.10359
LightGBM 0.94938 ± 0.00805 0.95004 ± 0.00766 0.95597 ± 0.00687 0.94938 ± 0.00805 8.07665 ± 1.28849
MLP
Train loss Val loss Test loss Test accuracy Test precision Test recall Test F1 Test ROC AUC Test LogLoss Test MCC Total params FLOPs MACs
0.01342 ± 0.01090 0.12372 ± 0.01638 0.08709 ± 0.01161 0.97037 ± 0.00805 0.97239 ± 0.00756 0.97037 ± 0.00805 0.96992 ± 0.00846 0.99961 ± 0.00018 0.10320 ± 0.01376 0.96701 ± 0.00891 706,057 1,411,072 705,024
ViT
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 24.18381 ± 0.14994 0.97334 ± 0.00296 0.82123 ± 0.03314 0.88654 ± 0.02193 0.86791 ± 0.03361 0.94138 ± 0.01966 0.90270 ± 0.01715 1,600,609 83,777,096
IGTD 26.78283 0.93827 ± 0.01310 0.93838 ± 0.01313 — — — — 1,269,455 12,377,946
REFINED 51.73347 ± 8.33281 0.97624 ± 0.01070 0.92404 ± 0.01751 0.92346 ± 0.02849 0.93110 ± 0.02637 0.99342 ± 0.00305 0.32760 ± 0.05950 1,269,455 12,377,946
FeatureWrap 78.42049 ± 11.15532 0.90424 ± 0.00418 0.79585 ± 0.01276 0.75926 ± 0.01746 0.79167 ± 0.01411 0.97051 ± 0.00461 0.77868 ± 0.08332 17,470,447 174,454,188
ViT + MLP
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 51.20903 ± 5.63295 0.98946 ± 0.00302 0.95172 ± 0.00838 0.96049 ± 0.01883 0.96218 ± 0.01829 0.99859 ± 0.00163 0.13929 ± 0.07387 3,893,025 464,977,088 ± 91,899,166.11703
IGTD 36.43950 ± 1.83848 0.98792 ± 0.00475 0.95294 ± 0.01026 0.95556 ± 0.01408 0.95874 ± 0.01229 0.99784 ± 0.00181 0.22177 ± 0.09474 1,218,401 269,517,528 ± 23,674,701.60606
REFINED 75.39355 ± 2.91066 0.99004 ± 0.00320 0.95299 ± 0.00580 0.96173 ± 0.00517 0.96512 ± 0.00333 0.99862 ± 0.00102 0.14492 ± 0.04696 1,989,455 648,339,608 ± 36,611,247.47466
FeatureWrap 94.19557 ± 3.95154 0.98666 ± 0.00948 0.93410 ± 0.03600 0.94568 ± 0.01598 0.95165 ± 0.01262 0.99831 ± 0.00069 0.21044 ± 0.05732 18,746,639 2,513,403,744 ± 496,755,031.91467
CNN
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 18.50549 ± 0.88259 0.70133 ± 0.00457 0.63837 ± 0.01416 0.68148 ± 0.01549 0.69369 ± 0.01499 0.93206 ± 0.00367 0.96299 ± 0.03732 77,937 3,614,528
IGTD 23.99849 0.93704 ± 0.01537 0.93723 ± 0.01538 — — — — 190,897 63,914,592
REFINED 39.24781 ± 1.83563 0.97247 ± 0.00675 0.91175 ± 0.02772 0.91358 ± 0.00756 0.92283 ± 0.00767 0.99358 ± 0.00182 0.37951 ± 0.03106 733,137 63,914,592
FeatureWrap 77.13566 ± 0.42878 0.96653 ± 0.00509 0.91007 ± 0.01701 0.88148 ± 0.01104 0.89928 ± 0.01478 0.99107 ± 0.00174 0.53426 ± 0.04917 11,985,833 1,669,530,816
CNN + MLP
Method Train time (s) Test Acc Test F1 Test Precision Test Recall Test ROC AUC Test LogLoss #Params FLOPs
TINTO 29.90100 ± 2.27126 0.98492 ± 0.00487 0.94350 ± 0.01763 0.96420 ± 0.00276 0.96609 ± 0.00225 0.99911 ± 0.00047 0.10987 ± 0.02834 820,465 56,082,752
IGTD 157.23238 0.95556 ± 0.00805 0.96572 ± 0.00999 — — — — 2,804,841 18,386,727,744
REFINED 42.14406 ± 2.49779 0.98654 ± 0.00336 0.94222 ± 0.01663 0.95185 ± 0.01187 0.95589 ± 0.01141 0.99915 ± 0.00040 0.15621 ± 0.02966 1,422,257 718,212,000
FeatureWrap 97.10382 ± 4.24365 0.98926 ± 0.00357 0.95299 ± 0.01137 0.93580 ± 0.01421 0.94377 ± 0.01086 0.99883 ± 0.00059 0.22689 ± 0.08034 12,981,545 18,386,727,744