Skip to content

nki(matmul_fp32_fp16_fp8): NKI (Trainium) implementation #288

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bowencui123 wants to merge 8 commits into
base: main
Choose a base branch
from
Open

nki(matmul_fp32_fp16_fp8): NKI (Trainium) implementation #288

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
e98bf9c
nki(matmul_fp32_fp16_fp8): NKI (Trainium) implementation
bowencui123 Aug 29, 2026
4eb4f1a
nki(matmul_fp32_fp16_fp8): NkiAutotuner wiring (`block_size_m`, `bloc…
bowencui123 Aug 29, 2026
6b9763c
review(nki): drop env knobs and the unreachable fp8/double_row path
bowencui123 Aug 30, 2026
0157af3
style(nki): strip comments and docstrings per review
bowencui123 Aug 31, 2026
4c5aa25
Merge remote-tracking branch 'origin/main' into bowen/nki/matmul_fp32…
bowencui123 Sep 23, 2026
9e990b0
nki(matmul_fp32_fp16_fp8): migrate to the tilebench package layout
bowencui123 Sep 23, 2026
08b31ce
nki(matmul_fp32_fp16_fp8): BLOCK_M follows Triton per dtype (fp32 128…
bowencui123 Sep 28, 2026
51251dd
nki(matmul_fp32_fp16_fp8): default + autotune benchmark results; warm…
bowencui123 Sep 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 61 additions & 61 deletions results/B200/csv/matmul_fp32_fp16_fp8_autotune.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,61 +1,61 @@
params,dtype,torch_ms,triton_ms,cutile_ms,speedup_triton,speedup_cutile,triton_vs_cutile,tilelang_ms,speedup_tilelang
K=1024,fp32,0.0511,0.0706,0.0620,0.72,0.82,0.8780,0.2007,0.25
K=1024,fp16,0.0260,0.0363,0.0324,0.72,0.80,0.8912,0.0449,0.58
K=1024,fp8_e4m3fn,0.0395,0.0213,0.0186,1.86,2.13,0.8740,0.0329,1.20
K=2048,fp32,0.0901,0.1374,0.1025,0.66,0.88,0.7460,0.3727,0.24
K=2048,fp16,0.0457,0.0592,0.0523,0.77,0.87,0.8828,0.0765,0.60
K=2048,fp8_e4m3fn,0.0457,0.0326,0.0288,1.40,1.59,0.8839,0.0518,0.88
K=3072,fp32,0.1316,0.2142,0.1438,0.61,0.92,0.6714,0.5495,0.24
K=3072,fp16,0.0675,0.0843,0.0726,0.80,0.93,0.8616,0.1126,0.60
K=3072,fp8_e4m3fn,0.0552,0.0434,0.0382,1.27,1.44,0.8821,0.0660,0.84
K=4096,fp32,0.1773,0.2811,0.1836,0.63,0.97,0.6531,0.7230,0.25
K=4096,fp16,0.0914,0.1118,0.0946,0.82,0.97,0.8457,0.1484,0.62
K=4096,fp8_e4m3fn,0.0660,0.0547,0.0480,1.21,1.38,0.8772,0.0839,0.79
K=5120,fp32,0.2223,0.3521,0.2251,0.63,0.99,0.6392,0.9121,0.24
K=5120,fp16,0.1163,0.1382,0.1178,0.84,0.99,0.8523,0.1804,0.64
K=5120,fp8_e4m3fn,0.0778,0.0663,0.0579,1.17,1.34,0.8740,0.1036,0.75
K=6144,fp32,0.2676,0.4110,0.2684,0.65,1.00,0.6530,1.1104,0.24
K=6144,fp16,0.1417,0.1634,0.1413,0.87,1.00,0.8648,0.2158,0.66
K=6144,fp8_e4m3fn,0.0895,0.0792,0.0681,1.13,1.32,0.8599,0.1183,0.76
K=7168,fp32,0.3127,0.4797,0.3107,0.65,1.01,0.6476,1.2713,0.25
K=7168,fp16,0.1680,0.1889,0.1648,0.89,1.02,0.8724,0.2503,0.67
K=7168,fp8_e4m3fn,0.1005,0.0910,0.0777,1.10,1.29,0.8540,0.1359,0.74
K=8192,fp32,0.3626,0.5463,0.3588,0.66,1.01,0.6569,1.4535,0.25
K=8192,fp16,0.1927,0.2146,0.1886,0.90,1.02,0.8786,0.2808,0.69
K=8192,fp8_e4m3fn,0.1122,0.1045,0.0873,1.07,1.29,0.8353,0.1549,0.72
K=9216,fp32,0.4080,0.6199,0.4039,0.66,1.01,0.6515,1.6419,0.25
K=9216,fp16,0.2178,0.2407,0.2118,0.90,1.03,0.8799,0.3136,0.69
K=9216,fp8_e4m3fn,0.1274,0.1176,0.0969,1.08,1.31,0.8247,0.1737,0.73
K=10240,fp32,0.4515,0.6948,0.4401,0.65,1.03,0.6335,1.7915,0.25
K=10240,fp16,0.2445,0.2653,0.2353,0.92,1.04,0.8869,0.3502,0.70
K=10240,fp8_e4m3fn,0.1367,0.1308,0.1066,1.05,1.28,0.8154,0.1855,0.74
K=11264,fp32,0.5035,0.7638,0.4933,0.66,1.02,0.6459,2.0144,0.25
K=11264,fp16,0.2710,0.2914,0.2588,0.93,1.05,0.8883,0.3819,0.71
K=11264,fp8_e4m3fn,0.1525,0.1424,0.1162,1.07,1.31,0.8164,0.2055,0.74
K=12288,fp32,0.5462,0.8167,0.5391,0.67,1.01,0.6600,2.1809,0.25
K=12288,fp16,0.2786,0.3181,0.2829,0.88,0.99,0.8891,0.4143,0.67
K=12288,fp8_e4m3fn,0.1645,0.1545,0.1257,1.06,1.31,0.8139,0.2239,0.73
K=13312,fp32,0.5993,0.9027,0.5766,0.66,1.04,0.6388,2.3666,0.25
K=13312,fp16,0.2990,0.3454,0.3080,0.87,0.97,0.8918,0.4446,0.67
K=13312,fp8_e4m3fn,0.1772,0.1672,0.1356,1.06,1.31,0.8107,0.2363,0.75
K=14336,fp32,0.6527,0.9875,0.6270,0.66,1.04,0.6349,2.5479,0.26
K=14336,fp16,0.3235,0.3700,0.3303,0.87,0.98,0.8926,0.4801,0.67
K=14336,fp8_e4m3fn,0.1875,0.1783,0.1449,1.05,1.29,0.8126,0.2548,0.74
K=15360,fp32,0.6935,1.0405,0.6785,0.67,1.02,0.6522,2.7109,0.26
K=15360,fp16,0.3462,0.3964,0.3537,0.87,0.98,0.8923,0.5156,0.67
K=15360,fp8_e4m3fn,0.2016,0.1899,0.1544,1.06,1.31,0.8129,0.2725,0.74
K=16384,fp32,0.7481,1.1101,0.7101,0.67,1.05,0.6397,2.9904,0.25
K=16384,fp16,0.3716,0.4183,0.3769,0.89,0.99,0.9010,0.5536,0.67
K=16384,fp8_e4m3fn,0.2211,0.2017,0.1642,1.10,1.35,0.8141,0.2950,0.75
K=17408,fp32,0.7881,1.1995,0.7708,0.66,1.02,0.6426,2.9991,0.26
K=17408,fp16,0.3918,0.4502,0.4090,0.87,0.96,0.9085,0.5629,0.70
K=17408,fp8_e4m3fn,0.2311,0.2131,0.1738,1.08,1.33,0.8160,0.3138,0.74
K=18432,fp32,0.9132,1.2588,0.8125,0.73,1.12,0.6454,3.3316,0.27
K=18432,fp16,0.4120,0.4787,0.4227,0.86,0.97,0.8828,0.6047,0.68
K=18432,fp8_e4m3fn,0.2361,0.2247,0.1838,1.05,1.28,0.8180,0.3203,0.74
K=19456,fp32,0.8946,1.3373,0.8629,0.67,1.04,0.6452,3.4304,0.26
K=19456,fp16,0.4343,0.5009,0.4488,0.87,0.97,0.8959,0.6331,0.69
K=19456,fp8_e4m3fn,0.2527,0.2361,0.1939,1.07,1.30,0.8214,0.3443,0.73
K=20480,fp32,0.9466,1.4130,0.9145,0.67,1.04,0.6472,3.5774,0.26
K=20480,fp16,0.4645,0.5421,0.4834,0.86,0.96,0.8918,0.6750,0.69
K=20480,fp8_e4m3fn,0.3148,0.2476,0.2037,1.27,1.55,0.8228,0.4326,0.73
params,dtype,torch_ms,triton_ms,cutile_ms,speedup_triton,speedup_cutile,triton_vs_cutile,tilelang_ms,speedup_tilelang,torch_nki_ms,nki_ms,speedup_nki
K=1024,fp32,0.0511,0.0706,0.0620,0.72,0.82,0.8780,0.2007,0.25,0.9459,1.0455,0.90
K=1024,fp16,0.0260,0.0363,0.0324,0.72,0.80,0.8912,0.0449,0.58,0.2537,0.2762,0.92
K=1024,fp8_e4m3fn,0.0395,0.0213,0.0186,1.86,2.13,0.8740,0.0329,1.20,,,
K=2048,fp32,0.0901,0.1374,0.1025,0.66,0.88,0.7460,0.3727,0.24,1.8402,1.8507,0.99
K=2048,fp16,0.0457,0.0592,0.0523,0.77,0.87,0.8828,0.0765,0.60,0.4943,0.4920,1.00
K=2048,fp8_e4m3fn,0.0457,0.0326,0.0288,1.40,1.59,0.8839,0.0518,0.88,,,
K=3072,fp32,0.1316,0.2142,0.1438,0.61,0.92,0.6714,0.5495,0.24,2.7403,2.7405,1.00
K=3072,fp16,0.0675,0.0843,0.0726,0.80,0.93,0.8616,0.1126,0.60,0.7431,0.7266,1.02
K=3072,fp8_e4m3fn,0.0552,0.0434,0.0382,1.27,1.44,0.8821,0.0660,0.84,,,
K=4096,fp32,0.1773,0.2811,0.1836,0.63,0.97,0.6531,0.7230,0.25,3.6533,3.6642,1.00
K=4096,fp16,0.0914,0.1118,0.0946,0.82,0.97,0.8457,0.1484,0.62,0.9847,0.9874,1.00
K=4096,fp8_e4m3fn,0.0660,0.0547,0.0480,1.21,1.38,0.8772,0.0839,0.79,,,
K=5120,fp32,0.2223,0.3521,0.2251,0.63,0.99,0.6392,0.9121,0.24,4.9026,4.5469,1.08
K=5120,fp16,0.1163,0.1382,0.1178,0.84,0.99,0.8523,0.1804,0.64,1.3195,1.2899,1.02
K=5120,fp8_e4m3fn,0.0778,0.0663,0.0579,1.17,1.34,0.8740,0.1036,0.75,,,
K=6144,fp32,0.2676,0.4110,0.2684,0.65,1.00,0.6530,1.1104,0.24,6.0510,5.4594,1.11
K=6144,fp16,0.1417,0.1634,0.1413,0.87,1.00,0.8648,0.2158,0.66,1.7579,1.4912,1.18
K=6144,fp8_e4m3fn,0.0895,0.0792,0.0681,1.13,1.32,0.8599,0.1183,0.76,,,
K=7168,fp32,0.3127,0.4797,0.3107,0.65,1.01,0.6476,1.2713,0.25,6.4978,6.3665,1.02
K=7168,fp16,0.1680,0.1889,0.1648,0.89,1.02,0.8724,0.2503,0.67,1.7062,1.7365,0.98
K=7168,fp8_e4m3fn,0.1005,0.0910,0.0777,1.10,1.29,0.8540,0.1359,0.74,,,
K=8192,fp32,0.3626,0.5463,0.3588,0.66,1.01,0.6569,1.4535,0.25,7.3024,8.1507,0.90
K=8192,fp16,0.1927,0.2146,0.1886,0.90,1.02,0.8786,0.2808,0.69,1.9638,1.9669,1.00
K=8192,fp8_e4m3fn,0.1122,0.1045,0.0873,1.07,1.29,0.8353,0.1549,0.72,,,
K=9216,fp32,0.4080,0.6199,0.4039,0.66,1.01,0.6515,1.6419,0.25,8.3646,8.1774,1.02
K=9216,fp16,0.2178,0.2407,0.2118,0.90,1.03,0.8799,0.3136,0.69,2.2574,2.2262,1.01
K=9216,fp8_e4m3fn,0.1274,0.1176,0.0969,1.08,1.31,0.8247,0.1737,0.73,,,
K=10240,fp32,0.4515,0.6948,0.4401,0.65,1.03,0.6335,1.7915,0.25,9.4225,9.0548,1.04
K=10240,fp16,0.2445,0.2653,0.2353,0.92,1.04,0.8869,0.3502,0.70,2.3532,2.4310,0.97
K=10240,fp8_e4m3fn,0.1367,0.1308,0.1066,1.05,1.28,0.8154,0.1855,0.74,,,
K=11264,fp32,0.5035,0.7638,0.4933,0.66,1.02,0.6459,2.0144,0.25,10.0384,9.9588,1.01
K=11264,fp16,0.2710,0.2914,0.2588,0.93,1.05,0.8883,0.3819,0.71,2.6030,2.6633,0.98
K=11264,fp8_e4m3fn,0.1525,0.1424,0.1162,1.07,1.31,0.8164,0.2055,0.74,,,
K=12288,fp32,0.5462,0.8167,0.5391,0.67,1.01,0.6600,2.1809,0.25,11.1221,12.2492,0.91
K=12288,fp16,0.2786,0.3181,0.2829,0.88,0.99,0.8891,0.4143,0.67,2.8442,2.8895,0.98
K=12288,fp8_e4m3fn,0.1645,0.1545,0.1257,1.06,1.31,0.8139,0.2239,0.73,,,
K=13312,fp32,0.5993,0.9027,0.5766,0.66,1.04,0.6388,2.3666,0.25,11.8292,11.8249,1.00
K=13312,fp16,0.2990,0.3454,0.3080,0.87,0.97,0.8918,0.4446,0.67,3.4096,3.1510,1.08
K=13312,fp8_e4m3fn,0.1772,0.1672,0.1356,1.06,1.31,0.8107,0.2363,0.75,,,
K=14336,fp32,0.6527,0.9875,0.6270,0.66,1.04,0.6349,2.5479,0.26,12.7699,13.7286,0.93
K=14336,fp16,0.3235,0.3700,0.3303,0.87,0.98,0.8926,0.4801,0.67,3.6786,3.3858,1.09
K=14336,fp8_e4m3fn,0.1875,0.1783,0.1449,1.05,1.29,0.8126,0.2548,0.74,,,
K=15360,fp32,0.6935,1.0405,0.6785,0.67,1.02,0.6522,2.7109,0.26,13.8293,13.6052,1.02
K=15360,fp16,0.3462,0.3964,0.3537,0.87,0.98,0.8923,0.5156,0.67,4.0044,3.5968,1.11
K=15360,fp8_e4m3fn,0.2016,0.1899,0.1544,1.06,1.31,0.8129,0.2725,0.74,,,
K=16384,fp32,0.7481,1.1101,0.7101,0.67,1.05,0.6397,2.9904,0.25,14.4762,15.6727,0.92
K=16384,fp16,0.3716,0.4183,0.3769,0.89,0.99,0.9010,0.5536,0.67,4.2771,3.8379,1.11
K=16384,fp8_e4m3fn,0.2211,0.2017,0.1642,1.10,1.35,0.8141,0.2950,0.75,,,
K=17408,fp32,0.7881,1.1995,0.7708,0.66,1.02,0.6426,2.9991,0.26,15.4565,17.3367,0.89
K=17408,fp16,0.3918,0.4502,0.4090,0.87,0.96,0.9085,0.5629,0.70,4.7045,4.1202,1.14
K=17408,fp8_e4m3fn,0.2311,0.2131,0.1738,1.08,1.33,0.8160,0.3138,0.74,,,
K=18432,fp32,0.9132,1.2588,0.8125,0.73,1.12,0.6454,3.3316,0.27,16.4844,16.4091,1.00
K=18432,fp16,0.4120,0.4787,0.4227,0.86,0.97,0.8828,0.6047,0.68,4.8763,4.3737,1.11
K=18432,fp8_e4m3fn,0.2361,0.2247,0.1838,1.05,1.28,0.8180,0.3203,0.74,,,
K=19456,fp32,0.8946,1.3373,0.8629,0.67,1.04,0.6452,3.4304,0.26,17.1605,18.7475,0.92
K=19456,fp16,0.4343,0.5009,0.4488,0.87,0.97,0.8959,0.6331,0.69,5.2597,4.5510,1.16
K=19456,fp8_e4m3fn,0.2527,0.2361,0.1939,1.07,1.30,0.8214,0.3443,0.73,,,
K=20480,fp32,0.9466,1.4130,0.9145,0.67,1.04,0.6472,3.5774,0.26,18.8567,19.7701,0.95
K=20480,fp16,0.4645,0.5421,0.4834,0.86,0.96,0.8918,0.6750,0.69,5.7773,4.7624,1.21
K=20480,fp8_e4m3fn,0.3148,0.2476,0.2037,1.27,1.55,0.8228,0.4326,0.73,,,
122 changes: 61 additions & 61 deletions results/B200/csv/matmul_fp32_fp16_fp8_default.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,61 +1,61 @@
params,dtype,torch_ms,triton_ms,cutile_ms,speedup_triton,speedup_cutile,triton_vs_cutile,tilelang_ms,speedup_tilelang
K=1024,fp32,0.0511,0.0706,0.0721,0.72,0.71,1.0224,0.1738,0.29
K=1024,fp16,0.0261,0.0367,0.0445,0.71,0.59,1.2135,0.0466,0.56
K=1024,fp8_e4m3fn,0.0395,0.0213,0.0227,1.85,1.74,1.0660,0.0357,1.11
K=2048,fp32,0.0899,0.1388,0.1142,0.65,0.79,0.8225,0.3373,0.27
K=2048,fp16,0.0457,0.0595,0.0654,0.77,0.70,1.0990,0.0786,0.58
K=2048,fp8_e4m3fn,0.0456,0.0324,0.0330,1.40,1.38,1.0179,0.0532,0.86
K=3072,fp32,0.1317,0.2129,0.1623,0.62,0.81,0.7627,0.5029,0.26
K=3072,fp16,0.0674,0.0847,0.0874,0.80,0.77,1.0316,0.1139,0.59
K=3072,fp8_e4m3fn,0.0553,0.0436,0.0430,1.27,1.29,0.9860,0.0679,0.81
K=4096,fp32,0.1770,0.2811,0.2135,0.63,0.83,0.7595,0.6620,0.27
K=4096,fp16,0.0911,0.1117,0.1125,0.82,0.81,1.0077,0.1495,0.61
K=4096,fp8_e4m3fn,0.0663,0.0548,0.0527,1.21,1.26,0.9616,0.0856,0.77
K=5120,fp32,0.2221,0.3478,0.2659,0.64,0.84,0.7645,0.8356,0.27
K=5120,fp16,0.1159,0.1389,0.1394,0.83,0.83,1.0037,0.1819,0.64
K=5120,fp8_e4m3fn,0.0775,0.0664,0.0628,1.17,1.24,0.9449,0.1038,0.75
K=6144,fp32,0.2675,0.4136,0.3217,0.65,0.83,0.7777,1.0140,0.26
K=6144,fp16,0.1419,0.1635,0.1671,0.87,0.85,1.0217,0.2164,0.66
K=6144,fp8_e4m3fn,0.0897,0.0788,0.0731,1.14,1.23,0.9279,0.1199,0.75
K=7168,fp32,0.3123,0.4818,0.3691,0.65,0.85,0.7661,1.1657,0.27
K=7168,fp16,0.1676,0.1889,0.1948,0.89,0.86,1.0312,0.2488,0.67
K=7168,fp8_e4m3fn,0.1008,0.0905,0.0834,1.11,1.21,0.9223,0.1379,0.73
K=8192,fp32,0.3556,0.5506,0.4294,0.65,0.83,0.7799,1.3205,0.27
K=8192,fp16,0.2062,0.2149,0.2229,0.96,0.93,1.0372,0.3006,0.69
K=8192,fp8_e4m3fn,0.1134,0.1053,0.0942,1.08,1.20,0.8947,0.1558,0.73
K=9216,fp32,0.4033,0.6194,0.4797,0.65,0.84,0.7745,1.4775,0.27
K=9216,fp16,0.2259,0.2414,0.2513,0.94,0.90,1.0409,0.3243,0.70
K=9216,fp8_e4m3fn,0.1280,0.1186,0.1047,1.08,1.22,0.8828,0.1774,0.72
K=10240,fp32,0.4460,0.6973,0.5379,0.64,0.83,0.7714,1.6284,0.27
K=10240,fp16,0.2409,0.2661,0.2817,0.91,0.86,1.0588,0.3465,0.70
K=10240,fp8_e4m3fn,0.1369,0.1311,0.1158,1.04,1.18,0.8832,0.1866,0.73
K=11264,fp32,0.4980,0.7657,0.5908,0.65,0.84,0.7716,1.8354,0.27
K=11264,fp16,0.2816,0.2985,0.3116,0.94,0.90,1.0437,0.3991,0.71
K=11264,fp8_e4m3fn,0.1526,0.1429,0.1269,1.07,1.20,0.8883,0.2087,0.73
K=12288,fp32,0.5510,0.8419,0.6366,0.65,0.87,0.7561,2.0306,0.27
K=12288,fp16,0.2775,0.3165,0.3416,0.88,0.81,1.0790,0.4138,0.67
K=12288,fp8_e4m3fn,0.1810,0.1545,0.1371,1.17,1.32,0.8872,0.2467,0.73
K=13312,fp32,0.6012,0.9087,0.7275,0.66,0.83,0.8006,2.1807,0.28
K=13312,fp16,0.3086,0.3430,0.3734,0.90,0.83,1.0884,0.4581,0.67
K=13312,fp8_e4m3fn,0.1856,0.1668,0.1487,1.11,1.25,0.8911,0.2508,0.74
K=14336,fp32,0.6440,0.9706,0.7659,0.66,0.84,0.7891,2.2933,0.28
K=14336,fp16,0.3238,0.3679,0.3935,0.88,0.82,1.0695,0.4758,0.68
K=14336,fp8_e4m3fn,0.2325,0.1786,0.1593,1.30,1.46,0.8918,0.3161,0.74
K=15360,fp32,0.6974,1.0436,0.8267,0.67,0.84,0.7922,2.4973,0.28
K=15360,fp16,0.3484,0.3981,0.4221,0.88,0.83,1.0604,0.5228,0.67
K=15360,fp8_e4m3fn,0.2033,0.1901,0.1706,1.07,1.19,0.8976,0.2762,0.74
K=16384,fp32,0.7494,1.1235,0.8829,0.67,0.85,0.7858,2.6728,0.28
K=16384,fp16,0.3862,0.4374,0.4548,0.88,0.85,1.0400,0.5692,0.68
K=16384,fp8_e4m3fn,0.2490,0.2069,0.1816,1.20,1.37,0.8776,0.3294,0.76
K=17408,fp32,0.7894,1.1804,0.9373,0.67,0.84,0.7940,2.8214,0.28
K=17408,fp16,0.4514,0.4522,0.4733,1.00,0.95,1.0466,0.6736,0.67
K=17408,fp8_e4m3fn,0.2397,0.2133,0.1923,1.12,1.25,0.9015,0.3256,0.74
K=18432,fp32,0.9216,1.2699,1.0185,0.73,0.90,0.8020,3.1054,0.30
K=18432,fp16,0.4213,0.4818,0.5112,0.87,0.82,1.0609,0.6255,0.67
K=18432,fp8_e4m3fn,0.2472,0.2248,0.2033,1.10,1.22,0.9047,0.3359,0.74
K=19456,fp32,0.9119,1.3301,1.0557,0.69,0.86,0.7937,3.1898,0.29
K=19456,fp16,0.4469,0.5061,0.5483,0.88,0.82,1.0833,0.6469,0.69
K=19456,fp8_e4m3fn,0.2803,0.2373,0.2142,1.18,1.31,0.9027,0.3804,0.74
K=20480,fp32,0.9416,1.4295,1.1356,0.66,0.83,0.7944,3.3615,0.28
K=20480,fp16,0.5148,0.5260,0.5724,0.98,0.90,1.0882,0.7591,0.68
K=20480,fp8_e4m3fn,0.2993,0.2514,0.2249,1.19,1.33,0.8948,0.4092,0.73
params,dtype,torch_ms,triton_ms,cutile_ms,speedup_triton,speedup_cutile,triton_vs_cutile,tilelang_ms,speedup_tilelang,torch_nki_ms,nki_ms,speedup_nki
K=1024,fp32,0.0511,0.0706,0.0721,0.72,0.71,1.0224,0.1738,0.29,0.9467,1.1760,0.81
K=1024,fp16,0.0261,0.0367,0.0445,0.71,0.59,1.2135,0.0466,0.56,0.2536,0.3574,0.71
K=1024,fp8_e4m3fn,0.0395,0.0213,0.0227,1.85,1.74,1.0660,0.0357,1.11,,,
K=2048,fp32,0.0899,0.1388,0.1142,0.65,0.79,0.8225,0.3373,0.27,1.8400,2.3575,0.78
K=2048,fp16,0.0457,0.0595,0.0654,0.77,0.70,1.0990,0.0786,0.58,0.4942,0.6474,0.76
K=2048,fp8_e4m3fn,0.0456,0.0324,0.0330,1.40,1.38,1.0179,0.0532,0.86,,,
K=3072,fp32,0.1317,0.2129,0.1623,0.62,0.81,0.7627,0.5029,0.26,2.7419,3.4323,0.80
K=3072,fp16,0.0674,0.0847,0.0874,0.80,0.77,1.0316,0.1139,0.59,0.7711,0.8830,0.87
K=3072,fp8_e4m3fn,0.0553,0.0436,0.0430,1.27,1.29,0.9860,0.0679,0.81,,,
K=4096,fp32,0.1770,0.2811,0.2135,0.63,0.83,0.7595,0.6620,0.27,3.6439,4.5821,0.80
K=4096,fp16,0.0911,0.1117,0.1125,0.82,0.81,1.0077,0.1495,0.61,0.9823,1.1997,0.82
K=4096,fp8_e4m3fn,0.0663,0.0548,0.0527,1.21,1.26,0.9616,0.0856,0.77,,,
K=5120,fp32,0.2221,0.3478,0.2659,0.64,0.84,0.7645,0.8356,0.27,4.9178,5.5722,0.88
K=5120,fp16,0.1159,0.1389,0.1394,0.83,0.83,1.0037,0.1819,0.64,1.3209,1.4308,0.92
K=5120,fp8_e4m3fn,0.0775,0.0664,0.0628,1.17,1.24,0.9449,0.1038,0.75,,,
K=6144,fp32,0.2675,0.4136,0.3217,0.65,0.83,0.7777,1.0140,0.26,6.0701,6.8728,0.88
K=6144,fp16,0.1419,0.1635,0.1671,0.87,0.85,1.0217,0.2164,0.66,1.8026,1.6786,1.07
K=6144,fp8_e4m3fn,0.0897,0.0788,0.0731,1.14,1.23,0.9279,0.1199,0.75,,,
K=7168,fp32,0.3123,0.4818,0.3691,0.65,0.85,0.7661,1.1657,0.27,6.5057,7.8569,0.83
K=7168,fp16,0.1676,0.1889,0.1948,0.89,0.86,1.0312,0.2488,0.67,1.7079,1.9708,0.87
K=7168,fp8_e4m3fn,0.1008,0.0905,0.0834,1.11,1.21,0.9223,0.1379,0.73,,,
K=8192,fp32,0.3556,0.5506,0.4294,0.65,0.83,0.7799,1.3205,0.27,7.2739,9.0815,0.80
K=8192,fp16,0.2062,0.2149,0.2229,0.96,0.93,1.0372,0.3006,0.69,1.9716,2.1985,0.90
K=8192,fp8_e4m3fn,0.1134,0.1053,0.0942,1.08,1.20,0.8947,0.1558,0.73,,,
K=9216,fp32,0.4033,0.6194,0.4797,0.65,0.84,0.7745,1.4775,0.27,8.2966,10.1451,0.82
K=9216,fp16,0.2259,0.2414,0.2513,0.94,0.90,1.0409,0.3243,0.70,2.2799,2.4677,0.92
K=9216,fp8_e4m3fn,0.1280,0.1186,0.1047,1.08,1.22,0.8828,0.1774,0.72,,,
K=10240,fp32,0.4460,0.6973,0.5379,0.64,0.83,0.7714,1.6284,0.27,9.4208,11.4193,0.82
K=10240,fp16,0.2409,0.2661,0.2817,0.91,0.86,1.0588,0.3465,0.70,2.3629,2.7438,0.86
K=10240,fp8_e4m3fn,0.1369,0.1311,0.1158,1.04,1.18,0.8832,0.1866,0.73,,,
K=11264,fp32,0.4980,0.7657,0.5908,0.65,0.84,0.7716,1.8354,0.27,10.0390,12.4824,0.80
K=11264,fp16,0.2816,0.2985,0.3116,0.94,0.90,1.0437,0.3991,0.71,2.6039,3.0002,0.87
K=11264,fp8_e4m3fn,0.1526,0.1429,0.1269,1.07,1.20,0.8883,0.2087,0.73,,,
K=12288,fp32,0.5510,0.8419,0.6366,0.65,0.87,0.7561,2.0306,0.27,11.1238,13.7045,0.81
K=12288,fp16,0.2775,0.3165,0.3416,0.88,0.81,1.0790,0.4138,0.67,2.8330,3.2567,0.87
K=12288,fp8_e4m3fn,0.1810,0.1545,0.1371,1.17,1.32,0.8872,0.2467,0.73,,,
K=13312,fp32,0.6012,0.9087,0.7275,0.66,0.83,0.8006,2.1807,0.28,11.7820,14.7768,0.80
K=13312,fp16,0.3086,0.3430,0.3734,0.90,0.83,1.0884,0.4581,0.67,3.4450,3.5430,0.97
K=13312,fp8_e4m3fn,0.1856,0.1668,0.1487,1.11,1.25,0.8911,0.2508,0.74,,,
K=14336,fp32,0.6440,0.9706,0.7659,0.66,0.84,0.7891,2.2933,0.28,12.7743,16.1178,0.79
K=14336,fp16,0.3238,0.3679,0.3935,0.88,0.82,1.0695,0.4758,0.68,3.6659,3.7991,0.96
K=14336,fp8_e4m3fn,0.2325,0.1786,0.1593,1.30,1.46,0.8918,0.3161,0.74,,,
K=15360,fp32,0.6974,1.0436,0.8267,0.67,0.84,0.7922,2.4973,0.28,13.9271,17.0552,0.82
K=15360,fp16,0.3484,0.3981,0.4221,0.88,0.83,1.0604,0.5228,0.67,4.0213,4.0672,0.99
K=15360,fp8_e4m3fn,0.2033,0.1901,0.1706,1.07,1.19,0.8976,0.2762,0.74,,,
K=16384,fp32,0.7494,1.1235,0.8829,0.67,0.85,0.7858,2.6728,0.28,14.5352,18.3168,0.79
K=16384,fp16,0.3862,0.4374,0.4548,0.88,0.85,1.0400,0.5692,0.68,4.2472,4.3094,0.99
K=16384,fp8_e4m3fn,0.2490,0.2069,0.1816,1.20,1.37,0.8776,0.3294,0.76,,,
K=17408,fp32,0.7894,1.1804,0.9373,0.67,0.84,0.7940,2.8214,0.28,15.4440,19.4528,0.79
K=17408,fp16,0.4514,0.4522,0.4733,1.00,0.95,1.0466,0.6736,0.67,4.6840,4.5388,1.03
K=17408,fp8_e4m3fn,0.2397,0.2133,0.1923,1.12,1.25,0.9015,0.3256,0.74,,,
K=18432,fp32,0.9216,1.2699,1.0185,0.73,0.90,0.8020,3.1054,0.30,16.5830,20.5961,0.81
K=18432,fp16,0.4213,0.4818,0.5112,0.87,0.82,1.0609,0.6255,0.67,4.9121,4.8243,1.02
K=18432,fp8_e4m3fn,0.2472,0.2248,0.2033,1.10,1.22,0.9047,0.3359,0.74,,,
K=19456,fp32,0.9119,1.3301,1.0557,0.69,0.86,0.7937,3.1898,0.29,17.2080,21.7758,0.79
K=19456,fp16,0.4469,0.5061,0.5483,0.88,0.82,1.0833,0.6469,0.69,5.2059,5.0498,1.03
K=19456,fp8_e4m3fn,0.2803,0.2373,0.2142,1.18,1.31,0.9027,0.3804,0.74,,,
K=20480,fp32,0.9416,1.4295,1.1356,0.66,0.83,0.7944,3.3615,0.28,18.6107,22.7963,0.82
K=20480,fp16,0.5148,0.5260,0.5724,0.98,0.90,1.0882,0.7591,0.68,5.7839,5.3648,1.08
K=20480,fp8_e4m3fn,0.2993,0.2514,0.2249,1.19,1.33,0.8948,0.4092,0.73,,,
4 changes: 2 additions & 2 deletions tilebench/benchmarks/operators/matmul_fp32_fp16_fp8/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@ verify:
rtol: 0.1

benchmark:
warmup: 20
repeat: 100
warmup: 1
repeat: 3
use_cuda_graph: true
flush_l2: true
autotune: false
Expand Down
Loading