File size: 91,159 Bytes
936ce55 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 |
-------------------------- DeepSpeed Flops Profiler --------------------------
Profile Summary at step 2:
Notations:
data parallel size (dp_size), model parallel size(mp_size),
number of parameters (params), number of multiply-accumulate operations(MACs),
number of floating-point operations (flops), floating-point operations per second (FLOPS),
fwd latency (forward propagation latency), bwd latency (backward propagation latency),
step (weights update latency), iter latency (sum of fwd, bwd and step latency)
world size: 32
data parallel size: 32
model parallel size: 1
batch size per GPU: 16
params per GPU: 3.05 B
params of model = params per GPU * mp_size: 3.05 B
fwd MACs per GPU: 9.86 TMACs
fwd flops per GPU: 19.73 T
fwd flops of model = fwd flops per GPU * mp_size: 19.73 T
fwd latency: 158.91 ms
fwd FLOPS per GPU = fwd flops per GPU / fwd latency: 124.15 TFLOPS
bwd latency: 559.86 ms
bwd FLOPS per GPU = 2 * fwd flops per GPU / bwd latency: 70.48 TFLOPS
fwd+bwd FLOPS per GPU = 3 * fwd flops per GPU / (fwd+bwd latency): 82.34 TFLOPS
step latency: 167.69 ms
iter latency: 886.46 ms
FLOPS per GPU = 3 * fwd flops per GPU / iter latency: 66.77 TFLOPS
samples/second: 577.58
----------------------------- Aggregated Profile per GPU -----------------------------
Top 1 modules in terms of params, MACs or fwd latency at different model depths:
depth 0:
params - {'DiT': '3.05 B'}
MACs - {'DiT': '9.86 TMACs'}
fwd latency - {'DiT': '158.73 ms'}
depth 1:
params - {'ModuleList': '3.02 B'}
MACs - {'ModuleList': '9.8 TMACs'}
fwd latency - {'ModuleList': '154.18 ms'}
depth 2:
params - {'DiTLayer': '3.02 B'}
MACs - {'DiTLayer': '9.8 TMACs'}
fwd latency - {'DiTLayer': '154.18 ms'}
depth 3:
params - {'GemmaMLP': '1.51 B'}
MACs - {'GemmaMLP': '6.18 TMACs'}
fwd latency - {'DiTSelfAttention': '81.24 ms'}
------------------------------ Detailed Profile per GPU ------------------------------
Each module profile is listed after its name in the following order:
params, percentage of total params, MACs, percentage of total MACs, fwd latency, percentage of total fwd latency, fwd FLOPS
Note: 1. A module can have torch.nn.module or torch.nn.functional to compute logits (e.g. CrossEntropyLoss). They are not counted as submodules, thus not to be printed out. However they make up the difference between a parent's MACs (or latency) and the sum of its submodules'.
2. Number of floating-point operations is a theoretical estimation, thus FLOPS computed using that could be larger than the maximum system throughput.
3. The fwd latency listed in the top module's profile is directly captured at the module forward function in PyTorch, thus it's less than the fwd latency shown above which is captured in DeepSpeed.
DiT(
3.05 B = 100% Params, 9.86 TMACs = 100% MACs, 158.73 ms = 100% latency, 124.29 TFLOPS
(layers): ModuleList(
(0): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.24 ms = 3.3% latency, 124.79 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 628.95 us = 0.4% latency, 1.28 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 36.24 us = 0.02% latency, 904.2 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 184.54 us = 0.12% latency, 4.36 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.76 ms = 1.74% latency, 87.27 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 238.42 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 276.8 us = 0.17% latency, 248.26 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 160.93 us = 0.1% latency, 106.75 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.87 us = 0.1% latency, 113.12 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.09 us = 0.09% latency, 125.32 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 251.05 us = 0.16% latency, 273.72 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.18 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 336.48 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 343.8 us = 0.22% latency, 399.76 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.1 us = 0.2% latency, 426.69 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.36 us = 0.19% latency, 456.06 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 86.55 us = 0.05% latency, 387.71 GFLOPS)
)
)
(1): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.23% latency, 127.56 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.56 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.38 us = 0.02% latency, 981.71 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.98 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.56 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.76 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 248.19 us = 0.16% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 248.67 us = 0.16% latency, 276.35 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.64 us = 0.1% latency, 109.68 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.73 us = 0.09% latency, 114.74 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.61 us = 0.09% latency, 125.75 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.95 us = 0.15% latency, 282.86 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 341.33 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 334.02 us = 0.21% latency, 411.46 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.96 us = 0.2% latency, 429.55 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.36 us = 0.19% latency, 456.06 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.68 us = 0.05% latency, 400.96 GFLOPS)
)
)
(2): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.23% latency, 127.59 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.08 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.33 us = 0.02% latency, 954.44 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.69 us = 0.1% latency, 5.01 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.83 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.06 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 254.15 us = 0.16% latency, 270.38 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.26 us = 0.1% latency, 111.37 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 159.03 us = 0.1% latency, 108.03 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.38 us = 0.09% latency, 125.97 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 238.9 us = 0.15% latency, 287.66 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.03 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.46 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 333.55 us = 0.21% latency, 412.05 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.82 us = 0.2% latency, 425.75 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 299.93 us = 0.19% latency, 458.24 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 85.59 us = 0.05% latency, 392.03 GFLOPS)
)
)
(3): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.97 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 602.25 us = 0.38% latency, 1.34 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 164.27 us = 0.1% latency, 4.9 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.68 ms = 1.69% latency, 89.59 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.29 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.84 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.48 us = 0.16% latency, 277.68 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.26 us = 0.1% latency, 111.37 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.16 us = 0.1% latency, 113.66 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 233.41 us = 0.15% latency, 294.41 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 342.82 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 331.88 us = 0.21% latency, 414.12 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.81 us = 0.2% latency, 432.45 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.41 us = 0.19% latency, 457.51 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.49 us = 0.05% latency, 406.76 GFLOPS)
)
)
(4): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.09 ms = 3.21% latency, 128.43 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.98 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.98 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.53 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.13 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.24 us = 0.16% latency, 277.95 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.36 us = 0.1% latency, 109.18 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.63 us = 0.1% latency, 113.3 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 237.46 us = 0.15% latency, 289.39 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 343.16 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 329.73 us = 0.21% latency, 416.82 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.81 us = 0.2% latency, 432.45 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 302.55 us = 0.19% latency, 454.26 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.25 us = 0.05% latency, 407.93 GFLOPS)
)
)
(5): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.09 ms = 3.21% latency, 128.42 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.5 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.33 us = 0.02% latency, 954.44 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.55 us = 0.1% latency, 5.08 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.56 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.58 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.58 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.84 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.23 us = 0.15% latency, 283.69 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.4 us = 0.1% latency, 109.84 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.16 us = 0.1% latency, 113.66 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.62 us = 0.09% latency, 121.31 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.14 us = 0.09% latency, 126.2 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 239.85 us = 0.15% latency, 286.51 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 342.68 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 335.93 us = 0.21% latency, 409.13 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.2 us = 0.2% latency, 429.23 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 297.78 us = 0.19% latency, 461.54 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.06 us = 0.05% latency, 413.93 GFLOPS)
)
)
(6): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.16 ms = 3.25% latency, 126.68 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 582.22 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.38 us = 0.02% latency, 981.71 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.03 us = 0.1% latency, 5.06 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.51 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.56 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.86 us = 0.16% latency, 275.03 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 159.98 us = 0.1% latency, 107.39 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 152.11 us = 0.1% latency, 112.94 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 143.05 us = 0.09% latency, 120.1 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 236.99 us = 0.15% latency, 289.97 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.53 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 336.65 us = 0.21% latency, 408.26 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 330.69 us = 0.21% latency, 415.62 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 304.7 us = 0.19% latency, 451.06 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 85.59 us = 0.05% latency, 392.03 GFLOPS)
)
)
(7): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.19 ms = 3.27% latency, 126.06 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 600.58 us = 0.38% latency, 1.34 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.81 us = 0.02% latency, 941.36 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 169.99 us = 0.11% latency, 4.74 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.39 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.01 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 257.25 us = 0.16% latency, 267.13 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.16 us = 0.1% latency, 110.01 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.95 us = 0.09% latency, 122.76 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.91 us = 0.09% latency, 121.92 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.72 us = 0.16% latency, 277.41 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.72 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 340.7 us = 0.21% latency, 403.4 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.63 us = 0.2% latency, 427.32 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 311.14 us = 0.2% latency, 441.73 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.45 us = 0.05% latency, 402.11 GFLOPS)
)
)
(8): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.24% latency, 127.04 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 592.95 us = 0.37% latency, 1.36 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.09 us = 0.02% latency, 961.11 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 154.02 us = 0.1% latency, 5.23 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.56 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.62 us = 0.16% latency, 275.29 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.06 us = 0.1% latency, 112.24 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.73 us = 0.09% latency, 114.74 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.86 us = 0.09% latency, 121.11 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.53 us = 0.09% latency, 115.66 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.62 us = 0.15% latency, 280.93 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.18 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 339.19 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 334.98 us = 0.21% latency, 410.29 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 318.29 us = 0.2% latency, 431.81 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 306.13 us = 0.19% latency, 448.96 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.21 us = 0.05% latency, 403.26 GFLOPS)
)
)
(9): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.17 ms = 3.26% latency, 126.46 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 590.8 us = 0.37% latency, 1.36 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.86 us = 0.02% latency, 967.88 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 161.17 us = 0.1% latency, 5 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.76 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.44 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.01 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 255.11 us = 0.16% latency, 269.37 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 161.17 us = 0.1% latency, 106.59 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.53 us = 0.09% latency, 115.66 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.67 us = 0.09% latency, 122.13 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 255.58 us = 0.16% latency, 268.87 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.75 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.14 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.84 us = 0.21% latency, 406.82 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 338.55 us = 0.21% latency, 405.96 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.17 us = 0.19% latency, 457.87 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.45 us = 0.05% latency, 402.11 GFLOPS)
)
)
(10): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.93 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.98 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.86 us = 0.02% latency, 967.88 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 161.89 us = 0.1% latency, 4.97 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.31 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.58 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.62 us = 0.15% latency, 280.93 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.12 us = 0.1% latency, 109.34 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.33 us = 0.09% latency, 125.1 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.48 us = 0.16% latency, 277.68 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.39 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 336.17 us = 0.21% latency, 408.84 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.48 us = 0.2% latency, 430.19 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.12 us = 0.19% latency, 456.42 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.06 us = 0.05% latency, 413.93 GFLOPS)
)
)
(11): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.07 ms = 3.19% latency, 128.95 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 577.21 us = 0.36% latency, 1.4 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.66 us = 0.02% latency, 1 GFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.03 us = 0.1% latency, 5.06 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.67 ms = 1.68% latency, 90.06 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.1 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.6 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.14 us = 0.15% latency, 281.47 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 152.59 us = 0.1% latency, 112.59 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.97 us = 0.09% latency, 114.56 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.14 us = 0.09% latency, 121.72 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 134.71 us = 0.08% latency, 127.54 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 235.32 us = 0.15% latency, 292.03 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.99 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 342.48 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 334.98 us = 0.21% latency, 410.29 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.57 us = 0.2% latency, 432.78 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.17 us = 0.19% latency, 457.87 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 80.82 us = 0.05% latency, 415.15 GFLOPS)
)
)
(12): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 128.03 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 587.22 us = 0.37% latency, 1.37 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.33 us = 0.02% latency, 954.44 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 163.32 us = 0.1% latency, 4.93 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.42 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.7 ms = 1.7% latency, 89.2 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.29 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 233.41 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 245.33 us = 0.15% latency, 280.11 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 159.26 us = 0.1% latency, 107.87 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.16 us = 0.1% latency, 113.66 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.91 us = 0.09% latency, 121.92 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.38 us = 0.09% latency, 125.97 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 235.32 us = 0.15% latency, 292.03 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 343.16 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 332.12 us = 0.21% latency, 413.83 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.57 us = 0.2% latency, 432.78 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.41 us = 0.19% latency, 457.51 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.54 us = 0.05% latency, 411.51 GFLOPS)
)
)
(13): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.94 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 582.7 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.19 us = 0.02% latency, 1.02 GFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.79 us = 0.1% latency, 5.07 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.56 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.68 ms = 1.69% latency, 89.82 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 232.93 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 248.43 us = 0.16% latency, 276.61 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.73 us = 0.1% latency, 111.03 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 138.04 us = 0.09% latency, 124.45 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 134.94 us = 0.09% latency, 127.31 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 234.84 us = 0.15% latency, 292.62 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.99 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.98 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 349.76 us = 0.22% latency, 392.95 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.82 us = 0.2% latency, 425.75 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 302.79 us = 0.19% latency, 453.91 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.73 us = 0.05% latency, 405.58 GFLOPS)
)
)
(14): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.87 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 584.13 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.38 us = 0.02% latency, 981.71 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.69 us = 0.1% latency, 5.01 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.25 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.34 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 244.14 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.62 us = 0.15% latency, 280.93 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.97 us = 0.1% latency, 110.86 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.53 us = 0.09% latency, 115.66 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 138.04 us = 0.09% latency, 124.45 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 241.52 us = 0.15% latency, 284.53 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.99 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.12 us = 0.21% latency, 407.68 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.72 us = 0.2% latency, 429.87 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 299.45 us = 0.19% latency, 458.97 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.3 us = 0.05% latency, 412.72 GFLOPS)
)
)
(15): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.93 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 578.88 us = 0.36% latency, 1.39 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.66 us = 0.02% latency, 1 GFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.55 us = 0.1% latency, 5.08 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.7% latency, 88.87 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.25 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 231.74 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 246.29 us = 0.16% latency, 279.02 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.45 us = 0.1% latency, 110.52 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.86 us = 0.09% latency, 121.11 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.44 us = 0.09% latency, 114.2 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 239.37 us = 0.15% latency, 287.08 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 342.07 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 335.22 us = 0.21% latency, 410 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319 us = 0.2% latency, 430.84 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 298.74 us = 0.19% latency, 460.06 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.3 us = 0.05% latency, 412.72 GFLOPS)
)
)
(16): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.1 ms = 3.22% latency, 128.09 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 578.64 us = 0.36% latency, 1.39 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.26 us = 0.1% latency, 5.06 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.9 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.54 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.06 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 231.27 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 245.33 us = 0.15% latency, 280.11 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.54 us = 0.1% latency, 111.89 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 147.58 us = 0.09% latency, 116.41 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.19 us = 0.09% latency, 122.55 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.14 us = 0.15% latency, 281.47 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 339.12 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 338.79 us = 0.21% latency, 405.67 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.96 us = 0.2% latency, 429.55 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.6 us = 0.19% latency, 455.7 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.54 us = 0.05% latency, 411.51 GFLOPS)
)
)
(17): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.1 ms = 3.21% latency, 128.11 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 573.16 us = 0.36% latency, 1.41 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.9 us = 0.02% latency, 995.93 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 156.88 us = 0.1% latency, 5.13 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.75 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.43 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.82 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.13 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.72 us = 0.16% latency, 277.41 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.59 us = 0.1% latency, 109.01 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.3 us = 0.09% latency, 115.85 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.66 us = 0.09% latency, 126.64 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.47 us = 0.15% latency, 283.41 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 339.06 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 339.03 us = 0.21% latency, 405.39 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.67 us = 0.2% latency, 428.6 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.6 us = 0.19% latency, 455.7 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.02 us = 0.05% latency, 409.12 GFLOPS)
)
)
(18): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.19 ms = 3.27% latency, 125.93 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 584.6 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.22 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.76 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.74 ms = 1.72% latency, 87.86 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 469.68 us = 0.3% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 239.85 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 245.33 us = 0.15% latency, 280.11 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 158.79 us = 0.1% latency, 108.19 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.78 us = 0.1% latency, 111.72 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 143.77 us = 0.09% latency, 119.5 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.91 us = 0.09% latency, 121.92 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.86 us = 0.16% latency, 275.03 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.24 ms = 0.78% latency, 332.22 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 349.28 us = 0.22% latency, 393.49 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 328.06 us = 0.21% latency, 418.94 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 306.61 us = 0.19% latency, 448.26 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.92 us = 0.05% latency, 399.82 GFLOPS)
)
)
(19): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.26 ms = 3.32% latency, 124.18 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 604.15 us = 0.38% latency, 1.33 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 37.67 us = 0.02% latency, 869.87 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 167.61 us = 0.11% latency, 4.8 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.78 ms = 1.75% latency, 86.61 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 465.63 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 250.34 us = 0.16% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 255.11 us = 0.16% latency, 269.37 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 166.18 us = 0.1% latency, 103.38 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.88 us = 0.1% latency, 109.51 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.01 us = 0.09% latency, 115.29 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.38 us = 0.09% latency, 121.51 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 253.68 us = 0.16% latency, 270.89 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.25 ms = 0.79% latency, 330.25 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 349.76 us = 0.22% latency, 392.95 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 326.4 us = 0.21% latency, 421.08 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 311.61 us = 0.2% latency, 441.06 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.92 us = 0.05% latency, 399.82 GFLOPS)
)
)
(20): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.13 ms = 3.23% latency, 127.45 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 586.75 us = 0.37% latency, 1.37 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.57 us = 0.02% latency, 947.85 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 161.89 us = 0.1% latency, 4.97 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.53 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.25 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 232.93 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 243.43 us = 0.15% latency, 282.3 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.97 us = 0.1% latency, 110.86 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 147.1 us = 0.09% latency, 116.79 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 142.57 us = 0.09% latency, 120.5 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 241.99 us = 0.15% latency, 283.97 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 335.37 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 342.37 us = 0.22% latency, 401.44 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.82 us = 0.2% latency, 425.75 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 307.56 us = 0.19% latency, 446.87 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.97 us = 0.05% latency, 404.42 GFLOPS)
)
)
(21): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.25% latency, 126.85 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 586.03 us = 0.37% latency, 1.37 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.09 us = 0.02% latency, 961.11 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.22 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.73 ms = 1.72% latency, 88.03 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.72 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 256.06 us = 0.16% latency, 268.37 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 163.79 us = 0.1% latency, 104.89 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.87 us = 0.1% latency, 113.12 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.62 us = 0.09% latency, 121.31 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.61 us = 0.09% latency, 125.75 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 248.67 us = 0.16% latency, 276.35 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.9 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.59 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.6 us = 0.21% latency, 407.11 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.67 us = 0.2% latency, 428.6 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.41 us = 0.19% latency, 457.51 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.78 us = 0.05% latency, 410.31 GFLOPS)
)
)
(22): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.14 ms = 3.24% latency, 127.25 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.79 us = 0.37% latency, 1.37 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.09 us = 0.02% latency, 961.11 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 163.08 us = 0.1% latency, 4.94 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.52 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.84 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 258.21 us = 0.16% latency, 266.14 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.5 us = 0.1% latency, 111.2 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.01 us = 0.09% latency, 115.29 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.95 us = 0.09% latency, 122.76 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.33 us = 0.09% latency, 125.1 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 237.94 us = 0.15% latency, 288.81 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.86 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 340.22 us = 0.21% latency, 403.97 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 323.53 us = 0.2% latency, 424.81 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.12 us = 0.19% latency, 456.42 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.25 us = 0.05% latency, 407.93 GFLOPS)
)
)
(23): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.25% latency, 126.88 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 595.81 us = 0.38% latency, 1.35 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.81 us = 0.02% latency, 941.36 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 169.52 us = 0.11% latency, 4.75 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.66 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 466.35 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 238.9 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.15 us = 0.16% latency, 275.82 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.54 us = 0.1% latency, 111.89 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.2 us = 0.09% latency, 114.38 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 142.34 us = 0.09% latency, 120.7 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.09 us = 0.09% latency, 125.32 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 243.66 us = 0.15% latency, 282.03 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.27 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.34 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 342.61 us = 0.22% latency, 401.16 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.43 us = 0.2% latency, 428.91 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 303.27 us = 0.19% latency, 453.19 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.73 us = 0.05% latency, 405.58 GFLOPS)
)
)
(24): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.23% latency, 127.67 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.98 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.29 us = 0.02% latency, 928.64 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.22 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.29 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.34 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 246.76 us = 0.16% latency, 278.48 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.83 us = 0.1% latency, 108.85 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.01 us = 0.09% latency, 115.29 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.67 us = 0.09% latency, 122.13 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.86 us = 0.15% latency, 280.65 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.6 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 340.22 us = 0.21% latency, 403.97 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 326.16 us = 0.21% latency, 421.39 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 303.51 us = 0.19% latency, 452.84 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.25 us = 0.05% latency, 407.93 GFLOPS)
)
)
(25): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.22% latency, 127.78 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.56 us = 0.37% latency, 1.38 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.86 us = 0.02% latency, 967.88 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.74 us = 0.1% latency, 5.04 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.35 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 246.29 us = 0.16% latency, 279.02 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.21 us = 0.1% latency, 110.69 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.87 us = 0.1% latency, 113.12 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.38 us = 0.09% latency, 121.51 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.95 us = 0.15% latency, 282.86 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 339.86 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.12 us = 0.21% latency, 407.68 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.63 us = 0.2% latency, 427.32 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.88 us = 0.19% latency, 456.78 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.78 us = 0.05% latency, 410.31 GFLOPS)
)
)
(26): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.18 ms = 3.26% latency, 126.29 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 589.61 us = 0.37% latency, 1.37 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.62 us = 0.02% latency, 974.74 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 165.22 us = 0.1% latency, 4.87 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.78 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.82 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 250.1 us = 0.16% latency, 274.77 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.69 us = 0.1% latency, 110.35 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.73 us = 0.09% latency, 114.74 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 250.34 us = 0.16% latency, 274.51 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.24 ms = 0.78% latency, 332.92 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 346.66 us = 0.22% latency, 396.47 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 329.02 us = 0.21% latency, 417.73 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 307.32 us = 0.19% latency, 447.22 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.45 us = 0.05% latency, 402.11 GFLOPS)
)
)
(27): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.25% latency, 126.81 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 590.09 us = 0.37% latency, 1.36 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.57 us = 0.02% latency, 947.85 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 165.22 us = 0.1% latency, 4.87 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.7 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.7 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.25 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.13 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 252.25 us = 0.16% latency, 272.43 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 160.46 us = 0.1% latency, 107.07 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.4 us = 0.1% latency, 113.48 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.38 us = 0.09% latency, 121.51 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 243.66 us = 0.15% latency, 282.03 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 336.03 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 341.65 us = 0.22% latency, 402.28 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 325.44 us = 0.21% latency, 422.32 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 304.7 us = 0.19% latency, 451.06 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.21 us = 0.05% latency, 403.26 GFLOPS)
)
)
(28): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.17 ms = 3.26% latency, 126.45 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 592.71 us = 0.37% latency, 1.36 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.29 us = 0.02% latency, 928.64 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 166.18 us = 0.1% latency, 4.85 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.73 ms = 1.72% latency, 88.07 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 464.92 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 245.33 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 252.96 us = 0.16% latency, 271.66 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 158.55 us = 0.1% latency, 108.36 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.12 us = 0.1% latency, 109.34 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 143.53 us = 0.09% latency, 119.7 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 241.04 us = 0.15% latency, 285.09 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.8 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 342.37 us = 0.22% latency, 401.44 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.87 us = 0.2% latency, 427.01 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 302.55 us = 0.19% latency, 454.26 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.49 us = 0.05% latency, 406.76 GFLOPS)
)
)
(29): DiTLayer(
100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.16 ms = 3.25% latency, 126.68 TFLOPS
(input_layernorm): AdaLayerNormZero(
25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 577.21 us = 0.36% latency, 1.4 TFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.05 us = 0.02% latency, 934.96 MFLOPS)
(linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.79 us = 0.1% latency, 5.07 TFLOPS, in_features=2048, out_features=12288, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.75 us = 0.15% latency, 0 FLOPS)
)
(self_attn): DiTSelfAttention(
25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.73 ms = 1.72% latency, 88 TFLOPS
(q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.72 us = 0.29% latency, 0 FLOPS)
(k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS)
(q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.48 us = 0.16% latency, 277.68 TFLOPS, in_features=2048, out_features=4096, bias=False)
(k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 165.94 us = 0.1% latency, 103.53 TFLOPS, in_features=2048, out_features=1024, bias=False)
(v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.21 us = 0.1% latency, 110.69 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.14 us = 0.09% latency, 121.72 TFLOPS, in_features=2048, out_features=1024, bias=False)
(text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.95 us = 0.09% latency, 122.76 TFLOPS, in_features=2048, out_features=1024, bias=False)
(o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 252.25 us = 0.16% latency, 272.43 TFLOPS, in_features=4096, out_features=2048, bias=False)
)
(post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.7 us = 0.15% latency, 0 FLOPS)
(mlp): GemmaMLP(
50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 336.09 TFLOPS
(gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 341.42 us = 0.22% latency, 402.56 TFLOPS, in_features=2048, out_features=8192, bias=False)
(up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.15 us = 0.2% latency, 427.96 TFLOPS, in_features=2048, out_features=8192, bias=False)
(down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.84 us = 0.19% latency, 455.34 TFLOPS, in_features=8192, out_features=2048, bias=False)
(act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.49 us = 0.05% latency, 406.76 GFLOPS)
)
)
)
(patch_embed): PatchEmbed(
133.12 K = 0% Params, 536.87 MMACs = 0.01% MACs, 487.8 us = 0.31% latency, 2.22 TFLOPS
(proj): Conv2d(133.12 K = 0% Params, 536.87 MMACs = 0.01% MACs, 300.17 us = 0.19% latency, 3.61 TFLOPS, 16, 2048, kernel_size=(2, 2), stride=(2, 2))
)
(rotary_emb): GemmaRotaryEmbedding(0 = 0% Params, 0 MACs = 0% MACs, 0 s = 0% latency, 0 FLOPS)
(time_proj): Timesteps(0 = 0% Params, 0 MACs = 0% MACs, 248.43 us = 0.16% latency, 0 FLOPS)
(timestep_embedder): Sequential(
4.72 M = 0.15% Params, 75.5 MMACs = 0% MACs, 493.53 us = 0.31% latency, 306.02 GFLOPS
(0): Linear(526.34 K = 0.02% Params, 8.39 MMACs = 0% MACs, 227.93 us = 0.14% latency, 73.61 GFLOPS, in_features=256, out_features=2048, bias=True)
(1): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 50.78 us = 0.03% latency, 645.25 MFLOPS)
(2): Linear(4.2 M = 0.14% Params, 67.11 MMACs = 0% MACs, 144.72 us = 0.09% latency, 927.43 GFLOPS, in_features=2048, out_features=2048, bias=True)
)
(context_embedder): Sequential(
4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 451.56 us = 0.28% latency, 76.09 TFLOPS
(0): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 171.18 us = 0.11% latency, 0 FLOPS)
(1): Linear(4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 226.97 us = 0.14% latency, 151.38 TFLOPS, in_features=2048, out_features=2048, bias=True)
)
(norm_out): AdaLayerNormOut(
8.39 M = 0.28% Params, 134.22 MMACs = 0% MACs, 570.77 us = 0.36% latency, 470.36 GFLOPS
(silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.05 us = 0.02% latency, 934.96 MFLOPS)
(linear): Linear(8.39 M = 0.28% Params, 134.22 MMACs = 0% MACs, 148.77 us = 0.09% latency, 1.8 TFLOPS, in_features=2048, out_features=4096, bias=True)
(norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS)
)
(proj_out): Linear(131.14 K = 0% Params, 536.87 MMACs = 0.01% MACs, 173.09 us = 0.11% latency, 6.2 TFLOPS, in_features=2048, out_features=64, bias=True)
(repa_projector): Sequential(
9.97 M = 0.33% Params, 40.8 GMACs = 0.41% MACs, 698.57 us = 0.44% latency, 116.84 TFLOPS
(0): Linear(4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 206.47 us = 0.13% latency, 166.41 TFLOPS, in_features=2048, out_features=2048, bias=True)
(1): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.57 us = 0.02% latency, 242.65 GFLOPS)
(2): Linear(4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 171.9 us = 0.11% latency, 199.88 TFLOPS, in_features=2048, out_features=2048, bias=True)
(3): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 36.24 us = 0.02% latency, 231.48 GFLOPS)
(4): Linear(1.57 M = 0.05% Params, 6.44 GMACs = 0.07% MACs, 148.06 us = 0.09% latency, 87.03 TFLOPS, in_features=2048, out_features=768, bias=True)
)
)
------------------------------------------------------------------------------
|