| -------------------------- DeepSpeed Flops Profiler -------------------------- | |
| Profile Summary at step 2: | |
| Notations: | |
| data parallel size (dp_size), model parallel size(mp_size), | |
| number of parameters (params), number of multiply-accumulate operations(MACs), | |
| number of floating-point operations (flops), floating-point operations per second (FLOPS), | |
| fwd latency (forward propagation latency), bwd latency (backward propagation latency), | |
| step (weights update latency), iter latency (sum of fwd, bwd and step latency) | |
| world size: 32 | |
| data parallel size: 32 | |
| model parallel size: 1 | |
| batch size per GPU: 16 | |
| params per GPU: 3.05 B | |
| params of model = params per GPU * mp_size: 3.05 B | |
| fwd MACs per GPU: 9.86 TMACs | |
| fwd flops per GPU: 19.73 T | |
| fwd flops of model = fwd flops per GPU * mp_size: 19.73 T | |
| fwd latency: 158.91 ms | |
| fwd FLOPS per GPU = fwd flops per GPU / fwd latency: 124.15 TFLOPS | |
| bwd latency: 559.86 ms | |
| bwd FLOPS per GPU = 2 * fwd flops per GPU / bwd latency: 70.48 TFLOPS | |
| fwd+bwd FLOPS per GPU = 3 * fwd flops per GPU / (fwd+bwd latency): 82.34 TFLOPS | |
| step latency: 167.69 ms | |
| iter latency: 886.46 ms | |
| FLOPS per GPU = 3 * fwd flops per GPU / iter latency: 66.77 TFLOPS | |
| samples/second: 577.58 | |
| ----------------------------- Aggregated Profile per GPU ----------------------------- | |
| Top 1 modules in terms of params, MACs or fwd latency at different model depths: | |
| depth 0: | |
| params - {'DiT': '3.05 B'} | |
| MACs - {'DiT': '9.86 TMACs'} | |
| fwd latency - {'DiT': '158.73 ms'} | |
| depth 1: | |
| params - {'ModuleList': '3.02 B'} | |
| MACs - {'ModuleList': '9.8 TMACs'} | |
| fwd latency - {'ModuleList': '154.18 ms'} | |
| depth 2: | |
| params - {'DiTLayer': '3.02 B'} | |
| MACs - {'DiTLayer': '9.8 TMACs'} | |
| fwd latency - {'DiTLayer': '154.18 ms'} | |
| depth 3: | |
| params - {'GemmaMLP': '1.51 B'} | |
| MACs - {'GemmaMLP': '6.18 TMACs'} | |
| fwd latency - {'DiTSelfAttention': '81.24 ms'} | |
| ------------------------------ Detailed Profile per GPU ------------------------------ | |
| Each module profile is listed after its name in the following order: | |
| params, percentage of total params, MACs, percentage of total MACs, fwd latency, percentage of total fwd latency, fwd FLOPS | |
| Note: 1. A module can have torch.nn.module or torch.nn.functional to compute logits (e.g. CrossEntropyLoss). They are not counted as submodules, thus not to be printed out. However they make up the difference between a parent's MACs (or latency) and the sum of its submodules'. | |
| 2. Number of floating-point operations is a theoretical estimation, thus FLOPS computed using that could be larger than the maximum system throughput. | |
| 3. The fwd latency listed in the top module's profile is directly captured at the module forward function in PyTorch, thus it's less than the fwd latency shown above which is captured in DeepSpeed. | |
| DiT( | |
| 3.05 B = 100% Params, 9.86 TMACs = 100% MACs, 158.73 ms = 100% latency, 124.29 TFLOPS | |
| (layers): ModuleList( | |
| (0): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.24 ms = 3.3% latency, 124.79 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 628.95 us = 0.4% latency, 1.28 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 36.24 us = 0.02% latency, 904.2 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 184.54 us = 0.12% latency, 4.36 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.76 ms = 1.74% latency, 87.27 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 238.42 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 276.8 us = 0.17% latency, 248.26 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 160.93 us = 0.1% latency, 106.75 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.87 us = 0.1% latency, 113.12 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.09 us = 0.09% latency, 125.32 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 251.05 us = 0.16% latency, 273.72 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.18 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 336.48 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 343.8 us = 0.22% latency, 399.76 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.1 us = 0.2% latency, 426.69 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.36 us = 0.19% latency, 456.06 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 86.55 us = 0.05% latency, 387.71 GFLOPS) | |
| ) | |
| ) | |
| (1): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.23% latency, 127.56 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.56 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.38 us = 0.02% latency, 981.71 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.98 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.56 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.76 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 248.19 us = 0.16% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 248.67 us = 0.16% latency, 276.35 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.64 us = 0.1% latency, 109.68 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.73 us = 0.09% latency, 114.74 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.61 us = 0.09% latency, 125.75 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.95 us = 0.15% latency, 282.86 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 341.33 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 334.02 us = 0.21% latency, 411.46 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.96 us = 0.2% latency, 429.55 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.36 us = 0.19% latency, 456.06 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.68 us = 0.05% latency, 400.96 GFLOPS) | |
| ) | |
| ) | |
| (2): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.23% latency, 127.59 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.08 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.33 us = 0.02% latency, 954.44 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.69 us = 0.1% latency, 5.01 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.83 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.06 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 254.15 us = 0.16% latency, 270.38 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.26 us = 0.1% latency, 111.37 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 159.03 us = 0.1% latency, 108.03 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.38 us = 0.09% latency, 125.97 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 238.9 us = 0.15% latency, 287.66 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.03 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.46 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 333.55 us = 0.21% latency, 412.05 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.82 us = 0.2% latency, 425.75 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 299.93 us = 0.19% latency, 458.24 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 85.59 us = 0.05% latency, 392.03 GFLOPS) | |
| ) | |
| ) | |
| (3): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.97 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 602.25 us = 0.38% latency, 1.34 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 164.27 us = 0.1% latency, 4.9 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.68 ms = 1.69% latency, 89.59 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.29 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.84 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.48 us = 0.16% latency, 277.68 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.26 us = 0.1% latency, 111.37 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.16 us = 0.1% latency, 113.66 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 233.41 us = 0.15% latency, 294.41 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 342.82 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 331.88 us = 0.21% latency, 414.12 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.81 us = 0.2% latency, 432.45 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.41 us = 0.19% latency, 457.51 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.49 us = 0.05% latency, 406.76 GFLOPS) | |
| ) | |
| ) | |
| (4): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.09 ms = 3.21% latency, 128.43 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.98 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.98 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.53 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.13 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.24 us = 0.16% latency, 277.95 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.36 us = 0.1% latency, 109.18 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.63 us = 0.1% latency, 113.3 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 237.46 us = 0.15% latency, 289.39 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 343.16 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 329.73 us = 0.21% latency, 416.82 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.81 us = 0.2% latency, 432.45 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 302.55 us = 0.19% latency, 454.26 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.25 us = 0.05% latency, 407.93 GFLOPS) | |
| ) | |
| ) | |
| (5): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.09 ms = 3.21% latency, 128.42 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.5 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.33 us = 0.02% latency, 954.44 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.55 us = 0.1% latency, 5.08 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.56 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.58 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.58 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.84 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.23 us = 0.15% latency, 283.69 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.4 us = 0.1% latency, 109.84 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.16 us = 0.1% latency, 113.66 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.62 us = 0.09% latency, 121.31 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.14 us = 0.09% latency, 126.2 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 239.85 us = 0.15% latency, 286.51 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 342.68 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 335.93 us = 0.21% latency, 409.13 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.2 us = 0.2% latency, 429.23 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 297.78 us = 0.19% latency, 461.54 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.06 us = 0.05% latency, 413.93 GFLOPS) | |
| ) | |
| ) | |
| (6): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.16 ms = 3.25% latency, 126.68 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 582.22 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.38 us = 0.02% latency, 981.71 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.03 us = 0.1% latency, 5.06 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.51 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.56 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.86 us = 0.16% latency, 275.03 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 159.98 us = 0.1% latency, 107.39 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 152.11 us = 0.1% latency, 112.94 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 143.05 us = 0.09% latency, 120.1 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 236.99 us = 0.15% latency, 289.97 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.53 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 336.65 us = 0.21% latency, 408.26 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 330.69 us = 0.21% latency, 415.62 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 304.7 us = 0.19% latency, 451.06 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 85.59 us = 0.05% latency, 392.03 GFLOPS) | |
| ) | |
| ) | |
| (7): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.19 ms = 3.27% latency, 126.06 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 600.58 us = 0.38% latency, 1.34 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.81 us = 0.02% latency, 941.36 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 169.99 us = 0.11% latency, 4.74 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.39 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.01 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 257.25 us = 0.16% latency, 267.13 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.16 us = 0.1% latency, 110.01 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.95 us = 0.09% latency, 122.76 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.91 us = 0.09% latency, 121.92 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.72 us = 0.16% latency, 277.41 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.72 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 340.7 us = 0.21% latency, 403.4 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.63 us = 0.2% latency, 427.32 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 311.14 us = 0.2% latency, 441.73 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.45 us = 0.05% latency, 402.11 GFLOPS) | |
| ) | |
| ) | |
| (8): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.24% latency, 127.04 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 592.95 us = 0.37% latency, 1.36 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.09 us = 0.02% latency, 961.11 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 154.02 us = 0.1% latency, 5.23 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.56 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.62 us = 0.16% latency, 275.29 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.06 us = 0.1% latency, 112.24 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.73 us = 0.09% latency, 114.74 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.86 us = 0.09% latency, 121.11 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.53 us = 0.09% latency, 115.66 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.62 us = 0.15% latency, 280.93 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.18 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 339.19 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 334.98 us = 0.21% latency, 410.29 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 318.29 us = 0.2% latency, 431.81 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 306.13 us = 0.19% latency, 448.96 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.21 us = 0.05% latency, 403.26 GFLOPS) | |
| ) | |
| ) | |
| (9): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.17 ms = 3.26% latency, 126.46 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 590.8 us = 0.37% latency, 1.36 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.86 us = 0.02% latency, 967.88 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 161.17 us = 0.1% latency, 5 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.76 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.72 ms = 1.71% latency, 88.44 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.01 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 255.11 us = 0.16% latency, 269.37 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 161.17 us = 0.1% latency, 106.59 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.53 us = 0.09% latency, 115.66 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.67 us = 0.09% latency, 122.13 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 255.58 us = 0.16% latency, 268.87 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.75 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.14 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.84 us = 0.21% latency, 406.82 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 338.55 us = 0.21% latency, 405.96 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.17 us = 0.19% latency, 457.87 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.45 us = 0.05% latency, 402.11 GFLOPS) | |
| ) | |
| ) | |
| (10): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.93 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.98 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.86 us = 0.02% latency, 967.88 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 161.89 us = 0.1% latency, 4.97 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.31 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.58 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.62 us = 0.15% latency, 280.93 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.12 us = 0.1% latency, 109.34 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.33 us = 0.09% latency, 125.1 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.48 us = 0.16% latency, 277.68 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.39 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 336.17 us = 0.21% latency, 408.84 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.48 us = 0.2% latency, 430.19 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.12 us = 0.19% latency, 456.42 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.06 us = 0.05% latency, 413.93 GFLOPS) | |
| ) | |
| ) | |
| (11): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.07 ms = 3.19% latency, 128.95 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 577.21 us = 0.36% latency, 1.4 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.66 us = 0.02% latency, 1 GFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.03 us = 0.1% latency, 5.06 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.67 ms = 1.68% latency, 90.06 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.1 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.6 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.14 us = 0.15% latency, 281.47 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 152.59 us = 0.1% latency, 112.59 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.97 us = 0.09% latency, 114.56 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.14 us = 0.09% latency, 121.72 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 134.71 us = 0.08% latency, 127.54 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 235.32 us = 0.15% latency, 292.03 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.99 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 342.48 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 334.98 us = 0.21% latency, 410.29 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.57 us = 0.2% latency, 432.78 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.17 us = 0.19% latency, 457.87 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 80.82 us = 0.05% latency, 415.15 GFLOPS) | |
| ) | |
| ) | |
| (12): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 128.03 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 587.22 us = 0.37% latency, 1.37 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.33 us = 0.02% latency, 954.44 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 163.32 us = 0.1% latency, 4.93 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.42 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.7 ms = 1.7% latency, 89.2 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.29 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 233.41 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 245.33 us = 0.15% latency, 280.11 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 159.26 us = 0.1% latency, 107.87 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.16 us = 0.1% latency, 113.66 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.91 us = 0.09% latency, 121.92 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.38 us = 0.09% latency, 125.97 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 235.32 us = 0.15% latency, 292.03 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.2 ms = 0.76% latency, 343.16 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 332.12 us = 0.21% latency, 413.83 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 317.57 us = 0.2% latency, 432.78 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.41 us = 0.19% latency, 457.51 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.54 us = 0.05% latency, 411.51 GFLOPS) | |
| ) | |
| ) | |
| (13): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.94 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 582.7 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.19 us = 0.02% latency, 1.02 GFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.79 us = 0.1% latency, 5.07 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.56 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.68 ms = 1.69% latency, 89.82 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.53 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 232.93 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 248.43 us = 0.16% latency, 276.61 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.73 us = 0.1% latency, 111.03 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 138.04 us = 0.09% latency, 124.45 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 134.94 us = 0.09% latency, 127.31 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 234.84 us = 0.15% latency, 292.62 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.99 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.78% latency, 334.98 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 349.76 us = 0.22% latency, 392.95 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.82 us = 0.2% latency, 425.75 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 302.79 us = 0.19% latency, 453.91 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.73 us = 0.05% latency, 405.58 GFLOPS) | |
| ) | |
| ) | |
| (14): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.87 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 584.13 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.38 us = 0.02% latency, 981.71 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.69 us = 0.1% latency, 5.01 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.25 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.34 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 244.14 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.62 us = 0.15% latency, 280.93 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.97 us = 0.1% latency, 110.86 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.53 us = 0.09% latency, 115.66 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 138.04 us = 0.09% latency, 124.45 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 241.52 us = 0.15% latency, 284.53 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.99 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.12 us = 0.21% latency, 407.68 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.72 us = 0.2% latency, 429.87 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 299.45 us = 0.19% latency, 458.97 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.3 us = 0.05% latency, 412.72 GFLOPS) | |
| ) | |
| ) | |
| (15): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.11 ms = 3.22% latency, 127.93 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 578.88 us = 0.36% latency, 1.39 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.66 us = 0.02% latency, 1 GFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.55 us = 0.1% latency, 5.08 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.7% latency, 88.87 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.25 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 231.74 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 246.29 us = 0.16% latency, 279.02 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.45 us = 0.1% latency, 110.52 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.68 us = 0.09% latency, 114.02 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.86 us = 0.09% latency, 121.11 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.44 us = 0.09% latency, 114.2 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 239.37 us = 0.15% latency, 287.08 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 342.07 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 335.22 us = 0.21% latency, 410 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319 us = 0.2% latency, 430.84 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 298.74 us = 0.19% latency, 460.06 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.3 us = 0.05% latency, 412.72 GFLOPS) | |
| ) | |
| ) | |
| (16): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.1 ms = 3.22% latency, 128.09 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 578.64 us = 0.36% latency, 1.39 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.26 us = 0.1% latency, 5.06 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.9 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.54 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.06 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 231.27 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 245.33 us = 0.15% latency, 280.11 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.54 us = 0.1% latency, 111.89 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 147.58 us = 0.09% latency, 116.41 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.19 us = 0.09% latency, 122.55 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.14 us = 0.15% latency, 281.47 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 339.12 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 338.79 us = 0.21% latency, 405.67 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 319.96 us = 0.2% latency, 429.55 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.6 us = 0.19% latency, 455.7 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.54 us = 0.05% latency, 411.51 GFLOPS) | |
| ) | |
| ) | |
| (17): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.1 ms = 3.21% latency, 128.11 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 573.16 us = 0.36% latency, 1.41 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 32.9 us = 0.02% latency, 995.93 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 156.88 us = 0.1% latency, 5.13 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.75 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.43 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.82 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.13 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.72 us = 0.16% latency, 277.41 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.59 us = 0.1% latency, 109.01 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 148.3 us = 0.09% latency, 115.85 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.66 us = 0.09% latency, 126.64 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.47 us = 0.15% latency, 283.41 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 339.06 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 339.03 us = 0.21% latency, 405.39 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.67 us = 0.2% latency, 428.6 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.6 us = 0.19% latency, 455.7 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.02 us = 0.05% latency, 409.12 GFLOPS) | |
| ) | |
| ) | |
| (18): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.19 ms = 3.27% latency, 125.93 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 584.6 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.14 us = 0.02% latency, 988.77 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.22 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.76 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.74 ms = 1.72% latency, 87.86 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 469.68 us = 0.3% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 239.85 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 245.33 us = 0.15% latency, 280.11 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 158.79 us = 0.1% latency, 108.19 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.78 us = 0.1% latency, 111.72 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 143.77 us = 0.09% latency, 119.5 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.91 us = 0.09% latency, 121.92 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.86 us = 0.16% latency, 275.03 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.24 ms = 0.78% latency, 332.22 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 349.28 us = 0.22% latency, 393.49 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 328.06 us = 0.21% latency, 418.94 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 306.61 us = 0.19% latency, 448.26 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.92 us = 0.05% latency, 399.82 GFLOPS) | |
| ) | |
| ) | |
| (19): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.26 ms = 3.32% latency, 124.18 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 604.15 us = 0.38% latency, 1.33 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 37.67 us = 0.02% latency, 869.87 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 167.61 us = 0.11% latency, 4.8 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.78 ms = 1.75% latency, 86.61 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 465.63 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 250.34 us = 0.16% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 255.11 us = 0.16% latency, 269.37 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 166.18 us = 0.1% latency, 103.38 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 156.88 us = 0.1% latency, 109.51 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.01 us = 0.09% latency, 115.29 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.38 us = 0.09% latency, 121.51 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 253.68 us = 0.16% latency, 270.89 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.25 ms = 0.79% latency, 330.25 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 349.76 us = 0.22% latency, 392.95 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 326.4 us = 0.21% latency, 421.08 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 311.61 us = 0.2% latency, 441.06 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.92 us = 0.05% latency, 399.82 GFLOPS) | |
| ) | |
| ) | |
| (20): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.13 ms = 3.23% latency, 127.45 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 586.75 us = 0.37% latency, 1.37 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.57 us = 0.02% latency, 947.85 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 161.89 us = 0.1% latency, 4.97 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.69% latency, 89.53 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.25 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 232.93 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 243.43 us = 0.15% latency, 282.3 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.97 us = 0.1% latency, 110.86 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 147.1 us = 0.09% latency, 116.79 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 142.57 us = 0.09% latency, 120.5 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 241.99 us = 0.15% latency, 283.97 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 335.37 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 342.37 us = 0.22% latency, 401.44 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 322.82 us = 0.2% latency, 425.75 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 307.56 us = 0.19% latency, 446.87 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.97 us = 0.05% latency, 404.42 GFLOPS) | |
| ) | |
| ) | |
| (21): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.25% latency, 126.85 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 586.03 us = 0.37% latency, 1.37 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.09 us = 0.02% latency, 961.11 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.22 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.33 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.73 ms = 1.72% latency, 88.03 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.72 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 256.06 us = 0.16% latency, 268.37 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 163.79 us = 0.1% latency, 104.89 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.87 us = 0.1% latency, 113.12 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.62 us = 0.09% latency, 121.31 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.61 us = 0.09% latency, 125.75 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 248.67 us = 0.16% latency, 276.35 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.9 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 340.59 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.6 us = 0.21% latency, 407.11 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.67 us = 0.2% latency, 428.6 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.41 us = 0.19% latency, 457.51 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.78 us = 0.05% latency, 410.31 GFLOPS) | |
| ) | |
| ) | |
| (22): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.14 ms = 3.24% latency, 127.25 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.79 us = 0.37% latency, 1.37 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.09 us = 0.02% latency, 961.11 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 163.08 us = 0.1% latency, 4.94 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.52 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.84 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 258.21 us = 0.16% latency, 266.14 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 154.5 us = 0.1% latency, 111.2 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.01 us = 0.09% latency, 115.29 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.95 us = 0.09% latency, 122.76 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.33 us = 0.09% latency, 125.1 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 237.94 us = 0.15% latency, 288.81 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.37 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.86 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 340.22 us = 0.21% latency, 403.97 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 323.53 us = 0.2% latency, 424.81 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.12 us = 0.19% latency, 456.42 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.25 us = 0.05% latency, 407.93 GFLOPS) | |
| ) | |
| ) | |
| (23): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.25% latency, 126.88 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 595.81 us = 0.38% latency, 1.35 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.81 us = 0.02% latency, 941.36 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 169.52 us = 0.11% latency, 4.75 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.66 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 466.35 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 238.9 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 249.15 us = 0.16% latency, 275.82 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 153.54 us = 0.1% latency, 111.89 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 150.2 us = 0.09% latency, 114.38 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 142.34 us = 0.09% latency, 120.7 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.09 us = 0.09% latency, 125.32 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 243.66 us = 0.15% latency, 282.03 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.27 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.34 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 342.61 us = 0.22% latency, 401.16 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 320.43 us = 0.2% latency, 428.91 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 303.27 us = 0.19% latency, 453.19 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.73 us = 0.05% latency, 405.58 GFLOPS) | |
| ) | |
| ) | |
| (24): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.23% latency, 127.67 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 581.98 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.29 us = 0.02% latency, 928.64 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 160.22 us = 0.1% latency, 5.03 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.29 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.34 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 246.76 us = 0.16% latency, 278.48 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.83 us = 0.1% latency, 108.85 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.01 us = 0.09% latency, 115.29 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 140.67 us = 0.09% latency, 122.13 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 136.85 us = 0.09% latency, 125.54 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 244.86 us = 0.15% latency, 280.65 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.6 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 340.22 us = 0.21% latency, 403.97 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 326.16 us = 0.21% latency, 421.39 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 303.51 us = 0.19% latency, 452.84 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.25 us = 0.05% latency, 407.93 GFLOPS) | |
| ) | |
| ) | |
| (25): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.12 ms = 3.22% latency, 127.78 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 585.56 us = 0.37% latency, 1.38 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.86 us = 0.02% latency, 967.88 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 159.74 us = 0.1% latency, 5.04 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 241.04 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.69 ms = 1.7% latency, 89.35 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 462.77 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 235.08 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 246.29 us = 0.16% latency, 279.02 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.21 us = 0.1% latency, 110.69 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.87 us = 0.1% latency, 113.12 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.38 us = 0.09% latency, 121.51 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 242.95 us = 0.15% latency, 282.86 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.21 ms = 0.76% latency, 339.86 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 337.12 us = 0.21% latency, 407.68 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.63 us = 0.2% latency, 427.32 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 300.88 us = 0.19% latency, 456.78 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 81.78 us = 0.05% latency, 410.31 GFLOPS) | |
| ) | |
| ) | |
| (26): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.18 ms = 3.26% latency, 126.29 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 589.61 us = 0.37% latency, 1.37 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 33.62 us = 0.02% latency, 974.74 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 165.22 us = 0.1% latency, 4.87 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 239.13 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.78 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 461.82 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 239.61 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 250.1 us = 0.16% latency, 274.77 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.69 us = 0.1% latency, 110.35 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 149.73 us = 0.09% latency, 114.74 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.47 us = 0.09% latency, 123.18 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 135.9 us = 0.09% latency, 126.42 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 250.34 us = 0.16% latency, 274.51 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 240.09 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.24 ms = 0.78% latency, 332.92 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 346.66 us = 0.22% latency, 396.47 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 329.02 us = 0.21% latency, 417.73 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 307.32 us = 0.19% latency, 447.22 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.45 us = 0.05% latency, 402.11 GFLOPS) | |
| ) | |
| ) | |
| (27): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.15 ms = 3.25% latency, 126.81 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 590.09 us = 0.37% latency, 1.36 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.57 us = 0.02% latency, 947.85 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 165.22 us = 0.1% latency, 4.87 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.7 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.71 ms = 1.71% latency, 88.7 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.25 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 234.13 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 252.25 us = 0.16% latency, 272.43 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 160.46 us = 0.1% latency, 107.07 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 151.4 us = 0.1% latency, 113.48 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.38 us = 0.09% latency, 121.51 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.24 us = 0.09% latency, 123.39 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 243.66 us = 0.15% latency, 282.03 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 336.03 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 341.65 us = 0.22% latency, 402.28 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 325.44 us = 0.21% latency, 422.32 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 304.7 us = 0.19% latency, 451.06 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 83.21 us = 0.05% latency, 403.26 GFLOPS) | |
| ) | |
| ) | |
| (28): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.17 ms = 3.26% latency, 126.45 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 592.71 us = 0.37% latency, 1.36 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.29 us = 0.02% latency, 928.64 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 166.18 us = 0.1% latency, 4.85 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.73 ms = 1.72% latency, 88.07 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 464.92 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 245.33 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 252.96 us = 0.16% latency, 271.66 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 158.55 us = 0.1% latency, 108.36 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 157.12 us = 0.1% latency, 109.34 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 143.53 us = 0.09% latency, 119.7 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 137.57 us = 0.09% latency, 124.88 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 241.04 us = 0.15% latency, 285.09 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.46 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.22 ms = 0.77% latency, 337.8 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 342.37 us = 0.22% latency, 401.44 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.87 us = 0.2% latency, 427.01 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 302.55 us = 0.19% latency, 454.26 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.49 us = 0.05% latency, 406.76 GFLOPS) | |
| ) | |
| ) | |
| (29): DiTLayer( | |
| 100.68 M = 3.3% Params, 326.82 GMACs = 3.31% MACs, 5.16 ms = 3.25% latency, 126.68 TFLOPS | |
| (input_layernorm): AdaLayerNormZero( | |
| 25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 577.21 us = 0.36% latency, 1.4 TFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.05 us = 0.02% latency, 934.96 MFLOPS) | |
| (linear): Linear(25.18 M = 0.83% Params, 402.65 MMACs = 0% MACs, 158.79 us = 0.1% latency, 5.07 TFLOPS, in_features=2048, out_features=12288, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 236.75 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (self_attn): DiTSelfAttention( | |
| 25.17 M = 0.83% Params, 120.26 GMACs = 1.22% MACs, 2.73 ms = 1.72% latency, 88 TFLOPS | |
| (q_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 463.72 us = 0.29% latency, 0 FLOPS) | |
| (k_norm): GemmaRMSNorm(128 = 0% Params, 0 MACs = 0% MACs, 237.94 us = 0.15% latency, 0 FLOPS) | |
| (q_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 247.48 us = 0.16% latency, 277.68 TFLOPS, in_features=2048, out_features=4096, bias=False) | |
| (k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 165.94 us = 0.1% latency, 103.53 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 155.21 us = 0.1% latency, 110.69 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_k_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 141.14 us = 0.09% latency, 121.72 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (text_v_proj): Linear(2.1 M = 0.07% Params, 8.59 GMACs = 0.09% MACs, 139.95 us = 0.09% latency, 122.76 TFLOPS, in_features=2048, out_features=1024, bias=False) | |
| (o_proj): Linear(8.39 M = 0.28% Params, 34.36 GMACs = 0.35% MACs, 252.25 us = 0.16% latency, 272.43 TFLOPS, in_features=4096, out_features=2048, bias=False) | |
| ) | |
| (post_attention_layernorm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 237.7 us = 0.15% latency, 0 FLOPS) | |
| (mlp): GemmaMLP( | |
| 50.33 M = 1.65% Params, 206.16 GMACs = 2.09% MACs, 1.23 ms = 0.77% latency, 336.09 TFLOPS | |
| (gate_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 341.42 us = 0.22% latency, 402.56 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (up_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 321.15 us = 0.2% latency, 427.96 TFLOPS, in_features=2048, out_features=8192, bias=False) | |
| (down_proj): Linear(16.78 M = 0.55% Params, 68.72 GMACs = 0.7% MACs, 301.84 us = 0.19% latency, 455.34 TFLOPS, in_features=8192, out_features=2048, bias=False) | |
| (act_fn): PytorchGELUTanh(0 = 0% Params, 0 MACs = 0% MACs, 82.49 us = 0.05% latency, 406.76 GFLOPS) | |
| ) | |
| ) | |
| ) | |
| (patch_embed): PatchEmbed( | |
| 133.12 K = 0% Params, 536.87 MMACs = 0.01% MACs, 487.8 us = 0.31% latency, 2.22 TFLOPS | |
| (proj): Conv2d(133.12 K = 0% Params, 536.87 MMACs = 0.01% MACs, 300.17 us = 0.19% latency, 3.61 TFLOPS, 16, 2048, kernel_size=(2, 2), stride=(2, 2)) | |
| ) | |
| (rotary_emb): GemmaRotaryEmbedding(0 = 0% Params, 0 MACs = 0% MACs, 0 s = 0% latency, 0 FLOPS) | |
| (time_proj): Timesteps(0 = 0% Params, 0 MACs = 0% MACs, 248.43 us = 0.16% latency, 0 FLOPS) | |
| (timestep_embedder): Sequential( | |
| 4.72 M = 0.15% Params, 75.5 MMACs = 0% MACs, 493.53 us = 0.31% latency, 306.02 GFLOPS | |
| (0): Linear(526.34 K = 0.02% Params, 8.39 MMACs = 0% MACs, 227.93 us = 0.14% latency, 73.61 GFLOPS, in_features=256, out_features=2048, bias=True) | |
| (1): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 50.78 us = 0.03% latency, 645.25 MFLOPS) | |
| (2): Linear(4.2 M = 0.14% Params, 67.11 MMACs = 0% MACs, 144.72 us = 0.09% latency, 927.43 GFLOPS, in_features=2048, out_features=2048, bias=True) | |
| ) | |
| (context_embedder): Sequential( | |
| 4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 451.56 us = 0.28% latency, 76.09 TFLOPS | |
| (0): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 171.18 us = 0.11% latency, 0 FLOPS) | |
| (1): Linear(4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 226.97 us = 0.14% latency, 151.38 TFLOPS, in_features=2048, out_features=2048, bias=True) | |
| ) | |
| (norm_out): AdaLayerNormOut( | |
| 8.39 M = 0.28% Params, 134.22 MMACs = 0% MACs, 570.77 us = 0.36% latency, 470.36 GFLOPS | |
| (silu): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 35.05 us = 0.02% latency, 934.96 MFLOPS) | |
| (linear): Linear(8.39 M = 0.28% Params, 134.22 MMACs = 0% MACs, 148.77 us = 0.09% latency, 1.8 TFLOPS, in_features=2048, out_features=4096, bias=True) | |
| (norm): GemmaRMSNorm(2.05 K = 0% Params, 0 MACs = 0% MACs, 238.66 us = 0.15% latency, 0 FLOPS) | |
| ) | |
| (proj_out): Linear(131.14 K = 0% Params, 536.87 MMACs = 0.01% MACs, 173.09 us = 0.11% latency, 6.2 TFLOPS, in_features=2048, out_features=64, bias=True) | |
| (repa_projector): Sequential( | |
| 9.97 M = 0.33% Params, 40.8 GMACs = 0.41% MACs, 698.57 us = 0.44% latency, 116.84 TFLOPS | |
| (0): Linear(4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 206.47 us = 0.13% latency, 166.41 TFLOPS, in_features=2048, out_features=2048, bias=True) | |
| (1): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 34.57 us = 0.02% latency, 242.65 GFLOPS) | |
| (2): Linear(4.2 M = 0.14% Params, 17.18 GMACs = 0.17% MACs, 171.9 us = 0.11% latency, 199.88 TFLOPS, in_features=2048, out_features=2048, bias=True) | |
| (3): SiLU(0 = 0% Params, 0 MACs = 0% MACs, 36.24 us = 0.02% latency, 231.48 GFLOPS) | |
| (4): Linear(1.57 M = 0.05% Params, 6.44 GMACs = 0.07% MACs, 148.06 us = 0.09% latency, 87.03 TFLOPS, in_features=2048, out_features=768, bias=True) | |
| ) | |
| ) | |
| ------------------------------------------------------------------------------ | |