layer 1280 x 1280: full update has 1,638,400 numbers
rank r   LoRA numbers   share of the full update   reduction
1        2,560          0.0016                     640x
4        10,240         0.0063                     160x
16       40,960         0.0250                     40x
64       163,840        0.1000                     10x

relative error |BA - DW*| / |DW*| of a trained LoRA, against the best rank-r matrix
change DW*                  r=1              r=2              r=4              r=8              r=16           
rank 4 (4, 3, 2, 1)         0.6831 / 0.6831  0.4082 / 0.4082  0.0000 / 0.0000  0.0000 / 0.0000  0.0000 / 0.0000
decaying (2 x 0.75^i)       0.7500 / 0.7500  0.5625 / 0.5625  0.3164 / 0.3164  0.1001 / 0.1001  0.0100 / 0.0100
(each cell: trained / best possible)

LoRA at step 0: largest difference from the base layer 0.0
step   |grad A|   |grad B|
0      0.0000    0.1637
1      0.0023    0.1631
2      0.0046    0.1626
ControlNet at step 0: largest difference from the locked block 0.0, with a condition of standard deviation 10
step   |grad Z2|   |grad copy|   |grad Z1|
0      0.1267      0.0000       0.0000
1      0.1252      0.0003       0.0022
2      0.1237      0.0006       0.0043

merged against side path, float32: largest difference 4.3e-06 on outputs of size 8
extra multiply-adds per token for the side path, rank 16: 40,960 on top of 1,638,400, which is 2.5%
