Signs of individual dimensions in transformers carry semantic information and enable feature detection without training or rotation, opening a new path to mechanistic interpretability.
Different layers perform different roles and could therefore enable non-uniform distribution of parameters and computational resources as an alternative to constant architectural width.
