Add RoPE positional encoding - llama3 feature branch

[gordicaleksa]

Implemented RoPE - rotary position embedding from the RoFormer paper.

Note:

1. I do not conditionally remove the allocation of our learnable position embedding buffer (wpe) as that would require touching many parts of the codebase that rely on the particular order inside the parameter buffer (e.g. wpe has index 1).
2. I do turn off fwd/bwd computation / grad norm computation / update for the wpe buffer.
   The explicit tradeoff is: suffer a minimal memory bloat (maxT * C) but the PR has minimal impact on the readability of the codebase.

Tests:
I ran an A/B experiment: trained a 124M GPT-2 on 10B tokens (FineWeb subset) with:
a) learnable positional embeddings (default, -er == 0)
b) RoPE (-er == 1)
c) no positional embedding at all
all other settings being the same same.

Results:

Conclusions:

• The validation loss is significantly better with RoPE
• RoPE implementation slightly decresed the performance (consistent with what EleutherAI folks observed). I observed a drop ~1_632_000 -> ~1_603_000 tok/s (~1.7% perf hit).

[ademeure]

LGTM, we could reduce the number of GPU instructions for the kernels a bit by moving use_rope to being known at compile time, but it's debatable whether that's worth the small compile time increase of having more kernel versions given the kernel should be DRAM limited, see: https://godbolt.org/z/es6GzeePq

(it turns out the bigger optimisation by far is using a 3D grid to get rid of modulus/division/etc. for btc calculation, which is orthogonal and doesn't really belong in this PR)