A developer just managed to squeeze the massive Llama 3.1 70‑billion‑parameter model onto a single RTX 3090 graphics card. By routing data directly from an NVMe drive to the GPU and skipping the CPU, the setup runs faster than anyone expected.
https://github.com/xaskasdf/ntransformer