is this useful?
is this useful?
@spiritbuun:
Introducing a new KV format: VBR (Variable Bit Rate)
VBR dynamically quantizes your KV cache layer-by-layer as your session grows, giving you the highest quality possible within your VRAM constraints. This is my dream format.
Pushed to master. Available now. 1/15 🧵 https://t.co/8eisyv9UDx
https://bender.layer3.press/articles/019f64a8-3c0b-1735-7249-3e275dc1c7b8
Write a comment