I sorta see the argument, llama-32b-r1 scores 88% on MMLU vs gpt-4's 86.4, you could run it on a 3090 with offloading or fp4

I imagine the weights need to fit in VRAM; does that sound right? Or can the solution involve moving weights on/off the GPU

@NoaNabeshima weights have to fit in vram

@sylv ram and not necessarily vram?

*vram sorry

new market for 4090: https://manifold.markets/sylv/an-llm-as-capable-as-gpt4-runs-on-o-8da33ccbe4b1

@sylv The difference between the 3090 market and the 4090 market is because.. the 4090 supports fp8??

@NoaNabeshima I mean it has better specs in general but fp8 is pretty cool

@NoaNabeshima Oh it's different dates