Both MoEs and dense models are always getting better, so I don't think this comparison is meaningful across generations. But still for a first approximation, this tends to hold (you wouldn't expect a lot from a 10B model in coding yet).

Don't use ollama. The entire project is just a series of stupid decisions like this.

I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.

With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.

I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.

LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.

Maybe we'll have model weights being shared like game cartridges.

This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.

Many on HN are in deep denial about this imo.

Smaller models are suitable for simple tasks like classification / summarisation while larger models are better in agentic capabilities.

But yeah I'm skeptical all this overhead is worth it.

Looks like your great question doesn’t have an answer, but looking at the routing strategies things get even more confused, since the proposed ones tend to rely on extra llm calls to determine which model to pick.

The nice thing is that it makes sense for specific setups, less conversation oriented.

As an example, you need to classify batches of data, and have many fine tuned models. Or you need to do speed to text and need to pick which whisper to use.

You can write your own strategy, in that case an harness with subagents would be able to leverage this, picking the right model and then keeping its session sticky, but overall the lack of concern for caching points towards use cases where you do not gain much from it.

Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.

I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.

Communicating an idea concisely is difficult. Most people struggle to get ideas across at all, asking them to do it well with fewer words is often out of reach.

We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.

If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.

Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).

Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).

but

Qwen 3.8 27B is dropping this week...