What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context
About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me.
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017 (max). I hope AA adds other reasoning effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects and what I'm getting charged for them isn't what I'm thinking about.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision
then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
I think it's because of the fact that Deepseek at 16B can be comparable enough in tasks to Mimo with 42B
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
At around 75% of the full model intelligence
They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
Almost no one is running DeepSeek V4.1 Flash on their laptop, so comparing based on API pricing seems perfectly reasonable to me.
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects and what I'm getting charged for them isn't what I'm thinking about.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
Does deepseek v4.1 flash 1017 really take that much longer per task?
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
It feels like that would be the expected result, all else equal.
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
[0]: https://artificialanalysis.ai/models/comparisons/nemotron-3-...
[1]: https://artificialanalysis.ai/models/k2-horizon-375b-a23b & https://ifm.ai/blog/k2/