Many people are commenting that AI is just a scapegoat here and really this is a story about military incompetence. What I think is probably more interesting is how AI enables incompetent people to do more damage than they would otherwise. Two things can be true at once, the military can be incompetent, and AI can be used to make targeting decisions that are wrong - we can understand both the new world that we live in, and the absolution of responsibility it offers without ignoring these nuances.
The concern in this story is whether the targeting was actually a mistake, or if we intentionally killed a school full of children. If we take AI out of the equation, and it's just a soldier who made the call claiming it was an accident, we might not just believe that on its face. But throw AI in the mix and suddenly the media loses all ability to question anything they read in a press release.
AI enables more mistakes to be made faster and more convincingly. You have more stuff being generated which means more of it (by absolute quantity) is probably wrong. You have less time to review it because there is more material. Of course everything the wrong stuff included can have analysis making an argument for it attached (courtesy of an LLM). And finally people writ large tend to treat computer output as infalliable.
This increases not just the risk of war crimes and general moral bankruptcy but also of escalation (see the recent Chinese ship close call or the older, and not AI related, bombing of the Chinese embassy in Belgrade due to faulty intelligence) Also dropping multi million dollar missiles on non legitimate targets is a giant waste of money and more human analysis is likely cheaper.
The danger of AI has always been the recklessness and maliciousness of the humans who wield it, rather than some Skynet or paperclip maximization fantasies. So yes, just like with guns, with mass media, with any other powerful tool.
Is that true though? It seems like the iphones are doing some type of skin smoothing algorithm but the Huawei is the only one that actually seems to produce actual skin texture and blemishes. My initial read was that the Huawei was the least processed of the bunch.
They Huawei currently are the best when it comes to still pictures, it is the one area/feature they can win, nothing wrong with that. Where Apple wins is in the total hardware and software package, which is why they are better at the video end of the picture in comparison to their competition.
Samsung in the past has also held that distinction, being able to take very good still pictures, but they too fell down when it came to hardware and software, particularly when you start adding in video performance, which requires better software/hardware chops. Nothing is gonna beat Apple chip hardware right now.
It is really interesting to see this claim, because i thought the current theory was that typesafe actually repackaged the work from GLiNER[1] - which does seem to be a closer match, and their original paper[2] predates yours by several years. Curious if you had heard of it before? It is also open source[3] and I think also has some good usage.
"Ternary Bonsai 2 27B reaches up to 143 tokens/second on NVIDIA GeForce RTX 5090 and 46.8 tokens/second on M5 Max. On an RTX 4090, Ternary Bonsai 2 27B consumes just 0.714 mWh/token, making it 40% more energy-efficient than an 8B model running in full-precision."
150 tokens per second on a ternary model implies that it’s GPU bound, I’d bet a Q6 model is even faster because it’s existed longer and seen more optimization. You’d have to be insane to not run an NVFP4 quant over a ternary quant on Blackwell if they both fit.
Ninfer is compatible with a nvfp4 model for the 27b. Also nowadays I prefer to use the byteshape one, I get less loops, and I am not sure if I really see a difference in speed or quality. Pure vibe agentic coding on a C++ codebase or ocaml one, ocaml one has codex as reviewer as I am more interested by that project, the other is more for fun.
Love this for the folks with 16gb graphics cards - 3.8 27b has been incredible but not quite runnable on anything less than 32gb - will try loading this up on my 16gb intel b50 and see how it goes - not sure these quants can be accelerated by the XPU cores yet but maybe in time!
You can run the ~4 bit quant(s) on 24gb, if you're not _too_ picky on context size.
This will hopefully be better, though it'd be a _very_ surprising increase in performace at the size they say. Would love to see more about how it benchmarks.
I run Unsloth's UD-Q4_K_S on 20 GB of VRAM (RX 7900 XT) and I get ~90k tokens of context without quantizing KV cache. With 8-bit quantization, I get about a 134k token context window. That's with only one slot, but for me, it works pretty darn well, with 20-35 tok/s depending on how full that window is.
7900 XT is a sleeper card. When I initially bought it, it was priced at the lowest wattage per $ per GB VRAM (not normalized for token speeds...) Although I ended up swapping for the XTX because that 4GB means everything in just increasing the context window. At 8bit KV my window is over 200k, and although qwen3.8 loves vomiting out tokens as part of its reasoning chain I trust it enough to get assigned tasks done eventually, which I could not say of any model before its release.
How has software/driver support been? I got burned hard by AMD last generation or the one before. Things smoother now, or do you have to baby it like hell and pick and choose software that works?
I don't do anything fancier than inference, and I only use llama.cpp, which supports rOCM. I've had few issues; most GGUFs I download work right out of the box. Nearly any popular model has a quant that just works. But as you can see I don't use my GPU for anything weird or nonstandard.
I'm doing the same with a context of about 128-150k Surprisingly, I get subjectively better results with Unsloth's 3 bit quants (UD-Q3-XL something), than their 4 bit quants (S or M)
You can trivially run 131k on 24GB 4bit, and there are repos with tweaks that allow you to get the full 262k but idk if there's degradation with their approach.
I run qwen 27b on an old-ass 16gb gpu. It’s very possible using unsloth 2bit and 3bit quants, tho there are a bunch of interesting quants that let you run closer to 4bit on 16gb. This article that’s currently also on the front page mentions a bunch of them while discussing their own quant https://byteshape.com/blogs/Qwen3.8-27B/
Is nobody using structured outputs? They use constrained decoding at the generation stage to ensure the probability of tokens that would break the format are set to 0. I kinda figured everyone was doing this at this point.
Most serious providers are now supporting structured outputs in a reasonable way for all model configs. But for example on ollama structured outputs are still incompatible with tool calling and with reasoning
If you use a large enough volume, you will know this isn't fully reliable. You might get json, and it might not match what the model actually sent because the last layer cut it up to match what you want. At the end, not json, or json but not really matching what the model wanted, it's sort of the same issue: when you use them you HAVE to assume they can have a brain fart. That's fine, just code around it.
We use structured output and To my knowledge it has never failed (millions of data points). There seem to be two classes of people: those doing productive work with LLMs, and those who only get replies insulting their mothers…
I'm curious if you've tried dwarfstar and decided to move to llama.cpp and 3 bit quants or what made you go that route instead? I've been using ds4 for months now and it's already got support for the new vision model, haven't tried it yet, still on 0731 but it's been very solid for me.
I tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar.
Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actually get much better metal performance on my llama.cpp branch than on dwarfstar (plus all the extra llama.cpp features). Here it is in case you want to give it a shot: https://github.com/tarruda/llama.cpp/tree/qwen4exp-dsv4-opti...
I follow llama.cpp pretty closely as I use either llama.cpp itself or projects that depend on it all the time, and one thing that I don't think gets talked about is the sheer scale of community involvement. It seems like a logistical nightmare, but somehow thousands of different contributors are opening hundreds of PR's every week and getting them merged in to support various hardware or implement a new pattern or algorithm from a recent research paper. It's really quite awe inspiring for me to see, and think it is in no small part because of the leadership of ggerganov - so I'm happy to see that he is sticking around and plans to keep building this incredibly useful tool that has grown into a huge community at this point.
Here's why I think it's working so well, as someone who's participating in llama.cpp development and who has participated in many other open source projects:
1) Reviews are super fast. Sometimes I get the first review 5 minutes after submitting a PR. Today I had a PR merged with two reviews within 42 minutes. This is incredible work by the whole team. It keeps contributors motivated. The pace is intoxicating. On other projects, I've sometimes been waiting months for a review.
2) Georgi motivates people by giving them responsibility. There's no gatekeeping as with other projects. He happily delegates. You do good work? It's appreciated, and you get the freedom you need to make an impact. That feels awesome.
I've rarely become addicted to an open-source project so quickly.
That’s sounds so cool! I wonder how they get to do high throughput contributions. Is the structure very flat, or plugin based? I’ve always admired this project from the outside and glad to learn it’s doing so well. Looks like they have something really good going on. Kudos to the team!
I have been using a PR branch to run GLM 5.3 flash locally so I have been watching the dueling PRs develop to implement it and all the comments and reviews. You're right the scale of it is huge. The speed with which maintainers and community are responding is also super impressive.
I know there's a lot of discussion about AI overwhelming OS maintainers and I can see that. It does seem like llama.cpp is successfully riding that dragon right now though.
Until 1-2 months ago, AI-written code was not allowed in llama.cpp. That changed, but it's still not allowed to have AI write your posts for you (issues, PR descriptions, etc.). In my experience, banning AI-written prose leads to much better contributions than on projects where it's allowed.
Of course, llama.cpp also gets lots of AI slop PRs from first-time contributors, but from what I've seen, maintainers simply ignore the bad submissions and prioritize the good contributions. I think that's probably one of the best ways to handle the slop overwhelm.
reply