Plenty of measurements exist, but the trick, as the article said, is finding good measurements.
Cyclomatic complexity has been pretty solidly discredited within the maintainability research community for decades.
Sonar's cognitive complexity metric is a bit better, but here's a study that found that it still only has about a 0.5 correlation with how much difficulty programmers actually had reading code as measured by multiple methods.
They found that the most accurate way to measure code complexity that didn't involve something like an eye tracker or EEG is still basically just vibes - asking programmers if they thought it was hard to understand.
Halstead Effort came in second, and scored pretty well, but here's another one where it doesn't do so well, either. And it scores the SonarQube metrics even worse, with only a 0.35 correlation: https://www.sciencedirect.com/science/article/abs/pii/S01641...
Yeah, we have this rule in our SQ and I absolutely hate it because of the number of times I'm forced to break something up. Complexity that matters is usually at a higher, more architectural level.
Yeah. Looking at the code samples in that first page was pretty interesting. They're cherry-picked, of course, but still it's a little galling to see which ones SQ gave a lower score.
I definitely wouldn't want AI to be autonomously using that as a guide without doing some fairly serious internal A/B testing first. Kind of like for cyclomatic complexity, it's just too easy to find ways to maliciously comply. And if that's what you ask AI to do then that's likely what you're going to get.
Copyrights are such constrained monopolies, though. More or less on the same level as how Nike has a monopoly on Air Jordans, but not on shoes in general.
One of the most interesting things about media consumerism is the extent to which people fail to realize that watching something else is always an option.
That's beyond hyperbolic. Yes there has been consolidation in the mainstream media industry, but I still have better access to more distribution channels than I did in the early 2000s. Especially when it comes to indie stuff.
Lately I've been helping my mom downsize. She's got boxes and boxes of household goods that she held on to because it belonged to her mother or grandmother or great aunt or whoever. Basically just, when the time came to clean out the old house, dump the kitchen cabinets into a box, put the box in the basement, and leave it there for 40 years.
And the thing that's struck me is how much of this stuff is cheaply made and barely holds together. Sure, there's the occasional well-built item that resembles the stuff I'm used to seeing in antique stores. But mostly I'm just getting a lesson in how much survivorship bias informs my perception of 20th century consumer products.
If anything the real trend is the shift to plastic, which can be more durable but is also more prone to complete instead of partial failure and generally isn't repairable. e.g. a tin measuring cup might dent easily, but can also be bent back into shape. A plastic one probably won't suffer anything more than a scratch when you drop it, but when it finally does crack that's the end of it.
Speaking of plastic, so many people nostalgically remember Tupperware as a quality product that's much nicer than what you can get nowadays. But the truth is it was a multi-level marketing scam and their products were often contaminated with heavy metals. It used to be a lot easier to prop up a brand image in the pre-Internet days, when consumers largely couldn't talk to each other en masse like we do nowadays.
Say what you want about tupperware, but I just bought a set of ancient tupperware from a thrift store and even the second hand quality of it beats anything I have bought in the last 20 years hands down.
This is part of why last year I mostly switched to just borrowing DVDs from my public library. Avoiding ads by not paying seems to do the trick rather nicely. The quality and variety is better than what I can find on any one streaming service, too.
Every year it gets more and more important to learn how to not be sucked in by a small amount of convenience. "Capitalism" has figured out how to abuse your desire for maximum convenience. You need to figure out what your price is for it, and stick to it like glue.
The funny thing is the sooner you get off of some particular convenience treadmill the easier it is to stay off.
The funny thing is, it may be less convenient but it's also more enjoyable.
Endlessly scrolling some algorithmic feed hunting for something to watch can be done without getting off the couch, but it isn't actually fun. Walking to the library with my family to return a stack of books and movies and check out some new stuff is one of the highlights of our weekend.
For anyone who's wondering where to start, I'd like to suggest that your library probably has a copy of WALL-E.
I doubt they really care about coal miners’ actual votes. The US coal mining industry employs fewer than 50,000 people. I live in a major coal producing state and while it was an important industry in my grandparents’ time, these days it isn’t a particularly noteworthy share of our economy.
I would say if it’s specifically about coal at all, it’s more because of the GOP’s tendency toward weird nostalgic obsessions.
I should have included voters in all the communities in coal mining areas, not just the miners - all the families and everyone of voting age in coal mining areas who now might think that coal mining is coming back (and it just might under trump). Revoking limits on pollution by power plants benefits coal more than any other form of energy production.
I’ll grant that for specialized applications like coding agents and mathematics, but even there I suspect that most the real gains are actually taking place in the harness.
But I suspect returns may have already diminished into negative territory for at least some other use cases. One of my least favorite job responsibilities in this brave new era is figuring out how to avoid performance and behavior regressions when an older model were using for some application reaches end of life. It’s getting uncommon for me to look at our benchmark results and say, “Oh, good, it does better on one of the newer models!”
That is true, but the eventual realization that more machines doing more coin flips in parallel does not mean "more work gets done" might.
LLMs are amazing tech, but they're terrible without oversight. More agents faster just makes reality collapse on them quicker.
But yeah, you're right, temporarily, this will still push demand. But the topic was about "diminishing returns" as in "tech getting better". Not as in "customer spending".
It's kind of weird because more machines working together does mean more work gets done. Coin flips and weighted coin flips are totally different things. Any biases weights towards reality push you closer to reality when you use them.
New models keep being able to use more and more agents on longer time frames. Your hypothesis doesn't look like what we're measuring.
But high demand for LLM time isn't sufficient to keep customers at the frontier LLM SaaS providers. That demand can be satisfied locally or at non-frontier outlets, absent hardware shortages at least. The Tier 1 providers (and the would-be Tier 1s) presumably need to open up a much bigger lead in model quality, one that doesn't simply get distilled away this time, and/or continue to be protected by ongoing (or worsening!) hardware shortages. (And that's overlooking the revenue shortfalls which OpenAI and Anthropic seem to be facing already.)
I had actually been thinking more about all the non-LLM functionality that go into the harnesses. I'm not going to name names and I haven't done any rigorous testing, but my general impression is that choice of harness matters more than choice of model. In terms of basic task completion success specifically, not code aesthetics.
A perfect harness will not extract gold from a dumb model. It's a system that builds on each other, though we've not probed that frontier much to have a good intuition on what effects what.
One thing that I really want to know - the better models from today vs a year ago - what has changed. They have already pre-trained on all available public data. Scooping up the last percentage of archaic texts which were never digitized is not going to move the needle.
Is it just that the providers are generating tons of synthetic datasets on coding tasks so that the models get more exposure to the right thing to do? Every time someone points out an LLM stupidity they add some training data to patch over the weakness (trivial to generate "there are two 'l's in llama")?
I’m not so sure about that. Already AI vendors are back to cutting prices to try and keep customers from cutting back on their usage. My own employer is working hard at pivoting to much smaller fine-tuned models for established use cases, and seeing model performance improvement in addition to large inference cost reductions. Being able to run them locally hasn’t exactly been a disaster for devex, either.
It may turn out that demand for SOTA frontier models isn’t so limitless after all.
Every product follows demand curves. At a price of 0 you could find infinite usage. This has nearly zero relation to how much it costs to provide the product.
Infinite demand isn't a thing. Even if they'd offer free compute forever (not likely possible) I and many others would still use local models that we have full control over and that do not harvest our personal data.
Except of course it relates. All else being equal, we will prefer $X COGS over $2X COGS because that helps us with both profit margins and price competition.
Framing it in terms of the price people might be willing to pay for a single product in isolation frames the point I was making, which was about price competition, right out of the picture.
Maybe I'd be willing to pay $10 for product A if I had other options. But if there's a product B for $3 that's not quite as nice but still ticks all my boxes, then product instantly becomes a lot less attractive.
More channels or soldered memory? Channels are basically RAID 0 so it depends what you're measuring. Soldering memory down was the only way to use LPDDR5X so if you wanted the best memory you had to solder it down. LPCAMM2 exists though so newer devices can use that instead of soldering them down, but not all devices would be able to fit the required LPCAMM2 slots.
That's tiny! But it still depends a lot on form factor. LPDDR5X is used in phones and making memory removable would have its compromises. You may even have compromises in laptops. Look at how tiny a MacBook Air's mainboard is and you'll see the RAM modules on the same package as the SoC. SOCAMM2 is too large for that but a variant with only two modules could possibly work.
I don’t really know much about how this stuff works, but I have a feeling we wouldn’t even have to do anything punitive. We might just need to find and close whatever weird loophole allows the folks who are participating in this gold rush to feel confident enough that they’ve externalized their risks to be willing to engage in speculative data center buildout projects on such a grand scale in the first place.
And this is why I mildly dislike the term “software engineer”. If a mechanical engineer took your colleague’s approach toward their work, they would be legally liable for engineering malpractice.
Cyclomatic complexity has been pretty solidly discredited within the maintainability research community for decades.
Sonar's cognitive complexity metric is a bit better, but here's a study that found that it still only has about a 0.5 correlation with how much difficulty programmers actually had reading code as measured by multiple methods.
They found that the most accurate way to measure code complexity that didn't involve something like an eye tracker or EEG is still basically just vibes - asking programmers if they thought it was hard to understand.
https://www.frontiersin.org/journals/neuroscience/articles/1...
Halstead Effort came in second, and scored pretty well, but here's another one where it doesn't do so well, either. And it scores the SonarQube metrics even worse, with only a 0.35 correlation: https://www.sciencedirect.com/science/article/abs/pii/S01641...
reply