I can't see a pathway for these things to be able to learn from experience in any meaningful way - the energy budget seems to prohibit it. Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks. Despite this, large nervous systems are very energy intensive as well. For example in humans 20% of the energy budget goes to the brain which is 5% of the body weight.
So I refrer to LLMs as language extrusion confabulation machines. Language extrusion was a term I heard the linguist Emily Bender use. Confabulation because my observation is that talking to an LLM is very much similar to my experience of interacting with Korsakov syndrome patients some years ago.
I look forward to the hype settling down to see what we end up with.
> Artificial neural networks are already many orders of magnitude more energy intensive than natural neural networks.
What metric are you using to compare? By most counts, the energy budget of an instance of an LLM in a datacenter is lower than the energy a person uses. Of course, if you count energy per neuron connections vs weights then you'll likely get a quite different number. But then again LLMs do a lot of things with far fewer weights than the brain does neuron connections, even if you only count neurons in some parts of the brain. And of course you can point to capabilities that the brain has but LLMs lack, but on the whole it feels like it's pretty difficult to make a useful like-for-like comparison here.
I can't make sense of your comment. Firstly because of the obvious massive over-build of GPU infrastructure the AI companies are engaging in. Secondly because the instance of the LLM in the data centre that users interact with is only a small part of the story. The training phase is clearly prohibitively expensive, thus the fact that these things have no way to learn from experience except by smoke and mirrors.
Also your comment feels like the classic climate denial discourse - say something a bit complicated and a bit difficult to follow that looks at a very small out of context part of the story to cast doubt.
Maybe I shouldn't have been pissed of by your commient. In which case - the excess costs of the training, output, feedback, and training cycle of artificial neural networks seem to prohibit the nightly training consolidation cycle (i.e. sleep) that is an important part of natural neural networks.
There's two sides to efficiency: one of them is how much you put in and the other is how much you get out. I was asking the question because the answer of the relative efficiency of a human vs an LLM strongly depends on what you put on either side of the equation. It's a really easy pitfall to look at one side of one version of that equation and presume a really high inefficiency when that's not necessarily the case. It doesn't help that the data necessary to fully answer any version of this isn't publicly available. That's why I was asking for something more concrete in terms of how you were making the comparison.
If you're looking at training costs, there is definitely one aspect in which LLMs are obviously significantly less efficient: the amount of information needed for the initial training. This does translate into some pretty high costs but it only needs to be paid once for the amount of work that any given LLM does. In terms of fine-tuning LLMs can get significantly more data-efficient than the initial model, which also means energy-efficient, and it's not obvious to me that it would be drastically worse than a human (though again, only thinking in terms of doing the energy input for the kind of work that an LLM is good at).
(The increased efficiency in comparison to humans is part of the reason why you see Jevon's paradox mentioned a lot whenever concerns about the resources used by LLMs are mentioned: more efficiency can easily mean more resource use in total)
I'm only concerned with the input efficiency, but the output energy efficiency is quite bad too. I can not see a pathway to generating economies of scale for this stuff.
Meta's bots are spectatularly awful, and if your organisation needs to use facebook advertising, they're also unblockable. There are ways around it that doesn't exactly block them but does clearly communicate they should GTFO, but it just shows you what an awful company Meta is.
It was brought to my attention that the phrase "terminal, degenerative, sexually transmitted condition" returned no results from my search engine. So I tried to fix it.
as a rule the interviewee should be assessing for goodness of fit, as well as the interviewer. If my simplified nuanced answer to why I limit my usage of LLMs was poorly received, I wouldn't want to take a job somewhere likely to make me miserable.
Unless I'm looking after about 6 months and my savings isn't as healthy as it once was. Or one of my kids is sick. Or pretty much anything happens costing money.
Then I'll be miserable for a little while as I continue my job search.
Practicality, sometimes, is more important than ethics when other people rely on you.
"Unless I was really excited for flipping burgers, the menu options, the new McKidney, non-working ice-cream machines, and the legacy of Ronald McDonald, I wouldn't want to take a job somewhere likely to make me miserable".
A lot of jobs don't work based on those criteria. In a bad job market, IT jobs don't either. Especially if it's your 20th application with no response.
aka LLMs can not learn from experience - this is a fundamental limitation. c.f - individuals with Korsakov's syndrome - who also confabulate in a similar manner.
I've had some impressive customer service from Yamaha concerning decades old saxophones that they have zero prospect of generating revenue from in the future, for an unrelated (musical) data point.
Japanese companies don't give you a good service because of future revenue prospects.
They give a good service because they respect the thing they built and the person who cares for it. They want their products to live and bring joy to people who guards and cares for them.
This is a completely different and much deeper philosophy.
This is not limited to Japanese companies though. A Lamy representative told me that the factory in Germany restored an out of production fountain pen to mint condition, for free.
Want a bit for your early 80s Mercedes? Your local dealer probably has it, or knows where to get it.
Want a bit for your early 50s Mercedes? Your local dealer knows a guy in Stuttgart who will send it over.
Want a bit for your early 30s Mercedes? Your local dealer knows a guy in Stuttgart who knows a guy who will *just go and make you one*, albeit it will be priced accordingly.
In the early 80s, a friend restored a 1936 Mercedes-Benz 170V Saloon that he found in pieces in the attic of an old downtown Seattle body shop. It took five pickup truck loads to get the entire car to his shop and about three years to completely restore it to original condition. He had almost every part except for the steel wheel that was for the spare tire, the instrument dial backgrounds, and the rubber window surround moldings.
Despite combing the Earth for that fifth wheel for the spare, he was never able to find one but, since the spare tire was kept underneath a trimmed cover on the back of the car, it wasn't missed. The instrument dials he faked by carefully photographing, and printing on photographic paper, a set of dial faces.
Those rubber window moldings?
He was able to have a well-connected German friend contact Mercedes in Germany where they responded by breaking out the original molds and producing two complete sets for the car.
The year after the car was complete, it won a local Mercedes-Benz of America concours show by 4/100ths of one point over a Gullwing that had just gone through a $200,000 restoration.
I think both them and BMW have all the tools to make any part they used on production models. Not sure exactly how far back either catalog goes, but from what I've heard it is total for even 60s & 70s and maybe 50s models. Of course the way out of date stuff will cost but they could make something for you if your really want it.
Dealing with BMW is hilarious because you will end up speaking to some guy either looks about 14 or like God's older brother, and he will say either "Ja so this part, I will show you how you take the beer can and you cut a piece like this, oder so, and it will fit over this and it is ready, ja? Ausgezeichnett, ja?" or he will say "Ja so this part, it is a special order from Munich and it will be four thousand euros but we will have it tomorrow" and the customer being German will just pay it and pick the car up after work tomorrow, and that's it, end of the story, done.
Or the customer will be Austrian and there is nothing for it but assassinating archdukes and bloody warfare over a 1.50EUR gasket.
Both Yamaha and Roland keep a long back catalogue of reasonably priced parts for at least their mid-range pianos and up as well. I should check Korg too I guess.
Of course they do - it's the same as bookmakers or other gambling syndicates, it only makes sense to operate at a certain scale otherwise the rounding errors and stochasticity will kill you.
Depends on the HF. They're talking largely about multi-strat or macro firms here. If you're a HF that fills a niche, then size actually works against you as the pool of opportunities available reduces as you get bigger.
If you want to just stop yourself from shooting yourself in the foot in your current checkout and maintain privacy around your own stupidity, you can add patterns to `.git/info/exclude`
dumb, sick, stupid, other adjectives - generally not negative although could be in the right context - both in the Australian and North West England (where Matt was from) context. Just watch out for being called a fucking cunt. In almost all cases that one is pretty unambiguous. I do remember at one point having a conversation with Matt about how handling East Coast and West Coast North Americans was quite a different proposition from a cultural perspective, also related to this thread.
Matt would have liked this discussion. And given that his and my mutual friend has some actual legitimate serious expertise on obscenity in the English language even more so.
So I refrer to LLMs as language extrusion confabulation machines. Language extrusion was a term I heard the linguist Emily Bender use. Confabulation because my observation is that talking to an LLM is very much similar to my experience of interacting with Korsakov syndrome patients some years ago.
I look forward to the hype settling down to see what we end up with.
reply