I feel like the current “reasoning” that LLMs are doing has got to be a dead end eventually. Every time I have to read another answer with “but wait” and “Actually,” as they “reason” their way to a (sometimes) better answer, I feel like there’s got to be a way to just shortcut to the actual correct answer instead of burning all these token going in circles mimicking actual thought
One line of evolution seems to be toward some form of latent-space reasoning, as in [1]. Natural language seems like a relatively low-bandwidth channel for intermediate reasoning.
While I was reading the top level message I was thinking on something wrong those lines. That being said I think it's somewhat useful to be able to see the reasoning in a meaningful format. There's already enough hidden layers of unfathomable meaning.
What if instead of a latent they used at a shorter form of note-taking-like reasoning, using more symbol to achieve a denser CoT. We'd get the best of both worlds. WDYT ?
"Latent" means it (the space) is there all the time but usually not exposed or apparent. Latent space is a high-dimensional space that is not legible to humans, so normally reasoning gets constrained to progress in tokens instead so that it can be evaluated.
Models already reason in latent space, but they have to keep encoding-decoding their "thoughts" from/to human interpretable tokens, and notably those tokens are sampled from a distribution. The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.
It's as if the only way you could think was by writing down a word, erasing all the thoughts from your head, then reading the word you just wrote down and deciding on the next word, etc.
Reasoning purely in latent space means that the model would still produce an output equivalent to tokens but unconstrained e.g. the output could be raw and opaque vectors. A significant downside is that you lose the ability to inspect the reasoning trace. It would also make the reasoning trace potentially larger which has operational issues.
> The model cannot output a vector and have that same vector fed back in at the next step, it only sees what token the sampler collapsed its vector into.
Not completely true: KV is a projection of the activation at each layer's input, so attention heads see (a representation of) all previous tokens' activations at that layer. The hard decision at the LM head doesn't change that.
Chain-of-thought output shouldn't be taken literally. The tokens are a substrate for computation, not necessarily evidence that the model is wasting time and electricity by gratuitously second-guessing itself over and over.
You can see evidence of this phenomenon in models dating back to the OG Deepseek R1. It was common to see the model talk itself out of the correct solution in the <thinking> block, or fail to reach it at all, only to produce a correct answer in the response. And vice versa; it was also common to see it reason its way to the right answer and then fail to follow through in the response.
I can see the reasoning being a substrate for computation, but in which space should we interpret this computation to be happening? The vector representations of individual tokens are completely different (and even the way the reasoning traces are broken up into tokens will be pretty different) between Qwen and Claude e.g.. The only way I can see this being effective (which it is) is thus that we SHOULD interpret the model to be "computing" in natural language and thus we can indeed take the chain-of-though somewhat literally.
The Deepseek R1 behaviour you describe is from a model from january last year, are you sure this is not pathological behaviour rather than an indication of the reasoning not needing to be taken literally?
I do however agree with the point that it is not necessarily a dead-end. That Qwen loops almost at an OCD-like level, but retains accuracy on the times it does answer, shows that. Yes ideally it loops less, but I am for now happy to accept that this is what it takes to run models locally. At least it is available for our inspection.
Not the person you replied to, but I think a more accurate description of the reasoning we see is proof of effort, not necessarily great insight into how the reasoning is occurring. For the most part, researchers currently describe the intent and motives (in however one may define them for LLMs) as black boxes right now. Even the mechanics of the cognitive process is not well understood. Depending on the model and harness, the thinking will often look like gibberish. I suspect they've invested considerable effort into presenting thinking as a reasonable approximation of what they imagine it to be. Claude and OpenAI have also begun encouraging multi-step problem solving (or the models themselves decide this), and we can see their more accurate responses at the conclusion of each phase.
Fine tuning or post-training is effectively biasing certain outcomes: making them more likely to occur. This comes with trade-offs. A coding LLM will bias technical language, which would harm a model for general use.
This opens a really interesting field of research. Our brains use specialised regions because specialisation turned out to be the most energy efficient method for biological compute. It might also be the best performant. We don't want to activate 100% of our prefrontal cortex to breath. What a stupendous waste of the organ. I think we see incredible advancements in model clusters in the future, using specialised models for specialised tasks. We have the appearance of this today in some harnesses, but they are shallow imitations. The real innovation will be low-cost, accurate routing. Existing solutions are woefully inadequate for many reasons.
It needs to argue with itself to extract most of the knowledge embedded in the weights into the context. Asking it to synthesize ideas directly in a single go is simply unreasonable. And MoE models need to walk multiple experts to extract all the knowledge on top of that. So you need to give them the reasoning trace to first spill all the associations into.
Yeah. It's "thinking" in absurd massive vectors. It needs to assess a couple to weigh out. That's the compression. That's the nature. It looks ridiculous when thinking traces render out such simple statements ('reassessing ..') but I expect this is far deeper an assessment than it can fully reflect to us on, and I expect its a huge part of their thinking.
Now though I'm considering all the hidden "thinking" in the models layers that happens for each token output. It is a wild amount of waste! We just can't see it.
This kind of stupid excessive computation is fundamentally how these models are so good.
One day hopefully not so soon someone smart or a foundation model will come up with a more efficient architecture. That's when things get really scary.
I don't think that's true. If a context contains a statement followed by something opposing that statement, that will confuse the model. So "actually wait, I really need to" is there to signal that the previous thinking may be flawed and that what follows is a new attempt.
It's good to remember that LLMs have no more state then what they can derive from the context up til any point. So if that context is hard to interpret, that will reduce effectiveness.
As long as the model itself understands that when it says "foo bar" in its own reasoning trace it means "pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar", where's the loss in clarity?
How could it understand, if there's no distinction between that and "from foo logically follows bar"? If the notes it's taking for itself don't make sense, it might just as well not take them.
Obviously it's going to have to come up with unambiguous shorthand for itself during gradient descent. My point is that that shorthand doesn't necessarily have to be unambiguous to human observers.
I feel you have both missed the point of caveman reasoning.
LLMs are trained on human natural language, not a specialised internal-only monologue to make syntactic shortcuts. Their response should make grammatical sense to a human reader because they are mimicking human speech.
"foo bar" is ambiguous.
"pursue theory foo; no, this didn't lead anywhere, let's backtrack and pursue theory bar" is accurate, should be meaningful to LLM attention, but is too verbose.
The minimal caveman way to say this is "not foo. instead do bar".
This really should provide the LLM attention with everything it needs to grasp the intention, but with far fewer tokens used.
I was pretty happy with Depseek Pro in Opencode util I discovered I can see the thinking trace by clicking on the "thinking..." communicate. All those seemingly unnecessary "but wait" messages are frustrating to read. But at least to some extent it's just model taking time thinking through the problem, and the trace produced doesn't have to be representative of what happens internally: https://arxiv.org/abs/2404.15758
It is MoE. It needs to engage multiple experts when the problem is complex or unclear. So you naturally see more of those simply as a primitive it learns to use to page in more diverse set of weights. Remember that each token is just 6 experts out of 256. So it literally needs to tell its router that it needs a different set the next time.
And this memory control primitive leaks into the reasoning chain, because it has no other channel for it available and we do not know how to train any other channel.
On the flip side, it tends to converge quickly, roughly proportional to the actual difficulty / clarity of the task.
Has this hypothesis that reasoning tokens help the model engage a wider range of experts been tested? I'm a little skeptical that it's the main driver of extended reasoning traces, particularly because MoE models are generally already trained so that expert activations are as uniformly distributed as possible. But it would be interesting to know to what extent this happens.
I think you should avoid reading the thinking blocks unless you spot errors in the output.
I am very confident the reason we get all these second guessing and "but wait" and "actually" is they train them on collapsed corrected sessions. i.e they take sessions that look like this:
user: Do x.
agent: the user wants me to do x. I think I need to do a and b first.
agent: does a.
agent: does b.
user: No no no doing a was wrong you should do c before b.
agent: undoes a. does c.
agent: does x
And they turn it to a session where the user correction shows up in the thinking. i.e
user: do x.
agent: the user wants me to do x. I think I need to do a and b first.
agent: but wait maybe I should do c instead of a
agent: does c
agent: does b
agent: does x
