(April 2024)
I think a good model for the program learned by an LLM is this:
Imagine starting out with a correct model of human reasoning, implemented as a computer program. Now imagine taking that code and "spaghettifying" it: create a lot of duplication, turn table lookups into huge "case" statements, unroll loops and truncate them after a couple iterations, do (a lot of) random edits, introduce nondeterminism, and so on. Soon you have a much larger piece of code that can successfully reason some of the time, but also makes a lot of mistakes.
I would argue this is essentially what an LLM is—it's just that the LLM got there by a different route.
Unfortunately spaghettification is very difficult to invert, so it's not clear where this leaves us if we want to improve on LLM reasoning performance. But I think the "spaghetti model" at least provides a better basis for believing that something fundamentally new is required than just saying "we tried some stuff and it didn't work".
