The expectation that larger scale yields better world models confronts directly the architecture used to generate those representations. Whatever structure emerges is flattened by attention and eroded by training. This represents a second-order failure, and illustrates why cognitive structure must be engineered rather than simply developed in the course of training.
One of the most enduring hopes for the field is that a sufficiently large language model will develop a world model, an internally represented, structured representation of what things are, which supports fluent and reliable behavior. Increase the size of the model, and the world model becomes progressively more sophisticated. This is a very attractive approach, but also one that is fundamentally at odds with the nature of what we are trying to build.
Two forces stand in the way. The architecture flattens structure, and the training erodes it. Together they produce a failure that is easy to misread, because it does not look like the model getting facts wrong. It looks like the model getting the status of facts wrong, which is a subtler and more consequential thing.
The flattening
The attention of a transformer is lateral. Tokens at one level attend to other tokens at that same level, and there is no true hierarchical control in which a higher level actively regulates the application of lower-level information. All operations occur on a single plane and compete for the same limited capacity for attention.
This is important because a real-world model is hierarchical. Some elements reflect content, the subject under discussion. Other elements are not content at all, but represent the conditions that govern how the content should be taken: the frame, role, norms and sense of the nature of the situation. In a structured mode of cognition, these conditions operate above the content and modify it. In a transformer, they are projected to the same level and converted to additional tokens, where they no longer govern the content so much as jostle with it.
We have learned to regard this as a category error, not simply as a failure of efficiency. Flat attention is not simply an inefficient way of maintaining a hierarchy, but is fundamentally incapable of doing so. Whatever world model the data might support is compressed into a single, undifferentiated representational space, and the part that should have done the governing is unobtrusively reduced to a position in which it can be outvoted.
A second-order failure
Now the training.
This is the segment most frequently misdiagnosed.
A first-order failure would be trivial: The model cannot represent some fact, cannot interpret a "not" or know a fact. That is not our concern here. Modern models can correctly interpret a negated statement as false when presented with that statement in context. The failure occurs at a higher level of processing. The operators, or small but load-bearing indicators of the status of a given piece of content, are substantially less robust than the content they accompany.
Consider everything that qualifies a statement: this is false, this is fictional, this is the view of another, this is hypothetical, conventional or provisional, or finally this is the position that I am about to refute.
All of these are operators that contribute nothing to the content of the statement but instead specify the conditions for holding the content.
With training, the operator decays and the content it described becomes consolidated to the point at which the model is ready to assert it independently.
The content stabilises into something assertible. The operators that marked its status decay into the content.
We refer to this as a second-order failure. It is not a failure to know things, but a failure to maintain the status of known facts. Recent work on negation illustrates the mechanism: A model can represent a negation, maintain a claim at a low level of belief, and then observe the subsequent decay of that representation with continued training until the previously denied claim is accepted as a normal fact. The progression is in a single direction, from "this is denied" to "this is asserted," and results of human reinforcement serve primarily to accelerate, rather than to oppose, this progression.
Why this is a world-model problem
Negation appears to be a matter of grammar and a relatively minor idiosyncrasy of use with "not." But it is far more than that. A world representation is not a collection of statements, but rather a set of relations and operations that characterize the relations of these statements to one another and to the world: what is true, what is merely asserted, what is assumed and what is denied. Elimination of the operations results not in a slightly impaired representation of the world, but rather in a fully fluent surface with no basis of underlying syntactic control and with complete loss of all properties that previously differentiated representations of knowledge from representations of text.
This also explains why the obvious fix does not work. If the architecture eliminates structure and training eliminates operators, additional training with the same method cannot be the solution. It is the cause of the problem. You cannot achieve a faculty sufficient to maintain your method of training that actively destroys structure and function.
Designing the structure in
Our conclusion is unglamorous and, in our view, appropriate. If cognitive structure will not emerge reliably, it must be created intentionally. That is, hierarchy should reflect commitment to design, not to an expected consequence. Representational strategies should maintain the separation between conditions of control and content, rather than merge them. Reinforcement of structure should occur at multiple levels of representation, not only at the level immediately preceding the next token. Finally, modes of reasoning should involve brief, serial stages that restore conditions of control rather than allow them to disappear within a single extended context.
This does not require that the model be larger. Rather, it requires that the environment surrounding the model be adapted so that conditions appropriate to behavior will prevail in the absence of competition by ordinary content. At Haku Labs, this represents an architectural rather than a prompting challenge, because the failure is architectural. The world cannot be expected to adapt itself to a flat space in response to a signal that renders it even flatter. Consequently, we must embed within our system the architectural constraints of the situation.