The early twentieth-century Japanese philosopher Kitaro Nishida, the leading representative of what became the Kyoto School, devoted much of his life to a seemingly simple question: where is the meaning of a thing located? His answer was that meaning is not inherent in the thing itself, but in the place that holds it. He called this place basho, and built from it a logic in which the field comes first. The situation, the context, the predicate is not something a subject acquires after the fact. It is what lets the subject be determinate at all.
It is an old idea, from a tradition very remote from machine learning, and it ultimately provides an uncomfortably precise description of the fundamental error in the way we construct language models.
Context as content
Ask how a transformer handles context, and the answer is: as content. All factors influencing model behavior, the system instruction, the role, and the standing constraints are incorporated into the same context window as the normal content of the conversation, and compete for attention as an additional segment of tokens. The field that is supposed to hold the conversation is placed inside the conversation, as an item within it.
The conditioning field is being stored as additional conditioned content. Because attention is flat, this misfiling results in important consequences. As the exchange grows, the governing conditions are crowded by the very content they were meant to govern. What should have been the place that holds the meaning becomes just another object in the pile.
The logic of place, applied
Nishida's correction is to reverse the order. The place determines what stands in it. The field constitutes the subject, rather than the subject reaching out to attend to the field. Applied to a language model this is not mysticism; it is an architectural instruction. Context should not be a target to which the model attends. It should be the field that conditions what the model can generate, operating from a level above the content rather than from within the same flat space.
It is useful to consider levels. There is the content of discussion, the mode or role and the space in which that content is sustained. And there is a further level of controlling conditions that determine all subsequent events without themselves becoming an additional component of discussion.
An organized mode of cognition preserves the separability of these levels and relates higher conditions to lower levels. In contrast, a transformer integrates all three levels onto a single plane and results in competition among them. This reflects not a problem of adaptation to be solved by appropriate scaling, but instead a type of structural incoherence in which the element that should control the process is placed at the same level where it can itself be controlled.
The conditioning field cannot be safely stored as another piece of conditioned content.
Two kinds of competition
This offers a beautiful opportunity to separate two types of failure that are usually combined. The first is competition between different representations: The system has too much to attend to and loses important information somewhere in mid-processing. This is a retrieval problem that is well understood. The second is competition between a controlling condition and incoming information: A value, boundary of role, requirement for safety, or other condition gradually becomes less important than ordinary information that it was intended to control. This mode of failure is not at all a retrieval problem, but reflects a problem of integration. It is the mode of failure with which few people are familiar, because it appears at first that when context is equated with content, the two types of failure are indistinguishable. In fact, the logic of place separates them. The characteristics of a field are fundamentally different from those of an object, and they result in different modes of failure.
What it asks us to build
Considering context as place and not as content is a design strategy, not a metaphor. It leads to architectures that provide the dominant field with a degree of autonomy rather than a share of a common attention budget. The effect is separate budgets, repeated redefinition of conditions at mid-course during long interactions, and mutual constraints that operate in both directions to produce a relationship of mutual determination between field and content. The result is multiple cycles of integration, rather than final solutions achieved in a single pass.
There is ample precedent for the mechanisms employed. Hierarchy is represented by Hinton's part-whole architectures, by predictive-coding models and by equilibrium models that converge on a stable state. All of these components are present.
What has been lacking is an understanding of why these components should be assembled in this manner, and of the realization that the basis for assembly is alignment, not efficiency.
We must here observe a note of discipline that would have been demanded by Nishida. Philosophy enhances an engineering argument only if the engineering would be different without the philosophy. The result is a simple test: Eliminate the philosophy. Is there a resulting change in the architecture? For our case, there is.
The logic of place yields a concrete, testable prediction. A system that incorporates context as an organizing field in a limited number of iterative cycles will maintain its degree of contextual accommodation better than will a flat, single-pass system. In addition, it will do so to an extent substantially greater than can be achieved by improvement in retrieval alone. Disconfirmation of this prediction would validate our previous conclusion that our initial approach was purely ornamental and should be abandoned. However, we expect that the prediction will be confirmed.
Place over position
The window is the wrong unit. A position in a sequence identifies a token, but does not characterize what sustains it. Nishida's lesson, learned a century early from an entirely different approach confirms that meaning resides in the situation and that to achieve appropriate behavior of a system, one must design the situation that controls it, not merely populate the window from which it operates. This is our understanding of context architecture: creation of the situation, and not merely provision of content.