Tag
The paper introduces Graph Machine, a method to replace dense attention layers in transformers with sparse layers using dynamic pointers, improving efficiency and maintaining or enhancing performance during pretraining.
This paper introduces Graph Machine, an architecture with explicit edge-based mechanisms (edge-augmented attention and edge-centric referral) to improve iterative relational reasoning. Experiments on Sudoku show it outperforms Transformer baselines, with ablations and mechanistic analysis attributing gains to the edge mechanisms.