@HistedLab: I have a lot of respect for @nickcammarata and Goodfire, but I will add some thoughts from brains/neuroscience. First, …

I have a lot of respect for @nickcammarata and Goodfire, but I will add some thoughts from brains/neuroscience.

First, polysemanticity and distributed representations are widely seen in biological brains. (In some ways this is what I have spent my career on.) 1/

In neuroscience we talk about “mixed representations,” (see @StefanoFusi2 @MillerLabMIT ) which is closely related to whether neurons are clean or not. In basically every brain area we look at, there is strong polysemanticity.

Even in primary visual cortex, perhaps the most well understood cortical area, the notion of a receptive field that does only one thing (roughly, a clean representation), is rare.

In V1, simple cells exist when they are recorded singly and simple grating patches are used. But as we and others (C Gilbert etc) have shown, once natural vis stimuli are used, population/net effects make neural responses dirty

C Deveau et al from our lab

Mixed selectivity! Maps directly to polysemanticity in mech interp.

Also, the fact that polysemanticity and non-clean representations are seen in brains doesn’t necessarily mean that AI systems can’t be trained to produce clean representations. Maybe they can. But my feeling is that it’s the nature of highly nonlinear function-fitting with

connected networks / transformer blocks that by its nature leads to neurons that are hard to interpret. We don’t yet know enough about brains and LLM network computation to know for sure.

In the coming years this will be a focus of research in both fields, I believe.

An interesting point in this post: that the higher activation levels of neurons may be monosemantic (implication that the other activation levels are also important for computation). I think there are strong hints this is true in brains too

Pulling this up from the reply chain too