Oof, right at the top it points to how people were apparently convinced of this situation from a single crappy example

You work in data science and you over indexed on a singular example to explain away an entire set of phenomenon??

My experience with more.... directly testable NN models that are small seems to be that they absolutely can be far too interconnected to make strong explanations. The structure and pathways they fall into don't have to make inherent sense to anything or anyone.

Is it possible that almost all nodes in a network being explainable with a singular feature that makes obvious sense to us is evidence of poor efficiency? The network hasn't been made to collapse learning or discover more effective organizational structures than what might make sense to us.

I don't think I have the fundamentals here to contribute though.

Working with some neural networks, you can somewhat feel the structure when you read between the lines of the output. You can sometimes feel that a threshold somewhere tripped and set the signals down an alternative path.

But confirmation bias and all that.

I hope interpretability has a breakthrough. It would make my job far easier and cooler.

I had figured the "Recursive generalization" was like, understood? Maybe I'm just too pop-sci brained. "Split categories into more on this new boundary" and "Combine two separate categories that aren't different enough" I thought were always the core verbs for working with bulk data. My mental model has never been formalized though, so is invalid.

I guess I never got a good sense of the examples they had which made it look like what was "obvious" or made sense to me wasn't always a good explanation in general. Neat.

It's interesting that meaningful examples both ways care about shapes so much, as your eyes encode edge finding and shapes really early in the signal path. It's why some optical illusions are so effective. It's awesome.

The whole "how do you test a neuron: Find a group of nouns and find where you can split them" is just how I find I think learning works. You build your mental model, and find the split points where it diverges from reality and you are wrong, and you refine. I feel like this is how I learned Random Forests conceptually, though surely that's not a valid comparison?

Apologies, not sure I have actually contributed anything here! I bet I misunderstood the entire article somehow. Specifically, I do not have the fundamentals of looking into visual models that he is really working from. But I always figured that when Data Scientists talk of "Features" in larger models, they explicitly meant these far more abstract representational "features". Doesn't the premise of model hijacking and abbliteration sort of presuppose that you have abstract "Reject" or "Safety" neuron paths? I've always understood this stuff as that neuron path, without weighting too much on individual neurons.

>I used to think of neurons as human-shaped nouns: one is a car, another is a curve.

I didn't think anyone was ever supposed to think this for human neurons! The way humans identify something as even a "Chair" is insanely abstract and cultural! Very specifically, the "building sub branches to disambiguate" is just, like, how you would expect a hierarchical human brain to self organize, right?

Some of this is why I've been so displeased with the interpretability research from Anthropic, where it felt like they just pointed new NNs at the disparate neurons of another NN and post hoc named them. I felt the paths were of importance at least but had no idea how to convey, interpret, or model that. Learn you a good math kids! Not having any formalisms hurts.

Neural networks are clean