Please don’t go building these monstrosities.
I wrote a LinkedIn post that began with the sentence “This is what's wrong with neural networks.” More than one response demanded to know what in the world my message was or what alternative I was offering. The ultimate point is indeed deeper than “neural networks never deal with what defines or constitutes things or concepts.”
The ultimate point I want to make is that the grand project of what’s been called “artificial intelligence” has failed, and will continue to fail if the goal is still going to be “artificial intelligence.”
Let’s take a look at what “artificial intelligence” even means. From Britannica (bolded by me):
The term is frequently applied to the project of developing systems endowed with the intellectual processes characteristic of humans, such as the ability to reason, discover meaning, generalize, or learn from past experience.
If that’s your idea of what machines are supposed to do then congratulations- You’ve set yourself up for perennial failure, as I’ve thoroughly explained how.
Subpoint Number One: Imitation is wrongheaded and seldom yield optimal results
Let’s say you want to design a machine that flies like a bird and carries a multitude of people. Do you literally build a mechanical thing that’s in the shape of bird and have it flap its wings? Do you just build a huge ornithopter? No; It’s going to scale like shit. (see illustration above made with Microsoft Copilot. Sorry that it looks like typical generated crap but I can’t manually draw exactly what I had in mind, though the result wasn’t exactly what I had in mind either)
Also, the term “iron horse” refers to a steam locomotive and not a mechanical horse.
Mathematical and engineering breakthroughs aren’t dependent on imitations of human and animal morphology. We’re designing and making tools, not breeding plants and animals which are descended from evolutionary survival mechanics instead of optimal design. Efforts to double down and insert “self-interested” mechanisms into machines are just desperate attempts at saving a failing project that’s founded upon anthropomorphic principles.
Subpoint Number Two: Living things aren’t results of any design in the first place so why even try “copying" the non-design?
Three things to point out here regarding any alleged “brain design” or “organic neural network design” that anyone would attempt to emulate:
The process of evolution isn’t that of design but that of creation. The successive retirement of organisms that change over time through genetic mutation isn’t a process of “designing” anything. Please see what the word “design” means. Even if the meaning of the word changes, why copy what could be suboptimal? Suboptimal design is widespread in nature, and attempts at emulation could make things even worse than nature (This could be evident in MLP vs. KAN which I will talk about later).
As I’ve previously pointed out, there isn’t a exhaustive model of an underdetermined entity such as the organic neuronal network that is the brain:
There isn’t even really an explicit hypothesis for it:
Subpoint Number Three: Having universal solutions looking for problems (AGI) is bass-ackwards
The first rule to approaching a computational problem should be to have a well-defined problem to solve in the first place. Define the problem that is supposed to be solved by a machine. Look at what the problem is and isn't.
Is the problem "make something that works like the human mind?" Obviously not, because the human mind works via cognitive biases: https://en.wikipedia.org/wiki/List_of_cognitive_biases Does anyone seriously want to engineer all that in?
Everyone needs to accept the fact that we're engineering machine tools in service of practical problem solving, and act accordingly. Many are either in denial of the actual mission of designing and building machines (e.g. to solve problems requiring machine operations), have no idea what the actual mission is, or just don't care what the practical end is.
Trying to gain insight into the workings of the brain by doing pure science is fine, but let’s not confuse the task at hand if what you want is coaxing performance out of machine tools.
If your mission is practically aimless, your project is bound to go nowhere.
Here's what a machine optimization project shouldn't be:
It shouldn't be about coaxing intelligent-seeming behaviors out of a system. A system will look intelligent until it looks stupid.
It shouldn't be about constructing models that imitate supposed brain functions, because that's simulating theoretical intelligence, not implementing solutions to a practical performance problem.
It shouldn't be about trying to reconstruct the brain, because reverse engineering the brain is a fool's errand (aforementioned underdetermination of scientific theory, "all models are wrong," etc.) Again, what is the problem at hand? Is it "we can't engineer the brain" or is it "machines are not performant?" What are people trying to do? Ersatz natural science, or actual engineering?
In sum, as someone involved in a machine computation project, you'd have to decide whether you're engineering solutions for problem solving, or engineering behaviors and supposed theoretical functions based on observed behaviors. Ask yourself this: Am I simulating intelligence, or getting to the root of the problems being solved? How is a solution not up to the task, and why? Don't just treat symptoms (e.g. utilizing various forms of human guidance); Get to the root problem. Start over if you must.
It looks as fun as whack-a-mole, as just as effective.
”Okay, I’m convinced. I’m ready to ditch the hopeless anthropomorphic project and start over. Where do I move my efforts and renew them?”
From the KAN paper. Garbage-looking crap on the left, elegance on the right.
Imitation leads to solutions that are clumsy and overwrought. The perfect example of this comes from the superiority of a Kolmogorov–Arnold Network compared to a "neural network" utilizing multilayer perceptron. The simplicity and elegance of KAN make an MLP neural network look messy in comparison. Let's compare and contrast the origins of each design:
MLP: The idea of perceptron took its inspiration from biological neurons https://towardsdatascience.com/what-is-a-perceptron-basics-of-neural-networks-c4cfea20c590
KAN: Inspired by Kolmogorov–Arnold representation theorem, which was a solution to a variant of one of the 23 problems listed by mathematician David Hilbert as crucial for the advancement of mathematics
Given their respective origins it is no surprise that the mathematically-inspired solution is the more efficient and better performing one.
Machine optimization only becomes an imitation game when people think they're completely out of ideas to solve problems with. There are plenty of ideas left in engineering and mathematics.
Example of 4-stage pipeline from Wikimedia Commons
Let’s look at engineering, specifically techniques to enable instruction-level parallelism in modern superscaler microprocessors. Is instruction pipelining inspired by the brain? Nope. Branch prediction? No again. Processor architects don’t go leafing through biology textbooks when it comes time to break through processing bottlenecks; They rely on engineering ingenuity to work around bottlenecks.
Moving on to mathematics, there are so many unresolved problems in mathematics, yet all the funding is getting funneled to water-guzzling generative AI server farms running things that enterprises are starting to find serious issues with (Thankfully the FOMO bumrush seems to be subsiding.) Some of these unresolved issues begging for actual time and money include:
Herbert’s Problems that remain unresolved to this day
Long lists of unproven mathematical conjectures
tldr; anthropomorphic principles that came with the failed project of artificial intelligence need to be tossed into the open flames with extreme prejudice, and resources reinvested into ideas in engineering and mathematics into a renewed project of optimized computational methodologies.
This is one heck of a long list from Wikipedia to scroll through. I don’t see how the world is hurting for sources of inspiration when it comes to designing machines “inspired by” certain things.