I recently had an interesting discussion about the enduring classic Gödel, Escher and Bach: An Eternal Gold Braid (GEB) by Douglas Hofstadter. The book is so famous and influential that High School classes have been modeled after it; it won the Pulitzer Prize for non fiction in 1980. To group it in with other classics is a well-deserved sorting as it’s a marvel covering a great many interesting topics in math, art and music. But what I remember best is it’s Lewis Carol like spirit of exploration of discrete computer science concepts such as recursion and symbolic representation. It reminds me how some suggest that recursion, hierarchies of data structures and other computer science concepts are useful building blocks for AI. In the book Hofstadter points out how cognition/thinking bootstraps from simple processing constructs, e.g. neurons.
Tools like recursion have always had an appeal for AI developers given their economy of expression and power they leverage from a surprisingly few statements. It’s very slick how complex problems can be broken down by recursion into simpler versions of the same thing and ultimately solved via a few statements nested at the bottom of a stack the Operating System so naturally and invisibly manages. Any hierarchy and any direction of traversal can be embodied in a subroutine consisting of just a few lines of self referential code.
This same quest for algorithmic parsimony seems central to the idea that thought will ultimately be modeled by a handful of recursive-like, Platonic principles which simply haven’t been pinned down yet. Perhaps, but as I became a better programmer I tended to leave recursion alone. Maintaining recursive decent involved increasing amounts of time when adding a new twist for a customer – what once shined as a perfect expression of concise power often would iteratively evolve into a multi-threaded knot of confusion. At some point it’s simply easier to completely spread the work out since RAM/CPUs are ever faster and Disks/Labor can be found ever cheaper as long as the coding chore is made simple. Moore’s the law and the pity.
The ancient question of whether we live in a world that can be modeled with pure logic or is stochastic by nature extends back to the time of Plato and formal logic/proofs as codified by Aristotle. This is in contrast to the probabilistic view characterized by the later and more colorful Roman society with their leaders throwing dice for decisions and ultimately consummated with Neils Bohr telling Einstein to ‘stop telling God what to do with his dice’ at the birth of quantum physics. The probabilistic view’s relationship to consciousness is taken to the limit by some very capable people who assert that there are quantum effects influencing neurons and as such they may never be suitably modeled. I tend to think the self-organizing view requires neither formal proofs nor quantum physics as the former I expect will lead to the same locus bound systems that have stumped AI for the last several decades and the latter seems a bit of an overkill, as well as a convenient way to thwart attempts to model consciousness, perhaps based more on ethics and a not unfounded pandora’s box fear than reality.
GEB goes from fun and fanciful to welcoming the casual reader into the mathematically serious world of irrationality ad infunitum as it playfully starts to tap and then kick the stuffing out of symbolic logic using Godel’s Incompleteness Theorem. Everything gets wrapped up in atomic statements like “I’m lying” and its recursive contradiction. But the arguments he makes are so carefully iterated and shown so to be so clearly unavoidable in formal systems that when I first read it I reflected back that I probably wasted time chasing the self defining world in the Theory of Formal Languages class I struggled with in college assuming it led to something bigger. Actually that class was very useful in simply helping me understanding compiler front end parsers like LEX/YACC, which I’ve used in a dozen Unix projects in my career, so my advice to CS students is to try to enjoy it as there’s obviously great value there, but production rules, meta-symbols and grammars are not a panacea for AI to me.
After GEB made samurai sandwiches out of so much of the discrete, finite math stuff I was fed in Computer Science courses, it was conveniently easy to rationalize a new direction. Now did I really need to read that book to realize a better direction for AI lies in taking a bit more inspiration from nature? No, of course not, lots of folks just use common sense to conclude its a bit silly to attempt to model an infinite universe using rules in what we used to call a ‘finite core.’ The unconscious serves up a lot of stuff from a lot of different directions for what we perceive as a single thread of consciousness. That tip of the iceberg awareness has led to years of folks starting out over and over with systems that try to capture the decision making we label as thought but its just that, its just our perception of what’s happening in our minds. Cascades of pattern recognition start transforming light into nascent concepts as soon as it starts moving down the optic nerve. We say say ‘cup’ and then fool ourselves by designing forward object oriented databases that seem to be a complete model of the ‘cup’ system but of course its not. Its the billions of neurons firing in parallel and transforming information into awareness while moving across a signalling distance equivalent to a few hundred milliseconds of relatively slow, electrochemical synapses which is why we can recognize a cup no matter what light, angle or portion of it we see. We may believe we think in discrete fashion from our awareness of the thread of our consciousness’s trail of thought, but we’re built from billions of neurons that are constantly producing this environment and we’re unaware of any of it.
Picking through some of the other common data structures though, it seems to me casting neurons in a hierarchical template is pretty fundamental if not essential. Jeff Hawkins argues this skillfully in “On Intelligence” and the fact there are very roughly some six identified layers of the neocortex divided between instructing downward towards the cerebellum and upwards for what seems to me as imagination and pure consciousness seems prima facie evidence the neocortex leverages a hierarchy. An interesting twist is that many of the leaf neurons involve a recurrent connection to input neurons to enable one to essentially “hold that thought’ or provide a continuity of cascading memories or predict the future or any one of a number of other such temporal capabilities; in this way the hierarchies seem to me to be loops. The branches encapsulate loops of branches of loops and so on limited physically but in theory like some connection nested ever expanding Mandelbrot set.
A fractal like structure of loops within hierarchies could be boundless but a pared down version of this structure seems to exist everywhere across regions of the brain giving them their characteristic Golgi stained look of an sloppily artistic contour copy of M. C. Escher’s Cubic space division. This odd construct can apparently provide the plasticity brains demand: for example, the hippocampus can immediately store a short list of numbers, such as a taxi driver sorting away a large number of addresses, and is speculated as a cause of thickening of this area in the brains of folks who have to memorize a lot of information. At night sleep seems to act like a kind of genetic algorithm as represented by the neocortex slowly sifting through and deciding which recent short-term memories should merge with the larger, longer term, auto-associative memories stored there.
As such its become a bit of a corollary for me that custom design variants on auto-associative memories are vital components of starting successful AI architectures. Its obvious new insights stemming from non binary architectures, similar to a guitar player switching to Drop D tuning and finding his compositions gain a completely new style, have a creative effect that designers learn to quickly make use of and represent a markedly apparent contrast in style from indexed sequential memory. Of course the only thing worthy of long term interest which means money will be working code which is precisely how it should be. Its an understatement to say its a financial risk for very small companies to skip ahead and turn extremely complex, revolutionary hardware architectures directly into silicon; however, I’m very encouraged that IBM appears to be taking the risk , now with help from DARPA, and attempting to create a new kind of CPU inspired from biological systems that is fundamentally different from the Turing Machine architecture we’ve all used since the end of WWII. Still, designing 3D matrices of inexpensive, microcontrollers to emulate neurons is one fun alternative to custom silicon or purchasing a data center for raw computing power to emulate massively parallel computation. Besides, all those cool 70s era heuristics like circular queues for real-time communication are relevant again on small systems with no OS, and the physical dimensions and weight of Arduinos even with 9V batteries attached can drive real stuff you need a transistor to step up power for like legs and arms.
To sum up this overextended blog entry its my belief the essential data structures for some realistic attempt at biologically inspired brain emulation are probably already well known: neurons with ensembles of simple nonlinear transfer functions and hierarchies encapsulating loops everywhere reading/writing information via a very distant cousin to what wiki and texts usually describe as auto-associative memory. That’s basically one way of describing the neocortex as I interpret neuroscientists; multiplied by a few billion neurons, with about 8,000 connections on average between each neuron, wired everywhere near and far across it’s fabric. We probably won’t see them on the shelves of Frys for a while though as the white matter wiring sitting beneath the neocortex alone is beyond any technology of today. Jeff Hawkins, whom I’m obviously a big fan of, in On Intelligence proposes a fiber, presumably wave division multiplexing, emulation for this serious communication bump in the road to AI together with ever faster machines to simulate the billions of neurons that we obviously could never use metal solutions, e.g. solder, to hold together. Finally, everything in biological systems fire in parallel with no direct equivalent of a system clock but a collection of overlapping electromagnetic ‘waves’ whose eigenvectors somehow wash over neurons in a combinatorial dance conducive to thought. There could be many more layers to the onion we’re not yet aware of.
Apart from the tall order of instantiating this machine an important step will involve linking what is likely already a sentient system to deterministic systems we’ll hopefully still be in control of so as to focus their attention on interesting problems. As Carl Sagan said so beautifully “we’re star stuff seeking to know itself”: whether carbon based, silicon or mixed I believe that’s a forever true statement. Someday there may be a ‘Google’ pondering a million times faster than a chemically bound human neuron’s transfer function in orchestration with several billion parallel processing units involved in much more than just finding cool links on the web. I feel there’s an incredibly challenging, mesmerizing and beguiling development ahead of us for self organizing, massively parallel AI and the answers resulting systems provide might well help us with some of the most significant questions of all time. I guess we should all stay tuned and of course pick up a copy of GEB, its really a wonderfully fun book to explore.
