The Technology

Overview

Neuromorphic technology refers to the science of understanding structures and processing mechanisms of the brain.

HNeT technology applies the power of holography to the modeling of synthetic neuron cells (through the application of non-linear phase coherence/decoherence principles). Neuromorphic structures and the underlying holographic principles of operation provide a vast increase in capability for machine learning.

To provide a practical example, a neuromorphic assembly will track a human face in real time. This assembly learns facial images by direct exposure, building within its memory all observed forms of an individual, and can subsequently identify that individual within a crowd. It can even determine facial expressions such as smiling or frowning. The functionality demonstrated by HNeT assemblies of this type approaches and even exceeds the limit of current technological capabilities.

HNeT technology is not limited to facial recognition and may be applied to numerous areas within other sectors, such as medicine, process control, automation/robotics, finance, etc.

HNeT

We have constructed an application development for this technology, allowing one to build neuromorphic assemblies and integrate their capabilities into real-world applications. This system is called HNeT.

The HNeT API provides a high level of flexibility in the construction and execution of neuromorphic assemblies, as well as application-level integration, using Windows-based development tools, such as Visual C and Java. This application development system consists of the following two primary components.

Set of desktop computers

The HNeT Supervised Learning (SL) Platform

Supervised Learning (SL) Platform

The HNeT Supervised Learning (SL) Platform provides an advanced GUI environment for constructing and optimizing feedforward assemblies from data files provided in standard formats. Neuromorphic assemblies generated by the SL platform may be integrated into applications through OLE/OCX interfaces or directly through the HNeT API.

The HNeT Supervised Learning (SL) Platform is an assembly generation tool structured for feedforward supervised learning. Developers can load standard ASCII or binary files containing stimulus-response training, and testing and validation data. Optimization of assembly performance may be performed in a fully automated manner. Generated assemblies may be deployed through the HNeT Application Programming Interface.

Automation tools facilitate assembly generation using the SL Platform executable as a local or remote server, allowing the analyst to allocate a server farm across an array of networked computers. These automation tools allow the analyst to centrally batch process SL Platform project files and provide a central monitoring/analysis facility for the entire SL Platform server farm.

Numerous utilities are provided in this environment, such as performance testing, graphical analysis aids, data preparation wizards, etc. Data files may be segmented into training, testing, and validation sets for automated performance optimization and verification. Work sessions may be saved as projects, whereby all related data files, assembly configurations, and program settings are managed by the system and reloaded through a click of a button. Various screen views of the HNeT SL Platform are shown to the left.

The HNeT Application Programming Interface (API)

Application Programming Interface

The HNeT Application Programming Interface (API) allows developers to allocate specific cell types and interconnect these cells to form cerebellar, neo-cortical, spatial-temporal, hyperincursive assemblies, or any combination/hybrid of these primary models.

The developer is able to independently modify any property associated with each cell, such as memory decay, learning rate, neural plasticity, I/O conversion operations, execution sequence, and a host of others. The HNeT API has been optimized for SIMD and hyper-threaded operation. Cell assemblies may be structured to form supervised learning, unsupervised learning, spatial-temporal, and other more sophisticated models.

The HNeT API provides over 100 functions for allocating and configuring neuromorphic assemblies, controlling related properties, and integrating assembly operations into the application layer. The API applies the concept of objects and uses handles in referencing these objects. Cell/assembly handles are applied in adjusting cell properties, controlling execution, modifying synaptic connections, etc. Properties associated with individual cells or assemblies (learning rate, neural plasticity, synaptic interconnections, memory decay, among others) may be separately adjusted.

The HNeT API facilitates the integration of neuron cells into simple or highly elaborate neuromorphic structures. The library permits the allocation of an unrestricted number of assemblies and allows integration of the primary neuron cell groups shown.

Primary assembly structures are illustrated for the cerebellar and neo-cortical models.

The HNeT API provides an array of functions for the customization of individual cell architectures, assembly architectures, synaptic interconnectivity, neural plasticity (synaptic pruning and regrowth), construction of supervised, unsupervised, spatial-temporal and hyper-incursive neural models, data conversion, and preprocessing. A function reference sheet for the HNeT API language is available upon request.

Performance Aspects

The following sections summarize a number of performance aspects that are characteristic of holographic neural processing. This material is intended for a technical or engineering audience.

Learning Capacity

Summarizes performance aspects pertaining to the speed of stimulus-response learning and related memory storage capacity in comparison to conventional neural networks.

The random statistical or "Monte Carlo" test is commonly applied in engineering analysis and provides one of the more rigorous techniques for evaluating system performance. The comparison table below illustrates convergence (reduction in recall error) given five variables for the stimuli (five input dimensions). Random test values are uniformly distributed within the boundary of 0.0 to 10. Convergence results are shown for learning of 500 stimulus-response patterns.

Learning within an HNeT cerebellar assembly, for this instance, is compared against a conventional genetic neural network. As is typical within the non-linear realm, conventional neural networks are unable to converge (cannot learn).

Random statistical tests and many real-world problems are highly non-linear, as the number of associative patterns learned can greatly exceed the dimensionality of the stimulus input (in the above example, by two orders of magnitude). A problem colloquially termed "hitting the wall" occurs for conventional neural nets, and the application of more cells/layers/interconnections does not resolve this limitation.

For neuro-holographic assemblies, learning convergence given non-linear datasets occurs at a rate proportional to linear problem solutions, as shown by the graph to the right. Such rapid convergence in non-linear problem spaces is revolutionary, with implications for virtually all fields of engineering that are, mildly put, dramatic.

Convergence

Illustrates the convergence characteristics of holographic learning when applying multiple training exposures or "epochs."

Rapid convergence of non-linear solutions brings truly "real-time" learning to complex real-world problems. Holographic neural assemblies exhibit a logarithmic rate in learning convergence for both linear and non-linear problem spaces.

Non-linear spaces require the use of combinatorics, often referred to in the QIP world as "entangled states." Conventional neural networks encounter severe limitations in recall error reduction for non-linear problems, as indicated in the previous comparison.

Error convergence chart

The graph to the right illustrates convergence characteristics typical of an HNeT assembly trained on highly non-linear data, where the number of patterns learned greatly exceeds the number of stimulus pattern elements. In under 100 epochs, recall accuracy reaches the resolution of floating point numbers applied in the computation. This convergence characteristic of HNeT (logarithmic reduction in error - down to the computational resolution) is observed in both random test and real-world application data.

An important feature of holographic neural processing is that the above logarithmic convergence for non-linear problem sets degrades only at the saturation point where the number of learned associative patterns exceeds the number of cortical memory elements. This limit occurs irrespective of the dimensionality of the stimulus input pattern (the number of elements defining a stimulus pattern).

This property holds true, irrespective of whether the number of associative patterns learned is 1000 or 1.0 x 109

Generalization

Describes generalization aspects when the learning environment is highly complex or "non-linear."

The topology of a trained holographic neural assembly is shown below for a non-linear associative mapping, illustrated along three input dimensions (represented by x, y and time). In practice, the holographic neural method learns such non-linear topologies along an unrestricted number of input dimensions (up to 16 Million in the case of HNeT).

Each crest or valley within this topology defines an associative pattern (also called memory trace) that has been learned by the HNeT assembly. The HNeT assembly interpolates smoothly among all associative patterns that have been learned.

There is no known technique capable of achieving this convergence characteristic for non-linear systems other than the holographic neural method.

By comparison, the topology achieved through linear methods (such as linear regression) is defined by a flat plane within the N dimensions of the input space, restricting learning capacity for real-world systems and data.

The issue here is that despite significant efforts in applying conventional neural network technologies, often little advantage is gained over simpler and faster linear regression techniques.

Neural Plasticity

Describes the process of neural plasticity (synaptic pruning and regrowth) and the performance gained through optimization of complex combinatorics.

Neural plasticity refers to the operation of dynamically pruning cell processes (synaptic interconnections and related dendrites) and regrowing new cell processes. Neural plasticity within HNeT optimizes the combinatoric selection of higher-order input signals (granule cell processing) through autocorrelation with cortical memory magnitude.

Application of neural plasticity dramatically improves the generalization capability of holographic neural assemblies. The graph to the left illustrates typical error reduction when testing against independent validation data sets and applying neural plasticity. This example illustrates the error reduction curve for an assembly trained to segment facial images from complex background clutter.

Five function interfaces are provided within the HNeT API to control neural plasticity. A broad range of options is provided, and different modes of pruning and regrowth are achieved by applying plasticity operations in various combinations.

Computational Complexity

Defines the number of numerical operations or hardware registers (Complex Multiply and Accumulate) required to execute a neuro-holographic assembly.

The neuro-holographic process may be divided into three principal components of operation: A) preprocess conversion, B) combinatorics, and C) execution of the cortical cell in learning and recall. Computational complexity is established by the numeric operations performed in each of these three stages.

Preprocess conversion applies methods, such as mapping to phase through sigmoidal or histogram equalization functions; frequency domain conversions, such as Fourier and Wavelet-based transforms, and so on.

The remainder of the process consists of Complex Multiply (CM) operations in the combinatorial stage and Complex Multiply and ACcumulate operations (CMAC) within the cortical cell for learning and recall. As such, computational load is determined primarily by the number of cortical memory elements and the average product order in combinatorics generation.

For instance, an HNeT assembly comprised of 1000 cortical memory elements and processing, on average, 2nd order combinatorics would require the processing resource shown to the left. Computational overhead is linearly proportional to that required within a single neuron cell within traditional multi-cell, multi-layered ANS systems such as back-propagation (by a factor of 4).

Both the rate of learning convergence and associative storage capacity for the neuro-holographic method dramatically exceed that of conventional neural network architectures.

Mathematics

Although the mathematical basis behind HNeT is somewhat abstract, one does not require an in-depth understanding of the theory in order to design and build applications using the HNeT Application Development System. It is important that one understands how information is presented to the system and how the various classes of holographic-based neuron cells interface with each other.

A stimulus-response pattern, association, or "memory" may be represented by a set of values reflecting conditions or states measured within an external environment, such as pressure, temperature, brightness, etc.

During learning, neural cells associate or "map" one set of analog values (i.e., the stimulus fields) to an associated set of values (i.e., the responses). When stimuli are distributed over a time span, one has spatial-temporal or episodic learning.

The mathematical basis for HNeT permits vast numbers of stimulus-response or "associative" patterns to be learned and superimposed onto a vector comprised of complex scalars called the cell's cortical memory. In fact, the number of values used to store cortical memory is often no larger than the number of elements within a single stimulus.

The mechanism for holographic storage displays a capacity to achieve extremely large information densities due to the fact that large numbers of associative memories are superimposed or enfolded onto the same set of storage elements (in other words - computer RAM).

Again, the material provided in subsequent pages is intended for a technical audience.

Holographic Learning

Holographic neural operation is based upon phase interactions, following phase coherence/decoherence principles observed within electromagnetic wave theory and quantum mechanics (QM).

Operations performed at the synaptic level of the cortical cell (Purkinje or stellate) are of identical form to the QM wave equation if one follows the simplified linear framework of HNeT. However, in practice, we apply far more sophisticated non-linear processes.

The inner product shown below describes the learning process that takes place within each cortical memory element Xn of the cortical cell. Each cortical cell may possess many thousands of such cortical memory elements.

The learning operation stores phase information pertaining to each separate stimulus-response memory trace (indexed by t) onto the identically same set of complex scalars, again each represented by Xn. For clarity, the S vector indicates the stimulus input, and R is the associated response.

The response recall process within holographic neural processing follows a similar form; however, the inner product is performed across cortical memory elements (indexed by n) rather than time. During the above recall operation, a new stimulus S* is processed through each prior learned stimulus-response memory superimposed within the cell's cortical memory to produce a response recall R' as shown on the next page.

Phase Coherence

A recall operation invokes a deterministic response to each of the memory traces (indexed by t) that has been learned and superimposed within the cells' cortical memory. Prior learned memories that resemble the current stimulus input issue the dominant responses within R'. Mathematically, the generated response is shown as follows:

Complex scalars exhibit an interesting property in that multiply operations induce rotation in phase. What occurs during response recall is phase alignment through rotation. This alignment occurs, however, for only those memory traces that most resemble the current input stimuli.

Phase angle alignment (phase coherence) generates the dominant contribution in the response while remaining memory traces undergo misalignment (phase decoherence), producing a far smaller residual. The net effect is that the correct response is produced when the holographic assembly is exposed to any prior learned stimuli.

The phase coherence/decoherence principle is illustrated for stimulus learned at t1. Each prior learned stimulus-response pattern may be recalled quite accurately following even a single training exposure, and again, all memory traces reside concurrently within the same set of computer bytes (cortical memory elements).

HNeT operation is based upon the above phase coherence principle; however, it utilizes far more sophisticated and powerful non-linear methods.

Combinatorics

HNeT is a non-linear model that applies complex combinatorics (generation of higher-order relationships from input stimuli). The specific cell type that performs this function is the granule cell. The number of unique products increases as a factorial relationship to both the number of stimulus fields N and product order P, as indicated by the relationship:

Applying complex combinatorics, memory storage capacity remains directly proportional to the number of (now higher order) input fields and, thus, cortical memory elements. Combinatorics facilitates rapid convergence for non-linear systems, providing a mechanism whereby extremely large numbers of stimulus-response memories may be accurately learned within an assembly that receives far lower numbers of input (stimulus) variables.

For instance, consider an assembly that reads 16 stimulus input variables, say axial positions and rates of movement in a robotic control device. The assembly could employ downstream granules in the generation of up to 10th-order product combinations.

In this case, there are greater than 2 x 106 unique higher-order combinations, allowing a proportional number of stimulus-response patterns (i.e.,> 2 Million) to be rapidly and accurately learned. The table on the left provides some indication of the combinatorial explosion that occurs when stimulus size and product order are increased.

Applying combinatorics, there is little restriction in terms of complexity (non-linearity) and storage density that may be learned within neuro-holographic assemblies.

Commutativity

Commutativity is another intrinsic property of holographic processing. Stimulus-response pattern storage densities increase in direct proportion to the number of cortical memory elements across all neuron cells directly connected within an assembly.

This concept is illustrated by the following equality, considering [X]c as the cortical memory values stored within each cell (c) and [S]c as the stimuli processed through each of those memory elements:

The resultant increase in memory storage capacity is exponential and permits one to construct assemblies that are effectively unrestricted with respect to associative memory storage. Commutivity has important implications for neuromorphic structures that follow the neo-cortical architecture.

To illustrate, within a neo-cortical assembly, stellate cells relay axonal processes to proximal pyramidal cells. In human biology, a typical stellate cell receives up to 20,000 input signals, while a pyramidal cell receives in excess of 100,000 synaptic inputs.

Doing the math, commutativity facilitates storage densities in excess of 2 Billion associative patterns prior to saturation (degradation of learning rate and recall accuracy). In real terms, this storage density equates to one memory trace for every second across an 80-year lifespan.

The Quantum Analogy

Quantum mechanics image

At its most fundamental level, Quantum Mechanics (QM) is based on a mathematical abstraction referred to as the wave equation. The wave equation mathematically defines a superposition of harmonics possessing varying frequency and phase. Phase coherence within this superposition of harmonics results in an energy envelope defining elementary particles and photons.

The wave packet equation is shown to the right in discrete summation form, where ci is a complex number representing amplitude:

The above linear superposition may be written in a simpler form as follows:

Wave packet equation

A quantum state vector formed by a set of quantities ψ is expressed using the ket-vector, which is a column vector comprised of complex values.

The adjoint to the ket is the bra-vector, which is a row vector comprised of complex conjugate coefficients:

The quantum state vector that defines the learning process across a set of patterns indexed by i may be expressed in Dirac notation as follows:

Quantum state vector equation

In quantum mechanics, the amplitude coefficient cn (or, in the case of HNeT - the desired response for pattern n) is expressed by evaluating the following product:

HNeT amplitude coefficient

Other Aspects

There are many other features concerning the HNeT technology. A brief summary of some of these aspects is provided below:

Memory Decay

Adjusts the rate of memory decay within cortical cells. This is performed through controlled attenuation in the magnitude of cortical memory elements.

Learning Rate

Adjusts the rate of learning. By default, the learning rate is set to 100%, meaning a new stimulus-response pattern is mapped or learned precisely following one training exposure. Reduction of the learning rate may slow the learning process; however, it often improves generalization.

Intracellular Feedback

A much simplified description of the learning and recall process is provided on this website. The HNeT system employs intracellular feedback of the generated response signal to facilitate convergence (recall error reduction) over multiple training epochs.

Spatial-Temporal Learning and Neural Plasticity

Cells modeled on the hippocampal structure are provided within HNeT. These provide the ability to buffer temporally distributed signals, allowing downstream cortical structures to learn temporal patterns or episodic memory such as speech.

This feature allows cells to optimize synaptic interconnections, applying correlation in cortical memory magnitude. The cell actually builds an equation of state governing the learning environment.

Biology

Neuromorphic science attempts to understand the interconnectivity and signal-processing features of biological neuron assemblies.

Activities have been announced by a number of government and academic institutions regarding attempts to model neurological structures. Most projects focus on the emulation of anatomical form by applying traditional connective methods; however, they lack the performance benefits of holographic processing and information superposition.

We believe that principles of holographic learning and superposition are of primary importance in the realization of advanced capabilities demonstrated by biology.

If you would like to further understand how HNeT interprets the science of biology, this information is summarized in the following pages.

Overview

The following provides an overview of the HNeT neuromorphic architecture. We believe biological neural processes to be holographic. An advanced form of digital holography defines the core operational aspect of the HNeT technology that AND Corporation has developed and commercialized.

HNeT closely mimics principal regions of the brain. Our system allows one to construct cell assemblies ranging from supervised feedforward to more advanced spatial-temporal and hyper-incursive (cognitive) structures.

HNeT cell groups have been assigned biological names due to their similarity to specific classes of neuron cells (i.e., the granule, stellate / Martinotti, pyramidal, and Purkinje cells). The following pages provide the application engineer with an overview of the biology upon which HNeT is based. This material is intended for a technical audience.

Neo Cortex

The cerebral cortex is a layer of gray matter covering a white core over both hemispheres of the brain. This gray matter contains principally neuron cell bodies and areas of synaptic connection.

The subcortical white matter is predominantly comprised of cell axons directing neural pathways between various regions of the cortex (i.e., commissural and association fibers, thalamocortical, corticospinal, corticoreticular pathways, etc).

Neuron cells of the neocortex fall predominantly into three categories: pyramidal, stellate, and granule cells. Neuron cells of the neocortex fall predominantly into three categories: pyramidal, stellate, and granule cells. Neo-cortical assemblies form a three-staged construction (granule stellate pyramidal), as illustrated in the diagram to the right.

The following pages summarize the component cell features and assembly structure of the neocortex and describe how HNeT models this architecture. For clarity, neuron cell dendrites receive input signals, and neuron cell axons relay output signals.

Cerebellum

The structure of the cerebellum (reptilian brain) and the organization of cells is quite different from the more recently evolved neocortex (mammalian brain). The construction of the cerebellum is divided into the outmost molecular layer, the Purkinje layer, and the granule layer. The most predominant cell types are the Purkinje and granule cells.

The cerebellum is based on a two-stage cell assembly construction (granule Purkinje), as illustrated to the right. This may be compared to the three-staged construction of the neocortex.

The following pages summarize the component cell features and assembly structure of the cerebellum and describe how HNeT models this architecture.

Other Models

The hippocampus is located in the central region of the brain, forming a spiral of interconnected soma (cell bodies), as illustrated to the right. In humans, the number of layers along the spine of the hippocampal structure is estimated to be 40 million.

Studies of traumatic damage within this region suggest that the hippocampus is associated with temporal or episodic perception and short to medium-term memory.

The hippocampus consists of layers of rings arranged in a spiral configuration.

Signals received from the cortical spinal tracts and thalamus propagate through each lateral segment of this spiral structure, presenting a temporal history of the efferent (incoming) patterns.

These temporally distributed patterns subsequently propagate to assembly structures of the neocortex. An analogous structure is provided within HNeT by use of the AllocTemporalCell function. This structure is applied in the learning and recall of temporally based patterns, such as speech.

Neural Plasticity

The brain employs an auto-correlative feature called neural plasticity. This pertains to a process of "pruning" synaptic connections and related dendrites, as well as synaptic regrowth. In this manner, the brain is able to further adapt to its environment through "hardware" changes. Within HNeT, neural plasticity is also applied, whereby synapses and associated combinatorics are pruned through autocorrelation with the cortical memory element that receives the synaptic signal.

Cortical memory elements that display large magnitudes indicate the input combinatoric is highly correlated to the response. Thus, it is important to map out the stimulus-response environment they are exposed to. Low magnitudes indicate the correlation is low, and associated input synapses are less important.

Synaptic regrowth, triggered by lower magnitude, is performed given constraints assigned by the user (source cell, product order, spin). The source cell identifies the axons from which input signals are read (i.e., those cells whose outputs are used to generate the combinatoric). Spin provides control over the application of complex conjugates in the generation of combinatorics.

The HNeT system provides the research scientist/application developer with a high level of control over learning and the associated features of neural plasticity.