Leo's Home page -- Github Page -- License: CC BY-SA 4.0

An Introduction to Scalar Representations in Neural Models

Leonardo M. Rocha

Contact Me

Introduction

This document aims to present the global ideas of neural networks from the point of view of biological neurons.

The idea is once this basic knowledge is understood and starting from a Spiking Neural Network find possible ways of representing encodings in an Artificial Neural Network setup which does not rely in temporal spikes but in atemporal values instead to be used later to find alternative encoding posibilities for use as input in current Artificial Neuron Network Model (ANNM) technologies without the need to change the entire paradigm.

The present document does not intend to be a complete guide to deeply understand neural coding, but as a high level overview of the encoding mechanism and how to encode a physical property into neural values linking resources for a deeper understanding.

This work is inspired by the tuning curves of real neurons and the fact that an ensemble of neurons can represent an encoding of an input into a different vector space, being able to create a linear decoder that reconstructs the initial signal.

The work presented here inspired mainly by the following books:

• Neural Engineering (Eliasmith and Anderson- 2003)
• How to Build a Brain (Eliasmith - 2013)

Important Concepts:

These are the concepts that should be retained from the current document

• Neuron Spike: when a neuron fires
• Leaky Integrate and Fire Neuron (LIF): A set of neural models used to represent and study the brain
• Neural Ensemble: a set of neurons representing a computation
• Neural Response Curve: how a neuron encodes (in firing frenquency) to an input value
• Neural Tuning Curve: What the neural firing frequency response to an input means in the input domain; being the input domain the possible input values given to the neuron to encode.
• Sparse Distributed Representation: A vector that turns on a few values to encode an input

How neurons encode values

I am going to lie now, it is not a big lie but this is not a complete picture of what happens. Even if there is much (much) more to how a neuron works this description is enough for the present study.

Even if there are many neural representations, the set of models that is enough to understand the general mechanics are the Leaky Integrate and Fire Model

To simplify, a neuron can be in one of two states:

1. firing
2. not firing

For a neuron to fire it needs to accumulate enough charge to pass a firing threshold.

This threshold is attained by accumulating charge that comes from spikes of neurons connected to the dentrites of the neuron in question (the one we are analyzing). The leaky model means that it loses charge all the time, which means that it needs to be receiving more energy than the one it is losing to attain the firing threshold.

When a neuron fires it is called a spike, and after has a refractary period in which it can not fire while is charging again.

For the purposes of the current study of the neuron, what we care about more than if it fired or not is the firing frequency, which will represent a physical property in an encoder.

You can see an example for a single neuron in the Nengo documentation

Noise

There is another important point to be made: brains are noisy, there is a lot of noise and this implies that the brain has also mechanisms to cope with it.

Human Eye Horizontal Position Example

Let's start with an example focusing on the encoding of the horizontal position of the eye. This organ and how it encodes values has been deeply studied just to link a few studies on the subject.

There are many things that we could do to encode the position of an eye, let's start by what we can do in a robot.

In a robot we can encode the camera (eye) horizontal position as a voltage value, which we can set between a minimum and a maximum. Let's say that our interval is [0,1] being 0 the outmost left and 1 the outmost right. this might be good for a circuit where this values can be interpreted directly, but in a noisy environment which encodes values in spikes this is not the best thing to do.

Instead what the brain does is use a set of neurons to represent a single scalar value. In this setup, each neuron has a firing rate which can be represented visually in what is called a "Neural Tuning Curve", this has also been studied in the Primary Visual Cortex). You can see for example this study on Sensory Coding.

So in this case, what we have is a set of neurons each firing at a different rate, and covering a frequency spectrum that can later be decoded to a single scalar value.

While there are many publications on encoding and decoding mechanisms, we can decode a value from a neural ensemble (a set of neurons representing a value or computation that we care about) with a linear function which might seem counter intuitive as the encoders (and what makes the neural computation powerful) are non-linear.

For all the purposes now we'll consider only a linear decoder, while the encoders are of non-linear nature.

We can see here the screenshots of a scalar representation simulated by 50 spiking neurons done with Nengo.

For example, a simple unitary encoding function with 50 neurons produce the following tuning curves.

The curve on the top represents how each of the 50 neurons fires (in frequency) depending on the exitatory input value.

The bottom graph which represents the Tuning Curves of all the 50 neurons. Each curve relates the firing rate of the neuron with the eye position represented in a [-1,+1] value range, being 0 the center.

And a screenshot of nengo simulation of the same 50 neuron unitary function encoding:

This image shows:

• At the left: A bar that was moved during the simulation changing the input value
• at the bottom left graph: The input value (exitatory) in time
• Top Right graph: The decoded value (linear decoder), see the comparison with the input
• Center graph in blue: the spikes of each neuron in time
• Center right in gray and black squares: the 50 neurons current state, black ones are inactive, the closest to white the more active at the moment.

Sparse Distributed Representations

As you can see in the previous image, on the instant activity representation (the black-gray-white squares), only some neurons are firing at a certain input, not all of them. This is what is called a Sparse Representation, while Distributed Representation means that not only many neurons are inactive (actually most of them), but that those neurons are distributed among the entire neuron ensemble.

It is not shown here, but bigger the neuron ensemble the sparser we can make it, this leads to a better and more accurate reconstruction of the input signal in the noisy environment.

It is important to note that these representations contain strong redundancy, which means that a single (or many) failures due to noise, a neuron "breaking" and many other external factors can be handled, even if with a bit of loss of reconstruction accuracy. The more "damage" and noise a network receives, the less accurate the signal recontruction.

A few reference on the subject:

• Berkeley
• Scholarpedia
• Numenta has great resources on the subject

Importance of Sparse Distributed Representations

While it might seem counter-intuitive and wasting resources to represent one scalar input with many neurons, this approach has many advantages from the processing point of view in the brain. The main advantages are:

1. deal with noise
2. gracefully deal with failures (catastrophic failures are mostly avoided, unless a rock crushes the head)
3. one single neural ensemble can encode different inputs differently
4. sparse representations can be combined in different ways allowing for more complex representations on input set values

The possibility to combine higly dimensional vectors allows for complex non-linear computations in the input sets which is what the brain needs to do.

I higly recommend reading the following to understand better the subject:

• How do neurons operate on sparse distributed representations?
• Properties of Sparse Distributed Representations and their Application to Hierarchical Temporal Memory

This is the base on which I base studies that I will publish later.

Conclusion

• In the brain an input scalar is represented by a neural ensemble of many neurons, each having a different frequency firing response to different input values.
• A LIF (Leaky Integrate and Fire) Neuron is a model that gets charge while there is input and continuously loses charge, if the input is stronger than the leak it will accumulate enough charge to pass the firing threshold, making the neuron fire
• The graph showing the firing frequency response curves for all the neurons is the Response Curve Graph
• The graph showing how the firing frequency of each neuron relates to the input value (for example the eye position) represents Tuning Curves
• Most neurons are inactive or present low activity at a certain input value, this is a Sparse Distributed Representation
• Sparse Distributed Representation allow for noise and failure resilient complex computations in a defined vector space.