Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

1.2.1 Rate Encoding

Authors
Affiliations
Virginia Tech, USA
Technical University of Denmark
Innatera

Rate Encoding is a prevalent form of encoding continuous values to discrete spikes, and is quite easy to work with. This is because it relates well to the Deep Learning networks and training methodologies (that researchers leverage to train SNNs – discussed in the later chapters). The core idea behind Rate Encoding is to represent continuous values via a spike rate over time (i.e., number of spikes averaged over time), e.g., 20Hz, 25Hz, etc. – such that the spike rate is proportional to the input value. Here, we discuss two popular methods of rate encoding: Count Rate Encoding (also commonly known as Frequency Rate Encoding) and Population Rate Encoding.

1.2.1.1Count/Frequency Rate Encoding

This encoding method is a standard approach used in most Spiking Neural Networks. The idea is to have one spike generator per input dimension to encode continuous values to binary spikes. That one spike generator can either be a mathematical function (e.g., Poisson Encoding) or a neuron (e.g., Neuron Encoding); these are described below in more detail.

1.2.1.1.1Poisson Encoding

1.2.1.1.2Neuron Encoding

In the Neuron Encoding method, continuous values are encoded to spikes via a spiking neuron stimulation. Consider a continuous-valued scalar input x[t]x[t] that is to be encoded; the idea is to obtain a current J[t]J[t] from x[t]x[t] and use J[t]J[t] to stimulate a neuron (can be a LIF\texttt{LIF} or an IF\texttt{IF}) corresponding to x[t]x[t] to produce spikes. This method is popularly used in the Neural Engineering Framework (NEF) Eliasmith & Anderson, 2003, Stewart, 2012 that we shall look into in later sections. The following equation can be used to obtain J[t]J[t] from the input x[t]x[t] to an encoding neuron:

J[t]=α×e×x[t]+JbiasJ[t] = \alpha \times e \times x[t] + J_\text{bias}

where α\alpha is the encoding neuron’s gain, ee is the encoding neuron’s encoder coefficient, x[t]x[t] is the scalar input, and JbiasJ_\text{bias} is the bias current to the encoding neuron. Note that ee determines the sensitivity of the encoding neuron to input values; we will provide more details in the NEF appendix. After obtaining J[t]J[t], one can use Eqs. (2)(b\textsf{b}, c\textsf{c}, d\textsf{d}) or Eqs. (4)(b\textsf{b}, c\textsf{c}, d\textsf{d}) to encode a continuous signal x[t]x[t] to spikes by using an IF\texttt{IF} or a LIF\texttt{LIF} neuron, respectively; note that I[t]I[t] in these equations is replaced by J[t]J[t].

Looking at the values of the hyper-parameters in Eq. (1) above, one can set their values to αR+\alpha\in\mathbb{R}^+ and JbiasR0+J_\text{bias}\in\mathbb{R}^+_0. However, the value of ee depends not only on its intended sensitivity, but also on the dimensionality of the input (more in the NEF Appendix). For now, let us consider only scalar input, i.e., x[t]Rx[t]\in\mathbb{R}. This implies that ee should also be a scalar with e=1e=1 or e=1e=-1 if the input values that need to be encoded are only positive or negative, respectively. Overall, one should be careful in choosing appropriate values of α\alpha, ee, JbiasJ_\text{bias} in accordance with the input x[t]x[t] and the chosen VthrR+V_\text{thr}\in\mathbb{R}^+ or any other hyperparameter of the encoding neuron (e.g., voltage decay vdecayv_\text{decay} in LIF\texttt{LIF} neuron, i.e., in Eq. (4)).

1.2.1.2Population Rate Encoding

This is another standard but relatively less common encoding approach used in SNNs. The idea is to have a group of encoders per input dimension to encode continuous values to binary spikes. Why so? A population of differently characterized encoders is required when you want to capture different characteristics of the input signal, e.g., if your signal is composed of positive and negative values! In such a case, you would ideally like your encoders to be sensitive to the positive and negative values of your input signal. You can do this by specially tuning the implementation of your encoders. We next describe a special case of Population Rate Encoding, followed by the general case.

1.2.1.2.1Two-Neuron Encoding

Two-Neuron Encoding is the simplest version and a special case of Population Rate Encoding, where only two neurons are used for encoding the input. As hinted above, such an encoding system is commonly used when one has to encode an input signal composed of positive and negative values over time. If you use only one neuron sensitive to positive values, only the positive part of the signal will be encoded into spikes, and you will lose the information available from the negative part of the signal, and vice versa if you use only one neuron sensitive to negative values. To illustrate this, we take the following example of rate encoding a sine wave (oscillating between -1 and 1) using the abovementioned Neuron Encoding approach -- first by using a single neuron, then with two neurons. Consider the IF\texttt{IF} neuron’s current J[t]J[t] Eq. (1) and voltage Eq. (2)(b\textsf{b}, c\textsf{c}, and d\textsf{d}) below:

J[t]=α×e×x[t]+JbiasV[t]=V[t1]+J[t]S[t]=Θ(V[t]Vthr)V[t]Vrest\begin{align*} J[t] &= \alpha \times e \times x[t] + J_\text{bias} \\ V[t] &= V[t-1] + J[t] \\ S[t] &= \Theta(V[t] - V_\text{thr}) \\ V[t] &\leftarrow V_\text{rest} \end{align*}

where α(=1)\alpha (=1) is the gain of the IF\texttt{IF} neuron, Jbias(=0)J_\text{bias} (=0) is the bias and ee is the neuron’s encoder coefficient determining its sensitivity; x[t]x[t] is the input signal (i.e., sine wave in our example) to our IF\texttt{IF} neuron – the above rate-encoding equation is taken from the Neural Engineering Framework theory [Eliasmith & Anderson (2003), Stewart (2012)]. The encoding neuron produces a binary spike when V[t]>VthrV[t] > V_\text{thr}, and its voltage V[t]V[t] is reset to 0 upon spiking. Note that V[t]V[t] is constrained to be always >0> 0; there is no reason for it go negative – as for the neuron to spike, its V[t]V[t] should increase towards the positive VthrV_\text{thr}. The following code demonstrates the sine wave x[t]x[t] and its rate-encoded spikes when only one encoding neuron is used with e=1e = 1:

Sine Wave

The neuron is sensitive to only the positive part of the sine wave and encodes it to spikes, however, the information from the negative part is lost. If we allow negative V[t]V[t], and set another VthrV_\text{thr} in the negative direction - to produce spikes for the negative part of the sine wave, then that would go against the convention of V[t]V[t] and VthrV_\text{thr} being 0\geq 0. It would also complicate both the neuron design and hardware implementations thereof.

One option is to pre-process the sine wave by taking its absolute value, which would ensure that the input is always non-negative (as shown in Fig. x)

Let’s call this x[t]|x[t]| signal u1[t]u_1[t] and encode it with a single neuron (again with e=1e = 1), as done for x[t]x[t]. Then, the following spike train would be produced (Fig. 2b)

It can be easily inferred from the above that if another signal u2[t]u_2[t] with the same wave-form as the original signal u1[t]u_1[t] is encoded, then it would produce the same spike train as for x[t]|x[t]| in Fig. 2b. Note that pre-processing u2[t]u_2[t] by taking its absolute value will have no effect as u2[t]u_2[t] is already non-negative. Thus, there would be no difference between the spike trains obtained from the original sine wave x[t]x[t] and another non-negative signal u2[t]u_2[t] (after applying the same pre-processing step, i.e., absolute value operation).

Therefore, let us use another encoding neuron with e=1e = −1 to encode the original sine wave x[t]x[t]; analogous to the case of e=1e = 1, this would encode only the negative part of the sine wave, as can be seen in the Figs. 3a & 3b below:

Thus, if two neurons - one with e=1e = 1 and another with e=1e = −1 are used, then both the positive and the negative parts of the sine wave x[t]x[t] would be encoded faithfully, and no information would be lost, as shown in Fig. 4:

It is also noteworthy that if the signal u2[t]u_2[t] (of the same waveform as x[t]|x[t]|) is encoded via such a system of two neurons, then the neuron with e=1e = −1 would not spike at all, while the neuron with e=1e = 1 would spike and produce the spike train seen in Fig. 2b, thereby differentiating between the inputs x[t]x[t] and u2[t]u_2[t].

References
  1. Eliasmith, C., & Anderson, C. H. (2003). Neural engineering: Computation, representation, and dynamics in neurobiological systems. MIT press.
  2. Stewart, T. C. (2012). A technical overview of the neural engineering framework. University of Waterloo, 110.