Publications

In the natural world, the statistics of sensory stimuli fluctuate across a wide range. In theory, the brain could maximize information recovery if sensory neurons adaptively rescale their sensitivity to match their limited response bandwidth to the current range of inputs. Such adaptive coding has been observed in a variety of systems, but the premise that adaptation optimizes behavior has not been tested. Here we show that adaptation in cortical sensory neurons maximizes information about visual motion, and minimizes tracking errors in pursuit eye movements guided by that cortical activity. Thus efficient sensory coding is not simply an ideal standard but rather a compact description of real sensory computation that manifests in improved behavioral performance. The theory of efficient coding is linked to the idea that neural systems maximize information relevant to behavioral performance that can influence survival. Observations of neural responses in many organisms have demonstrated a capacity for efficient coding, but the consequences for behavior have not been explored. In our work, we demonstrate for the first time that efficient coding applies to a neural system as a whole, improving the accuracy of the movements it generates, and not solely to individual sensory neurons. We have exploited the close connection between cortical motion estimates and smooth pursuit eye movements to demonstrate parallel adaptation effects in sensory neurons and movement behavior. We find that adaptation to motion variance optimizes the encoding of motion information by MT neurons, with a behavioral impact of optimizing information in pursuit eye movements, minimizing visual tracking errors, and thereby improving vision of moving objects.

Delays in sensory processing give rise to a lag between a stimulus and the organism’s reaction. This presents a particular challenge to tracking behaviors like smooth pursuit where a difference in eye and target motion creates image motion blur on the retina. One strategy that might compensate for processing delays is to extrapolate the future target position and make anticipatory eye movements. We recorded eye position in humans and monkeys engaged in tracking tasks to test for predictive information in eye movements. We used tasks with randomized jumps and direction changes (1D and 2D) and we created a Pong-like video game in which the subject moves a paddle to keep a ball bouncing within an arena. Using information theory, we show that in most tracking tasks, gaze behavior is predictive on short ~200ms timescales. But while playing Pong, prediction extends over several seconds and is very close to the bound imposed by the predictability of target motion. We also develop a model of short time scale prediction based on retinal inputs that accounts for decisions to saccade or pursuit during tracking. We find that the brain incorporates prediction in gaze behavior in a context dependant manner. Prediction on short time scales (~200-400ms) helps to compensate for visual delays. A model based on extrapolation of retinal inputs (eye crossing velocity) predicts gaze decisions on shorter time scales (i.e. decision to saccade or pursue). Active tasks like playing Pong produce predictive movements on 5-10x longer time scales, and are nearly optimal. The next step is to model longer timescale gaze patterns (i.e. continuous gaze behavior) during Pong, and to consider patterns of motion over time in the analysis of predictive information.

Are sensory estimates formed centrally in the brain and then shared between perceptual and motor pathways, or is centrally represented sensory activity decoded independently to drive awareness and action? Questions about the brain's information flow pose a challenge because systems-level estimates of environmental signals are only accessible indirectly as behavior. Assessing whether sensory estimates are shared between perceptual and motor circuits requires comparing perceptual reports with motor behavior arising from the same sensory activity. Extrastriate visual cortex mediates both the perception of visual motion and it provides the visual inputs for behaviors like smooth pursuit eye movements. Pursuit has been a valuable testing ground for theories of sensory information processing because the neural circuits and physiological response properties of motion-responsive cortical areas are well-studied, sensory estimates of visual motion signals are formed quickly, and the initiation of pursuit is closely coupled to sensory estimates of target motion. Here we analyze variability in visually-driven smooth pursuit and perceptual reports of target direction and speed in human subjects while we manipulate the signal to noise level of motion estimates. Comparable levels of variability throughout viewing time and across conditions provide evidence for shared noise sources in the perception and action pathways arising from a common sensory estimate. We find that conditions that create poor, low-gain pursuit create a discrepancy between the precision of perception and that of pursuit. Differences in pursuit gain arising from differences in optic flow strength in the stimulus reconcile much of the controversy on this topic.

Direction and speed tracking errors during pursuit initiation are correlated. Eye direction-speed covariation is not predicted by vector-averaging decoding models or by the 2D structure of tuning in MT neurons. Despite the separability of MT tuning functions, direction and speed are encoded synergistically in single unit responses, and thus more information can be decoded about theta,v jointly (the motion vector) than the sum of information about the components individually. The synergy in direction-speed coding in MT arises from the tuning function itself and a Poisson model generates the same “synergy”. An analysis of two-spike patterns shows that MT neurons can be truly synergistic and encode more information with temporal patterns than in the firing rate. We conclude that a “ motion vector” rather than “component” (scalar) decoding model may recover more motion information and better predict behavioral data.

When an animal performs a visual tracking task, inherent temporal delay between target and eye motion is not avoidable. It is due to early sensory processing in the visual system and this delay introduces large position errors during the visual tracking of a stimulus that changes its trajectory abruptly. Therefore, smooth and relatively slow eye movements accumulate position and velocity errors as the eye tries to catch up to a fast moving object. One strategy that the brain might use to shorten the feedback loop is to extrapolate the future target position and make anticipatory eye movements. We studied whether neurons in the higher cortical area, in MT encode the future motion of the stimulus and how this information is carried out by anticipatory smooth eye movement. We find that Neurons in area MT and eye movements encode the future state of visual motion input. The timescale of prediction is equal to the temporal correlation of its visual motion input for neurons in cortical area MT and eye movement.

Despite the enduring interest in motion integration, a direct measure of the space-time filter that the brain imposes on a visual scene has been elusive. This is perhaps because of the challenge of estimating a three-dimensional function from perceptual reports in psychophysical tasks. We take a different approach. We exploit the close connection between visual motion estimates and smooth pursuit eye movements to measure stimulus-response correlations across space and time, computing the linear space-time filter for global motion direction in humans and monkeys. Although derived from eye movements, we find that the motion filter predicts perceptual motion estimates quite well. To test contributions of motor processing to the temporal duration of the motion filter, we recorded single unit responses in the monkey middle temporal cortical area (MT). We find that the duration of motion filter is consistent with motion integration by single cortical neurons, but that a pursuit-specific process determines the peak time delay. Remarkably, the visual system appears to pool motion signals across only a narrow range of foveal eccentricities rather than over the whole visual field, with a transiently enhanced contribution from the spatial segment in direction of motion. We find that the visual system is most sensitive to motion falling at roughly ½ the radius of the stimulus aperture. Hypothesizing that the visual drive for pursuit is related to the filtered motion energy in a motion stimulus, we compared measured and predicted eye acceleration across several other target forms.

Prediction is one of the fundamental problems in neural computation. Much of what we admire in expert human performance is predictive in character such as the point guard who passes the basketball to a place where his teammate will arrive in a split second, or the investor who buys a stock in anticipation that it will grow in the year to come. More generally, we gather sensory information not for its own sake but in the hope that this information will guide our actions. But acting takes time, and sense data can guide us only to the extent that those data inform us about the state of the world at the time of our actions. In short we learn the likelihood of future events and use those predictions to guide behavior. There are limits on the accuracy of prediction, and here we ask how close organisms come to optimal performance. We use pursuit eye movements, along with perceptual reports, as a model system in which to explore the learning of probabilities from examples. We created probability landscapes defined by the correlation in target turns over trials. We also find that both eye movements and reports of future target turns are modulated with the target odds over trials. We find that learning strategy is adaptive. The memory of past trials scales linearly with the correlation time of the probability landscape. Our data do not support the hypothesis that there is a central learning processes shared between perception and action pathways. Perceptual judgments and anticipatory eye movements are not highly correlated across trials and have different discrimination thresholds for changes in target motion odds.

Motion sensitive cortical neurons encode time-varying motion signals efficiently by rapidly adapting the gain of their responses to changes in stimulus variance. Neural sensitivity to fluctuations in motion (direction or velocity) is therefore context-dependent. When the variance of motion signals shifts, MT neurons rescale their gain to maintain a similar distribution of firing rates, avoid saturation, and optimize motion contrast sensitivity. Pursuit eye movement behavior -visual tracking of a moving target - benefits from efficient cortical coding of motion. When target motion variance is low, pursuit and cortex become more sensitive to perturbations and vice versa such that the information encoded about motion remains constant. The gain adaptation is rapid. Shifts in the joint distribution of stimulus and response are substantial within 20ms. Detecting the variance step from neural or pursuit responses within single trials requires a longer integration of sensory evidence. We find that these gain changes cannot be accounted for by saturation - they occur when the highest observed firing rate is below the peak firing rate of the neuron (or eye velocity), however this work does not identify a mechanism. These data suggest that feature selective cortical areas are themselves capable of efficient sensory coding and that efficiencies in cortical coding can be relevant to behavioral performance.

Pursuit demonstrates that the brain encodes time-varying motion signals efficiently. Changes in both the linear gain and in the information encoded about target motion are consistent with the system preserving information via gain adaptation. When target motion variance is low, pursuit becomes more sensitive to perturbations and vice versa such that the information encoded about motion remains constant. These results demonstrate a behavioral benefit for efficient sensory coding. Physiology data show that adaptation in MT cortical neurons drives adaptation in pursuit. MT neurons, like pursuit, become less sensitive to target direction fluctuations as motion variance increases, consistent with Barlow’s theory of sensory efficiency. Furthermore, this gain adaptation rescales rapidly after a step change in motion direction variance. We compared the eye direction distribution (or firing rate) in a 20 ms window preceding a step to that immediately following a step. The gain change is already apparent in the first time window after the step as a shift in the distribution of response vs stimulus values. These data suggest that feature selective cortical areas are themselves capable of efficient sensory coding and that efficiencies in cortical coding are relevant to behavioral performance.

Analysis of sensory-motor behaviors can be very useful for increasing our understanding of the computations being performed by circuits within the brain. Visual smooth pursuit, a task in which eyes move to stabilize the retinal image of a target, provides an ideal testbed for this type of analysis, as minimal motor noise is added to initial sensory estimates of an object's retinal image motion. Neurons in the middle temporal area (MT) of the primate extrastriate visual cortex have been shown to be selectively responsive to retinal image motion, and exhibit direction and speed preferential spike tuning. Here, we analyze the structure of smooth pursuit eye movements, and compare the results to the structures of estimates of decoding methods based on responses from model MT units. These methods differ in how the variance of neural responses is transformed as visual information propagates through the visual system, and we can use this property to determine which methods may be describing computations that the visual system is actually performing. We find that the correlation between direction and speed error at the end of smooth pursuit is significantly higher at oblique directions than at cardinal directions. Existing population encoding models, including maximum likelihood and multiple forms of vector averaging, fail to capture this phenomenon. Although a motor noise model adds signal-dependant variance to horizontal and vertical speed components, it recreates the oblique pattern of correlations between direction and speed noise, it also adds directional structure to direction and speed variance.

In order to stabilize a moving target’s retinal image, the brain must make continuous visual estimates of target motion and evaluate the trade-off between smoothly modulating eye movement and initiating a saccade. Smooth pursuit eye movement is used for continuous acquisition of visual information, whereas saccadic movement can reduce a large retinal error in a short period of time. Lefèvre and colleagues (2002) introduced the decision rule between pursuit and saccades (Eye-Crossing Time) during one dimensional visual tracking of moving stimuli in humans. Our goal for this study is to expand this notion to investigate if there exists general oculomotor computation for making eye movement decision. In order to achieve this goal, three different experimental paradigms have been done in human and monkeys: a 1D and 2D visual tracking with double step-ramp and a single-player version of the video game, Pong. Interestingly, we observed that in the highly predictive situation, such as during the pong game, saccadic eye movement is not captured with the same rule. Here, we apply information theoretic analysis to quantify the interaction between target, gaze, and time. We find that eye crossing time (TxE), the negative fraction of position error and the retinal slip, describes the likelihood of saccades and pursuit in both 1-D and 2-D visual tracking procedure. The predictability of target movement allows the brain to extrapolate future target position to influence gaze decision rules in both humans and non-human primates’.

Performance in sensory-motor behaviors guides our understanding of many of the key computational functions of the brain: the representation of sensory information, the translation of sensory signals to commands for movement, and the production of behavior. Eye movement behaviors have become a valuable testing ground for theories of neural computation because the neural circuitry has been well characterized and the mechanical control of the eye is comparatively simple. Here I review recent studies of eye movement behaviors that provide insight into sensory-motor computation at the single neuron and systems levels. They show that errors in sensory estimation dominate eye movement variability and that the motor system functions to reduce the behavioral impact of its own intrinsic noise sources.

Synaptic transmission involves the calcium dependent release of neurotransmitter from synaptic vesicles. Genetically encoded optical probes emitting different wavelengths of fluorescent light in response to neuronal activity offer a powerful approach to understand the spatial and temporal relationship of calcium dynamics to the release of neurotransmitter in defined neuronal populations. To simultaneously image synaptic vesicle recycling and changes in cytosolic calcium, we developed a red-shifted reporter of vesicle recycling based on a vesicular glutamate transporter, VGLUT1-mOrange2 (VGLUT1-mOr2), and a presynaptically localized green calcium indicator, synaptophysin-GCaMP3 (SyGCaMP3) with a large dynamic range. The fluorescence of VGLUT1-mOr2 is quenched by the low pH of synaptic vesicles. Exocytosis upon electrical stimulation exposes the luminal mOr2 to the neutral extracellular pH and relieves fluorescence quenching. Reacidification of the vesicle upon endocytosis again reduces fluorescence intensity. Changes in fluorescence intensity thus monitor synaptic vesicle exo- and endocytosis, as demonstrated previously for the green VGLUT1-pHluorin. To monitor changes in calcium, we fused the synaptic vesicle protein synaptophysin to the recently improved calcium indicator GCaMP3. SyGCaMP3 is targeted to presynaptic varicosities, and exhibits changes in fluorescence in response to electrical stimulation consistent with changes in calcium concentration. Using real time imaging of both reporters expressed in the same synapses, we determine the time course of changes in VGLUT1 recycling in relation to changes in presynaptic calcium concentration. Inhibition of P/Q- and N-type calcium channels reduces calcium levels, as well as the rate of synaptic vesicle exocytosis and the fraction of vesicles released.

What fascinates us about animal behavior is its richness and complexity, but understanding behavior and its neural basis requires a simpler description. Traditionally, simplification has been imposed by training animals to engage in a limited set of behaviors, by hand scoring behaviors into discrete classes, or by limiting the sensory experience of the organism. An alternative is to ask whether we can search through the dynamics of natural behaviors to find explicit evidence that these behaviors are simpler than they might have been. We review two mathematical approaches to simplification, dimensionality reduction and the maximum entropy method, and we draw on examples from different levels of biological organization, from the crawling behavior of Caenorhabditis elegans to the control of smooth pursuit eye movements in primates, and from the coding of natural scenes by networks of neurons in the retina to the rules of English spelling. In each case, we argue that the explicit search for simplicity uncovers new and unexpected features of the biological system and that the evidence for simplification gives us a language with which to phrase new questions for the next generation of experiments. The fact that similar mathematical structures succeed in taming the complexity of very different biological systems hints that there is something more general to be discovered.

To probe how the brain integrates visual motion signals to guide behavior, we analyzed the smooth pursuit eye movements evoked by target motion with a stochastic component. When each dot of a texture executed an independent random walk such that speed or direction varied across the spatial extent of the target, pursuit variance increased as a function of the variance of visual pattern motion. Noise in either target direction or speed increased the variance of both eye speed and direction, implying a common neural noise source for estimating target speed and direction. Spatial averaging was inefficient for targets with >20 dots. Together these data suggest that pursuit performance is limited by the properties of spatial averaging across a noisy population of sensory neurons rather than across the physical stimulus. When targets executed a spatially uniform random walk in time around a central direction of motion, an optimized linear filter that describes the transformation of target motion into eye motion accounted for approximately 50% of the variance in pursuit. Filters had widths of approximately 25 ms, much longer than the impulse response of the eye, and filter shape depended on both the range and correlation time of motion signals, suggesting that filters were products of sensory processing. By quantifying the effects of different levels of stimulus noise on pursuit, we have provided rigorous constraints for understanding sensory population decoding. We have shown how temporal and spatial integration of sensory signals converts noisy population responses into precise motor responses.

We have used a combination of theory and experiment to assess how information is represented in a realistic cortical population response, examining how motion direction and timing is encoded in groups of neurons in cortical area MT. Combining data from several single-unit experiments, we constructed model population responses in small time windows and represented the response in each window as a binary vector of 1s or 0s signifying spikes or no spikes from each cell. We found that patterns of spikes and silence across a population of nominally redundant neurons can carry up to twice as much information about visual motion than does population spike count, even when the neurons respond independently to their sensory inputs. This extra information arises by virtue of the broad diversity of firing rate dynamics found in even very similarly tuned groups of MT neurons. Additionally, specific patterns of spiking and silence can carry more information than the sum of their parts (synergy), opening up the possibility for combinatorial coding in cortex. These results also held for populations in which we imposed levels of nonindependence (correlation) comparable to those found in cortical recordings. Our findings suggest that combinatorial codes are advantageous for representing stimulus information on short time scales, even when neurons have no complicated, stimulus-dependent correlation structure.

To evaluate the nature and possible sources of variation in sensory-motor behavior, we measured the signal-to-noise ratio for the initiation of smooth-pursuit eye movements as a function of time and computed thresholds that indicate how well the pursuit system discriminates small differences in the direction, speed, or time of onset of target motion. Thresholds improved rapidly as a function of time and came close to their minima during the interval when smooth eye movement is driven only by visual motion inputs. Many features of the data argued that motor output and sensory discrimination are limited by the same noise source. Pursuit thresholds reached magnitudes similar to those for perception: <2-3 degrees of direction, approximately 11-15% of target speed, and 8 ms of change in the time of onset of target motion. Pursuit and perceptual thresholds had similar dependencies on the duration of the motion stimulus and showed similar effects of target speed. The evolution of information about direction of target motion followed the same time course in pursuit behavior and in a previously reported sample of neuronal responses from extrastriate area MT. Changing the form of the sensory input while keeping the motor response fixed had significant effects on the signal-to-noise ratio in pursuit for direction discrimination, whereas holding the sensory input constant while changing the combination of muscles used for the motor output did not. We conclude that noise in sensory processing of visual motion provides the major source of variation in the initiation of pursuit.

Suppose that the variability in our movements is caused not by noise in the motor system itself, nor by fluctuations in our intentions or plans, but rather by errors in our sensory estimates of the external parameters that define the appropriate action. For tasks in which precision is at a premium, performance would be optimal if no noise were added in movement planning and execution: motor output would be as accurate as possible given the quality of sensory inputs. Here we use visually guided smooth-pursuit eye movements in primates as a testing ground for this notion of optimality. In response to repeated presentations of identical target motions, nearly 92% of the variance in eye trajectory can be accounted for as a consequence of errors in sensory estimates of the speed, direction and timing of target motion, plus a small background noise that is observed both during eye movements and during fixations. The magnitudes of the inferred sensory errors agree with the observed thresholds for sensory discrimination by perceptual systems, suggesting that the very different neural processes of perception and action are limited by the same sources of noise.

We used the responses of neurons in extrastriate visual area MT to determine how well neural noise can be reduced by averaging the responses of neurons across time. For individual MT neurons, we calculated the time course of Shannon information about motion direction from sustained motion at constant velocities. Stimuli were random dot patterns moving at the preferred speed of the cell for 256 msec, in a direction chosen randomly with 15 degrees increments. Information about motion direction calculated from cumulative spike count rose rapidly from the onset of the neural response and then saturated, reaching 80% of maximum information in the first 100 msec. Most of the early saturation of information could be attributed to correlated fluctuations in the spike counts of individual neurons on time scales in excess of 100 msec. Thus, temporal correlations limit the benefits of averaging across time, much as correlations among the responses of different neurons limit the benefits of averaging across large populations. Although information about direction was available quickly from MT neurons, the direction discrimination by individual MT neurons was poor, with mean thresholds above 30 degrees in most neurons. We conclude that almost all available directional information could be extracted from the first few spikes of the response of the neuron, on a time scale comparable with the initiation of smooth pursuit eye movements. However, neural responses still must be pooled across the population in MT to account for the direction discrimination of the pursuit behavior.