Modeling Language, Cognition and Action

Preface.

List or Reviewers.

Contents.

This chapter introduces the research topic of the modeling language cognition and action in neural computation and psychology. It highlights the shift from classical connectionist simulations of isolated language and cognitive processes towards the modeling of autonomous cognitive systems in which linguistic, cognitive and sensorimotor abilities interact and co-exist. In the second part, a brief overview of the sections and individual chapters of the book is given.

Research from psycholinguistic investigations of the embodiment of language converges on two conclusions. First, models of real language use require flexible semantic features in contrast to a fixed set of features. Second, successful models cannot depend exclusively on frequency-based constraints. Instead, successful models of real language use will need to be sensitive to constraints imposed by bodily processes of perception, action, and emotion.

By using neurocognitive evidence on mirror neuron system concepts the MirrorBot project has developed neural models for intelligent robot behaviour. These models employ diverse learning approaches such as reinforcement learning, self-organisation and associative learning to perform cognitive robotic operations such as language grounding in actions, object recognition, localisation and docking. In this paper we describe architectures based on an associative self-organising framework which were designed to combine multimodal inputs of language, vision and motor programs to produce complex robot behaviours.

This paper presents a new connectionist model of spatial language based on real psycholinguistic data. It puts together various constraints on object knowledge (“what”) and on object localisation (“where”) in order to influence the comprehension of a range of linguistic terms, mirroring what participants do in experiments. The computational model consists of a vision processing module for input scenes, an Elman network module for the representation of object dynamics, and a dual-route network for the production of object names and linguistic prepositions describing the scene. Preliminary simulations on the prediction of spatial term ratings are presented, and extensions of the model to vague quantifiers and other syntactic categories are considered.

The Neural Theory of Language project aims to build structured connectionist models of language and cognition consistent with constraints from all domains and at all levels. These constraints include recent experimental evidence that details of neural computation and brain architecture play a crucial role in language processing. We focus in this paper on the computational level and explore the role of embodied representations and simulative inference in language understanding.

This article introduces a neural architecture which is capable of learning to use language in accordance with desires and to understand the intentions expressed by utterances of other people. We explain the architecture and describe simulation experiments we carried out in multi-agent game environments. We then discuss how compositionality is accomplished in our system. Finally, we relate the components of our architecture with the real brain by pointing out functional imaging and other experiments in which subjects had to carry out tasks related to the capabilities of our model.

The brain correlates of words and their referent actions and objects appear to be strongly coupled neuron ensembles or assemblies distributed over defined cortical areas. In this work we describe the implementation of a cell assembly-based model of several visual, language, planning, and motor areas to enable a robot to understand and react to simple spoken commands. The essential idea is that different cortical areas represent different aspects of the same entity, and that the long-range cortico-cortical projections represent hetero-associative memories that translate between these aspects or representations.

Using words to label categories is a true human universal. In addition to their public function in communication, labels may also serve private functions in shaping how concepts are represented. The present work explored the effects of assigning category labels on perceptual representations. A connectionist simulation is presented that examines the effects of labels on learning different types of categories. It is found that labels can augment perceptual information, and play an especially important role in shaping representations of entities whose perceptual features alone are insufficient for reliable classification.

The paper describes a neural network model of early language acquisition with an emphasis on how language positively influences the categories with which the child categorizes reality. Language begins when the two separate networks that are responsible for nonlinguistic sensory-motor mappings and for recognizing and repeating linguistic sounds become connected together at 1 year of age. Language makes more similar the internal representations of different inputs that must be responded to with the same action and more different the internal representations of inputs that must be responded to with different actions.

Two schemes for measuring the complexity of words (the Index of Phonetic Complexity and Analysis of Phonological Structure) were compared with regards to how well they predict which items in a stream of spontaneous speech produced by people who stutter, were likely to be dysfluent. Connectionist models were trained to output ‘stuttered’ or ‘fluent’ as a response to each word using either the IPC or the ANOPhS measure for that word. While the model trained using the IPC codes did not learn to discriminate between fluent and dysfluent items the ANOPhS model showed evidence of learning.

Comprehending a sentence requires the construction of a mental representation of the situation the sentence describes. Many researchers assume that, apart from such a situational representation, there is a level of representation at which the propositional structure of the sentence is encoded. This paper presents a simple sentence comprehension model, consisting of a neural network that transforms sentences into representations of the events they describe. During training, the network develops internal representations of the sentences. An investigation of these representations reveals that they can encode propositional information without implementing propositional structure.

We study how independent component analysis can be used to create automatically syntactic and semantic features based on analyzing words in contexts.

We study properties of morphemes by analyzing their use in a large Finnish text corpus using Independent Component Analysis (ICA). As a result, we obtain emergent linguistic representations for the morphemes. On a coarse level, main syntactic categories are observed. On a more detailed level, the components depict potential thematic roles of the morphemes. An interesting question is whether these discovered lower-dimensional representations could be directly utilized in language processing applications.

The Cognitive Linguistic Adaptive Resonant Network (CLAR-NET, Koutsomitopoulou 2004) is a biologically faithful neural network model of conceptual associations with input from English. Conceptual linguistic associations are analyzed as dynamic resonant patterns represented in terms of neuronal activation. The CLAR-NET model extends the line of research of Loritz 1999 to various linguistic phenomena in the realm of conceptual analysis, among which homonymy. This short presentation outlines the model using homonymy as an illustrative example.

We present simulations on how simple recurrent networks (SRNs) learn to represent prototypical centre-embedding and cross-serial languages, and analyse the emerging network dynamics. The networks find solutions for the tasks that can be characterised as simple dynamical systems and yield an explanation why cross-serial languages in an SRN setting are harder to aquire.

The present work considers the possibility that names play an active role in language, as a point of attention around which properties can be clustered. The contribution of neural network experiments of linguistic asks is here discussed in support of such an hypothesis.

We show how a population of simulated robots developed their communication capabilities in order to solve a collective navigation problem. The self-organized emergent vocabulary includes four different signals that influence both the motor and signalling behaviour of other robots. The analysis of the evolved behaviours also indicates: (a) the emergence of a simple form of communication protocol that allows individuals to switch signalling on and off, (b) the emergence of tightly co-adapted communicative and non-communicative behaviours, and (c) the exploitation of properties resulting from the dynamical interactions between motor and signalling behaviours produced by interacting robots.

Estimation of others' intention is essential for intelligent social behaviours. However, estimation of intention only from observation of another's actions is in general an ill-posed problem. Here, we propose a reinforcement MOSAIC framework for this difficult problem. We demonstrate the framework in a simulation of a cartpole swing-up task to show that the proposed estimation scheme can be used for imitation and cooperation.

The ability of building up mental representations of external situations to uncouple the behaviour from direct environmental control can be considered to be a prerequisite for cognitive, adaptive behaviour. The network we present can be used for generation of action but also – due to its attractor characteristics – as a basis for mental representations. In this context, a new learning algorithm is proposed, which leads to a self-organised weight distribution yielding stable states and allows building up several cell assemblies existing simultaneously within a larger network.

A preliminary model for the development of Spatial Cognition as Action (SCA) is proposed. Two propositions underpin the model, where intentional action is predominately driven by the nature of our understanding of the action in the world, and knowledge is considered in part as body schema in relation to the self, the other and interaction with artefacts, Expressive gesture-in-interaction is considered part of an emergent dynamic non-linear system. Neural network modelling techniques, video analysis and simulated animation were applied to the recognition of dynamic gesture of people with severe speech and motor impairment. This work re-examines the gestural corpora from an inter-modal interaction perspective. An action based annotation system uses feature sets that represent aspects of child spatial and kinaesthetic cognition. The model is informed by current thinking with regard to the motor theory, neural basis of language and evidence from imaging studies within a framework of narrative theory to understand spatial conceptual integration.

In this paper we will present the result of a set of experiments in which a robotic arm with a two-fingered hand is evolved for the ability to grasp objects. The robot is controlled by a neural network controller. Preliminary results demonstrate that evolutionary robotics techniques might scale up to situations involving robots with several degrees of freedom and problems that require an ability to produce sequential behaviour.

Hermer and Spelke (1994, 1996) have claimed that the merging of geometric (e.g the shape of the environment) and non-geometric information (e.g. the colour of landmarks) requires language. To support such an hypothesis, they consider evidences from children that, before language acquisition, can orient in space only using geometric cues. However, it has also been demonstrated that fish (Sovrano et al., 2002) are also able to integrate both kinds of cue in the same experimental setting. Here we propose the hypothesis that the observed stages in the development of spatial orientation abilities may reflect changes in the frequency of exposition to different classes of stimuli. We test this hypothesis using a genetic algorithm to train a population of simulated robots to solve the blue-wall task. By manipulating the frequency with which the robots come into contact with different classes of spatial information we show that, at specific frequencies, the sequence observed in the robots corresponds to the pattern observed in human beings.

In this paper, I will present the initial steps to a neurocognitive theory of semantic priming. This theory is applied to the debate on the automaticity of semantic activation and the modulatory role of attention in the spread of activation. I conclude with an outline of a programme of work that not only addresses the issue of semantic activation, but also relates to the internal structure of the semantic memory.

This paper describes a computational model of the Attentional Blink constructed using the dual-stage model proposed by Chun and Potter (1995) and also incorporating a token based account of working memory Kanwisher (1987). This model reproduces data from a number of blink paradigms and makes predictions that lag-1 sparing is temporal and not sequential in origin. A further prediction is that enhancing the distinctiveness of T2 can impair T1 performance and also provoke order inversions of T1 and T2. Experiments from our lab examined the validity of these predictions. Implications and results are discussed.

The biased competition hypothesis is of particular interest in the visual attention literature currently. We describe the first computational model to apply biased competition in active vision. At the cellular level, the model simulates both spatial and object-based attentional effects over time courses seen in single cell recordings in ventral stream areas. Such effects at the cellular level lead to systems level behaviour that replicates that observed during active visual search for orientation and colour conjunction targets, where the scan path is guided to target coloured locations in preference to locations containing the target orientation or blank areas.

One of the most controversial aspects of hemispatial neglect is its asymmetric nature. According to most theories of neglect, this asymmetry would depend on asymmetrical representations of space in the left and right hemisphere, although there is little agreement on their exact nature and their functional meaning. In the present paper, we show a series of neural networks simulations which implement different theories of neglect to verify their neuropsychological plausibility. The results show that asymmetries are best explained by a right hemispheric dominance for spatial representations, expressed in terms of a higher number of neurons involved in spatial tasks.

We describe an oscillatory neural model of attention that can be used to track a moving target among a set of distractors. The model works with a set of identical visual objects randomly moving on the screen. At the initial stage, the model selects into the focus of attention an arbitrary object that is considered as a target. Other objects are used as distractors. The model aims to preserve the target until one of the following events takes place: If a target moves outside the boundaries of the visual field, another target is selected in the focus of attention. If a target overlaps with a distractor, the focus of attention is spread over the composite object. After separation of the objects only one of them is kept in the attention focus. This scheme of attention focus formation and switching is implemented through a proper interplay of synchronizing and desynchronizing interactions in a neural network with a central element.

We analyzed the mechanisms for selective attention and action in an artificial, evolved agent. Our analysis shows the agent selectively responds to targets through reactive inhibition of nontarget items. Reactive inhibition, in which suppression of nontargets is in proportion to their salience, has been previously shown to be a means of selection in people. Our results suggest that reactive inhibition may be a fundamental process in selective control of action.

We describe a model of repetition blindness which draws on the dichotomous division of the visual system into two subsystems which process identity and location information. The model is constructed from self-organising networks of spiking neurons which are connected by plastic inhibitory and excitatory synapses. In particular, we describe how these networks are capable of learning translation invariant letter representations and learning to locate a stimulus in the input array using the SOM learning algorithm.

Our connectionist model provides a theoretical explanation for the existence of slow and fast emotional Stroop effects, and depicts them as independent but interacting phenomena. We build upon previous modelling work by Cohen et al (1990) and Botvinick et al (2001) among others, and incorporate data that suggest a functional division of the Anterior Cingulate Cortex (ACC) into Cognitive and Affective Divisions. This work suggests that slow emotional Stroop effects are caused by activation of the affective portion of the ACC, which inhibits the Cognitive division, reducing top-down cognitive control on the subsequent trial.

The last two decades have seen a great deal of theorising and speculation about the modular nature of human intelligence, as well as a rise in use of modular architectures in artificial intelligence. Nevertheless, whether such models of natural intelligence are well supported is still an issue of debate. In this paper, I propose that the most important criteria for modularity is specialised representations. I present a modular model of primate learning of the transitive inference task, and propose an extension to this model which would explain task-learning results in other domains. I also briefly relate this work to both neuroscience and established AI learning architectures.

A qualitative explanation for experimental observations of episodic, semantic and procedural memory and the differences between them is provided based on the different ways in which information derived from the senses is recorded and accessed in the recommendation architecture cognitive model.

It has been suggested that self-control can be represented as an internal conflict between the higher and lower centres of the brain. Based on this viewpoint we implemented a two-player model, simulating a top-level conscious processor for effortful reasoning, and an intuitive processor for heuristic problem solving, competing in a game theoretical situation. The results show that reducing the number of patterns of play promotes cooperation, which supports the theory that self-control can be explained in terms of learning to co-operate with oneself where behaviour variability is a factor.

‘Somatic marker’ theory proposes that body states act as a valence associated with potential choices based on prior outcomes; and thus aid decision-making. The main supporting evidence for this theory arose from clinical interviews of subjects with ventromedial prefrontal cortex (VMF) lesions and their performance on the Iowa ‘Gambling Task’ (IGT). VMF patient behaviour has been described as ‘myopia’ about future consequences. The aim of this paper is to investigate the implications of this description using an abstract simulation of the neural mechanisms that could underlie decision-making in this type of reinforcement learning task.

The automatic recognition of facial expressions of pain has potential medical significance: some patients are unable or unwilling to ask for analgesia, but it has been rather little studied. We report some preliminary work. The expressions were produced by actors. Human participants were able to distinguish the expressions intended to be pain from others with an average accuracy of around 80%, with disgust the most difficult to distinguish. A simple computer model consisting of a Gabor filtering front end and a linear discriminator network performed similarly well, suggesting that quite a simple model is able to emulate human performance.

Most computational models for gender classification use global information (the full face image) giving equal weight to the whole face area irrespective of the importance of the internal features. Here, we use a global and feature based representation of face images that includes both global and featural information. We use dimensionality reduction techniques and a support vector machine classifier and show that this method performs better than either global or feature based representations alone.

Quinn & Eimas (1996a) and Spencer, Quinn, Johnson & Karmiloff-Smith (1997) have shown that 3- to 4-month-old infants mainly use head information to categorize basic-level exemplars like cats and dogs. The question raised in this paper is the following: could this preference for head information be explained by the statistics of the input related to the head or was an attentional component necessary? In other words, it might be possible that the statistical information related to the head of the animals is more useful for a distributed model of visual categorization. We used the distributed model of visual categorization proposed by Mareschal & French (1997) & Mareschal, French & Quinn (2000) to confirm i) that the infants may well use mainly head information to create their category representations and ii) that this model, based on purely bottom-up processing, does not seem to be able to explain the preference for head-related information reported in 3- to 4-month-old infants. Finally, we suggest that an attentional focus on the head information might be necessary to simulate available behavioral data.

Children aged 3½ exhibit less backwards blocking effect than those aged 4½; the latter only are sensitive to probabilities (Sobel, Tenenbaum & Gopnik, 2004). The original account proposed by Sobel et al. (2004) is that children develop a mechanism for Bayesian structure learning. This account is problematic because it evades the explanation of the origins of the initial core of knowledge that is used by the posited Bayesian mechanism. I propose here an alternative explanation: Children’s differential performance stems from a memory limitation, with retroactive interference stronger in younger children, but adult-like in older children. This claim is supported by simulations with Ans and Rousset's (1997) memory self-refreshing neural networks architecture.

Attempts in the past to use evolutionary simulations to model the emergence of modular neural architectures has led to conflicting results, Here, we present some preliminary work on the use of computational embryogeny to model the evolution of neural architecture at a less coarse level of description. We believe that such an approach will lead to much more reliable and biologically realistic simulations of brain evolution.

Catastrophic forgetting is a well-known failing of many neural network systems whereby training on new patterns causes them to forget previously learned patterns. Humans have evolved mechanisms to minimize this problem, and in this paper we present our preliminary attempts to use simulated evolution to generate neural networks that suffer significantly less from catastrophic forgetting than traditionally formulated networks.

Human concepts are represented by neural circuits called Cell Assemblies (CAs). We have a simulator based on neurons that are fatiguing leaky integrators, with Hebbian learning rules to modify the synaptic strength. A neuron is recruited into a CA by having its synaptic weight both to and from the CA increased. This causes the neuron to fire whenever the CA is ignited, and this supports the increased synaptic strength. A neuron may be incorrectly recruited into a CA via spontaneous neural activation or overlapping CAs. This paper aims to resolve the problem of incorrectly recruited neurons. We need to find a mechanism that allows incorrectly recruited neurons to eventually leave a CA. One way to do this is to reduce the amount of energy that is retained. Unfortunately this reduces the persistence of a CA. Another way is to modify the fatigue mechanism. Modifying the fatigue mechanism so the fatigue is reduced proportionally to the current fatigue level reduces the likelihood that an incorrectly recruited neuron fires. In the long run this reduces the synaptic strength between it and the CA. This enables the removal of the neuron from the CA.

There have been many attempts to quantify visual similarity within different categories of objects, which a view to using such measures to predict impaired recognition performance. Although many studies have linked measures of visual similarity to behavioral outcomes associated with object recognition, there has been little research on whether these measures are associated with human ratings of perceived similarity. In this work, we compare similarity measures extracted from Principal Component Analysis, Isometric Feature Mapping and wavelets representations with ratings of human subjects. Our results show that features extracted by calculating the standard deviation of wavelet coefficients provides the closest fit to the human rating data of all the methods we applied here.

Enactive Distributed Associationism (EDA) is a simple and general purpose approach to the modeling of developmental learning, requiring no explicit supervision, specialization, or any artificially imparted symbolic base. These biologically plausible models implement unsupervised self-organization and exhibit ongoing classical conditioning leading to successful environmental adaptation in complex real environments. Producing multiple psychological phenomena, this approach is highly appropriate for cumulative cognitive modeling, especially where complex ontogenetically developing traits are to be modeled (such as phobic responses), Unlike many Artificial Life models, EDA provides a high level of interpretability and an understanding of the structures developed in relation to their formal psychological properties. This cognitive penetrability combines Transient Localism with an analogy to the localist symbolic networks to form an approach that is free from any assumptions regarding what is to be modeled.

Research on the psychology of reasoning has been dominated for the past 40 years by the deduction paradigm which assesses people's ability to reason logically. As a result, computational models developed to date implement syntactic or semantic forms of mental logic. More recently, however, theorists have given much greater emphasis to pragmatic influences on reasoning and developed the theory that human reasoning reflects the operation of two distinct cognitive systems. The challenges that this theory sets for computational modelling of human reasoning are identified and discussed.

Author Index.