• Humanoid robot ‘Dimitri’ with elastic coupling joints which can tolerate with physical interaction with environment and human during long-term developmental tutoring processes.


    Ahmadreza Ahmadi, Jun Tani and Collaboration with Rodrigo da Silva Guerra(Universidade Federal de Santa Maria)

  • Predictive Coding for Dynamic Vision: Development of Functional Hierarchy in a Multiple Spatio-Temporal Scales RNN Model


    Authors: Minkyu Choi and Jun Tani

    Abstract:
    The current paper presents a novel recurrent neural network model, the predictive multiple spatio-temporal scales RNN (P-MSTRNN), which can generate as well as recognize dynamic visual patterns in the predictive coding framework. The model is characterized by multiple spatio-temporal scales imposed on neural unit dynamics through which an adequate spatio-temporal hierarchy develops via learning from exemplars. The model was evaluated by conducting an experiment of learning a set of whole body human movement patterns which was generated by following a hierarchically defined movement syntax. The analysis of the trained model clarifies what types of spatio-temporal hierarchy develop in dynamic neural activity as well as how robust generation and recognition of movement patterns can be achieved by using the error minimization principle.

    Choi, M., & Tani, J. (2016). Predictive Coding for Dynamic Vision: Development of Functional Hierarchy in a MultipleSpatio-Temporal Scales RNN Model. arXiv preprint arXiv:1606.01672. [PDF]     [VIDEO]   

  • Deep Learning for Dynamic Vision:


    Authors: Minju Jung, Jungsik Hwang, and Jun Tani

    Abstract:
    It is well known that the visual cortex efficiently processes high-dimensional spatial information by using a hierarchical structure. Recently, computational models that were inspired by the spatial hierarchy of the visual cortex have shown remarkable performance in image recognition. Up to now, however, most biological and computational modeling studies have mainly focused on the spatial domain and do not discuss temporal domain processing of the visual cortex. Several studies on the visual cortex and other brain areas associated with motor control support that the brain also uses its hierarchical structure as a processing mechanism for temporal information. Based on the success of previous computational models using spatial hierarchy and temporal hierarchy observed in the brain, the current report introduces a novel neural network model for the recognition of dynamic visual image patterns based solely on the learning of exemplars. This model is characterized by the application of both spatial and temporal constraints on local neural activities, resulting in the self-organization of a spatio-temporal hierarchy necessary for the recognition of complex dynamic visual image patterns. The evaluation with the Weizmann dataset in recognition of a set of prototypical human movement patterns showed that the proposed model is significantly robust in recognizing dynamically occluded visual patterns compared to other baseline models. Furthermore, an evaluation test for the recognition of concatenated sequences of those prototypical movement patterns indicated that the model is endowed with a remarkable capability for the contextual recognition of long-range dynamic visual image patterns.

    Jung, M., Hwang, J., & Tani, J. (2015). Self-organization of spatio-temporal hierarchy via learning of dynamic visual image patterns on action sequences. PLoS One 10(7): e0131214. doi: 10.1371/journal.pone.0131214. [PDF]     [VIDEO]    [SOURCE CODE]

  • Achieving “Synergy” in Cognitive Behavior of Humanoids via Deep Learning of Dynamic Visuo-Motor-Attentional Coordination

    Authors: Jungsik Hwang, Minju Jung, Naveen Madapana, Jinhyung Kim, Minkyu Choi and Jun Tani

    Abstract:
    The current study examines how adequate coordination among different cognitive processes including visual recognition, attention switching, action preparation and generation can be developed via learning of robots by introducing a novel model, the Visuo-Motor Deep Dynamic Neural Network (VMDNN). The proposed model is built on coupling of a dynamic vision network, a motor generation network, and a higher level network allocated on top of these two. The simulation experiments using the iCub simulator were conducted for cognitive tasks including visual object manipulation responding to human gestures. The results showed that “synergetic” coordination can be developed via iterative learning through the whole network when spatio-temporal hierarchy and temporal one can be self-organized in the visual pathway and in the motor pathway, respectively, such that the higher level can manipulate them with abstraction.
    [VIDEO]    [PDF]   



  • Neuro-Robotics toward general Intelligence

    This project aims to understand the essential brain mechanisms for the higher-order cognition by synthesizing through robotics experiments. We build so-called the large scale brain network (LSBN) model by utilizing available data for connectivity matrix among local regions in the human brain under collaboration with Prof. Daeshik Kim in KAIST. A challenge is to show that humanoid robots can adapt to a wide range of cognitive tasks by flexibly combining various cognitive resources developed in the LSBN. Here, the aim is not just to train robots to be good at a particular cognitive task, but to educate them to be good at various cognitive tasks simultaneously where general intelligence would be required. The project may start with focusing on the main cortical areas including the prefrontal cortex, the premotor cortex, the temporal cortex, the parietal cortex and the sensory peripheral regions. It is presumed that general intelligence would appear in adequate harmony among those brain regions under specific connectivity. The project will utilize iCub as a humanoid robot platform.

  • Social Cognitive Neuro-Robotics

    This project aims to explore possible neuropsychological mechanisms for social cognition by conducting a set of experiments for robot-human as well as robot-robot interactions. The major research question is how human and robots can be cooperative each other in particular task by reading others' minds as well as by establishing joint attentions. Emergence of cooperative behaviors among individuals is not trivial phenomena because each individual possesses independent volition or free will. We speculate that cooperation could emerge by achieving a sort of coherence among neuronal activities of individuals by means of the mirror neuron mechanisms of reading others intentions. By conducting neuro-robotics experiments, we examine the underlying dynamic mechanisms accounting for spontaneous generation of cooperative behaviors as well as their breakdown. (This project is conducted collaboratively with Ogata and Sugano labs in Waseda University.) [PDF]     [VIDEO]    











  • Exploring Nontrivial Dynamic Computation by Recurrent Neural Network (RNN)

    Our group has investigated how nontrivial computation accounting for some aspects of the higher-order cognitive functions can be achieved in dynamics of RNN models. We have shown that a version of RNN model can function as a generative model which can both generate and recognize a set of temporal patterns via learning of the exemplar (see more here). We have also shown that multiple timescale dynamics characteristics introduced in RNN models can afford self-organization of functional hierarchy such that the fast dynamics part tends to form a set of primitive dynamics while the slow dynamics part does for macroscopic manipulation of those primitives (see more here). These results suggest that meaningful computation in terms of its compositionality and systematicity can be effectively developed in those RNN models. By extending these studies, the current project will examine how a particular RNN model can learn to extract stochastic structures hidden in observed continuous spatio-temporal patterns. We will investigate this problem by utilizing mathematical theory of chaos and symbolic dynamics. By challenging an essential mathematical question, deterministic dynamic systems (including chaos) versus stochastic systems for modeling and reconstructing stochastic complex signals, we attempt to show that there exists the middle between these two extremes. The model will be applied for recognition and regeneration of video as well as auditory streams with expectation of more fluid and contextual processing of the streams as compared to the conventional Bayesian approaches including Hidden Markov Model.



  • NeuroInteractive Arts Projects

    This project explores novel experiences of human participants while directly interacting with emergent dynamics developed in artificial brains. The continuous interactions take place between the participants and the artificial brains in a multimodal manner by utilizing haptic devices including an active force-control joystick and a vibrotactile stimulation system as well as auditory and visual streams while the artificial brains learn and regenerate the sensory images experienced in the past. The project will examine what sorts of interactive dynamics can appear between the two sides while the participants mental states go back and forth between unconscious and conscious states. The goal of the project is to find out if this type of interactive dynamics could bring new types of phenomenological experiences for human participants.