Visual Analysis of Crowds forSocio-Cognitive Behaviors Understanding

  • Mohammad Sami Zitouni

Student thesis: Doctoral Thesis


Monitoring and inferring socio-cognitive behaviors through crowd analysis can help us to understand many processes. Be it people in crowded environments, road traffic or even a flock of fish, situational awareness becomes critical for creating adequate disaster response, providing incident management, exercising control, etc. Recent researches have indicated that crowd modeling is conventionally based on density analysis. However, socio-cognitive behavior studies have demonstrated that crowds often display a wide variety of behaviors that arise spontaneously from the collective motions of unconnected individuals. Therefore, behavior analysis employing physics-based approaches only, thereby neglecting the socio-psychological aspects, may present diverse challenges to accurate inference. Thus, by identifying and modeling some of the interacting agents that underpin the evolution of such behaviors, we can deliver contexts that can help in the autonomous analysis of diversified behaviors in crowded environments. In this thesis, first a survey of recent works on crowd behavior analysis is conducted. The proposed behavioral categorization is used, based on the identified generalized behaviors taking into consideration the social aspects. Subsequently, mathematical relations between the modeling techniques and the behavior types are formulated using a probabilistic model of the proposed categories. Then, the crowd structure and its components (i.e. individuals and groups) are identified by detection and tracking techniques. This is used to model in a hierarchical manner relationships between the crowd components to differentiate between various behaviors and to verify validity of the probabilistic model. Additionally, an alternative CNN-based method is presented for extracting and classifying crowd components where automatic annotations of ground-truth data are obtained. Then, a data association scheme is proposed where detection and tracking results at both the individualistic and group levels are complementarily used to smoothly maintain data associations between subsequent video-frames, and to identify interactions between crowd components. Finally, a number of behavioral features are introduced for socio-cognitive behaviors classification. These features are estimated from detection and association results for crowd components, incorporating the corresponding probabilistic aspects. The features are calculated for the temporal patches and used to train a neural network to identify the socio-cognitive behavior types. The presented work can be considered a step towards a comprehensive framework that is not limited to a specific application, and can be applied for various scenarios and tasks of socio-cognitive behavior analysis of crowds.
Date of AwardSep 2019
Original languageAmerican English


  • Crowd Analysis
  • Behavior Understanding
  • Socio-cognitive Classification
  • Detection & Tracking
  • Probabilistic Models
  • Neural Networks.

Cite this