Introduction to Behavioral Imaging
CS 8803/4803 IBI

Spring 2016
MWF 2:05-2:55 pm
Klaus 1447

This course will provide an introduction to Behavioral Imaging, a new research field which encompasses the measurement, modeling, analysis, and visualization of behaviors from multi-modal sensor data. It is tailored for undergraduate and graduate students who are interested in this emerging field. The course is designed to provide:

  • A broad introduction to research questions in Behavioral Imaging (BI)
  • An in-depth understanding of the key technologies for BI
  • Hands-on experience in working with relevant sensor data (video, audio, wearable)
  • An overview of the psychology literature relating to behavior from a computational perspective

Beginning in infancy, individuals acquire the social and communicative skills which are vital for a healthy and productive life, through face-to-face interactions with caregivers and peers. However, children with developmental delays face great challenges in acquiring these skills, resulting in substantial lifetime risks. The goal of research in Behavioral Imaging is to develop computational methods that can support the fine-grained and large-scale measurement and analysis of social behaviors, with the potential to positively impact the diagnosis and treatment of developmental disorders such as autism.

Other medical domains can also benefit from a computational study of behavior. One example is chronic health conditions, in which health-related behaviors such as smoking or unhealthy eating habits play an important role. The ability to treat chronic conditions is hampered by an inability to reliably measure health-related behaviors, particularly in naturalistic (field) conditions. Advances in wearable sensing technologies have the potential to enable a new data-driven approach to the treatment of such conditions.


Jim Rehg
Office hours: TBD


As this is a new research area, there are no textbooks that cover the breadth of this course. Recommended readings will be provided for all lectures and class discussions.


This class will be self-contained with respect to the core concepts and class material. Background material in psychology and signal anlaysis will be provided, so there are no subject-matter prerequisites. Some of the projects will require either familiarity with Matlab and access to Matlab on a computer, or the ability to run precompiled apps and/or libraries. Extensive Matlab programming will not be required. A basic understanding of signal and image processing will be useful but not required.


The class will consist of both lectures and discussions of papers drawn from the relevant literature. Each discussion period will be led by a team of students and will cover 1 or 2 or more papers. (The exact details will be finalized based on the class size and paper contents.) All students are expected to read the papers before each meeting and participate in the class discussion. The course material is divided into units, and each unit will feature some combination of lectures on technology and psychology topics, and discussion periods covering technology and psychology papers. Each student will the have the chance to lead the discussion on at least two papers during the semester, one in technology and one in psychology. There will be a final project for which the class will work in teams and will address one broad area of BI technology.

  • 1 Presentation: 20%
  • Two Projects: 40% (20% x 2)
  • Final Project: 25%
  • Participation: 15%

There will be two projects that will give you hands-on experience in collecting a particular type of behavioral signal and analyzing it to extract relevant behavioral information. We will provide you with some existing analysis tools in each case.

Project 1: First Person Vision

For this project, each team will be provided with a GoPro Session (wearable camera) and the goal is explore the analysis of facial information during a face-to-face social interaction. You will make use of the IntraFace face analysis toolkit to perform analysis of the video you captured. It provides the capability to detect faces and track facial landmarks, such as eyes, mouth, etc. More details will be provided in the project writeup.

Project 2: Wearable Activity Recognition

For this project, each team will be issued a Microsoft Band 2 wrist sensor platform. The goal is to explore the capture and analysis of body-worn physiological sensors such as accelerometers/gyroscopes. You will be provided with software for capturing and analyzing signals from the band. You will have to design and implement an analysis approach for recognizing a behavior in the field, such as drinking, eating, physical activity, etc. More details will be provided in the writeup.

Final Project

Each team will be allowed to define their own final project. You may choose any topic related to the class material, including on-going research projects that you may be involved in, and we can help you to identify a suitable project based on your interests. It is also possible to simply continue the analyses you conducted in either of the two projects and define an extended analysis task that constitutes a final project. You will be able to make use of the various software tools that we will be providing throughout the semester.

Final Exam Period

Final project presentations will take place during final exam week, at a time TBD.

Final Project Presentation Requirements

Each team will have TBD minutes to present the outcome of their final project activities to the class during the Final Exam Period. All students are expected to attend the Project Presentation Period in its entirety. Presentations will consist of powerpoint slides or their equivalent, and multiple team members may present (as long as the presentation does not exceed the allowed time). Since questions and discussion are an important part of the presentation process, you should prepare a 15-20 minute presentation and reserve the remaining time for discussion. Your presentation should cover the following topics:

  • What was the goal of your project?
  • What was your approach and what methods did you develop?
  • How did you evaluate the success of your approach?
  • Which methods that you developed worked well, and which ones did not?
  • What was the primary reason for the success or failure of each method?
  • What are the fundamental challenges in your project?
  • If you were to continue to work on this problem in the future, what would be the most promising approaches to try next?
  • Your presentation should include a well-chosen set of video clips to illustrate the successes and failures of the methods you developed, including intermediate results from algorithms that will make it possible to understand your final outputs.

Please take the preparation of these slides seriously, as they will impact your final grade in the course (see below). It is not necessary to submit the slides in advance of the final project period. However, be sure that you bring a copy of your slides on a USB drive in case there is a problem with your laptop on the day of the presentation. You are encouraged to test your laptop in advance to make sure there are no issues with the projector.

Final Project Grading

A well-prepared and well-presented set of final project slides is required for a maximum grade on the final project. As in research life and real-life, you will get full credit for what you have done only if your work is presented effectively.

The rubric for the final project will describe a point system which will be used to determine the score for each team on the final project. Any deficiencies in the final project presentation will result in a reduction in the final grade for the team. Examples of possible deficiencies are failing to address some of the issues mentioned under “Final Project Presentation” above, failing to present relevant examples to illustrate the performance of your methods, etc.

Each team must submit the final version of their slides to the instructors during the Project Presentation period, for grading purposes.

Class Paper Presentations and Discussion

Students will be expected to prepare a presentation and lead the class discussion on a particular paper. Active participation in paper discussions is integral to learning, since so much of course material does not yet exist in textbook or tutorial form. Therefore we will learn by reading and discussing original sources. Students are expected to read the assigned materials prior to class, and to participate in class discussions. It consists of 20% of the course grade for each team that presents, and 15% of the grade for that who are listening and discussing.

The presentation of a paper should address the following points:
1. What is the main claimed contribution? What do you think is the real contribution?
2. What are the strengths and/or weaknesses in the methodology?
3. What new questions are raised by this paper?