Research

Our group works in three broad areas: Computer Vision, Behavioral Imaging, and Mobile Health (mHealth). Within vision, we are developing the paradigm of First Person Vision and addressing a variety of fundamental problems in segmentation, recognition, video analysis, and visual tracking. Our Behavioral Imaging work develops novel methods to objectively measure and analyze social and communicative behaviors, with a focus on developmental disorders such as autism. Our work in mHealth develops novel mobile vision methods for assessing behavior and quantifying visual exposure. We are also developing predictive analytic methods based on machine learning for predicting future states of health risk from on-body sensor data.

Research Projects

  • Behavioral Imaging

    Behavioral Imaging

    Naturalistic human behavior presents a complex, multidimensional signal that, while sometime difficult to quantify, is invaluable for understanding and improving human health, development, and everyday functioning. This project aims to develop novel computational methods for measuring and analyzing human behavior to promote scientific and practical advances across a range of disciplines, just as techniques developed for brain imaging have revolutionized neuroscience and medicine. A particular focus of this project is devising new ways to “image” social behavior in children, to better understand and address issues related to atypical patterns of social and cognitive development such as those experienced by children with autism.

    LINKS

     

    NEWS

     

    RECENT PUBLICATIONS

    • J. Hernandez, Yin Li, J. M. Rehg and R. W. Picard. “Physiological Parameter Estimation Using a Head-mounted Wearable Device.” In MOBIHEALTH 2014, Athens, Greece.
      PDF ] : [ Project Webpage ]
    • A. Ciptadi, M. S. Goodwin and J. M. Rehg. “Movement Pattern Histogram for Action Recognition and Retrieval.” In Proc of ECCV 2014, Zurich, Switzerland.
      PDF ]
    • J. M. Rehg, G. D. Abowd, A. Rozga, M. Romero, M. A. Clements, S. Sclaroff, I. Essa, O. Y. Ousley, Y. Li, C. Kim, H. Rao, J. C. Kim, L. L. Presti, J. Zhang, D. Lantsman, J. Bidwell, and Z. Ye. “Decoding Children’s Social Behavior.” In Proc of CVPR 2013, Portland, OR.
      PDF ]
  • Eye Contact

    Eye Contact

    Overt visual attention, or the ability to direct our gaze to different parts of the environment around us, plays a critical role in intelligent behavior, from perception and learning to action and communication. We are developing systems and algorithms to help capture important information about human visual attention in a variety of settings, including systems for automatically detecting social gaze behaviors and non-invasive, real-world gaze measurement systems.

    LINKS

     

    RECENT PUBS

    • Y. Li, X. Hou, C. Koch, J. M. Rehg, and A. L. Yuille. “The Secrets of Salient Object Segmentation .” In Proc of CVPR 2014, Columbus, OH.
      PDF ] : [ Project Webpage ]
    • Y. Li, A. Fathi, and J. M. Rehg. “Learning to Predict Gaze in Egocentric Video.” In Proc of ICCV 2013, Sydney, Australia. Oral Presentation.
      PDF ]
    • A. Fathi, Y. Li, and J. M. Rehg. “Learning to Recognize Daily Actions using Gaze.” In Proc of ECCV 2012, Florence, Italy.
      PDF ] : [ Project Webpage ]
    • A. Fathi, J. K. Hodgins, and J. M. Rehg. “Social Interactions: A First-Person Perspective.” In Proc of CVPR 2012, Providence, RI.
      PDF ] : [ Project Webpage ]
  • Disease Progression Modeling

    Disease Progression Modeling

    Understanding the progression of a disease in a person can be as important as having a yes/no diagnosis; modeling disease progression over time can provide earlier screening and diagnosis as well as improve options for treatment as well as prevention. We have developed a continuous-time hidden markov model (CT-HMM) with multi-dimensional (MD) state structures for disease progression modeling that better accounts for the irregular intervals at which most people have medical checkups. When combined with advanced visualization techniques, we can analyze the models more intuitively and interactively, and we are also developing methods to help identify sub-classes (or phenotypes) within diagnosed populations.

    RECENT PUBLICATIONS

    • A. Gupta, Y.-Y. Liu, J. Sun and J. M. Rehg. “Visualizing State-Based Hypertenstion Progression Models.” In IEEE Vis 2014 EHRVis Workshop, Paris, France.
      [ PDF ]
    • Y.-Y. Liu, H. Ishikawa, M. Chen, G. Wollstein, J. S. Schuman, and J. M. Rehg. “Longitudinal Modeling of Glaucoma Progression Using 2-Dimensional Continuous-Time Hidden Markov Model.” In Proc of MICCAI 2013, Nagaya, Japan.
      [ PDF ]
    • Y.-Y. Liu, M. Chen, H. Ishikawa, G. Wollstein, J. S. Schuman, and J. M. Rehg. “Automated Foveola Localization in Retinal 3D-OCT Images Using Structural Support Vector Machine Prediction.” In Proc of MICCAI 2012, Nice, France.
      [ PDF ] : [ Project Webpage ]
  • Egocentric Vision

    Egocentric Vision

    Egocentric vision focuses on analyzing images and videos taken from a first-person perspective, such as from a head-mounted wearable camera. An egocentric view of the world captures many interesting visual properties that are not easily studied using stationary or even handheld cameras, such as subtle temporal patterns in hand-eye coordination and nuances of social gaze and eye contact. Our work in this area includes action recognition, gaze prediction, and object understanding, as well as theoretical approaches to perception and learning in an egocentric framework.

    LINKS

     

     NEWS

     

    RECENT PUBLICATIONS

    • J. Hernandez, Yin Li, J. M. Rehg and R. W. Picard. “Physiological Parameter Estimation Using a Head-mounted Wearable Device.” In MOBIHEALTH 2014, Athens, Greece.
      PDF ] : [ Project Webpage ]
    • Y. Li, A. Fathi, and J. M. Rehg. “Learning to Predict Gaze in Egocentric Video.” In Proc of ICCV 2013, Sydney, Australia. Oral Presentation.
      PDF ]
    • A. Fathi, and J. M. Rehg. “Modeling Actions through State Changes.” In Proc of CVPR 2013, Portland, OR.
      This work was partially funded by the Intel Science and Technology Center for Pervasive Computing.
      PDF ]
    • A. Fathi, Y. Li, and J. M. Rehg. “Learning to Recognize Daily Actions using Gaze.” In Proc of ECCV 2012, Florence, Italy.
      PDF ] : [ Project Webpage ]
    • A. Fathi, J. K. Hodgins, and J. M. Rehg. “Social Interactions: A First-Person Perspective.” In Proc of CVPR 2012, Providence, RI.
      PDF ] : [ Project Webpage ]
    • A. Fathi, A. Farhadi, and J. M. Rehg. “Understanding Egocentric Activities.” In Proc of ICCV 2011, Barcelona, Spain.
      This work was partially funded by the Intel Science and Technology Center for Pervasive Computing.
      PDF ]
    • A. Fathi, X. Ren, and J. M. Rehg. “Learning to Recognize Objects in Egocentric Activities.” In Proc. of CVPR 2011, Colorado Springs, CO.
      This work was partially funded by the Intel Science and Technology Center for Pervasive Computing.
      PDF ] : [ Project Webpage ]
  • Segmentation

    Segmentation

    Generating object segmentations in images and videos is an important paradigm for visual perception. In images, our work concentrates on improving the quality and speed of discrete inference approaches. In videos, we investigate the problem of efficiently generating long term object proposals. Our overall goal is to build models and algorithms which enable fast inference in images and videos, while remaining accurate.

    LINKS

    • Segtrack v2 dataset: Link

     

    NEWS

    • 09/20/13: NSF Proposal awarded for “A Compositional Approach to Video Segmentation,” Jim Rehg (PI), Fuxin Li (co-PI).

     

    RECENT PUBLICATIONS

    • A. Humayun, F. Li, and J. M. Rehg. “RIGOR: Reusing Inference in Graph Cuts for generating Object Regions.” In Proc of CVPR 2014, Columbus, OH.
      [ PDF ] : [ Project Webpage ]
    • F. Li, T. Kim, A. Humayun, D. Tsai, and J. M. Rehg. “Video Segmentation by Tracking Many Figure-Ground Segments.” In Proc of ICCV 2013, Sydney, Australia.
      [ PDF ] : [ Project Webpage ] : [ SegTrack v2 Dataset ]
    • D. Tsai, M. Flagg, A. Nakazawa, and J. M. Rehg. “Motion Coherent Tracking Using Multi-label MRF Optimization.” In International Journal of Computer Vision, 2012.
      [ Springer Link ] : [ Project Webpage ]
  • High-Speed Autonomous Driving

    High-Speed Autonomous Driving

    Human race car drivers perform incredible feats of perception and coordination in controlling their vehicles at high speeds and during aggressive maneuvers. In robotics approaches that pipeline perception and then control, perception has traditionally been a resource bottleneck that hinders the achievement of rapid, robust, and real-time control actions. We are investigating systematic, online resource allocation to alleviate this computational bottleneck, including work in 3D scene understanding and course learning, vehicle modeling and state estimation, and active control during airborne maneuvers

    LINKS

     

    RECENT PUBLICATIONS

    • A. Kundu, Y. Li, F. Daellert, F. Li and J. M. Rehg. “Joint Semantic Segmentation and 3D Reconstruction from Monocular Video.” In Proc of ECCV 2014, Zurich, Switzerland.
      [ PDF ] : [ Project Webpage ]

Past Projects

  • Causality in Video Event Analysis – Methods for temporal causal analysis of audio-visual events
  • Large Scale Categorization and Retrieval of Images and Video – Methods for image and video analysis that are capable of scaling to and exploiting very large dataset sizes
  • Computer-Assisted Diagnosis of OCT Imagery – Research on automated analysis of OCT images of the retina to assist in the diagnosis of macular pathologies
  • Clutter Manipulation – Techniques for perception and manipulation of clutter in real-world environments, using the PR2 mobile manipulator
  • Place Categorization – Research on automatically categorizing places within a 3-D environment to support mobile robot navigation

Sponsors and Programs