Holistic Scene Understanding

CS294A
Projects in Holistic Scene Understanding
Autumn 2008



--------------------------------------------------------------------------------


Course Overview



One of the original goals of computer vision was to fully understand a natural scene. This requires solving several problems simultaneously, including object detection, labeling of meaningful regions, and 3d reconstruction. While great progress has been made in tackling each of these problems in isolation, only recently have researchers again been considering the difficult task of assembling various methods to the mutual benefit of all.

Although "Holistic Scene Understanding" is a somewhat ambiguous concept, we propose that it involves some or all of the following components:
Identify the entities present in the scene (object categories, materials, surfaces, etc.).
Enumerate properties of the entities (name, color, articulation, pose, object sub-type, etc.).
Characterize the actions or activities that the entities are doing (person walking, car driving, cow eating, etc.).
Describe the relationships between these entities (the person is next to the dog, the car is driving off-road).
Reconstructing the 3D layout or structure of the scene (this person is walking in front of the car, the car is 100 meters away, etc).
In this class, our goal will be to develop a series of computer vision models that can work together to form the framework for holistic understanding of the scene. In particular, we hope that these models will be able to work together to improve their outputs over what is possible in isolation and will provide most of the information necessary to answer many interesting questions about the scene, including those listed above.

As a student in CS294A, you will help spearhead advances towards this ambitious goal. You will be in charge of a particular vision sub-task within the full system. The vision sub-tasks that we will consider include:
Object Detection
Multi-class image segmentation
Boundary/Edge Detection
Surface Orientation
Object Outlining
3D Reconstruction
Image Captioning
Leveraging Text Data Sources
Descriptive Classification
Image In-painting
System Visualization
Our goal as a class will be to integrate all of the projects in the class by the end of the quarter and generate a scene fly-through.

If you were unable to attend the first meeting but would like to take CS294A, please email cs294a-qa@cs.stanford.edu to let us know.


Teaching Staff



Instructor: Daphne Koller
Office: Gates 142
Office hours: Mondays 11am-12pm
email: koller@cs.stanford.edu
Phone: (650)723-6598



TA: Stephen Gould
Office: Gates 126
Office hours: Tuesdays 11am-12pm
email: sgould@cs.stanford.edu
Phone: (650)725-8784



TA: Geremy Heitz
Office: Gates 116
Office hours: Wednesdays 3:30pm-4:30pm
email: gaheitz@cs.stanford.edu
Phone: (650)725-8790




Course Schedule



This is a project course. There will be no homeworks and no weekly lectures, and we will instead spend the quarter working in teams on the vision subtasks described above. There will be meetings every week on Thursday between 3:30 and 5:05 pm: We will meet 4 times as a full class and in other weeks you will meet in your project groups with the instructor and TAs. In addition to the meetings, there will be four deliverables. The first three are progress milestones during the quarter, and the fourth is a final presentation or poster session during the final week of class.

In addition to class hours, we will have office hours as listed in the staff information above.

Date Meeting Milestone
Thursday, September 25th Introduction to Class (Full Class)
Thursday, October 2nd Project Group Meetings
Thursday, October 9th Project Group Meetings
Thursday, October 16th Project Group Meetings Baseline Models
Thursday, October 23rd Project Group Meetings
Thursday, October 30th Project Group Meetings
Thursday, November 6th Status Meeting (Full Class) Improved Independent Models
Thursday, November 13th Project Group Meetings
Thursday, November 20th Status Meeting (Full Class) Integrated Models
Thursday, November 27th Thanksgiving Break
Thursday, December 4th Final Presentations (Full Class) Final Models



More Information




Project Descriptions
Reading List
Data
Resources/Code