Tracking and Multi-object Segmentation

CS 585 HW 4
Patrick W. Crawford
Tim Chong
October 27

Problem Definition

In this assignment, we are given two image sequences of partially segmented data of bats and fishes, and we need to track the movement of these animals across multiple frames.

Method and Implementation

To do the object tracking, we decided to utilize the kalman filter built-in method for predicting the area of a given object's next location, as well as a greedy tracking method where the closest object to the new predicted location is used in conjunction with a maximum distance threshold. The basic concept is that for each point in an image, the prediction is run to where it would be located in the next frame. We look for the closest still available point to track to in the next frame in a circle around this point, by checking all tracks not marked and ensuring it is less than the maximum threshold distance away from the predicted location. Then, the track data and new point is pushed into the data-structure. If it is a new object, then it is initialized with a new array of points and color. If the object died, then it's data is deleted - we do not try to re-connect lost tracks or same tracks that re-enter as separated objects.

Note: The dataset for the bat tracking is entirely contained in part1.cpp, and the dataset for the fish tracking is entirely contained in part2.cpp, and they both share an imported header file called "filterBundle.h" which implements the kalman filter tracking itself.




Describe your experiments, including the number of tests that you performed, and the relevant parameter values.

Detection rate: As segmentation was only provided to us in part 1, segmentation only is really applicable to compare for part 2, despite it more or less being provided. We can visually see below that the detection of object nearly perfectly matches the original image where red contours provided - places where the fish was not detected by the provided sequence were of course not detected.
Accuracy rate: For the tracking itself, it is reasonably accurate overall. It quite reliably tracks lone points that move and change directions without issues, and when there are many points nearby it is still able to function decently - however there are still of course errors.


Our primary results are shown in these two image sequences, the tracked bat and then tracked fish sequences as shown below. The bat sequence we are already provided with labeled objects, whereas the fish sequence we had to re-extract the contours for already segmented fish.


Dataset Source Image Result Image
Tracking the bats
Tracking the fish
Segmenting fish


Discuss your method and results:

Questions answered:


My conclusion is that the Kalman filter tracking method is very robust and can yield pretty good results. Most of the improvement I expect can come from the logic of how to match one track with the other, as the prediction of the kalman filter clearly works fairly well. Occlusion and high density points to track naturally are hard problems to tackle, but that would be what needs consideration to improve our method. That being said, despite going with a greedy track selection method, it still worked quite well.

Credits and Bibliography

OpenCV Documentation, October 26, 2014
Kalman Filter Reference,; October 26, 2014
Solutions for lab 3, 4 and 6

Work was done with Timothy Chong