Markers, I will use them.

Feature detection has broken me & I have mostly decided that I will incorporate markers to a limited extent. Well, it hasn't broken me, but motion tracking is only but a part of my dissertation, and I need to do things on top of that. So, I cannot devote all my resources to solving problems that have not been solved in motion capture yet. Here's the current version (still markerless..):

So basically, I need to be able to distinguish between heads, so that it creates a new tracker for a new person. It obviously still loses where its tracking, non-trivially -- although it's better than before. It updates with face detection every 250 frames or so & also when its detecting too many zero pixels (this is after depth segmentation & only looking at the top 25% of blobs for heads -- yes, the kinect data is noisy).

Facial recognition is too CPU-intensive I think. I attempted to try tracking markers using some AR libraries (aruco, ARma) -- just as a prototype -- they were really light-weight & easy to implement -- but they were not meant for applications such as mine (nothing comes cheap in my case). I think I'm also ready to do better depth segmentation... & perhaps there is a way to disgard some of the Kinect noise.

I am worried about varied lighting conditions, etc. I am half-thinking about just turning the Kinect into a cheap IR sensor sans depth -- since the resolution of the depth information is fairly low & noisy for my purposes. Or just buying really high quality & fast webcams.

Another problem to solve: right now, the kinect is sucking up CPU -- like 120%... eek. I've traced the problem to the libfreenect-driver, but replacing the driver with an up-to-date version (the one on homebrew is 2 iterations behind) either crashes or runs once in debug mode, using even more CPU than before...

face tracking tango dancers, cont.

Soooo.... I am using the Haar face detection to find the faces when I lose them and update them every 75 frames. I am using Zhang's Real-time Compressive Tracker for the tracking. It does not perform as well on tango dancers. Well, nothing does! I am using pretty crude depth segmentation, skin color tracking (also the crude version) & only looking at the upper fourth of any blob to track something or detect a face. Only looking at the upper fourth of any blob means it will not catch lifts and dips (unless I can separate the couple), but those are really rare in tango social dance. 

So, it does NOT have continuity between the faces at the moment. Or at least, every 75 frames it loses continuity. So the next step will be creating the continuity. So, that means when I lose a face, I need to find it again via face detect & check whether THAT image is similar. I think I may dig in the code for the tracker I'm using since it is obviously doing SOMETHING like that. Also, since the tracker updates & learns, that means it should drift less (I'm guessing).   That means that eventually I can trust it not to drift less...

I could also make my face detection faster....

Also, To be honest I'm a little horrified at some of my mistakes (dancing) in this video, especially after repeated viewing. I was tired! Nevertheless!