Markers, I will use them.

Feature detection has broken me & I have mostly decided that I will incorporate markers to a limited extent. Well, it hasn't broken me, but motion tracking is only but a part of my dissertation, and I need to do things on top of that. So, I cannot devote all my resources to solving problems that have not been solved in motion capture yet. Here's the current version (still markerless..):

So basically, I need to be able to distinguish between heads, so that it creates a new tracker for a new person. It obviously still loses where its tracking, non-trivially -- although it's better than before. It updates with face detection every 250 frames or so & also when its detecting too many zero pixels (this is after depth segmentation & only looking at the top 25% of blobs for heads -- yes, the kinect data is noisy).

Facial recognition is too CPU-intensive I think. I attempted to try tracking markers using some AR libraries (aruco, ARma) -- just as a prototype -- they were really light-weight & easy to implement -- but they were not meant for applications such as mine (nothing comes cheap in my case). I think I'm also ready to do better depth segmentation... & perhaps there is a way to disgard some of the Kinect noise.

I am worried about varied lighting conditions, etc. I am half-thinking about just turning the Kinect into a cheap IR sensor sans depth -- since the resolution of the depth information is fairly low & noisy for my purposes. Or just buying really high quality & fast webcams.

Another problem to solve: right now, the kinect is sucking up CPU -- like 120%... eek. I've traced the problem to the libfreenect-driver, but replacing the driver with an up-to-date version (the one on homebrew is 2 iterations behind) either crashes or runs once in debug mode, using even more CPU than before...

Face Detection using Skin Detection -- so much better!!

Edit: I accidentally deleted this post, but I was able to restore it here.... Yay...

Ok, so I really need a trained haar cascade for like, a 3/4's face, bc that's the one that I'm not detecting. But anyways, it is performing pretty well. The only thing is that I really just need to go to doing my depth segmentation & get the background out of there in a way that doesn't obscure my faces. I found an interesting paper which uses both depth AND color to segment -- & it is straight-forward to implement, so maybe I'll try that. I'm worried about the performance aspect of that, though, since it requires that I translate into IRL coordinates from depth.

Anyhow, here's my skin/face combo results. Narrowing down the search area definitely created a lot more correct positives... AND still some false positives. Note that once the background is gone, that this will improve the results immensely.

The blue is detected skin regions & the pink again, is faces. Also, my algorithm is (in theory) not racist. So, ya, that's important, right?!! Apparently human skin is more or less the same color once you disregard  luminance.... actually working on a better skin detection right now from the paper I mentioned earlier... THEN, off to image/depth segmentation.

face detection disappointing so far on tango couples (EDITED: not as bad as I thought)

This is just my first, naive try using the Haar cascades in OpenCV. I'm using one frontal training set & one profile.

It could be that if the kinect was closer, then it would get better results... but then of course, there would be more occlusion for the theoretical other dancers. I did enlarge the image in order to do the detection, but this didn't help enough. My problem might also be that the kinect is from an angle above, and so, the faces are not straight-on enough. Wonder if I could solve with rotations? Seems CPU-expensive. Le sighz.

Here are some shots of it half-way working. It is, very disappointingly, very abysmal. I'm wondering if I can tweak it into functionality, though. I did try a quick skin detection thing, and that did really well. I think I could even tinker with it and get rid of the false positives.

 The back of the head isn't bad, considering. It would work for my purposes... 

Close but no cigar

Just with the background subtraction [below].. I'm going to also use the depth to subtract the background when I get around to it... 

Checking out: http://blog.mashape.com/post/53379410412/list-of-50-face-detection-recognition-apis

http://www.semanticvisiontech.com/

http://chenlab.ece.cornell.edu/projects/FaceTracking/

http://pointclouds.org/blog/hrcs/aaldoma/index.php ??

I think I might go into the skin detection more... 

EDIT: So it turns out I forgot to draw the profile detections boxes... plus, the other kinect seemed to get a better view. It was mostly my face. I think I could do a mirror and get the other profile... can't do this for all angles though, and it doesn't catch my partner hardly at all.  This is still mostly open-embrace. Close embrace is going to be tricky.

EDIT #2: 

Ok, so actually the background subtraction was hurting more in this case than helping. I was skeptical, but... I think what I need to do is just put a bounding box around the couple, and detect the face in there -- would be faster.  Anyways, ya, so with the first kinect & only frontal face, the background helped A LOT (for the little positives that were created) but in this case, NOT having the background really helped. I mean, looking at the visuals, you can see where it would get confused, but it really was better in the other case. Anyways... I also fooled around with the brightness/contrast. I added the other side profile & I was actually detecting my partner a few times.

smaller face detections!

So, again I think this all has to be supplemented by skin detection. It will definitely fix the relatively few number of false negatives. However, it is still pretty iffy in close embrace...so I think think I need to find another way (in addition? instead?). Damn you close embrace!!! You keep on foiling me!! Also, I can do template matching for a while once I find the face. But I do need to have a robust way of finding face the first time. Again, I think I also could do a bounding box on the couple, and only search that image. Could do that from the depth image maybe? Plus, seeing where the skin color falls?

Brighter lighting would help but this is SUPPOSED to be for a tango milonga. Wonder if there is a way to make Kinect RGB part better in dim light?

Ideas for detecting separate dancers in a tango couple

1. Face detection

2. Skin detection -- used to supplement #1 & also to detect ladies' bare legs

3.  Shoe/Foot detection - template matching? look for shoe-like blobs in segmented image. The ones near skin are ladies. The other ones are men.

4. Background subtraction is going to be useful even having depth ( I KNEW IT)

5. Thinning to get skeletal shapes? If have head & feet, could get CoM & possibly extrapolate which blobs are legs...

6. Optical flow to estimate where things are if they go out of frame.

Slim hope that Kinect could detect skeletons on tango dancers dashed...