Tango practica con Jose, Dec. 11

1. Doing well with intention and embrace
2. Remember to push off from the heel during back ochos
3. Reviewed the 2-leg pivot
4. Remember to use the heel for rebounds.
5. Boleo is going well.

Tango practica con Marco

Notes:
1. Am doing well, with intention, for most part BUT:
2. Forward steps... make sure foot, leg goes first, then hip follows -- don't move hip back or whatever first.
3. Stay, stay with the leader. Esp. when moving out of parada, feel the distance of the step. Also, keep distance with forward ochos.

Technical Mishaps and Adventures....

So, I went & checked my motion recording setup and shiz had broken. Le sighz. Again, I am having problems with 32 vs 64bit. Upon checking my iphone as webcam setup for motion capture recording, I find I get this strange bug:

*** QTCaptureSession warning: Session received the following error while decompressing video: Error Domain=NSOSStatusErrorDomain Code=-12905 "The operation couldn’t be completed. (OSStatus error -12905.)". Make sure that the formats of all video outputs are properly configured.

After much googling, I couldn't really find a good answer. I do know that it is something in my build or project settings, but what, I don't know. I'd be interested in an answer of course...  It is definitely related to something I did to make the IT++ library work with the 64-bit dynamic libraries since it was working before then. (or perhaps something when I was integrating matlab/mex in C++ but now decided didn't need it) Yes I could check BUT: 

Also, the Mobiola Webcam does not work in 64-bit (the OS X desktop half hasn't been updated since 2012). Other iPhone as webcam software I have tried either doesn't work on OS X or has around a 1sec. delay... which is pretty horrible. Mobiola so far has had waaaay better performance, as a real-time webcam.

After some mucking around... (too much), I finally decided to simply create a 32-bit video capture recording application that received sync'ing messages from OSC so I could capture the information. Even if I got the video capture in my current main application, it would be 64-bit & I'd have to use an iPhone app with a lot of latency. I figure its a temporary fix until Mobiola updates or I find a better iPhone app**. & then I will go through the pain of trying to fix that weird error.

Secondly, I found a strange Android bug in the Shimmer API. At least, I'm pretty sure it is from their code since its basically a memory leak in the JNI & I am not touching that part of the code. What happens is gradually, memory leaks until it runs out of space, causes a JNI pinned array ref problem and Android restarts the app. This happens after 30-45 min. so its not a disaster... I can simply restart, I guess, but I will need to hunt that down and fix it at some point...it is kind of a pain to have to force a restart every 1/2 hour. First, I'll report it to the Shimmer pplz, tho. I suppose its possible I need to explicitly free something, but I would think the garbage collector would be doing that, since I'm not dealing with the JNI directly... Its like all unsafety of C++ with the annoyances of JAVA. Joyous.

 **Hahaha I am not going write one I have better things to do

Catching up on Tango Notes postings, Dec. 1 & 3

Tango Notes, Practica with Pio, Dec. 1

 ++More heel

++In back ocho, push back with heel to fully arrive before pivoting

++Allow to boleo from open side, follow chest exactly

++Keep knees collected for boleo and etc. allows for faster shiz.

Tango Notes, Clase con Sofia, Dec. 3

++pushing from the floor to build axis

++Resistence! Resistance from the torso going backwards, resist from the hips going forward

++Intention & active embrace ALL the time... I lose connection for a moment arriving backwards

++Transfer of weight – arrive with the hip back, opposite shoulder receives free knee

++send around, not diagonal during boleos

Argentine Tango Notes, Practica with Jose, DNI Tango

Started out feeling a little lost / fuzzy. That's what I get for staying up too late and skipping coffee before.

1. I feel like I am a lot better about keeping my axis, in general. I am not getting any complaints about falling backward while walking backward.

2. I need to work on hips and knees, so that my foot always stays in contact with the floor. Hips & knees create the space for this.

3. Also, I need to respond with my leg first, then my torso for ocho movements (etc.). I need to keep my torso with my partner and show initial resistance to torso movement.

4. From previous lessons: I need to use my heel more, have more connection to the floor

Gait Analysis as mapping strategy for Argentine tango.

After I deployed an initial prototype, it occurs to me that actual knowledge of steps (while, yes, walking) would be helpful in creating my mapping strategies (translation of dancer motion to music).

Right now, I am looking at RRACE:
http://www.researchgate.net/publication/257307889_RRACE_Robust_realtime_algorithm_for_cadence_estimation

It estimates cadence, which will be helpful. It introduced me to the Lomb normalized periodogram. It is like the FFT, but for non-uniformly sampled data.

I am also measuring stationary moments in each leg...Using these two tools, I hope to extrapolate with reasonable accuracy when people are stepping. I can then use the distance between steps (literal distance using dead-reckoning, using the algorithm from xIO technologies)... or simply the time between steps...  distance might be more interesting?)

I am back. More coding. More tango (not pictured).

Sooooo...

Supercollider Code to Compress a Control Signal (or do any ar - only UGEN thing to it) and then Forward it to Osculator which turns it into a MIDI ctrl signal to Change something in Ableton Live.

I. Kid. You. Not.

And yet, it was relatively simple to implement, and once I remembered that magic "SendReply" call I was able to send the modified signal on to Ableton. I know I know, I should just get Max4Live, but look, that costs money, and I'm spending all my money on sensors and stuff and also, I'm just... so much faster in other paradigms. I actually have a year's subscription to regular MAX/MSP, in case I really need it for this project, etc.

I hesitated using Ableton Live, but in terms of the project, I think it fits, and it lets me go faster. Tango is a popular song, and the whole Live paradigm isn't forcing me into a song structure that shouldn't already be there.... and I will need to loop and etc. At the very least, it is super valuable for prototyping.

//add this synth def

(

SynthDef('sendShimmerCompress', { 

| whichControlBus = 0, thresh1 = 0.8, makeUpAmp = 1, sID=1908 |

var cSig, kout, sigOut = In.kr(whichControlBus); 

cSig = K2A.ar(sigOut); 

cSig = Compander.ar(cSig, cSig,

thresh: thresh1,

slopeBelow: 1,

slopeAbove: 0.1,

clampTime: 0.01,

relaxTime: 0.01);

kout = A2K.kr(cSig * makeUpAmp); 

SendReply.kr(Impulse.kr(50), 'compressShimmer', [ kout ], sID);

}).add;

)

//call it, make sure to add your own sID for each different signal

v = Synth.new('sendShimmerCompress', [\sID, 2345]); 

//add the OSCResponder to forward the call on to wherever.

(

g.remove; 

g = OSCresponderNode(nil, '/compressShimmer', { |t, r, msg| 

var sID; 

var shimmer1X = 2345; 

var oscM; 

sID = msg[2]; //which shimmer & sensor value is it?

if(sID == shimmer1X, {    

oscM = "/1/fader1"; 

});

c = NetAddr.new("127.0.0.1", 8000); // create the NetAddr
c.sendMsg(oscM, msg[3]);

 }).add;

 )

drinking cava (Spanish sparking wine)

I passed my comprehensive exams! I am ABD! I am going to Buenos Aires in eleven days.

And this is some serious shit here:
http://www.x-io.co.uk/gait-tracking-with-x-imu/

Thinking of going with the x-osc instead of the shimmer sensing. Yup.

So, after much consultation

It is possible that I may not use optical motion capture at all for my project.

OK, I probably will.

I am looking into using Shimmer sensors. They have software which syncs the sensors (accelerometers, gyro, et. al.) to each other AND to cameras. So I could have everyone wear one (ideally across back) & get ID & location THAT way. Then I can still get some rich motion data from the legs using the live video motion capture.

Or else, I just use all Shimmer motion sensors, but I'd actually like to avoid that. I don't want something bulky on their ankles.

I looked into some active and passive marker optical motion capture systems, but ultimately they were too rich for my blood & my department's (I'm in the arts!). They recommended at least 20 cameras to deal with the occlusion, and that's well. I can already tell you that 20 motion capture cameras of that caliber is out of my price range.

Anyways, the folks at Phase Space & Qualisys & APDM were particularly helpful, and I would recommend them highly if you have the funding.

PS. Not that anyone follows this blog, but I did finish my Variations V paper, and I have feedback for it. I am hoping to do the edits and publish it AFTER Buenos Aires. Possibly after my dissertation defense. After that I will probably also dinosaur more (and again -- yes I use dinosaur as a verb now, you would, too). Oh, AND! My! Dinosaur! Got! Accepted! Into! NIME! as! an! installation! in! London! UK! Crazy trip since I'll be in BsAs & Sharif (my partner in this dinosaur business) will be bring the dino, so I have to send it to him (across the country) before I leave.

Markers, I will use them.

Feature detection has broken me & I have mostly decided that I will incorporate markers to a limited extent. Well, it hasn't broken me, but motion tracking is only but a part of my dissertation, and I need to do things on top of that. So, I cannot devote all my resources to solving problems that have not been solved in motion capture yet. Here's the current version (still markerless..):

So basically, I need to be able to distinguish between heads, so that it creates a new tracker for a new person. It obviously still loses where its tracking, non-trivially -- although it's better than before. It updates with face detection every 250 frames or so & also when its detecting too many zero pixels (this is after depth segmentation & only looking at the top 25% of blobs for heads -- yes, the kinect data is noisy).

Facial recognition is too CPU-intensive I think. I attempted to try tracking markers using some AR libraries (aruco, ARma) -- just as a prototype -- they were really light-weight & easy to implement -- but they were not meant for applications such as mine (nothing comes cheap in my case). I think I'm also ready to do better depth segmentation... & perhaps there is a way to disgard some of the Kinect noise.

I am worried about varied lighting conditions, etc. I am half-thinking about just turning the Kinect into a cheap IR sensor sans depth -- since the resolution of the depth information is fairly low & noisy for my purposes. Or just buying really high quality & fast webcams.

Another problem to solve: right now, the kinect is sucking up CPU -- like 120%... eek. I've traced the problem to the libfreenect-driver, but replacing the driver with an up-to-date version (the one on homebrew is 2 iterations behind) either crashes or runs once in debug mode, using even more CPU than before...

face tracking tango dancers, cont.

Soooo.... I am using the Haar face detection to find the faces when I lose them and update them every 75 frames. I am using Zhang's Real-time Compressive Tracker for the tracking. It does not perform as well on tango dancers. Well, nothing does! I am using pretty crude depth segmentation, skin color tracking (also the crude version) & only looking at the upper fourth of any blob to track something or detect a face. Only looking at the upper fourth of any blob means it will not catch lifts and dips (unless I can separate the couple), but those are really rare in tango social dance. 

So, it does NOT have continuity between the faces at the moment. Or at least, every 75 frames it loses continuity. So the next step will be creating the continuity. So, that means when I lose a face, I need to find it again via face detect & check whether THAT image is similar. I think I may dig in the code for the tracker I'm using since it is obviously doing SOMETHING like that. Also, since the tracker updates & learns, that means it should drift less (I'm guessing).   That means that eventually I can trust it not to drift less...

I could also make my face detection faster....

Also, To be honest I'm a little horrified at some of my mistakes (dancing) in this video, especially after repeated viewing. I was tired! Nevertheless!

Beatriz Ferreyra is awesome

Beatriz Ferreyra's electroacoustic tango MurmureIn

adventures in depth segmentation

k-means segmentation gone rogue. I just find this amusing. I think I might try out another way, though. I am awesome at finding the slowest ways of doing things. All CPU + RAM are belong to us.

Face Detection using Skin Detection -- so much better!!

Edit: I accidentally deleted this post, but I was able to restore it here.... Yay...

Ok, so I really need a trained haar cascade for like, a 3/4's face, bc that's the one that I'm not detecting. But anyways, it is performing pretty well. The only thing is that I really just need to go to doing my depth segmentation & get the background out of there in a way that doesn't obscure my faces. I found an interesting paper which uses both depth AND color to segment -- & it is straight-forward to implement, so maybe I'll try that. I'm worried about the performance aspect of that, though, since it requires that I translate into IRL coordinates from depth.

Anyhow, here's my skin/face combo results. Narrowing down the search area definitely created a lot more correct positives... AND still some false positives. Note that once the background is gone, that this will improve the results immensely.

The blue is detected skin regions & the pink again, is faces. Also, my algorithm is (in theory) not racist. So, ya, that's important, right?!! Apparently human skin is more or less the same color once you disregard  luminance.... actually working on a better skin detection right now from the paper I mentioned earlier... THEN, off to image/depth segmentation.

face detection disappointing so far on tango couples (EDITED: not as bad as I thought)

This is just my first, naive try using the Haar cascades in OpenCV. I'm using one frontal training set & one profile.

It could be that if the kinect was closer, then it would get better results... but then of course, there would be more occlusion for the theoretical other dancers. I did enlarge the image in order to do the detection, but this didn't help enough. My problem might also be that the kinect is from an angle above, and so, the faces are not straight-on enough. Wonder if I could solve with rotations? Seems CPU-expensive. Le sighz.

Here are some shots of it half-way working. It is, very disappointingly, very abysmal. I'm wondering if I can tweak it into functionality, though. I did try a quick skin detection thing, and that did really well. I think I could even tinker with it and get rid of the false positives.

 The back of the head isn't bad, considering. It would work for my purposes... 

Close but no cigar

Just with the background subtraction [below].. I'm going to also use the depth to subtract the background when I get around to it... 

Checking out: http://blog.mashape.com/post/53379410412/list-of-50-face-detection-recognition-apis

http://www.semanticvisiontech.com/

http://chenlab.ece.cornell.edu/projects/FaceTracking/

http://pointclouds.org/blog/hrcs/aaldoma/index.php ??

I think I might go into the skin detection more... 

EDIT: So it turns out I forgot to draw the profile detections boxes... plus, the other kinect seemed to get a better view. It was mostly my face. I think I could do a mirror and get the other profile... can't do this for all angles though, and it doesn't catch my partner hardly at all.  This is still mostly open-embrace. Close embrace is going to be tricky.

EDIT #2: 

Ok, so actually the background subtraction was hurting more in this case than helping. I was skeptical, but... I think what I need to do is just put a bounding box around the couple, and detect the face in there -- would be faster.  Anyways, ya, so with the first kinect & only frontal face, the background helped A LOT (for the little positives that were created) but in this case, NOT having the background really helped. I mean, looking at the visuals, you can see where it would get confused, but it really was better in the other case. Anyways... I also fooled around with the brightness/contrast. I added the other side profile & I was actually detecting my partner a few times.

smaller face detections!

So, again I think this all has to be supplemented by skin detection. It will definitely fix the relatively few number of false negatives. However, it is still pretty iffy in close embrace...so I think think I need to find another way (in addition? instead?). Damn you close embrace!!! You keep on foiling me!! Also, I can do template matching for a while once I find the face. But I do need to have a robust way of finding face the first time. Again, I think I also could do a bounding box on the couple, and only search that image. Could do that from the depth image maybe? Plus, seeing where the skin color falls?

Brighter lighting would help but this is SUPPOSED to be for a tango milonga. Wonder if there is a way to make Kinect RGB part better in dim light?

Ideas for detecting separate dancers in a tango couple

1. Face detection

2. Skin detection -- used to supplement #1 & also to detect ladies' bare legs

3.  Shoe/Foot detection - template matching? look for shoe-like blobs in segmented image. The ones near skin are ladies. The other ones are men.

4. Background subtraction is going to be useful even having depth ( I KNEW IT)

5. Thinning to get skeletal shapes? If have head & feet, could get CoM & possibly extrapolate which blobs are legs...

6. Optical flow to estimate where things are if they go out of frame.

Slim hope that Kinect could detect skeletons on tango dancers dashed...

DSP, Audio, & Motion Capture

I was thinking about the connections between DSP for audio and audio processing / synthesis techniques and motion capture. Because you can think of both video & audio in terms of signals, they have a lot of similarities and can use the same techniques. You often have to transform signals into feature spaces... (ie, extract features) then work from there.

One universal problem is defining a perceptual quality (whether it is an action, how how an action is done, or a timbre color or pitch) within a computational space. Sometimes there seems this cruel quality in both practices: after all my human brain can track the motion. I understand what the timbre is and when it occurs. But my software doesn't have access to the tools that my brain does (yet). Nor has it been exposed to the years and years of training my brain has had to distinguish these qualities. This is very obvious to anyone in my field, but still, when I step back, it seems a bit poignant.

Of course, my brain can't generate a real-time control signal from movement to send to my audio synthesis routines, so there's that. :) Although I can use the information I have in the form that I have it to make noise via my physical body.

EXC_BAD_ACCESS in OSC + 64-bit Cinder with Xcode 4.6.2 in OS X 10.7

This was enough of a headache that it deserves a post.

I'm using cinder on OSX, 64-bit in order to run the OpenNI 2 block. The official release and everything of Cinder is still 32-bit, but most of the libraries will work in 64-bit -- but it is a pain. I am using the osc block that comes with the official 0.85 release of Cinder.

So I kept getting this weird EXC_BAD_ACCESS (code=13) error in OutboundPacketStream whenever I tried to use the osc::Sender object to send anything.

Unfortunately, I was convinced this was something weird I was doing myself... since all the examples worked & I have been using this library in 32-bit for a very long time...  But I wasn't thinking that about the fact they were in 32-bit & my code was compiling to 64-bit... in any case, it looks like it is fixed in the current version of the oscpack library (someone else had this problem, though not with the Cinder implementation in particular), so I replaced the /ip & /osc parts of the library. It solved my problem. Case closed!

Btw, I should probably move to OS X 10.9, I know. But you can see from the above mess that upgrading anything is generally a big hassle...

Cinder does not get as much curating as other libraries, like say Processing and has a much smaller user base. However, Cinder is leaner, meaner, and faster. And I've been coding in C++ for so long that I can just code in it -- unlike Java where I'm still occasionally looking up the syntax for something relatively basic or I'm doing something illegal since hey, you can do whatever I'm trying in C++... plus, its pretty easy to incorporate outside libraries.  I looked at OpenFrameworks for a little, too, but it looks messier to me....

Musical Analysis -- Moving Beyond the technical...

See my previous post on my Variations V analysis for background. 

So, the main chunk of my analysis is actually not computational. I match up the musical and movement events in the various performances of excerpts Variations V (see tables below) & then highlight them on the similarity matrices of the music and movement (see similarity matrices below). I have conclusions and remarks in my paper, which I have a complete first draft of (!). 

Structure in Miller's Cage Centennial Performance of Variations V

Sample had two main sections,

1 louder and is movement and music are less similar – running through sensors

2. Starts with a transitional relative silence, is softer, much higher movement & audio similarity, especially at the end.

Event #	Movement	Music
1	Stillness	High sustaining tone, no onsets
2	Dancers break stillness and all move at once	Low honking sounds resume
3	Two dancers still, one dancer running in a wide circle	Noisy, percussive synth sounds that seem triggered by the running, at end glissandos start when all 3 are on stage
4	All three dancers fall down, dancers turn in unison	High downward glissando in sync with fall, silence for unison turn
5	Two dancers circle arms in sync, third dancer still	Silence
6	Two dancers jump & change levels, third dancer still	Low honking sounds in response to level changes, lower sounds that seem to be in response to dancers’ movements
7	Duo unison ends, all dancers change levels	Loud honking sounds
8	One dancer rocks back and forth, others move very little	Low sounds roughly in sync with rocking

Structure in decibel's Performance of Variations V

Less obvious relationships in music and audio than in Miller's version. The work starts out with one dancer moving in front of a sensor like (1), and these moments have the most audio & movement similarity. Then, she breaks up this by running around the space, and setting off a loud burst of sound. The last part involves some dancers moving very near to a sensor, while the other dancer moves away from the triggers. Thus, there are some indications of similarity between the matrices, but they are much less clear than in the beginning.

Event #	Movement	Music
1	1 dancer moving arms, in front of projection and sensor, 2 other dancers moving minimally	Glissando sine waves that seem to be affected by the dancer arm’s movements
2	2a – dancer in front of the projection moves away all dancers briefly pause or move very little	Much softer noise, soft phased, saw wave  sounds glissando
3	1 dancer runs around the stage twice, two dancers move in unison in different corners of the stage area	Quieter noises continue 3a – loud siren-ish burst, which sounds during the first circle of the dancer When the dancer passes again, another, slightly different and softer noise triggers
4	Running dancer stops, unison dancing continues. One of the unison dancers stops & begins to move towards her unison partner for a duo	High glissando sine waves continue, seemly following the movements of unison dancers
5	1 duo, 1 solo (previously the runner) all moving and changing levels, no unison	Again, high glissando continues seemingly related to duo movements, but not the solo dancers
6	Solo dancer moves to the side off camera, duo breaks up, and everyone is changing position, perhaps away from sensors	Less glissando until there is a moment of sustain
7	Two dancers exit from view of the camera, off the stage area	High glissandos begin again

Music / Relationship Structure in 1966 Hamburg Cage and Cunningham performance

The video analysis of this work contained much more noise than in the previous two samples, since there were small camera movements and images obscuring the main viewpoint of the dancers. However, the similarity matrices show less connection between audio and music, except for the last moments of the clip, when a third dancer enters, and they all seem close to sensors as they move. All the SSMs show this moment. (Interaction #1)

Also, while it is not clear in the similarity matrix, the dancer on the left seems to be controlling a white noise bursts by his movements for a short time, exhibiting interaction #1.

Four different sections

            +opening – 2 men, each near a pole

            +2^nd man exists

            +man & woman

            + short section when a second man enters, very apparent on similarity matrices

Cage –1^st part, structure and correspondences

Event #	Movement	Music
1	One dancer stays in position, but arms move. A second dancer enters.	Radio static, feedback sounds, one clank
2	First dancer lunges & turns, second stays at the same location, but moves back and forth quickly	More sounds of radio static tuning, more feedback
3	First dancer stays still and balances, 2nd dancer still moves from side to side	Rhythmic clanking sounds enter over the sounds of radios tuning
4	First dancer collapses then starts jumping up and down and occasionally turning. Second dancer continues in his side to side figures but also occasionally jumps	The clanking changes rhythmic pattern, and the radio sounds get louder. Occasionally there is a burst of white noise which seems related to the movements of the first dancer’s movements

Cage – 2nd part, structure and correspondences

Event #	Movement	Music
1	Left dancer on one leg holds still. On right, dancer also on one leg, moves arms while turns	Quiet white noise with some quiet tones beneath
X	Close-up obscures view, disregard	Disregard
2	Dancer spins moving slowly left until out of camera view. Meanwhile, dancer in front slowly turns and moves one arm out and back	Soft buzzy clicks
3	Dancer extends leg out and back	Low bell sound that  seems controlled by leg
4	Dancer from left enters again. Meanwhile, dancer in front leans and lunges	Mostly quiet noises. Loud honk as left dancer takes a backward step
5	Third dancer enters. All dancer move.	More loud, intermittent honks. Radio static-type noise crescendos.