More results

Amplitudes instead of MFCC's. Interestingly, the giselle music & movement are closer together using this measure... perhaps hinting that timbre wasn't such a structural element in the work, while variation in amplitude (beats?) were. Makes sense. Using self-similarity matrices since the recurrence plots were too much tweaking.

I used gifs as the inputs instead of text files, which seemed to increase the accuracy. Using text files, two matrices of the same data (movement data), which did look extremely similar, were not clustered together. Switching to gifs solved the problem. Will probably redo the MFCC's that way as well. Seems writing the files to matlab data might not be the best thing for this process.

Actual Progress!!! Clustering Self-Similarity Matrices via Normalized Compression Distance (NCD)

See last post for awesome paper that led me to this!

Ok, so what does that MEAN??! you may ask. Well, this is the result of my clustering analysis (via NCD) of all the similarity matrices I have so far, and they roughly correspond with how similar I perceive them to be. I used the NCD command from CompLearn.  I highly recommend it! Simple to use! You have to install graphviz, for the neato command -- which they seem to assume you already have in the CompLearn documentation.

***The closer the ovals are to each other, more similar they are perceived to be.

Why is .DS_Store there? BC I haven't taken the time to figure out how to exclude it. Grrrr!

Abbreviations are:
v5cover_move - Movement data from Cage 100 Festival @ Chicago performance of Variations V
v5cover_music - MFCC (audio) data from Cage 100 Festival @ Chicago performance of Variations V
giselle_move - Movement data from Giselle ballet
giselle_music - MFCC (audio) data from Giselle ballet
v5_original_move - Movement data from Cage/Cunningham's Variations V (1966) Hamburg perf.

v5_original_music - MFCC (audio) data from Cage/Cunningham's Variations V (1966) Hamburg perf.

decibel_v5_move - Movement data from decibel's Variations V

decibel_v5_music - MFCC (audio) data from decibel's Variations V

So, the Cage 100 movement/music data pair (ie, v5cover_), in which the movement corresponds heavily to the music appear super close to each other. The other music/movement are sort of spread out, but stilll kinda close... y'know. AND the furthest away music/movement pair is the Giselle data, the only non-interactive work in the analysis (the only work that isn't a performance of Variations V). So, this is good! If this is the only computational measure of similarity I have for my paper for the class, I think that would be good enough!

So, clearly, this is a good start. I would like to try this with recurrence plots, like the previous paper mentioned. However, apparently they screen people who have access to CRP toolkit, so I'll have to wait for the download link email (or reapply in 4 days?). If, on the off-chance I can't get access, I'll just make sure my similarity maps are up to snuff. These results aren't publication-ready, by any means, but OMG! a start. I definitely would also like more data for comparison.

Maybe I will be able to publish on some of my results and methods. I was feeling very gloomy about that prospect these past couple days until now. 

Finally! A paper for self-similarity matrix comparison in the Music Information Retrieval literature -- doing what I want it to do!

Measuring Structural Similarity in Music - Juan P. Bello

I can definitely use this to compare music to movement.... It must be sensitive to small variations since he is using it to compare different performances of the same piece of music. He is using it for tonal music, but I can just use MFCCs (& maybe amplitude or STSMPS?) instead of the chroma-based features and CENs he's using, since (except for the Giselle null comparison example, the music is not necessarily pitch-based that I am analyzing) But anyways, that won't matter. I just need the comparison methods.

He is also using Recurrence Plots instead of self-similarity matrices, but they are similar measures. He then uses the Normalized Comparison Difference (NCD) -- something that is common to use in bioinformatics for genetics comparison -- to come up with a similarity score between the recurrence plots. Then, it looks like he finds threshold values for pairwise similarity and then uses the measure to retrieve performances of the same work of music. The measure is not tolerant of global structure changes (eg, if someone repeats a section and another does not) but that actually doesn't even matter for my application.

Tools that he uses:

Toolbox for Recurrence plots:
http://tocsy.pik-potsdam.de/CRPtoolbox/

Normalized Compression Distance:
http://www.complearn.org/ncd.html

I also had this idea in the shower that perhaps I could define the relationship between the structure of dance and music by their distance matrices. So if I had enough data, I could do ANOTHER NCD on the NCD data and see if the different performances of Variations V music and movement relationship (ie, the distance matrix between their similarity measures  (whether RP or SSM) ) cluster together -- as opposed to other interactive and non-interactive dance.

I found this link which provides a lot of good information about music information retrieval -- mostly because I accessed the textbook for the course through my university's library (he had a lot of papers on structural analysis of audio via SSM so I just looked up his name). I thought about using the method described in the textbook for the segmentation of self-similarity matrices to determine what the actual repetitive structure is but I think that Bello's approach is more apropo to my musical analysis problem. I would definitely apply the segmentation / path method as well if I had time... but man, do I need this paper to be over.