Dear all,
I am trying to obtain the sensitivity values for all splits of the dataset
during leave-one-out cross-validation (classification using SVM). I found in
the tutorial "Classification Model Parameters – Sensitivity Analysis" (
http://www.pymvpa.org/tutorial_sensitivity.html ) that RepeatedMeasure(sensana,
NFoldPartitioner()) should give the sensitivity values for each fold. Here are
the code snippet I used in my script slightly adapted from the tutorial:
clf = LinearNuSVMC() cv = CrossValidation(clf,
NFoldPartitioner(),enable_ca=['stats']) sensana =
clf.get_sensitivity_analyzer() cv_sensana = RepeatedMeasure(sensana,
NFoldPartitioner()) error = cv(ds) sensmap_cv =
cv_sensana(ds)
'print sensmap_cv.shape' gave me: (14L, 87L).
I have 14 subjects and I am using leave-one-subject-out cross-validation, and
there are 87 features. So the data structure seems correct. However, when I
look at the values of this 14x87 array, all the rows in the array contain
exactly the same values (i.e., the first row looks the same with all the other
rows).
Is this the correct way to obtain the sensitivity values for each fold of
cross-validation classification? If not, any suggestions how to do?
A related question about normalizing the sensitivity values: in the "Closing
Words" of the tutorial on the same webpage, it says: "It should also be noted
that sensitivities can not be directly compared to each other, even if they
stem from the same algorithm and are just computed on different dataset splits.
In an analysis one would have to normalize them first." My question is: if we
cannot compare the sensitivity values from different data splits without
normalizing them first, why can we average them or take the maximum value
across data splits without applying any normalization (the example script
snippets in the tutorial seem to do so)? I would imagine that the average or
the max value would also be affected by the scale of the data.
Any help would be very much appreciated!
Best,Meng
_______________________________________________
Pkg-ExpPsy-PyMVPA mailing list
[email protected]
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
