Hi Ryan.
By now it is hard for me to imagine how to make grid search optimiser to have a
similar interface to already implemented optimisers like SGD since they work in
slightly different domains. I guess a reasonable interface for grid search
optimiser will allow such usage.
arma::mat data /* = ... */;
arma::Row<size_t> labels /* = ... */;
GridSearchOptimizer<SoftmaxRegression<>, Accuracy, KFoldCV>
softmaxOptimizer(data, labels);
std::array<size_t, 1> numClasses = {5};
arma::vec lambdas = arma::logspace(-3, 1); // {0.001, 0.01, 0.1, 1}
std::tuple<size_t, double> bestSoftmaxParams =
softmaxOptimizer.Optimize(numClasses, lambdas);
double bestSoftmaxAccuracy = softmaxOptimizer.BestMeasurement();
SoftmaxRegression<>& bestSoftmaxModel = softmaxOptimizer.BestModel();
Here when we call Optimize we pass a collection of possible values for each
hyper-parameter (each additional constructor argument) of SoftmaxRegression<>.
A similar interface of grid search for hyper-parameters is provided by
scikit-learn:
http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html
<http://scikit-learn.org/stable/auto_examples/model_selection/grid_search_digits.html>.
The result of the Optimize call is a tuple of selected values (one per
hyper-parameter) - I think it is good to return it here since only here we can
deduct what type the tuple should have (if we want to be able to provide the
tuple of selected hyper-parameters by another call after optimizing, we need to
pass information about the tuple type when constructing a GridSearchOptimizer
object).
On the other hand, SGT has the following signature for Optimize:
double Optimize(arma::mat& iterate);
which I think can’t be easily utilized by grid search. Maybe you have some
ideas?
Best regards,
Kirill Mishchenko
> On 4 Apr 2017, at 22:28, Ryan Curtin <[email protected]> wrote:
>
> On Sat, Apr 01, 2017 at 10:55:45AM +0500, Kirill Mishchenko wrote:
>> Hi Ryan.
>>
>> I’m planning to implement the following functionality as a GSoC project:
>> Measurements
>> Accuracy
>> Mean squared error
>> Precision
>> Recall
>> F1
>> Validation
>> Simple validation (splitting data once with validation set size specified by
>> a user)
>> K-fold cross validation
>> Hyper-parameter tuning
>> Grid search based tuning
>>
>> Does it seem as a reasonable set of functionality? I have decided to
>> include simple validation since it can be more appropriate when we
>> have a lot of training data or when training is a time consuming
>> process.
>
> Hi Kirill,
>
> That sounds good to me. It would be nice if the grid search optimizer
> could have a similar API to the mlpack optimizers, so in that way,
> possibly other optimizers could be used. (i.e. you could use SGD to
> find the best value of some continuous parameters. This strategy falls
> apart a little bit when the hyperparameters are not of continuous type,
> so clearly we'll need to extend the OptimizerType paradigm at least a
> little bit. But we can figure that out when we get there.)
>
> Thanks,
>
> Ryan
>
> --
> Ryan Curtin | "Do I sound like I'm ordering a pizza?"
> [email protected] <mailto:[email protected]> | - John McClane
_______________________________________________
mlpack mailing list
[email protected]
http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack
