Thanks for the RFC. I like the idea of the config library concept. Some
concerns/questions:
- Same to @kevinthesun, I'd prefer to keep the current AutoTVM dispatch context
instead of introducing a new one. For example, we can just overload
`apply_history_best` to take either a JSON file like now or a config library in
this proposal.
- Current AutoTVM has a database module that stores tuned configs and results
to Redis database (altough no one is using this feature AFAIK). Considering the
compatibility and usability, we should hide the resume logic from users and
make all storage mechanisms consist. For example, we can first move the "log
to" callback to tuner arguments:
```
tuner_obj.tune(
n_trial=min(n_trial, len(tsk.config_space)),
early_stopping=early_stopping,
measure_option=measure_option,
config_library="history.json", # Can be either a string (JSON file
name) or a DB object
callbacks=[
autotvm.callback.progress_bar(n_trial, prefix=prefix),
],
)
```
In this way, we can implement the resume logic you proposed in the constructor
of tuners.
- For the rules that determine if the history can be reused for resuming the
tuning, I think it should be a general question as the tophub faces to.
However, I would not suggest to refer to the tuning specific information such
as trial numbers for the following reasons. First, users may require different
trial numbers or different measure options. Second, this limits the history be
used only for the exactly same task. If we design this resume logic in a
general way, we can also extend it to tophub.
In my personal opinion, when loading the history, in addition to checking if
the history config matches the current task in terms of target, op name, shapes
and attributes, we also need to check the device as you mentioned. We can try
to retrieve the target device info using system call and add it to every record
when dumping to file/database.
Besides, we also need to invalid the history/records when TVM has been updated,
because it may result in different performance even for the same config on the
same device. The simplest way is checking the timestamp. For example, we let
users config an expiration time when creating a tuner:
```
library_option = {"library": "history.json", "expiration": 1440}
tuner_obj.tune(
n_trial=min(n_trial, len(tsk.config_space)),
early_stopping=early_stopping,
measure_option=measure_option,
library_option=library_option,
callbacks=[
autotvm.callback.progress_bar(n_trial, prefix=prefix),
],
)
```
In this example, a user wants the tuner to load `history.json` and use all
records generated within 1440 minutes (1 day). One advantage of using this
approach is that we don't need to touch the searching part at all, but only let
`Tuner.tune` bypass the measurement when the record is available already.
Again thanks for the proposal, and any of my suggestions can be
adjusted/discussed if they are too overwhelming or unnecessary.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmlc/tvm/issues/4150#issuecomment-543970017
