jimmylao edited a comment on issue #52:
URL:
https://github.com/apache/incubator-bluemarlin/issues/52#issuecomment-1049939996
@Bimlesh759-AI
More data does not always generate better model. High quality data usually
generates better model.
Most of the time, real world data is noisy and there exists a lot of
redundancy. You may need to have more insight of the data.
1. Break-down information regarding your training/testing data? i.e.
- the number of positive and negative training/testing samples of all 19
items
- the number of positive and negative training/testing samples of each of
the 19 items
you may want to balance the number of training/testing samples of these 19
items.
2. Since your dataset is huge, it's quite essential to do parallel training.
I think you may try to use TF 2 for multiple GPU training already, how's your
progress on this topic? There's another option: TF 1 + horovod, to my
understanding, you tried this approach too, can you sync up what is the current
status of parallel training with multiple GPUs or machines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
