GitHub user mars opened a pull request:
https://github.com/apache/incubator-predictionio/pull/424
Implement Storage app & channel name-to-ID cache
When stress testing the Universal Recommender with high-concurrency
HTTP/REST queries, we observed that Elasticsearch traffic was majority composed
of requests resolving the Storage app's name & channel, over and over and over
again! In this case, [each per-query call to
`LEventStore.findByEntity`](https://github.com/heroku/predictionio-engine-ur/blob/master/src/main/scala/URAlgorithm.scala#L694)
re-resolves the app name to an ID.
This changeset implements memoization for the function that performs these
name-to-ID lookups, so that only one set of lookups is performed per process
for each app+channel combination. As a result, we've seen overall throughput
increase ð and error rate drop dramatically ð.
This common optimization effects all storage backends, not just
Elasticsearch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mars/incubator-predictionio
cache-storage-name-to-id
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-predictionio/pull/424.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #424
----
commit 9825ae2a6981431ce49a6ea40ddabd82ab4121f2
Author: Mars Hall <[email protected]>
Date: 2017-08-22T18:48:04Z
Implement Storage app & channel name-to-ID cache
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
