Obviously I'm brain-dead, since I forgot to retry the write on failure. So
I fixed this now:
(defn advance
[message db]
{:pre [
(= (type message) durable_queue.Task)
]}
(let [
;; 2017-10-05 -- if this is successful, then the return will look
like this:
;; {:_index facts-over-time, :_type deduplicaton, :_id
investor-6757, :_version 1, :result created, :_shards {:total 2,
:successful 1, :failed 0}, :created true}
success-or-failure (persistence/push-item-to-persistence @message
db)
]
(if (= (get-in success-or-failure [:_shards :successful]) 1)
(durable/complete! message)
(durable/retry! message))
But my question remains: why would the app crawl at a glacial speed, till I
get those SocketConnection errors, and then it goes fast?
I'm not sure if this is a Clojure question or an ElasticSearch AWS
question. I'm also researching what sort of rate limits AWS imposes, so I
can figure out why I would get SocketConnection errors at first.
If I got zero writes at first, then I could believe that all 20 of my
threads were blocking, and then they all timeout, then they all start going
fast. I know AWS load balancing has an algorithm that only slowly adapts to
spikes in demand. I have trouble believing they would apply that kind of
load balancing to their ElasticSearch service, but that would explain what
I'm seeing. I have not been able to find a single article online that
suggests this.
On Thursday, October 5, 2017 at 12:32:11 PM UTC-4, [email protected]
wrote:
>
> One last thing, I should mention, if I go through and add " LIMIT 100 " to
> all the SQL queries, everything works great. That is, when dealing with a
> few hundred documents, the app seems to work perfectly, and there are no
> errors. It's only when I try to work with a few million documents that
> things fall apart. I suspect some kind of contention arises... somewhere.
> Possibly in several locations. But I'm not yet sure what the problem is.
>
>
>
> On Wednesday, October 4, 2017 at 12:49:12 AM UTC-4, [email protected]
> wrote:
>>
>> This is probably a stupid question, but is there an obvious way to get an
>> error message out of Elastisch? I had an app that was working with MongoDB
>> and I was then told I had to use ElasticSearch instead (something about
>> only using AWS for everything) so now I'm trying to get an error message,
>> because my code doesn't seem to work. I read through here without seeing
>> anything obvious:
>>
>> http://clojureelasticsearch.info/articles/getting_started.htm
>>
>> I rewrote my MongoDB function, so it should work with Elastisch:
>>
>> (defn push-item-to-persistence
>> [item db]
>> (let [
>> denormalized-id (get-in item [:denormalized-id] :no-id)
>> item (assoc item :updated-at (temporal/current-time-as-datetime))
>> item (assoc item :permanent-holding-id-for-item-instances
>> (java.util.UUID/randomUUID))
>> item (assoc item :instance-id-for-this-one-item
>> (java.util.UUID/randomUUID))
>> item (assoc item :item-type :deduplication)
>> ]
>> (if (= denormalized-id :no-id)
>> (slingshot/throw+ {
>> :type
>> ::no-denormalized-id-in-push-item-into-database
>> :item item
>> })
>> (slingshot/try+
>> (put conn index mapping-type id document)
>> (esd/put db "facts-over-time" "deduplicaton" (str denormalized-id)
>> item)
>> (println " done with put in push-item-to-persistence ")
>> (catch Object o
>> (slingshot/throw+ {
>> :type ::push-item-to-persistence
>> :error o
>> :item item
>> :db db
>> }
>> ))))))
>>
>>
>> It doesn't seem that any documents are getting into ElasticSearch. I was
>> hoping to the (throw) would reveal to me some useful debugging information,
>> but that doesn't seem to happen.
>>
>> The connection appears to be valid, as I can do this:
>>
>> (let [conn (persistence/multi-thread-start config)
>> res (esd/search conn "facts-over-time"
>> "deduplication" :query {:match_all {}})
>> n (esrsp/total-hits res)
>> hits (esrsp/hits-from res)]
>> (clojure.pprint/pprint res))
>>
>> and I get:
>>
>> {:took 1,
>> :timed_out false,
>> :_shards {:total 5, :successful 5, :failed 0},
>> :hits {:total 0, :max_score nil, :hits []}}
>>
>> So the connection is there. But no records are.
>>
>>
>>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
