Sorry, I am stupid. I misunderstood the problem entirely. Escaped quote
marks are never really the problem. The real issue is that this works for
us:
{ "money" : "100000", "contact_name" : "Martha Vena" }
and this doesn't work for us:
"{ \"money\" : \"100000\", \"contact_name\" : \"Martha Vena\" }"
but in the second case the whole thing is a string. In the first case I
suppose the structure is a hashmap. I'll look at Carmine to see if it can
help me get the hashmap structure consistently.
On Friday, July 10, 2015 at 6:04:01 PM UTC-4, [email protected] wrote:
>
>
> Hmm, well, I am grateful to you for running such a detailed test, and I
> should have tested this myself:
>
> >The actual bytes stored at the "test" key are 0x61 0x22 0x62
>
> However, something in our code fails when the quote marks are escaped, but
> everything works the way we expect when the quote marks do not appear to be
> escaped. I amazed by this:
>
> user=> (count *1)
> 3
>
> Perhaps the error is something entirely different from what I was
> assuming.
>
>
>
>
>
> On Thursday, July 9, 2015 at 7:52:54 PM UTC-4, Francis Avila wrote:
>>
>> This is what I am doing with Carmine and it seems to work properly.
>>
>> First some setup in the repl:
>>
>> user=>(require '[taoensso.carmine :as car])
>> nil
>> user=> (def conn {:pool {} :spec {}})
>> #'user/conn
>> user=> (defn raw-str-set [k ^String v] (car/set k (car/raw (.getBytes v
>> "UTF-8"))))
>> #'user/raw-str-set
>> user=> (defn raw-str-get [k] (car/parse (car/get k) #(String. ^bytes %
>> "UTF-8")))
>> #'user/raw-str-get
>>
>>
>> Now I save a test string:
>>
>> user=> (car/wcar conn (raw-str-set "test" "a\"b"))
>> "OK"
>>
>> In a redis-cli terminal:
>>
>> $ redis-cli
>> 127.0.0.1:6379> get "test"
>> "a\"b"
>> 127.0.0.1:6379>
>>
>> The escaped \" you see in redis-cli is just for printing. The actual
>> bytes stored at the "test" key are 0x61 0x22 0x62
>>
>> Now lets pull this out of redis with Carmine:
>>
>>
>> user=> (car/wcar conn (raw-str-get "test"))
>> "a\"b"
>> user=> (count *1)
>> 3
>>
>> As you can see, we round-tripped without any strange extra escaping.
>>
>> Probably your java app can just use Jedis client.get("test") and
>> client.set("test","a-string"), since I think it does UTF-8 encode and
>> decode of strings and no other serialization.
>>
>>
>> You still need to use JSON on top of this so you can get richer data
>> structures instead of just strings.
>>
>> On Thursday, July 9, 2015 at 4:18:51 PM UTC-5, [email protected]
>> wrote:
>>>
>>> I am sorry, I should have explained this sooner. When I do this:
>>>
>>> (.getBytes document-as-string "UTF-8")
>>>
>>> The quote marks are always escaped. When I do this:
>>>
>>> document-as-byte-array (bytes (byte-array (map (comp byte int)
>>> document-as-string)))
>>>
>>> The quote marks are sometimes escaped, but most of the time they are
>>> not.
>>>
>>> I need to avoid having those quote marks escaped. I am not clear what is
>>> causing the escaping.
>>>
>>>
>>>
>>>
>>> On Thursday, July 9, 2015 at 12:01:51 PM UTC-4, Francis Avila wrote:
>>>>
>>>> your document-as-byte-array is wrong--it only handles ascii, and very
>>>> inefficiently too.
>>>>
>>>> Just say (.getBytes document-as-string "UTF-8") to get a utf-8 encoding
>>>> of the string as a byte array.
>>>>
>>>> On Thu, Jul 9, 2015 at 10:41 AM, <[email protected]> wrote:
>>>>
>>>>>
>>>>> > As for the escaped quotes, you may be using pr or prn to print, or
>>>>> maybe you are
>>>>> > using pr-str to produce the string representation. I can't be sure.
>>>>>
>>>>> At the moment I create the string like this:
>>>>>
>>>>> document-as-string (str "{\"transaction-id\" : \""
>>>>> transaction-id "\", \"message\" : \"" message "\"}")
>>>>> document-as-byte-array (bytes (byte-array (map (comp byte int)
>>>>> document-as-string)))
>>>>>
>>>>> The crazy thing is that this sometimes works, but other times the
>>>>> quote marks appear in Redis as escaped quote marks. As near as I can
>>>>> tell,
>>>>> the important factor is the length of the string. A short string is
>>>>> likely
>>>>> to have its quote marks escaped. A long string does not have its quote
>>>>> marks escaped.
>>>>>
>>>>> My co-worker is working on the Java app. I am working on the Clojure
>>>>> app. We can both adjust out apps freely, just so long as we can get data
>>>>> to
>>>>> and from each other, in a manner that allows us to eventually cast the
>>>>> data
>>>>> to and from JSON.
>>>>>
>>>>> Any suggestions are welcome.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wednesday, July 8, 2015 at 8:58:53 PM UTC-4, Francis Avila wrote:
>>>>>>
>>>>>> Who is saving these strings, and who is reading them? Do you have
>>>>>> complete control over both apps, or does one of them need to be aligned
>>>>>> with the other?
>>>>>>
>>>>>> If the Java app is the baseline, you need to know the exact details
>>>>>> of the format of the data it saves. Just knowing "it's JSON" is not
>>>>>> enough,
>>>>>> because there may be other data types (e.g. dates) that don't have
>>>>>> native
>>>>>> JSON representations. (The go-to JSON de/encoder for clojure is
>>>>>> cheshire:
>>>>>> https://github.com/dakrone/cheshire )
>>>>>>
>>>>>> I took a quick look at Jedis and the easy, default way of using it is
>>>>>> with strings. It will encode strings to UTF-8 before sending to Redis
>>>>>> and
>>>>>> decode from UTF-8 on read. You can set raw byte arrays too (which will
>>>>>> not
>>>>>> be altered in any way before sending), but it's not clear to me how it
>>>>>> can
>>>>>> read out raw byte arrays. (I'm sure there's a way, but it's not
>>>>>> immediately
>>>>>> obvious.)
>>>>>>
>>>>>> As for the escaped quotes, you may be using pr or prn to print, or
>>>>>> maybe you are using pr-str to produce the string representation. I can't
>>>>>> be
>>>>>> sure.
>>>>>>
>>>>>> On Wednesday, July 8, 2015 at 5:31:27 PM UTC-5, [email protected]
>>>>>> wrote:
>>>>>>>
>>>>>>> And I have another stupid question. Using the above code, I am
>>>>>>> sometimes getting strings in Redis that have escaped quotation marks,
>>>>>>> like
>>>>>>> this:
>>>>>>>
>>>>>>> " \"transaction-id\" : \" 1ec47c2e-21ee-427c-841c-80a0f89f55d7 \"
>>>>>>> \"debrief\" : \" Susan Hilly at Citi called to get a quotation for
>>>>>>> discounted weekly car rental for approximately 3 cars per week, or 150
>>>>>>> rentals annually. \" "
>>>>>>>
>>>>>>> Why is that happening?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wednesday, July 8, 2015 at 5:38:20 PM UTC-4, [email protected]
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Francis Avila,
>>>>>>>>
>>>>>>>> Thank you for your response. The Java app is using Jedis and the
>>>>>>>> Clojure app is using Carmine. I'm wondering if you can suggest what
>>>>>>>> you
>>>>>>>> think would be the easiest way to allow these 2 apps to understand
>>>>>>>> each
>>>>>>>> other's strings?
>>>>>>>>
>>>>>>>> You were correct about how unsafe the above code was. I tested it
>>>>>>>> for less than 15 minutes and ran into the fact that a \n newline made
>>>>>>>> a
>>>>>>>> mess of everything.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wednesday, July 8, 2015 at 5:15:35 PM UTC-4, Francis Avila wrote:
>>>>>>>>>
>>>>>>>>> You are running into Carmine's automatic nippy serialization.
>>>>>>>>> https://github.com/ptaoussanis/carmine#serialization
>>>>>>>>>
>>>>>>>>> Redis only stores byte arrays (what it calls "strings"). Carmine
>>>>>>>>> uses the nippy library (the meaning of "NPY" in your byte stream) to
>>>>>>>>> represent rich types compactly as bytes.
>>>>>>>>> https://github.com/ptaoussanis/nippy
>>>>>>>>>
>>>>>>>>> If you give Carmine a byte array to store, it will store it
>>>>>>>>> directly without nippy-encoding it. E.g. (.getBytes "{}" "UTF-8")
>>>>>>>>>
>>>>>>>>> BTW your document-as-string example is extremely unsafe: how will
>>>>>>>>> you reliably read this message out again? e.g. what if the 'debrief'
>>>>>>>>> string
>>>>>>>>> contains a single quote? Use a proper serialization format.
>>>>>>>>>
>>>>>>>>> So the key is to have both your Clojure and Java app store *bytes*
>>>>>>>>> in Redis using the same serialization. You can store anything you
>>>>>>>>> want
>>>>>>>>> (nippy, utf-8-encoded json, fressian, bson, utf-8 xml, utf-16 java
>>>>>>>>> strings,
>>>>>>>>> whatever) as long as it's bytes and it's read and written the same
>>>>>>>>> way in
>>>>>>>>> all your apps.
>>>>>>>>>
>>>>>>>>> The Redis library your Java app is using may have its own
>>>>>>>>> automatic de/serialization, too. You need to find out what it's doing
>>>>>>>>> and
>>>>>>>>> either work with this or turn it off, just like with Carmine.
>>>>>>>>>
>>>>>>>>> Nippy unfortunately does not have a Java API out of the box:
>>>>>>>>> https://github.com/ptaoussanis/nippy/issues/66
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wednesday, July 8, 2015 at 3:35:49 PM UTC-5,
>>>>>>>>> [email protected] wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am not sure if this is a Clojure question or a Java question. I
>>>>>>>>>> don't know Java, so I could use whatever help folks can offer.
>>>>>>>>>>
>>>>>>>>>> "primitive string" here means what I can write when I am at the
>>>>>>>>>> terminal.
>>>>>>>>>>
>>>>>>>>>> We have 2 apps, one in Clojure, one in Java. They talk to each
>>>>>>>>>> other via Redis. I know the Java app can read stuff out of Redis,
>>>>>>>>>> using our
>>>>>>>>>> "transaction-id", if I use the terminal and open up "redis-clj" and
>>>>>>>>>> write a
>>>>>>>>>> string directly from the terminal. But I have this Clojure code,
>>>>>>>>>> which
>>>>>>>>>> depends on Peter Taoussanis's Carmine library:
>>>>>>>>>> (defn worker [document]
>>>>>>>>>> {:pre [(string? (:transaction-id document))]}
>>>>>>>>>> (let [transaction-id (:transaction-id document)
>>>>>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>>>>> redis-connection {:pool {} :spec {:host "127.0.0.1" :port
>>>>>>>>>> 6379 }}]
>>>>>>>>>> (timbre/log :trace " message we will send to NLP "
>>>>>>>>>> document-as-string)
>>>>>>>>>> (carmine/wcar redis-connection (carmine/set transaction-id
>>>>>>>>>> document))
>>>>>>>>>> (loop [document-in-redis (carmine/wcar redis-connection
>>>>>>>>>> (carmine/get transaction-id))]
>>>>>>>>>>
>>>>>>>>>> (if-not (.contains (first document-in-redis) "processed")
>>>>>>>>>> (recur (carmine/wcar redis-connection (carmine/get
>>>>>>>>>> transaction-id)))
>>>>>>>>>> (do
>>>>>>>>>> (carmine/wcar redis-connection (carmine/del
>>>>>>>>>> transaction-id))
>>>>>>>>>> document-in-redis)))))
>>>>>>>>>>
>>>>>>>>>> This line in particular, I have tried doing this several ways:
>>>>>>>>>>
>>>>>>>>>> document-as-string (str "{'transaction-id' : '"
>>>>>>>>>> transaction-id "', 'debrief' : '" (:debrief document) "'}" )
>>>>>>>>>>
>>>>>>>>>> In Redis, I expect to see:
>>>>>>>>>>
>>>>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358',
>>>>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig
>>>>>>>>>> responded
>>>>>>>>>> to our proposal and said his company is read to move forward. The
>>>>>>>>>> rate of
>>>>>>>>>> $400 per ton of shredded paper was acceptable to them, and they
>>>>>>>>>> shred about
>>>>>>>>>> 2 tons of documents every month. $96,000 in potential revenue
>>>>>>>>>> annually. I
>>>>>>>>>> will meet with him tomorrow and we will sign the contract.'}
>>>>>>>>>>
>>>>>>>>>> But if I then launch redis-cli, I see:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 127.0.0.1:6379> keys *
>>>>>>>>>> 1) "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>>>>>
>>>>>>>>>> 127.0.0.1:6379> get "42e574e7-3b80-424a-b9ff-01072f1e0358"
>>>>>>>>>> "\x00>NPY\b\x00\x00\x01\xfc\xf1\xfe\x1b\x00\x00\x00\nj\nip-addressi\x0e165.254.84.238j\x05tokeni$46b87d64-cff3-4b8b-895c-e089ac59544dj\x0bapi-versioni\x02v1j\x0etransaction-idi$42e574e7-3b80-424a-b9ff-01072f1e0358j\adebrief\r\x00\x00\x01YSmeek
>>>>>>>>>>
>>>>>>>>>> Hallie of Withers, Smeg, Harrington and Norvig responded to our
>>>>>>>>>> proposal
>>>>>>>>>> and said his company is rea-\x00\xf1\x06move forward. The
>>>>>>>>>> raty\x00\xf0\x0c$400 per ton of shredded pa\x16\x00\xf1\bwas
>>>>>>>>>> acceptable to
>>>>>>>>>> them,q\x00Bthey0\x00\x80 about 2E\x00\x10sF\x00\xf1Ldocuments every
>>>>>>>>>> month.
>>>>>>>>>> $96,000 in potential revenue annually. I will meet with him
>>>>>>>>>> tomorrow{\x00\"we#\x00@sign\x92\x00\xa0 contract."
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I don't know what all of those extra characters are. The Java app
>>>>>>>>>> is not picking this item up, so I assume the Java app is not seeing
>>>>>>>>>> this as
>>>>>>>>>> a string. I expected this to look the same as if I had written this
>>>>>>>>>> at the
>>>>>>>>>> terminal:
>>>>>>>>>>
>>>>>>>>>> {'transaction-id' : '42e574e7-3b80-424a-b9ff-01072f1e0358',
>>>>>>>>>> 'debrief' : 'Smeek Hallie of Withers, Smeg, Harrington and Norvig
>>>>>>>>>> responded
>>>>>>>>>> to our proposal and said his company is read to move forward. The
>>>>>>>>>> rate of
>>>>>>>>>> $400 per ton of shredded paper was acceptable to them, and they
>>>>>>>>>> shred about
>>>>>>>>>> 2 tons of documents every month. $96,000 in potential revenue
>>>>>>>>>> annually. I
>>>>>>>>>> will meet with him tomorrow and we will sign the contract.'}
>>>>>>>>>>
>>>>>>>>>> I assume it is easy to get a string into a format that can be
>>>>>>>>>> understood by both a Clojure app and a Java app. I don't care what
>>>>>>>>>> format
>>>>>>>>>> that is, but it needs to be consistent.
>>>>>>>>>>
>>>>>>>>>> Can anyone make suggestions about what I can do to make sure the
>>>>>>>>>> Clojure app and the Java app both write to Redis using a format that
>>>>>>>>>> the
>>>>>>>>>> other will understand? In particular, both apps need to see the
>>>>>>>>>> "'transaction-id".
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to [email protected]
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> [email protected]
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to a topic in the
>>>>> Google Groups "Clojure" group.
>>>>> To unsubscribe from this topic, visit
>>>>> https://groups.google.com/d/topic/clojure/9swfBxAbT90/unsubscribe.
>>>>> To unsubscribe from this group and all its topics, send an email to
>>>>> [email protected].
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>>
>>>>
>>>>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
