(Sent the below email earlier but for some reason didn't make it to the list)
I wasn't aware of the group-by function. Being new to clojure, I spent a lot of
time coming up with my complex function. Seeing the one line solution really
blew me away. Thanks!!
Follow up questions on my problem (my code again first):
;data is a list of records (maps)
(def data '({:a1 2 :a2 34 :a3 76 :a4 87},
{:a1 3 :a2 30 :a3 38 :a4 39},
{:a1 2 :a2 67 :a3 32 :a4 38},
{:a1 4 :a2 47 :a3 73 :a4 38},
{:a1 3 :a2 84 :a3 86 :a4 63}))
;result is a map, key = value of :a1, val = all records for that value of a1
;as pointed out, result can be obtained by (group-by :a1 data)
(def result
'{2 [{:a1 2 :a2 34 :a3 76 :a4 87}, {:a1 2 :a2 67 :a3 32 :a4 39}], ;grouped a1
= 2
3 [{:a1 3 :a2 30 :a3 38 :a4 39}, {:a1 3 :a2 84 :a3 86 :a4 63}], ;grouped a1
= 3
4 [{:a1 4 :a2 47 :a3 73 :a4 38}] ;grouped :a1 = 4
})
;groups the data by :a1 (see result above)
(def data-g (group-by :a1 data)
;for a key value in data-g returns the total of all :a4 values
(defn calc-tot-a4 [data-g k]
(reduce + (map :a4 (get data-g k))))
;adjust a4 values for a given key
(defn adj-a4 [data k]
(assoc data k (map #(assoc % :a4 (+ (:a3 %) (:a4 %))) (get data k))))
;adjust data
(defn adj-data [data f]
(reduce f data (keys data)))
;make adjustments to the grouped data
(def data-adj (adj-data data-g adj-a4))
I am expressing my domain specific problem in a very abstract way above. In
reality the data I am manipulating comes from a database. This is financial
performance data for an entire month. The data set is large - about 50,000
records. I need to group it by "day" and then perform manipulations on it. For
example, I need to make adjustments to the values loaded from the database
based on certain conditions. I need to aggregate values for a certain day etc.
My question is - am I on the right track with the adj-data/adj-a4 functions.
Secondly since I am constantly modifying the in-memory data (loaded using
contrib.sql functions which loads it into plain maps), is this inefficient
since the data structures are immutable. So every time I change a value,
clojure functions will return me a new data structure. Will this be inefficient
when I am dealing with a large data set and making constant
modifications/adjustments to the values?
Thanks for your great help.
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
Alan
Sent: Sunday, April 17, 2011 5:18 PM
To: Clojure
Subject: Re: Help with function to partition my data
Simpler still: group-by?
(group-by :a1 data) produces the result that Shoeb was looking for.
PS The original message isn't in Google's archives either.
On Apr 17, 7:54 am, Nate Austin <[email protected]> wrote:
> I didn't see this original email for some reason, but here's a
> simplified version:
>
> (reduce
> (fn [coll {a1 :a1 :as value}] (update-in coll [a1] conj value))
> {}
> data)
>
> This takes advantage of update-in passing nil if it doesn't have the
> key and conj with nil returning a seq of one value.
>
>
>
>
>
>
>
> On Sun, Apr 17, 2011 at 12:30 AM, Shoeb Bhinderwala <[email protected]> wrote:
> > Wrote the following function which did the trick:
>
> > (defn partn [data]
> > (let
> > [add-mapping
> > (fn [m v]
> > (assoc m v (filter #(= v (:a1 %)) data)))]
> > (reduce add-mapping {} (set (map #(:a1 %) data))))
>
> > On Sat, Apr 16, 2011 at 8:15 PM, shuaybi <[email protected]> wrote:
>
> >> I am trying to write a function but can't get it right. I need to loop
> >> through my data records and for each unique value of :a1 I need to
> >> partition the data and create a resulting map so that the key is value
> >> of :a1 and the associated value of this key is all records with that
> >> value of :a1. Better explained with code, where given the definition
> >> of data below, I need to write a function that will return the
> >> definition of "result" below.
>
> >> ;data is a list of records (maps)
> >> (def data '({:a1 2 :a2 34 :a3 76 :a4 87},
> >> {:a1 3 :a2 30 :a3 38 :a4 39},
> >> {:a1 2 :a2 67 :a3 32 :a4 38},
> >> {:a1 4 :a2 47 :a3 73 :a4 38},
> >> {:a1 3 :a2 84 :a3 86 :a4 63}))
>
> >> ;result should be a map, keyed by unique values of :a1 and containing
> >> all records for that value of :a1
> >> (def result '{2 ({:a1 2 :a2 34 :a3 76 :a4 87}, {:a1 2 :a2 67 :a3
> >> 32 :a4 39}), ;list of records where :a1 = 2
> >> 3 ({:a1 3 :a2 30 :a3 38 :a4 39}, {:a1 3 :a2 84 :a3
> >> 86 :a4 63}), ;list of records where :a1 = 3
> >> 4 ({:a1 4 :a2 47 :a3 73 :a4 38}) ;list of records
> >> where :a1 = 4
> >> })
>
> >> (defn get-val-a1
> >> "Return all records in data where :a1 is specified val"
> >> [data val]
> >> (filter
> >> #(= val (:a1 %)) ;filter function
> >> data))
>
> >> (defn unique-a1
> >> "Get set of all unique :a1 values"
> >> [m]
> >> (set (map #(:a1 %) m)))
>
> >> ;This is the function I am trying to write but can't get it right.
> >> (defn partn [data]
> >> (let [val-set (unique-a1 data)) ;get unique :a1 values
> >> ret-val {}]
> >> (loop [v val-set] ;loop through the unique values
> >> (assoc ret-val v (get-val-a1 data v))) ;build final map
> >> ret-val))
>
> > --
> > You received this message because you are subscribed to the Google
> > Groups "Clojure" group.
> > To post to this group, send email to [email protected]
> > Note that posts from new members are moderated - please be patient with your
> > first post.
> > To unsubscribe from this group, send email to
> > [email protected]
> > For more options, visit this group at
> >http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
