Hi Black Thanks for the reply but figured it out on my own. Posting the answer after this.
On Monday, July 11, 2016 at 11:42:10 PM UTC+5:30, Blake Miller wrote: > > Hi Punit > > The behavior you are referring to is a feature of the Scala compiler, > which is why it does not happen automatically when you try to use it from > Clojure. > > Please see the note here: > > > https://github.com/t6/from-scala/blob/4e1752aaa2ef835dd67a8404273bee067510a431/test/t6/from_scala/guide.clj#L161-L166 > > You may find that library a useful resource, either as a dependency or > simply as reference material. > > What you want to do is find the full method signature, including the > implicits, and invoke _that_ from clojure, passing values for all implicit > parameters (in this case, your custom ordering function. > > HTH > > On Saturday, July 9, 2016 at 6:13:17 AM UTC, Punit Naik wrote: >> >> Hi Ashish >> >> The "package" is indeed the full package name. >> On 09-Jul-2016 11:02 AM, "Ashish Negi" <[email protected]> wrote: >> >>> Should not be `package` in `:import` be the actual package name of ` >>> RFMCPartitioner` ? >>> >>> see examples at https://clojuredocs.org/clojure.core/import >>> >>> like : >>> >>> (ns foo.bar >>> (:import (java.util Date >>> Calendar) >>> (java.util.logging Logger >>> Level))) >>> >>> >>> >>> (ns xyz >>> (:import >>> [** RFMCPartitioner] >>> [** RFMCKey] >>> ) >>> ) >>> >>> >>> where ** is package full name. >>> >>> >>> >>> On Friday, 8 July 2016 21:31:27 UTC+5:30, Punit Naik wrote: >>>> >>>> >>>> >>>> >>>> I have a scala program in which I have implemented a secondary sort >>>> which works perfectly. The way I have written that program is: >>>> >>>> object rfmc { >>>> // Custom Key and partitioner >>>> >>>> case class RFMCKey(cId: String, R: Double, F: Double, M: Double, C: >>>> Double) >>>> class RFMCPartitioner(partitions: Int) extends Partitioner { >>>> require(partitions >= 0, "Number of partitions ($partitions) cannot be >>>> negative.") >>>> override def numPartitions: Int = partitions >>>> override def getPartition(key: Any): Int = { >>>> val k = key.asInstanceOf[RFMCKey] >>>> k.cId.hashCode() % numPartitions >>>> } >>>> } >>>> object RFMCKey { >>>> implicit def orderingBycId[A <: RFMCKey] : Ordering[A] = { >>>> Ordering.by(k => (k.R, k.F * -1, k.M * -1, k.C * -1)) >>>> } >>>> } >>>> // The body of the code >>>> // >>>> // >>>> val x = rdd.map(RFMCKey(cust,r,f,m,c), r+","+f+","+m+","+c) >>>> val y = x.repartitionAndSortWithinPartitions(new RFMCPartitioner(1))} >>>> >>>> I wanted to implement the same thing using clojure's DSL for spark >>>> called flambo. Since I can't write partitioner using clojure, I re-used >>>> the >>>> code defind above, compiled it and used it as a dependency in my Clojure >>>> code. >>>> >>>> Now I am importing the partitioner and the key in my clojure code the >>>> following way: >>>> >>>> (ns xyz >>>> (:import >>>> [package RFMCPartitioner] >>>> [package RFMCKey] >>>> ) >>>> ) >>>> >>>> But when I try to create RFMCKey by doing (RFMCKey. cust_id r f m c), >>>> it throws the following error: >>>> >>>> java.lang.ClassCastException: org.formcept.wisdom.RFMCKey cannot be cast >>>> to java.lang.Comparable >>>> at >>>> org.spark-project.guava.collect.NaturalOrdering.compare(NaturalOrdering.java:28) >>>> at >>>> scala.math.LowPriorityOrderingImplicits$$anon$7.compare(Ordering.scala:153) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter$$anon$8.compare(ExternalSorter.scala:170) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter$$anon$8.compare(ExternalSorter.scala:164) >>>> at >>>> org.apache.spark.util.collection.TimSort.countRunAndMakeAscending(TimSort.java:252) >>>> at org.apache.spark.util.collection.TimSort.sort(TimSort.java:110) >>>> at org.apache.spark.util.collection.Sorter.sort(Sorter.scala:37) >>>> at >>>> org.apache.spark.util.collection.SizeTrackingPairBuffer.destructiveSortedIterator(SizeTrackingPairBuffer.scala:83) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.partitionedIterator(ExternalSorter.scala:687) >>>> at >>>> org.apache.spark.util.collection.ExternalSorter.iterator(ExternalSorter.scala:705) >>>> at >>>> org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:64) >>>> at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92) >>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>>> at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:242) >>>> at >>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) >>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:244) >>>> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) >>>> at org.apache.spark.scheduler.Task.run(Task.scala:64) >>>> at >>>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> My guess is that its not able to find the ordering that I have defined >>>> after the partitioner. But if it works in Scala, why doesn't it work in >>>> Clojure? >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to [email protected] >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> [email protected] >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Clojure" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/clojure/ZoLWl_vbcdU/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to [email protected] Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
