I tried your idea of generating the size first, then passing it to the
matrix & vector generators. This does seem to work better. The shrunk cases
that return are actually worse, but so far it hasn't wedged itself, which
is a great improvement. They all return within a few seconds. I don't yet
understand the shrinking well enough to know why it's better.
Example shrunk state:
:samples ["0" "0" "0" "0" "0" "0" "0" "0" "0"], :probes ["0" "0" "0" "0"
"0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0"
"0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0"
"0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0" "0"
"0" "0" "0" "0"]
(def gen-matrix-size
(gen/tuple gen/s-pos-int gen/s-pos-int))
(def gen-mostly-ints
(gen/frequency [[9 gen/int] [1 (gen/return Double/NaN)]]))
(defn gen-matrix2 [x y]
(gen/vector (gen/vector gen-mostly-ints x) y))
(defn gen-names [n]
(gen/vector
(gen/such-that not-empty gen/string-alpha-numeric)
n))
(def gen-tsv2
(gen/bind
gen-matrix-size
(fn [[x y]]
(gen/hash-map
:probes (gen-names x)
:samples (gen-names y)
:matrix (gen-matrix2 x y)))))
On Wednesday, November 12, 2014 12:47:37 PM UTC-8, Lucas Bradstreet wrote:
>
> I'm pretty sure I did encounter the performance problem you're talking
> about, but I killed it and re-ran until I hit cases that shrink quickly.
> I'm afraid I'm not much help with those, although I agree that the bad
> shrinking is probably related to the performance issues.
>
> On 13 Nov 2014, at 04:22, Brian Craft <[email protected] <javascript:>>
> wrote:
>
> Interesting that you don't see a performance problem. What version did you
> try? I'm using 0.5.9.
>
> I just re-ran this example to make sure I wasn't imagining it. On the 11th
> run, it wedged, not returning, and burning a lot of cpu, presumably trying
> to shrink.
>
> It's a larger problem with the real test cases. One test ran for about
> eight hours yesterday before I killed it. Hard to tell what it's doing, and
> not sure what sort of times I should expect for shrinking larger test cases.
>
>
> On Wednesday, November 12, 2014 9:05:40 AM UTC-8, Lucas Bradstreet wrote:
>>
>> I've also had some tricky shrinking type issues with recursive generators
>> using bind. I had a play with your generators, using such-that to reduce
>> the row/column name length and also preventing some generator shrinking by
>> using no-shrink, but I didn't have much luck improving the resulting
>> shrinks (though the shrinks did finish in a timely fashion).
>>
>> You might have more luck if you generated an m x n matrix size
>> independently of the matrix itself and feed the size into the matrix
>> generator and the samples and probes generators. This may allow the matrix
>> size to be shrunk down more easily, while also shrinking the samples and
>> probes vectors without them having to be regenerated for ever new matrix
>> shrink (i.e. where the matrix size is maintained but the gen/int values
>> within it are shrunk).
>>
>> However, this is all guess work and conjecture, as I don't understand the
>> shrinking algorithm well enough, and I haven't tried my suggestion. Also,
>> modifying your generators in this way may make your model less general if
>> you wish to use it in other ways later.
>>
>> Cheers
>>
>> Lucas
>>
>>
>>
>> On 12 Nov 2014, at 02:43, Brian Craft <[email protected]> wrote:
>>
>> Using test.check, I'm finding the shrinking to be very, very slow.
>> Running a hundred cases takes a few seconds, unless it hits an error, in
>> which case it takes 40-60 minutes to shrink, and the shrinking is not very
>> effective (resulting test case is much larger than necessary). Sometimes
>> the shrinking is much faster. It behaves a bit like it's occasionally
>> getting into a pathological state, or a difficult shrinking scenario.
>>
>> Are there any docs on generators or the shrinking algorithm that would
>> help build tests that shrink more effectively?
>>
>> The problematic generator builds a randomly-sized matrix of integers,
>> with randomly assigned names for the rows and columns. The failure case is
>> when either a column or row name is repeated. I expect the slow shrinking
>> has something to do with it being rare for the generator to emit the same
>> name twice.
>>
>> ; Generator of randomly sized matrices of random numbers.
>> (def gen-matrix
>> (gen/bind
>> gen/s-pos-int
>> (fn [x] (gen/bind
>> gen/s-pos-int
>> (fn [y] (gen/vector (gen/vector gen/int x) y))))))
>>
>> ; Generator of matrix with probe and sample ids.
>> (def gen-tsv
>> (gen/bind
>> gen-matrix
>> (fn [m]
>> (gen/hash-map
>> :probes (gen/vector
>> (gen/such-that not-empty gen/string-alpha-numeric)
>> (count m))
>> :samples (gen/vector
>> (gen/such-that not-empty gen/string-alpha-numeric)
>> (count (first m)))
>> :matrix (gen/return m)))))
>>
>> Shrinking will result in a case like
>>
>> {:matrix [[1 4 -3] [-4 -3 -5] [-5 2 3] [4 -5 -5] [1 -2 3] [1 4 1]],
>> :samples ["0" "0" "0"], :probes ["0" "0" "0" "0" "0" "0"]}
>>
>> where :samples ["0" "0"] :probes["0"] would do.
>>
>> The following test will exhibit the behavior, sometimes succeeding,
>> sometimes failing quickly, sometimes shrinking for a very long time:
>>
>> (tc/quick-check 100 (prop/for-all [tsv gen-tsv] (= (count (set (:probes
>> tsv))) (count (:probes tsv)))))
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to [email protected]
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> [email protected]
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> For more options, visit https://groups.google.com/d/optout.
>>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to [email protected] <javascript:>
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> [email protected] <javascript:>
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected] <javascript:>.
> For more options, visit https://groups.google.com/d/optout.
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
