I have a plain text file containing an English-language essay I want to
split into sentences, based on common punctuation.
I wrote this function, which examines a character and determines if it's an
end of sentence punctuation mark:
(defn ispunc? [c]
(> (count (filter #(= % c) '("." "!" "?" ";"))) 0))
I know this is no grammatically perfect, and that some text such as "U.S.",
etc. will be mis-parsed, but this is just an experiment and I don't need
that level of precision.
So I loaded my file using slurp and tried using the partition-by function
with ispunc? like this:
(def my-text (slurp "mytext.txt"))
(def my-sentences (partition-by ispunc? my-text))
Unfortunately, this returns a sequence of 1, where the only element is the
entire string.
So I tried splitting the string into a list of characters, and applying
partition-by with ispunc? like this:
(def my-text-chars (partition (count my-text) my-text))
(def my-sentences (partition-by ispunc? (nth my-text-chars 0)))
This worked, because it is logically correct, but I get
a java.lang.OutOfMemoryError when I try to access any of the elements in
my-sentences (the plain text "mytext.txt" file is 1.3 mb in size).
So is there a way to do this more idiomatically, without splitting into
single chars and recombining?
While 1.3 mb is not small, it's also not so large that it can't be slurped,
so there must be a simpler way of splitting on punctuation into sentences.
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
