[
https://issues.apache.org/jira/browse/LUCENE-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050413#comment-17050413
]
Michael McCandless commented on LUCENE-9191:
--------------------------------------------
I plan to commit this soon ... it should improve the efficiency of tests using
{{LineFileDocs}} since the up-front random seeking is much more efficient now.
These docs are derived from the [Europarl parallel corpus
v7|https://www.statmt.org/europarl/], and then randomly split into smallish
documents, one per line, and then broken into 20 MB, 200 MB and 2000 MB source
files (before compression). I'll commit the 20 MB file here, along with the
Python script that creates the random files.
I also copied all the files up to {{home.apache.org}}: [200
MB|http://home.apache.org/~mikemccand/200mb.txt.gz] (and its [.seek
file|http://home.apache.org/~mikemccand/200mb.txt.seek]), and [2000
MB|http://home.apache.org/~mikemccand/2000mb.txt.gz] (and its [.seek
file|http://home.apache.org/~mikemccand/2000mb.txt.seek]), in case developers
want to test on a wider set of random docs :)
> Fix linefiledocs compression or replace in tests
> ------------------------------------------------
>
> Key: LUCENE-9191
> URL: https://issues.apache.org/jira/browse/LUCENE-9191
> Project: Lucene - Core
> Issue Type: Task
> Reporter: Robert Muir
> Assignee: Michael McCandless
> Priority: Major
> Attachments: LUCENE-9191.patch, LUCENE-9191.patch
>
>
> LineFileDocs(random) is very slow, even to open. It does a very slow "random
> skip" through a gzip compressed file.
> For the analyzers tests, in LUCENE-9186 I simply removed its usage, since
> TestUtil.randomAnalysisString is superior, and fast. But we should address
> other tests using it, since LineFileDocs(random) is slow!
> I think it is also the case that every lucene test has probably tested every
> LineFileDocs line many times now, whereas randomAnalysisString will invent
> new ones.
> Alternatively, we could "fix" LineFileDocs(random), e.g. special compression
> options (in blocks)... deflate supports such stuff. But it would make it even
> hairier than it is now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
