Package: lrzip Version: 0.552+20110217+gitcd8b086-1 Severity: normal [NOTE: This still seems to be a problem with the most recent upstream version of 0.603 - try with "lrzip -lvv" - hence Cc:ing to upstream Con Kolivas.]
Hi,
While trying to compress a big (172 GiB) file (with lrzip -nvv), I ran
into this error:
------------------------------------------------------------
Starting thread 7 to compress 19381057 bytes from stream 1
Starting thread 0 to compress 19381057 bytes from stream 1
pthread_createResource temporarily unavailable
Fatal error - exiting
------------------------------------------------------------
It turns out lrzip does not pthread_join() the threads it creates,
which in turn eventually exhausts the threads. You can verify this by
modifying it to print some debug information:
------------------------------------------------------------
--- lrzip-0.552+20110217+gitcd8b086/stream.c 2011-02-17 00:32:01.000000000
+0200
+++ lrzip.mod/stream.c 2011-04-22 19:16:04.000000000 +0300
@@ -71,15 +71,21 @@
fatal("pthread_cond_broadcast failed");
}
+static int threads_created=0, threads_joined=0;
+
void create_pthread(pthread_t * thread, pthread_attr_t * attr,
void * (*start_routine)(void *), void *arg)
{
+ print_maxverbose("create_pthread: created %d, joined %d\n",
+ ++threads_created, threads_joined);
if (pthread_create(thread, attr, start_routine, arg))
fatal("pthread_create");
}
void join_pthread(pthread_t th, void **thread_return)
{
+ print_maxverbose("join_pthread: created %d, joined %d\n",
+ threads_created, ++threads_joined);
if (pthread_join(th, thread_return))
fatal("pthread_join");
}
------------------------------------------------------------
Now it is evident that join_pthread() is not called (and there is no
other place in the source that would call pthread_join()):
------------------------------------------------------------
$ lrzip -nvv root.dump
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 8
Detected 8362065920 bytes ram
Compression level 7
Nice Value: 19
Show Progress
Max Verbose
Compression mode is: RZIP pre-processing only
Output filename is: root.dump.lrz
File size: 184045373440
Enabling sliding mmap mode and using mmap of 2787352576 bytes with window of
5574709248 bytes
Succeeded in testing 2787352576 sized mmap for rzip pre-processing
Will take 34 passes
Chunk size: 5574709248
Byte width: 5
Succeeded in testing 3498805584 sized malloc for back end compression
Using 8 threads to compress up to 218675349 bytes each.
Beginning rzip pre-processing phase
hashsize = 4194304. bits = 22. 64MB
Starting sweep for mask 1
Starting sweep for mask 3
Starting sweep for mask 7
Starting sweep for mask 15
Starting sweep for mask 31
Starting sweep for mask 63
Starting thread 0 to compress 218675349 bytes from stream 1
create_pthread: created 1, joined 0
Thread 0 writing 218675349 compressed bytes from stream 1
Starting thread 1 to compress 218675349 bytes from stream 1
create_pthread: created 2, joined 0
Thread 1 writing 218675349 compressed bytes from stream 1
Starting sweep for mask 127
Starting thread 2 to compress 218675349 bytes from stream 1
create_pthread: created 3, joined 0
Thread 2 writing 218675349 compressed bytes from stream 1
Starting thread 3 to compress 218675349 bytes from stream 1
create_pthread: created 4, joined 0
Thread 3 writing 218675349 compressed bytes from stream 1
Starting thread 4 to compress 218675349 bytes from stream 1
create_pthread: created 5, joined 0
Thread 4 writing 218675349 compressed bytes from stream 1
Starting sweep for mask 255
Starting thread 5 to compress 218675349 bytes from stream 1
create_pthread: created 6, joined 0
Thread 5 writing 218675349 compressed bytes from stream 1
Starting thread 6 to compress 218675349 bytes from stream 1
create_pthread: created 7, joined 0
Thread 6 writing 218675349 compressed bytes from stream 1
Starting thread 7 to compress 218675349 bytes from stream 1
create_pthread: created 8, joined 0
Thread 7 writing 218675349 compressed bytes from stream 1
Sliding main buffer to offset 2787352576
Starting thread 0 to compress 218675349 bytes from stream 1
create_pthread: created 9, joined 0
Thread 0 writing 218675349 compressed bytes from stream 1
Starting thread 1 to compress 218675349 bytes from stream 1
create_pthread: created 10, joined 0
Thread 1 writing 218675349 compressed bytes from stream 1
Starting thread 2 to compress 218675349 bytes from stream 1
create_pthread: created 11, joined 0
[...]
Pass 5 / 34 -- Elapsed Time: 00:17:24. ETA: 02:06:7092. Compress Speed:
20.766MB/s.
Succeeded in testing 2834032518 sized malloc for back end compression
Using 8 threads to compress up to 177127033 bytes each.
Beginning rzip pre-processing phase
Thread 4 writing 10575901 compressed bytes from stream 0
Thread 5 writing 176603041 compressed bytes from stream 1
Starting sweep for mask 1
Starting sweep for mask 3
Starting sweep for mask 7
Starting sweep for mask 15
Starting sweep for mask 31
Starting thread 6 to compress 177127033 bytes from stream 1
create_pthread: created 55, joined 0
Thread 6 writing 177127033 compressed bytes from stream 1
Starting sweep for mask 63
Starting thread 7 to compress 177127033 bytes from stream 1
create_pthread: created 56, joined 0
Thread 7 writing 177127033 compressed bytes from stream 1
[...]
------------------------------------------------------------
Sami
-- System Information:
Debian Release: wheezy/sid
APT prefers oldstable
APT policy: (500, 'oldstable'), (500, 'unstable'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.38.4 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages lrzip depends on:
ii bash 4.1-3 The GNU Bourne Again SHell
ii libbz2-1.0 1.0.5-6 high-quality block-sorting file co
ii libc6 2.11.2-13 Embedded GNU C Library: Shared lib
ii libgcc1 1:4.6.0-4 GCC support library
ii liblzo2-2 2.04-1 data compression library
ii libstdc++6 4.6.0-4 The GNU Standard C++ Library v3
ii zlib1g 1:1.2.3.4.dfsg-3 compression library - runtime
lrzip recommends no packages.
lrzip suggests no packages.
-- no debconf information
signature.asc
Description: Digital signature
