Hi, Hugh.
To answer some of your questions. The total count of triples in our Quad
Store is around 10.6billion. The server machine has 6 cores and 59GB RAM.
The actual .sql script run is this:
log_enable(2);
SPARQL DELETE { GRAPH ?graph { ?st
<http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }}
INSERT { GRAPH ?graph { ?st
<http://purl.org/vocab/changeset/schema#subject> ?subject . }} WHERE {
GRAPH ?graph { ?st <http://www.w3.org/1999/02/22-rdf-syntax-ns#subject>
?subject . }} LIMIT 10000000;
checkpoint;
log_enable(1);
And the script is called out like this:
isql-vt 1111 dba dba update.sql
The script was called out 5 times in a row and then another checkpoint
was made like this:
isql-vt 1111 dba dba exec="checkpoint;"
No other process was using virtuoso when we were executing this script
and when the crash happened.
In addition we have a second instance of virtuoso, which often is making
many concurrent insertions. We noticed that making integrity check in
that instance fails after some time with the following error:
...
09:42:19 /usr/bin/virtuoso-t() [0x5db66c]
09:42:19 /usr/bin/virtuoso-t(sf_sql_execute_w+0x74) [0x5db884]
09:42:19 /usr/bin/virtuoso-t() [0x8d8f38]
09:42:19 /usr/bin/virtuoso-t() [0x8df6fc]
09:42:19 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7ff65fd23182]
09:42:19 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7ff65f32d47d]
09:42:19 GPF: page.c:899 out of order
Is it a problem if we keep using that instance, when the integrity check
fails with this error? Should we definitely do a crash dump and restore
from that? The reason I am asking is that this problem occurs almost
every week and restoring from crash dump takes some time. So processes
using that virtuoso instance are down during that time.
Best Regards.
Piret Lattikas
On 19.10.15 17:41, Hugh Williams wrote:
Hi Piret,
I note you are using an older Virtuoso 7.2.3212 build from Apr 2015
where as the current stable release build from git or source-forge is
7.2.3214, thus I would suggest updating to this later release or even
the current git develop/7 head to get all the latest fixes and see if
the problem persists.
The “??”’s in gdb back trace output would indicate the Virtuoso
binary is stripped of symbols thus to get a meaningful back trace a
debug build with symbols intact needs to be compiled as follows:
1. Configure using the —with debug option:
./configure —with-debug
2. In the Makefile, check to ensure CFLAGS has the “-g” option set
which generates debug information:
CFLAGS = -g -O2
3. Then do "make clean all deinstall reinstall” to build a new debug
unstripped binary (virtuoso-t)
4. Start database with this new binary and force the crash condition
again to generate a new core file
5. Use gdb to load core file
gdb virtuoso-t core
6. At the prompt, type "bt" or “backtrace” to backtrace through stack
and provide the output when top of stack is reached.
You indicate having 1.7billion triples to be updated of which about
500 million are still to be updated, but what is the total count of
triples in the Quad Store ( select * where {?s ?p ?o} ) ? Also what
are the specs of the server machine being used in terms of number of
cores and memory in particular and what does the output of the
following command run from the “isql” command line tool, report when
the database is “warm” :
status();
Can you provide the actual content of the SQL script being run ?
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 19 Oct 2015, at 11:37, Piret Lattikas <piret.latti...@zerotech.ee
<mailto:piret.latti...@zerotech.ee>> wrote:
Hi,
We were running a simple update SPARQL query for a large set of data.
The update query for replacing one predicate with another, itself is
this:
SPARQL DELETE { GRAPH ?graph { ?st
<http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }}
INSERT { GRAPH ?graph { ?st
<http://purl.org/vocab/changeset/schema#subject> ?subject . }} WHERE
{ GRAPH ?graph { ?st
<http://www.w3.org/1999/02/22-rdf-syntax-ns#subject> ?subject . }}
LIMIT 10000000;
The initial count of triples that needed to be updated was ~1 700 000
000. The .sql script used for update set the log_enable(2) at the
beginning, before executing the actual update. And a checkpoint was
called out in the same .sql script after the update query finished.
We were able to run that script for quite a while without any
problems. Suddenly running the script made virtuoso crash, the
remaining triples count that needed updating was around 500 000 000.
The error in virtuoso.log file shows the following:
09:13:37 In delete or rb of insert on RDF_QUAD dp 38144031 seg 6,
col 3 is 10909 values and next 8276
09:13:37 /usr/bin/virtuoso-t() [0x8d4afd]
...
09:13:37 /usr/bin/virtuoso-t() [0x53f19b]
09:13:37 /usr/bin/virtuoso-t(delete_node_input+0x13b) [0x5cbb3b]
09:13:37 /usr/bin/virtuoso-t() [0x5c7fed]
09:13:37 /usr/bin/virtuoso-t() [0x5c8416]
09:13:37 /usr/bin/virtuoso-t(table_source_input_unique+0x2a9) [0x5cfe69]
09:13:37 /usr/bin/virtuoso-t() [0x5c7fed]
...
09:13:37 /usr/bin/virtuoso-t() [0x5c826e]
09:13:37 /usr/bin/virtuoso-t(skip_node_input+0x466) [0x5cc9e6]
09:13:37 /usr/bin/virtuoso-t() [0x5c7fed]
...
09:13:37 /usr/bin/virtuoso-t() [0x5db26a]
09:13:37 /usr/bin/virtuoso-t(sf_sql_execute_w+0x74) [0x5db884]
09:13:37 /usr/bin/virtuoso-t() [0x8d8f38]
09:13:37 /usr/bin/virtuoso-t() [0x8df6fc]
09:13:37 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f69a6f65182]
09:13:37 /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f69a656f47d]
09:13:37 GPF: collock.c:1410 uneven length cols after delete, update
or rollback of insert
Do get the core dump of the crash set 'ulimit -c unlimited' and
started the script again. The crash is reproducable with every run.
The core dump shows the following:
# gdb virtuoso-t core
...
[New LWP 15882]
[New LWP 15879]
[Thread debugging using libthread_db enabled]
Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/virtuoso-t +wait +configfile
/etc/virtuoso-opensource-7/virtuoso.ini'.
Program terminated with signal SIGSEGV, Segmentation fault.
(gdb) bt
#0 0x00000000008d4bde in ?? ()
...
#6 0x000000000053f19b in ?? ()
#7 0x00000000005cbb3b in delete_node_input ()
#8 0x00000000005c7fed in ?? ()
#9 0x00000000005c8416 in ?? ()
#10 0x00000000005cfe69 in table_source_input_unique ()
#11 0x00000000005c7fed in ?? ()
...
#30 0x00000000005c826e in ?? ()
#31 0x00000000005cc9e6 in skip_node_input ()
#32 0x00000000005c7fed in ?? ()
...
#43 0x00000000005db26a in ?? ()
#44 0x00000000005db884 in sf_sql_execute_w ()
#45 0x00000000008d8f38 in ?? ()
#46 0x00000000008df6fc in ?? ()
#47 0x00007f69a6f65182 in start_thread (arg=0x7f691a3bd700) at
pthread_create.c:312
#48 0x00007f69a656f47d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111
We run the integrity check of virtuoso with "backup '/dev/null'" and
that passed successfully, so this indicated for us that the database
itself is intact.
The version of virtuoso we are using is:
Virtuoso Open Source Edition (Column Store) (multi threaded)
Version 7.2.0.3212-pthreads as of Apr 8 2015
Compiled for Linux (x86_64-pc-linux-gnu)
Copyright (C) 1998-2015 OpenLink Software
Do you have any suggestions why this happened and how can it be fixed?
Best regards,
Piret Lattikas
------------------------------------------------------------------------------
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
<mailto:Virtuoso-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
