HI Sebastian,
Looking at the Virtuoso configuration file params you provided I
not you
are running in LiteMode, with only the following set:
[Parameters]
LiteMode=1
ServerPort=1111
DisableTcpSocket=1
PrefixResultNames=0
ServerThreads=5
CheckpointInterval=10
MaxDirtyBuffers=50
SchedulerInterval=5
FreeTextBatchSize=1000
Which means the Virtuoso NumberOfBuffers which is not set and
controls
the amount of RAM used by the server will be the default of 2000 * 8K
buffers = 16K . So you entire Virtuoso Server is running within
16K of
memory and you are attempting to insert a 1MB plus triple, I think it
will have problems as the server will then having to be swapping
between
memory and disk like crazy. I would suggest you set the
NumberOfBuffers
to something like 200000 to give the Virtuoso server a reasonable
amount
of RAM to perform such inserts ...
I only have 3gb of main memory so 200000 does not work here:
GPF: disk.c:1294 Cannot allocate memory for Database buffers, try to
decrease NumberOfBuffers INI setting
But I tried 100000 and killed virtuoso after two hours.
Also you introduced the lite mode for me to be able to run Virtuoso on
desktop machines where people are already going crazy if you use 100MB
of memory. So raising the number of buffers cannot really be the end
solution.
Yes, but you got to realize that it requires some tuning in order for
the system to be able to still perform. If you try to stuff data
larger than your buffer size into the system, then it will require
the database to make a lot of swaps to get the relevant data pages
into memory, constantly swapping data between memory and db. Then
updating the freetext index etc.
Since i do not know all the details of your machine, what other
processes are running etc, i cannot tell you exactly what settings
you can try at this point.
I fully appreciate the fact you want to keep the memory footprint
down, but this does not mean that you can expect all lite mode will
just be able to handle arbitrary commands without some performance
costs.
Now i am prepared to help analyze your current statements and see if
we can come up with a method that will return in a more reasonable
time, without dramatically increase the footprint of virtuoso.
I don't really know how to continue now.
The only solution I can see ATM is to split the big literals into
several statements. After all they are only used for the full text
index
anyway (it is the plain text representation of files in the user's
home
directory.)
Any other (hopefully better) ideas?
Can you dump the sample data in some format like .n3 or similar so i
can have a look at this. Or just dump the sparql commands you tried
to use to a text file and compress it.
Contact me privately so we can arrange pickup.
Patrick
