Hi Hugh,
the query workload is almost equally distributed.
The errors in the log regarding the lock file and “missed delete of name id
cache” occurred both not before updating to Version 7.2.2.1. Maybe some indexes
aren’t updated…
The status of the Virtuoso instance running at Port 8895 seems to remain
indefinitely.
Only the issue regarding memory leaks doesn’t occurred (until now) since I
updated to the new version. Don’t know if this is solved…
By the way, I have an additional question:
Since I have a lot of queries comprising parts like
?x <http://purl.org/dc/terms/partOf> ?a .
FILTER ( ?x >=
<http://bibsonomy.org/uri/bibtexkey/conf/icmcs/NagatomoYOIT02/dblp> ) .
FILTER ( ?x <
<http://bibsonomy.org/uri/bibtexkey/conf/wpmc/EndoMA14/dblp> ) .
Is it possible to speed up the evaluation time of those queries, e.g. by enable
some additional indexing?
Cheers
Andy
Von: Hugh Williams [mailto:hwilli...@openlinksw.com]
Gesendet: Dienstag, 23. Februar 2016 01:05
An: Nolle, Andreas <no...@hs-albsig.de>
Cc: virtuoso-users@lists.sourceforge.net
Betreff: Re: [Virtuoso-users] infrequent errors on parallel querying
Hi Andreas,
Missed that you were running 4 Virtuoso instances, how is the query workload
split across the 4 instances ?
I noticed the following errors in all the logs during done attempt to restart:
09:23:02 ERROR: Unable to lock file ../var/lib/virtuoso/db/virtuoso.lck
(Resource temporarily unavailable).
09:23:02 ERROR: Virtuoso is already runnning (pid 1264)
09:23:02 ERROR: This probably means you either do not have permission to start
09:23:02 ERROR: this server, or that virtuoso-t is already running.
09:23:02 ERROR: If you are absolutely sure that this is not the case, please try
09:23:02 ERROR: to remove the file ../var/lib/virtuoso/db/virtuoso.lck and
start again.
09:26:41 INFO: ERRS_0 01V01 QW004 <unspec>:2891:
WS.WS.SPARQL_ENDPOINT_GENERATE_FORM: Incompatible types INTEGER (189) and
VARCHAR (182) in = for debug and <constant>
09:34:13 DEBUG: missed delete of name id cache
benchset/d4e1a90786ac524b9f96f9d2f0506a12 0 (0x20e975b )
09:34:16 DEBUG: missed delete of name id cache
benchset/265b6be68f2030063c4c17e3778dbbe1 0 (0x2122444 )
09:35:50 DEBUG: missed delete of name id cache
benchset/8d0299f2a49cf965ac7c98101f55afaa 0 (0x2181ac6 )
But then did not occur on the following startup, do you know what happened here
as it seem there was an attempt to start an already running server, but I have
not seen the "DEBUG: missed delete of name id cache …” errors before …
In the status_8895 output I see many occurrences of:
Pending:
1126400: IER 141.87.4.9
156: IER 141.87.4.9
152: IER 141.87.4.9
148: IER 141.87.4.9
144: IER 141.87.4.9
.
.
.
Do these remain indefinitely or do they get cleanup after a while ?
Also you indicated that this errors had not been seen with the 3215 build
upgrade, is this still the case ?
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 21 Feb 2016, at 13:46, Nolle, Andreas
<no...@hs-albsig.de<mailto:no...@hs-albsig.de>> wrote:
Hi Hugh,
typical queries that are executed (each at any Virtuoso instance) are like the
following:
SELECT ?x ?P0src ?P1src
WHERE {
{
SERVICE <http://141.87.4.8:8899/sparql> {
?x rdf:type <http://purl.org/ontology/bibo/Collection> .
} .
BIND (<http://141.87.4.8:8899/sparql> AS ?P0src)
}
UNION
{
?x rdf:type <http://purl.org/ontology/bibo/Collection> .
BIND (<http://141.87.4.8:8891/sparql> AS ?P0src)
}
UNION
{
SERVICE <http://141.87.4.8:8893/sparql> {
?x rdf:type <http://purl.org/ontology/bibo/Collection> .
} .
BIND (<http://141.87.4.8:8893/sparql> AS ?P0src)
}
UNION
{
SERVICE <http://141.87.4.8:8895/sparql> {
?x rdf:type <http://purl.org/ontology/bibo/Collection> .
} .
BIND (<http://141.87.4.8:8895/sparql> AS ?P0src)
} .
?x rdf:type <http://swrc.ontoware.org/ontology#Article> .
BIND (<http://141.87.4.8:8891/sparql> AS ?P1src)
}
Since it is not possible to use the keyword OFFSET for queries, i.e. query
parts, that would return more than 1048576 results, there are also queries like:
SELECT ?x ?a ?b ?P0src ?P1src
WHERE {
{
SERVICE <http://141.87.4.8:8899/sparql> {
?x <http://purl.org/dc/terms/partOf> ?a .
FILTER ( ?x >=
<http://bibsonomy.org/uri/bibtexkey/conf/icmcs/NagatomoYOIT02/dblp> ) .
FILTER ( ?x <
<http://bibsonomy.org/uri/bibtexkey/conf/wpmc/EndoMA14/dblp> ) .
} .
BIND (<http://141.87.4.8:8899/sparql> AS ?P0src)
}
UNION
{
SERVICE <http://141.87.4.8:8891/sparql> {
?x <http://purl.org/dc/terms/partOf> ?a .
FILTER ( ?x >=
<http://bibsonomy.org/uri/bibtexkey/conf/icmcs/NagatomoYOIT02/dblp> ) .
FILTER ( ?x <
<http://bibsonomy.org/uri/bibtexkey/conf/wpmc/EndoMA14/dblp> ) .
} .
BIND (<http://141.87.4.8:8891/sparql> AS ?P0src)
}
UNION
{
?x <http://purl.org/dc/terms/partOf> ?a .
FILTER ( ?x >=
<http://bibsonomy.org/uri/bibtexkey/conf/icmcs/NagatomoYOIT02/dblp> ) .
FILTER ( ?x <
<http://bibsonomy.org/uri/bibtexkey/conf/wpmc/EndoMA14/dblp> ) .
BIND (<http://141.87.4.8:8893/sparql> AS ?P0src)
}
UNION
{
SERVICE <http://141.87.4.8:8895/sparql> {
?x <http://purl.org/dc/terms/partOf> ?a .
FILTER ( ?x >=
<http://bibsonomy.org/uri/bibtexkey/conf/icmcs/NagatomoYOIT02/dblp> ) .
FILTER ( ?x <
<http://bibsonomy.org/uri/bibtexkey/conf/wpmc/EndoMA14/dblp> ) .
} .
BIND (<http://141.87.4.8:8895/sparql> AS ?P0src)
BIND (<http://141.87.4.8:8895/sparql> AS ?P0src)
} .
?b <http://www.aktors.org/ontology/portal#article-of-journal> ?x .
FILTER ( ?x >=
<http://bibsonomy.org/uri/bibtexkey/conf/icmcs/NagatomoYOIT02/dblp> ) .
FILTER ( ?x < <http://bibsonomy.org/uri/bibtexkey/conf/wpmc/EndoMA14/dblp> )
.
BIND (<http://141.87.4.8:8893/sparql> AS ?P1src)
}
where the filter parts are generated beforehand by querying all individuals of
a specific concept piece by piece for each 1048576 results.
The error that occur is usually:
Virtuoso HTCLI Error HC001: Read Error in HTTP Client
But as I remember other errors that occurred during several tests over the last
weeks are:
- No Http Response
- Transaction aborted
- Timeout
- target server failed to respond
- …
Please find the output of status() for each Virtuoso instance (running at port
8891, 8893, 8895, 8899) as well as the corresponding log in the zip file at
https://www.dropbox.com/s/yzmrybj7jf8m8az/files.zip?dl=0. The call of status()
was done after I received a http error from Virtuoso instance 8893. For the
same Virtuoso instance (running at port 8893) I got a timeout on connecting by
isql. Only after a restart it was possible to connect via isql.
The reason why I assign only 16 GB per Virtuoso instance (4 x 16 GB in total)
is that each instance is take still more than 16 GB after some hours or days.
In my case I run up to 25000 queries (like the one above) at each Virtuoso
instance, but maximal 16 at one time over all instances. If you look at the
memory usage it starts normally but it increases over time as long as the swap
space is overcrowded… If this will be the case my code (running on another
server) restarts the whole server hosting the Virtuoso instances before new
SPARQL queries will be send. Please not that this behavior occurred especially
on previous versions (<7.2.2.3215) and I’m not sure if this is still the case
for version 7.2.2.3215 since I updated as recently as two weeks ago…
My Virtuoso instances hosting the following triple counts:
- Virtuoso instance 8891: 17765873
- Virtuoso instance 8893: 27897291
- Virtuoso instance 8895: 168888956
- Virtuoso instance 8899: 72372256
Best
Andy
Von: Hugh Williams [mailto:hwilli...@openlinksw.com]
Gesendet: Sonntag, 21. Februar 2016 02:51
An: Nolle, Andreas <no...@hs-albsig.de<mailto:no...@hs-albsig.de>>
Cc:
virtuoso-users@lists.sourceforge.net<mailto:virtuoso-users@lists.sourceforge.net>
Betreff: Re: [Virtuoso-users] infrequent errors on parallel querying
Hi Andy,
What is a typical query being executing that is result in a HTTP error and what
is the actual error ?
When the error occurs what is the status of the server at that point , please
provide the output of running the command:
status();
Are any errors reported in the “virtuoos.log” file, please provide a copy for
review ?
You indicate having a 128GB RAM machine but memory allocated to Virtuoso is
that for a 16GB RAM machine ie NumberOfBuffer = 1360000 in the INI file, was
there a specific reason for this ?
How many triples are hosted in the Virtuoso instance , please provide the
output of the count query:
select count(*) where {?s ?p ?o}
Best Regards
Hugh Williams
Professional Services
OpenLink Software, Inc. // http://www.openlinksw.com/
Weblog -- http://www.openlinksw.com/blogs/
LinkedIn -- http://www.linkedin.com/company/openlink-software/
Twitter -- http://twitter.com/OpenLink
Google+ -- http://plus.google.com/100570109519069333827/
Facebook -- http://www.facebook.com/OpenLinkSoftware
Universal Data Access, Integration, and Management Technology Providers
On 20 Feb 2016, at 15:47, Nolle, Andreas
<no...@hs-albsig.de<mailto:no...@hs-albsig.de>> wrote:
Hi folks,
I’ve set up four Virtuoso instances (Version 7.2.2.3215) on an Ubuntu 14.04.4
LTS server with 12 cores and 128 GB memory.
If I run several (federated) queries in parallel (maximal 16 at a time), where
some of the queries may take several minutes, it may happened that a SPARQL
endpoint return some HTTP error. If I rerun the same query some seconds later,
it will ordinarily be evaluated correct and the requested results are returned.
I’ve already tried to solve this infrequent issue but without any success. Can
you please give me some recommendations?
Maybe my configuration may comprise some not well-chosen parameter?
Please find below my virtuoso.ini:
[Database]
DatabaseFile = ../var/lib/virtuoso/db/virtuoso.db
ErrorLogFile = ../var/lib/virtuoso/db/virtuoso.log
LockFile = ../var/lib/virtuoso/db/virtuoso.lck
TransactionFile = ../var/lib/virtuoso/db/virtuoso.trx
xa_persistent_file = ../var/lib/virtuoso/db/virtuoso.pxa
ErrorLogLevel = 7
FileExtend = 200
MaxCheckpointRemap = 150000
Striping = 0
TempStorage = TempDatabase
[TempDatabase]
DatabaseFile = ../var/lib/virtuoso/db/virtuoso-temp.db
TransactionFile = ../var/lib/virtuoso/db/virtuoso-temp.trx
MaxCheckpointRemap = 9000
Striping = 0
[Parameters]
ServerPort = 1112
LiteMode = 0
DisableUnixSocket = 1
DisableTcpSocket = 0
MaxClientConnections = 10000
ServerThreads = 1000
CheckpointInterval = 60
O_DIRECT = 0
CaseMode = 2
MaxStaticCursorRows = 5000
CheckpointAuditTrail = 0
AllowOSCalls = 0
SchedulerInterval = 10
DirsAllowed = ., ../share/virtuoso/vad,../saleem/dumps,
/opt/benchset/bibsonomy
ThreadCleanupInterval = 1
ThreadThreshold = 1000
ResourcesCleanupInterval = 1
FreeTextBatchSize = 100000
SingleCPU = 0
VADInstallDir = ../share/virtuoso/vad/
PrefixResultNames = 0
RdfFreeTextRulesSize = 100
IndexTreeMaps = 512
MaxMemPoolSize = 200000000
PrefixResultNames = 0
MacSpotlight = 0
MaxQueryMem = 8G ; memory allocated to query processor
VectorSize = 1000 ; initial parallel query vector
(array of query operations) size
MaxVectorSize = 2000000 ; query vector size threshold.
AdjustVectorSize = 0
ThreadsPerQuery = 24
AsyncQueueMaxThreads = 36
NumberOfBuffers = 1360000
MaxDirtyBuffers = 1000000
TraceOn = errors
CallstackOnException = 1
MaxSortedTopRows = 10000000
DefaultIsolation = 2
[HTTPServer]
ServerPort = 8891
ServerRoot = ../var/lib/virtuoso/vsp
MaxClientConnections = 10000
DavRoot = DAV
EnabledDavVSP = 0
HTTPProxyEnabled = 0
TempASPXDir = 0
DefaultMailServer = localhost:25
ServerThreads = 1000
MaxKeepAlives = 100
KeepAliveTimeout = 10
MaxCachedProxyConnections = 10
ProxyConnectionCacheTimeout = 15
HTTPThreadSize = 280000
HttpPrintWarningsInOutput = 0
Charset = UTF-8
MaintenancePage = atomic.html
EnabledGzipContent = 1
[AutoRepair]
BadParentLinks = 0
[Client]
SQL_PREFETCH_ROWS = 100
SQL_PREFETCH_BYTES = 16000
SQL_QUERY_TIMEOUT = 0
SQL_TXN_TIMEOUT = 0
[VDB]
ArrayOptimization = 0
NumArrayParameters = 10
VDBDisconnectTimeout = 1000
KeepConnectionOnFixedThread = 0
[Replication]
ServerName = db-AKSWDESKTOP
ServerEnable = 1
QueueMax = 50000
[Striping]
Segment1 = 8G, db-seg1-1.db, db-seg1-2.db
Segment2 = 8G, db-seg2-1.db, db-seg2-2.db
[Zero Config]
ServerName = virtuoso (AKSWDESKTOP)
[Mono]
[URIQA]
DynamicLocal = 0
DefaultHost = localhost:8890
[SPARQL]
DefaultGraph = http://test.hs-albsig.de/benchset
ResultSetMaxRows = 10000000
DefaultQuery = select distinct ?Concept where {[] a ?Concept} LIMIT
100
DeferInferenceRulesInit = 0 ; controls inference rules loading
[Plugins]
LoadPath = ../lib/virtuoso/hosting
Best regards
Andy Nolle
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net<mailto:Virtuoso-users@lists.sourceforge.net>
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
