The size of Dbpedia.nq file is: 150 GB.
I tried to a bulk load using:
http://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtBulkRDFLoaderExampleDbpedia
*The message now change to: I am not able to understand as to why is
mp_mmap_clocks() taking over 10% of real time.*
02:20:05 PL LOG: Loader started
02:25:10 * Monitor: The mp_mmap_clocks over 10% of real time
02:27:10 * Monitor: The mp_mmap_clocks over 10% of real time
02:29:10 * Monitor: The mp_mmap_clocks over 10% of real time
02:31:11 * Monitor: The mp_mmap_clocks over 10% of real time
02:33:11 * Monitor: The mp_mmap_clocks over 10% of real time
02:35:11 * Monitor: The mp_mmap_clocks over 10% of real time
02:37:13 * Monitor: The mp_mmap_clocks over 10% of real time
02:39:14 * Monitor: The mp_mmap_clocks over 10% of real time
02:41:14 * Monitor: The mp_mmap_clocks over 10% of real time
02:43:14 * Monitor: The mp_mmap_clocks over 10% of real time
02:45:14 * Monitor: The mp_mmap_clocks over 10% of real time
*The output of my status() command is:*
$ ./isql 1111 dba dba
Connected to OpenLink Virtuoso
Driver: 07.20.3212 OpenLink Virtuoso ODBC Driver
OpenLink Interactive SQL (Virtuoso), version 0.9849b.
Type HELP; for help and EXIT; to exit.
SQL> status();
REPORT
VARCHAR
_______________________________________________________________________________
OpenLink Virtuoso Server
Version 07.20.3212-pthreads for Linux as of Apr 8 2015
Started on: 2015-04-09 02:18 GMT+5
Database Status:
File size 0, 33536 pages, 12718 free.
2720000 buffers, 20721 used, 18719 dirty 13 wired down, repl age 0 0 w.
io 0 w/crsr.
Disk Usage: 68 reads avg 0 msec, 0% r 0% w last 0 s, 2471 writes flush
3.523 MB,
0 read ahead, batch = 0. Autocompact 302 in 225 out, 25% saved.
Gate: 0 2nd in reads, 0 gate write waits, 0 in while read 0 busy scrap.
Log = /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.trx,
31992 bytes
2041 pages have been changed since last backup (in checkpoint state)
Current backup timestamp: 0x0000-0x00-0x00
Last backup date: unknown
Clients: 2 connects, max 2 concurrent
RPC: 13 calls, 2 pending, 2 max until now, 0 queued, 0 burst reads (0%), 0
second 9862M large, 12500M max
Checkpoint Remap 38 pages, 0 mapped back. 2 s atomic time.
DB master 33536 total 12718 free 38 remap 10 mapped back
temp 256 total 251 free
Lock Status: 0 deadlocks of which 0 2r1w, 9 waits,
Currently 8 threads running 0 threads waiting 1 threads in vdb.
Pending:
Client 1111:2: Account: dba, 203 bytes in, 256 bytes out, 1 stmts.
PID: 52667, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: PENDING, 1 threads.
Locks:
Client 1111:1: Account: dba, 693 bytes in, 1148 bytes out, 1 stmts.
PID: 52428, OS: unix, Application: unknown, IP#: 127.0.0.1
Transaction status: ROLLED BACK, 1 threads.
Running Statements:
Time (msec) Text
1011793 rdf_loader_run()
160 status()
Hash indexes
42 Rows. -- 162 msec.
top command shows the output:
top - 02:51:28 up 35 days, 7:43, 4 users, load average: 2.80, 2.76, 2.85
Tasks: 664 total, 1 running, 663 sleeping, 0 stopped, 0 zombie
Cpu(s): 8.0%us, 3.8%sy, 0.0%ni, 88.2%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st
Mem: 65927576k total, 18200988k used, 47726588k free, 7000k buffers
Swap: 33030136k total, 95788k used, 32934348k free, 681480k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
52343 A 20 0 34.3g 15g 12m S 294.3 24.3 99:38.37 virtuoso-t
2106 root 20 0 0 0 0 S 0.7 0.0 2:44.24 kondemand/3
47924 A 20 0 5857m 479m 24m S 0.7 0.7 42:31.05 java
52823 A 20 0 15436 1664 892 R 0.7 0.0 0:00.18 top
107 root 20 0 0 0 0 S 0.3 0.0 2:03.77 events/8
207 root 20 0 0 0 0 D 0.3 0.0 136:08.97 khubd
640 root 20 0 0 0 0 S 0.3 0.0 22:56.00 usb-storage
2078 root 20 0 249m 1964 836 S 0.3 0.0 5:42.84 rsyslogd
2104 root 20 0 0 0 0 S 0.3 0.0 4:27.97 kondemand/1
2107 root 20 0 0 0 0 S 0.3 0.0 2:28.72 kondemand/4
2123 root 20 0 0 0 0 S 0.3 0.0 4:12.28 kondemand/20
2124 root 20 0 0 0 0 S 0.3 0.0 2:41.25 kondemand/21
2148 root 20 0 18352 628 428 S 0.3 0.0 22:32.29 irqbalance
2602 root 20 0 22516 728 648 S 0.3 0.0 15:36.21
hald-addon-stor
On Wed, Apr 8, 2015 at 7:48 AM, Hugh Williams <hwilli...@openlinksw.com>
wrote:
> Hi Maria,
>
> What do you mean by “virtuoso halts” ?
>
> What is the size of the “Dbpedia.nq” file ?
>
> Are you able to make another isql connection to the database and run the
> “status();” command ?
>
> What does the Linux “top” command report as to the resources being
> consumed by the process ?
>
> Best Regards
> Hugh Williams
> Professional Services
> OpenLink Software, Inc. // http://www.openlinksw.com/
> Weblog -- http://www.openlinksw.com/blogs/
> LinkedIn -- http://www.linkedin.com/company/openlink-software/
> Twitter -- http://twitter.com/OpenLink
> Google+ -- http://plus.google.com/100570109519069333827/
> Facebook -- http://www.facebook.com/OpenLinkSoftware
> Universal Data Access, Integration, and Management Technology Providers
>
> On 7 Apr 2015, at 23:00, Maria Jackson <maria.jackson....@gmail.com>
> wrote:
>
> *I am trying to bulk load dbpedia.nq in Virtuoso using the following
> command:*
>
> *$ ./isql 1111 dba dba*
> Connected to OpenLink Virtuoso
> Driver: 07.20.3212 OpenLink Virtuoso ODBC Driver
> OpenLink Interactive SQL (Virtuoso), version 0.9849b.
> Type HELP; for help and EXIT; to exit.
> SQL> DB.DBA.TTLP_MT (file_to_string_output
> ('/home/Virtuoso/virtuosoInstalled/share/virtuoso/vad/Dbpedia.nq'), '', '<
> http://localhost:8890/DAV/dbpedia>',512);
>
> *Whenever I try to bulk load in Virtuoso I get the following error and
> Virtuoso halts. Can someone please help me as to how can I bulk load .nq
> files in Virtuoso:*
>
> *$ ./virtuoso-t -f -c
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.ini*
>
> Wed Apr 08 2015
> 01:44:17 { Loading plugin 1: Type `plain', file `wikiv' in
> `/home/Virtuoso/virtuosoInstalled/lib/virtuoso/hosting'
> 01:44:17 FAILED plugin 1: Unable to locate file }
> 01:44:17 { Loading plugin 2: Type `plain', file `mediawiki' in
> `/home/Virtuoso/virtuosoInstalled/lib/virtuoso/hosting'
> 01:44:17 FAILED plugin 2: Unable to locate file }
> 01:44:17 { Loading plugin 3: Type `plain', file `creolewiki' in
> `/home/Virtuoso/virtuosoInstalled/lib/virtuoso/hosting'
> 01:44:17 FAILED plugin 3: Unable to locate file }
> 01:44:17 OpenLink Virtuoso Universal Server
> 01:44:17 Version 07.20.3212-pthreads for Linux as of Apr 8 2015
> 01:44:17 uses parts of OpenSSL, PCRE, Html Tidy
> 01:44:27 Database version 3126
> 01:44:27 SQL Optimizer enabled (max 1000 layouts)
> 01:44:28 Compiler unit is timed at 0.000291 msec
> 01:44:30 Roll forward started
> 01:44:30 Roll forward complete
> 01:44:32 Checkpoint started
> 01:44:32 Checkpoint finished, log reused
> 01:44:32 HTTP/WebDAV server online at 8890
> 01:44:32 Server online at 1111 (pid 28041)
> 02:44:33 Checkpoint started
> 02:44:33 Checkpoint finished, log reused
> 03:05:37 * Monitor: Locks are held for a long time
> 03:05:55 * Monitor: Should read for update because lock escalation from
> shared to exclusive fails frequently (1)
> 03:07:37 * Monitor: Locks are held for a long time
> 03:18:55 * Monitor: Many lock waits
>
> I downloaded Virtuoso from: git clone git://
> github.com/openlink/virtuoso-opensource.git
> Git checkout gives me: $ git checkout -t remotes/origin/develop/7
> fatal: git checkout: branch develop/7 already exists
> The parameters of my virtuoso.ini file are:
> ;
> ; virtuoso.ini
> ;
> ; Configuration file for the OpenLink Virtuoso VDBMS Server
> ;
> ; To learn more about this product, or any other product in our
> ; portfolio, please check out our web site at:
> ;
> ; http://virtuoso.openlinksw.com/
> ;
> ; or contact us at:
> ;
> ; general.informat...@openlinksw.com
> ;
> ; If you have any technical questions, please contact our support
> ; staff at:
> ;
> ; technical.supp...@openlinksw.com
> ;
>
> ;
> ; Database setup
> ;
> [Database]
> DatabaseFile =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.db
> ErrorLogFile =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.log
> LockFile =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.lck
> TransactionFile =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.trx
> xa_persistent_file =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso.pxa
> ErrorLogLevel = 7
> FileExtend = 200
> MaxCheckpointRemap = 170000
> Striping = 0
> TempStorage = TempDatabase
>
>
> [TempDatabase]
> DatabaseFile =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso-temp.db
> TransactionFile =
> /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/db/virtuoso-temp.trx
> MaxCheckpointRemap = 2000000
> Striping = 0
>
>
> ;
> ; Server parameters
> ;
> [Parameters]
> ServerPort = 1111
> LiteMode = 0
> DisableUnixSocket = 1
> DisableTcpSocket = 0
> ;SSLServerPort = 2111
> ;SSLCertificate = cert.pem
> ;SSLPrivateKey = pk.pem
> ;X509ClientVerify = 0
> ;X509ClientVerifyDepth = 0
> ;X509ClientVerifyCAFile = ca.pem
> MaxClientConnections = 10
> CheckpointInterval = 60
> O_DIRECT = 0
> CaseMode = 2
> MaxStaticCursorRows = 5000
> CheckpointAuditTrail = 0
> AllowOSCalls = 0
> SchedulerInterval = 10
> DirsAllowed = ., /home/Virtuoso/virtuosoInstalled/share/virtuoso/vad
> ThreadCleanupInterval = 0
> ThreadThreshold = 10
> ResourcesCleanupInterval = 0
> FreeTextBatchSize = 100000
> SingleCPU = 0
> VADInstallDir = /home/Virtuoso/virtuosoInstalled/share/virtuoso/vad/
> PrefixResultNames = 0
> RdfFreeTextRulesSize = 100
> IndexTreeMaps = 256
> MaxMemPoolSize = 200000000
> PrefixResultNames = 0
> MacSpotlight = 0
> IndexTreeMaps = 64
> MaxQueryMem = 2G ; memory allocated to query processor
> VectorSize = 1000 ; initial parallel query vector (array of query
> operations) size
> MaxVectorSize = 1000000 ; query vector size threshold.
> AdjustVectorSize = 0
> ThreadsPerQuery = 4
> AsyncQueueMaxThreads = 10
> ;;
> ;; When running with large data sets, one should configure the Virtuoso
> ;; process to use between 2/3 to 3/5 of free system memory and to stripe
> ;; storage on all available disks.
> ;;
> ;; Uncomment next two lines if there is 2 GB system memory free
> ;NumberOfBuffers = 170000
> ;MaxDirtyBuffers = 130000
> ;; Uncomment next two lines if there is 4 GB system memory free
> ;NumberOfBuffers = 340000
> ; MaxDirtyBuffers = 250000
> ;; Uncomment next two lines if there is 8 GB system memory free
> ;NumberOfBuffers = 680000
> ;MaxDirtyBuffers = 500000
> ;; Uncomment next two lines if there is 16 GB system memory free
> ;NumberOfBuffers = 1360000
> ;MaxDirtyBuffers = 1000000
> ;; Uncomment next two lines if there is 32 GB system memory free
> ;NumberOfBuffers = 2720000
> ;MaxDirtyBuffers = 2000000
> ;; Uncomment next two lines if there is 48 GB system memory free
> NumberOfBuffers = 4000000
> MaxDirtyBuffers = 3000000
> ;; Uncomment next two lines if there is 64 GB system memory free
> ;NumberOfBuffers = 5450000
> ;MaxDirtyBuffers = 4000000
> ;;
> ;; Note the default settings will take very little memory
> ;; but will not result in very good performance
> ;;
> ;NumberOfBuffers = 10000
> ;MaxDirtyBuffers = 6000
>
>
> [HTTPServer]
> ServerPort = 8890
> ServerRoot = /home/Virtuoso/virtuosoInstalled/var/lib/virtuoso/vsp
> MaxClientConnections = 10
> DavRoot = DAV
> EnabledDavVSP = 0
> HTTPProxyEnabled = 0
> TempASPXDir = 0
> DefaultMailServer = localhost:25
> ServerThreads = 10
> MaxKeepAlives = 10
> KeepAliveTimeout = 10
> MaxCachedProxyConnections = 10
> ProxyConnectionCacheTimeout = 15
> HTTPThreadSize = 280000
> HttpPrintWarningsInOutput = 0
> Charset = UTF-8
> ;HTTPLogFile = logs/http.log
> MaintenancePage = atomic.html
> EnabledGzipContent = 1
>
>
> [AutoRepair]
> BadParentLinks = 0
>
> [Client]
> SQL_PREFETCH_ROWS = 100
> SQL_PREFETCH_BYTES = 16000
> SQL_QUERY_TIMEOUT = 0
> SQL_TXN_TIMEOUT = 0
> ;SQL_NO_CHAR_C_ESCAPE = 1
> ;SQL_UTF8_EXECS = 0
> ;SQL_NO_SYSTEM_TABLES = 0
> ;SQL_BINARY_TIMESTAMP = 1
> ;SQL_ENCRYPTION_ON_PASSWORD = -1
>
> [VDB]
> ArrayOptimization = 0
> NumArrayParameters = 10
> VDBDisconnectTimeout = 1000
> KeepConnectionOnFixedThread = 0
>
> [Replication]
> ServerName = db-BTP
> ServerEnable = 1
> QueueMax = 50000
>
>
> ;
> ; Striping setup
> ;
> ; These parameters have only effect when Striping is set to 1 in the
> ; [Database] section, in which case the DatabaseFile parameter is ignored.
> ;
> ; With striping, the database is spawned across multiple segments
> ; where each segment can have multiple stripes.
> ;
> ; Format of the lines below:
> ; Segment<number> = <size>, <stripe file name> [, <stripe file name> ..
> ]
> ;
> ; <number> must be ordered from 1 up.
> ;
> ; The <size> is the total size of the segment which is equally divided
> ; across all stripes forming the segment. Its specification can be in
> ; gigabytes (g), megabytes (m), kilobytes (k) or in database blocks
> ; (b, the default)
> ;
> ; Note that the segment size must be a multiple of the database page size
> ; which is currently 8k. Also, the segment size must be divisible by the
> ; number of stripe files forming the segment.
> ;
> ; The example below creates a 200 meg database striped on two segments
> ; with two stripes of 50 meg and one of 100 meg.
> ;
> ; You can always add more segments to the configuration, but once
> ; added, do not change the setup.
> ;
> [Striping]
> Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
> Segment2 = 100M, db-seg2-1.db
> ;...
>
> ;[TempStriping]
> ;Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
> ;Segment2 = 100M, db-seg2-1.db
> ;...
>
> ;[Ucms]
> ;UcmPath = <path>
> ;Ucm1 = <file>
> ;Ucm2 = <file>
> ;...
>
>
> [Zero Config]
> ServerName = virtuoso (BTP)
> ;ServerDSN = ZDSN
> ;SSLServerName =
> ;SSLServerDSN =
>
>
> [Mono]
> ;MONO_TRACE = Off
> ;MONO_PATH = <path_here>
> ;MONO_ROOT = <path_here>
> ;MONO_CFG_DIR = <path_here>
> ;virtclr.dll =
>
>
> [URIQA]
> DynamicLocal = 0
> DefaultHost = localhost:8890
>
>
> [SPARQL]
> ;ExternalQuerySource = 1
> ;ExternalXsltSource = 1
> ;DefaultGraph = http://localhost:8890/dataspace
> ;ImmutableGraphs = http://localhost:8890/dataspace
> ResultSetMaxRows = 10000
> MaxQueryCostEstimationTime = 400 ; in seconds
> MaxQueryExecutionTime = 60 ; in seconds
> DefaultQuery = select distinct ?Concept where {[] a
> ?Concept} LIMIT 100
> DeferInferenceRulesInit = 0 ; controls inference rules loading
> ;PingService = http://rpc.pingthesemanticweb.com/
>
>
> [Plugins]
> LoadPath = /home/Virtuoso/virtuosoInstalled/lib/virtuoso/hosting
> Load1 = plain, wikiv
> Load2 = plain, mediawiki
> Load3 = plain, creolewiki
> ;Load4 = plain, im
> ;Load5 = plain, wbxml2
> ;Load6 = plain, hslookup
> ;Load7 = attach, libphp5.so
> ;Load8 = Hosting, hosting_php.so
> ;Load9 = Hosting,hosting_perl.so
> ;Load10 = Hosting,hosting_python.so
> ;Load11 = Hosting,hosting_ruby.so
> ;Load12 = msdtc,msdtc_sample
>
> ------------------------------------------------------------------------------
> BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
> Develop your own process in accordance with the BPMN 2 standard
> Learn Process modeling best practices with Bonita BPM through live
> exercises
> http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual-
> event?utm_
>
> source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF_______________________________________________
> Virtuoso-users mailing list
> Virtuoso-users@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>
>
>
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
