Hi Jerven,
Thanks for the reply. Both solutions seems to work like a charme, however
the the Jena way of turning of log and auto checkpointing still needs some
minor fine tuning, but the isql calling through a script is doing the trick for
me now.
Andra
On 15-jan.-2014, at 22:30, Jerven Bolleman <jerven.bolle...@isb-sib.ch> wrote:
> Hi Andra,
>
> In my virtuoso branch I do something similar but with Sesame. However, if you
> want good speed over large loads
> you need to turn off “log” and “auto checkpointing”
>
> This is more JDBC level actions and I don’t know how you do it in Jena but in
> sesame you can do it with utility methods like this.
>
>
> public class CheckpointLoggingUtil
> {
> private static Logger log =
> Logger.getInstance(CheckpointLoggingUtil.class);
>
> public static void autoCheckpointOff(RepositoryConnection connection)
> throws SQLException
> {
> if (connection instanceof VirtuosoRepositoryConnection)
> {
>
> VirtuosoRepositoryConnection
> virtuosoRepositoryConnection = (VirtuosoRepositoryConnection) connection;
> Connection vrc =
> virtuosoRepositoryConnection.getQuadStoreConnection();
> CallableStatement checkpointOff =
> vrc.prepareCall("checkpoint_interval (0)");
> checkpointOff.execute();
> log.debug("Virtuoso auto checkpointing off");
> }
> }
>
> public static void autoCheckpointOn(RepositoryConnection connection)
> throws SQLException
> {
> if (connection instanceof VirtuosoRepositoryConnection)
> {
> log.debug("Virtuoso checkpoint starting");
> VirtuosoRepositoryConnection
> virtuosoRepositoryConnection = (VirtuosoRepositoryConnection) connection;
> Connection vrc =
> virtuosoRepositoryConnection.getQuadStoreConnection();
> CallableStatement checkpointOff =
> vrc.prepareCall("checkpoint_interval (60)");
> checkpointOff.execute();
> log.debug("Virtuoso auto checkpointing on");
> }
> }
>
> public static void checkpoint(RepositoryConnection connection)
> throws SQLException
> {
> if (connection instanceof VirtuosoRepositoryConnection)
> {
> log.debug("Virtuoso checkpoint starting");
> VirtuosoRepositoryConnection
> virtuosoRepositoryConnection = (VirtuosoRepositoryConnection) connection;
> Connection vrc =
> virtuosoRepositoryConnection.getQuadStoreConnection();
> CallableStatement checkpointOff =
> vrc.prepareCall("checkpoint");
> checkpointOff.execute();
> log.debug("Virtuoso checkpoint finished");
> }
> }
>
> public static void disableLogging(RepositoryConnection connection)
> throws SQLException
> {
> if (connection instanceof VirtuosoRepositoryConnection)
> {
> VirtuosoRepositoryConnection
> virtuosoRepositoryConnection = (VirtuosoRepositoryConnection) connection;
> Connection vrc =
> virtuosoRepositoryConnection.getQuadStoreConnection();
> CallableStatement loggingOff =
> vrc.prepareCall("log_enable (0)");
> loggingOff.execute();
> }
> }
>
> public static void enableLogging(RepositoryConnection connection)
> throws SQLException
> {
> if (connection instanceof VirtuosoRepositoryConnection)
> {
> VirtuosoRepositoryConnection
> virtuosoRepositoryConnection = (VirtuosoRepositoryConnection) connection;
> Connection vrc =
> virtuosoRepositoryConnection.getQuadStoreConnection();
> CallableStatement loggingOff =
> vrc.prepareCall("log_enable (3)");
> loggingOff.execute();
> }
> }
> }
>
> Then in my sesame code I do something like this
>
> connection = getConnection();
> connection.setAutoCommit(false);
> CheckpointLoggingUtil.disableLogging(connection);
> CheckpointLoggingUtil.autoCheckpointOff(connection);
>
> connection.add(some statements)
> Once in a while I might call a checkpoint
>
> and then commit everything and turn the “log” and “checkpointing" back on.
>
> connection.commit();
> CheckpointLoggingUtil.checkpoint(connection);
> CheckpointLoggingUtil.autoCheckpointOn(connection);
> CheckpointLoggingUtil.enableLogging(connection);
>
>
> Of course you can also just script the DBA connection with something like
> this.:
>
> isql -U dba -P dba -S 1111 “EXEC=DB.DBA.TTLP_MT (file_to_string_output
> ('wpContent_v0.0.73237_20140115.ttl'), ‘’,'http://rdf.wikipathways.org')”
>
> See http://docs.openlinksw.com/virtuoso/isql.html for more details
>
> Regards,
> Jerven
>
> On 15 Jan 2014, at 21:29, Jim McCusker <mccus...@gmail.com> wrote:
>
>> I do something like this:
>>
>> OntModel memoryModel = ...;
>>
>> VirtModel model = null;
>> try {
>> model =
>> VirtModel.openDatabaseModel(graphURI,virtuosoURL,virtuosoUser,virtuosoPassword);
>> model.begin();
>> model.removeAll();
>> model.add(memoryModel);
>> model.commit();
>> } catch (Exception e) {
>> if (model != null) model.abort();
>> } finally {
>> if (model != null) model.close();
>> }
>>
>>
>>
>> On Wed, Jan 15, 2014 at 3:02 PM, Andra Waagmeester
>> <andra.waagmees...@gmail.com> wrote:
>> Hi,
>>
>> I am looking into a solution to automatically load triples into our
>> triples store. We use jena to convert our data into triples. Our current
>> workflow is that we dump the triples in a file (e.g. content.ttl) after
>> which this file is loaded through the isql command line:
>>
>> "DB.DBA.TTLP_MT (file_to_string_output
>> ('wpContent_v0.0.73237_20140115.ttl'), '', 'MailScanner has detected a
>> possible fraud attempt from "rdf.wikipathways.org'" claiming to be
>> http://rdf.wikipathways.org'); “
>>
>> Loading the file takes minutes, but it still requires some manual steps. I
>> would very much like to make both the authoring and the subsequent loading
>> in the triple store a automatic process.
>>
>> I have tried the following in java:
>>
>> InputStream in = new FileInputStream(args[1]);
>> InputStream in = new
>> FileInputStream("wpContent_v0.0.73237_20140115.ttl");
>> String url ="jdbc:virtuoso://localhost:4444";
>>
>> /* STEP 1 */
>> VirtGraph wpGraph = new VirtGraph
>> ("http://rdf.wikipathways.org/";, url, “dba", “<dba-pw>");
>>
>> wpGraph.clear();
>> VirtuosoUpdateRequest vur =
>> VirtuosoUpdateFactory.read(in,wpGraph);
>> vur.exec();
>>
>>
>>
>> or loading them triple by triple:
>>
>> VirtGraph wpGraph = new VirtGraph
>> ("http://rdf.wikipathways.org/";, url, “dba", “<dba-pw>");
>>
>> //wpGraph.getTransactionHandler().begin();
>> wpGraph.clear();
>> StmtIterator iter = model.listStatements();
>> while (iter.hasNext()) {
>> Statement stmt = iter.nextStatement(); // get
>> next statement
>> wpGraph.add(new Triple(stmt.getSubject().asNode(),
>> stmt.getPredicate().asNode(), stmt.getObject().asNode()));
>> System.out.println( stmt.getSubject()+" -
>> "+stmt.getPredicate()+" - "+stmt.getObject());
>> }
>>
>> Both of these approached work, they only take hours to proceed.
>>
>> Can I get the minutes needed for loading through the isql command line in
>> an automated pipeline process? Can I trigger the DB.DBA.TTLP_MT through java
>> or a shell script?
>>
>> Any guidance is much appreciated
>>
>>
>> Kind regards,
>>
>>
>> Andra Waagmeester
>>
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> CenturyLink Cloud: The Leader in Enterprise Cloud Services.
>> Learn Why More Businesses Are Choosing CenturyLink Cloud For
>> Critical Workloads, Development Environments & Everything In Between.
>> Get a Quote or Start a Free Trial Today.
>> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
>> _______________________________________________
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>>
>>
>>
>>
>> --
>> Jim McCusker
>>
>> Data Scientist
>> 5AM Solutions
>> jmccus...@5amsolutions.com
>> http://5amsolutions.com
>>
>> PhD Student
>> Tetherless World Constellation
>> Rensselaer Polytechnic Institute
>> mcc...@cs.rpi.edu
>> http://tw.rpi.edu
>> ------------------------------------------------------------------------------
>> CenturyLink Cloud: The Leader in Enterprise Cloud Services.
>> Learn Why More Businesses Are Choosing CenturyLink Cloud For
>> Critical Workloads, Development Environments & Everything In Between.
>> Get a Quote or Start a Free Trial Today.
>> http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk_______________________________________________
>> Virtuoso-users mailing list
>> Virtuoso-users@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/virtuoso-users
>
> -------------------------------------------------------------------
> Jerven Bolleman jerven.bolle...@isb-sib.ch
> SIB Swiss Institute of Bioinformatics Tel: +41 (0)22 379 58 85
> CMU, rue Michel Servet 1 Fax: +41 (0)22 379 58 58
> 1211 Geneve 4,
> Switzerland www.isb-sib.ch - www.uniprot.org
> Follow us at https://twitter.com/#!/uniprot
> -------------------------------------------------------------------
------------------------------------------------------------------------------
CenturyLink Cloud: The Leader in Enterprise Cloud Services.
Learn Why More Businesses Are Choosing CenturyLink Cloud For
Critical Workloads, Development Environments & Everything In Between.
Get a Quote or Start a Free Trial Today.
http://pubads.g.doubleclick.net/gampad/clk?id=119420431&iu=/4140/ostg.clktrk
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
