I pulled the plug on it after > 48 hours. The file is 14Mb at this point.
I may start the Postgres server and try the export again. Perhaps if puppetdb isn't running this will be more efficient. On Friday, October 10, 2014 6:13:04 PM UTC-7, Wyatt Alt wrote: > > Hey Jon, > > Thanks for the update and the ticket: > https://tickets.puppetlabs.com/browse/PDB-947 > > We've been trying to reproduce this today and have had relatively > little luck. As you alluded earlier, it seems like a possible > contributor might be the number of reports per node. Would you mind > giving us the sizes of your exported tarball and contained folders, as > well as the average number of reports per node? > > I'm currently running a 200 node hsql export with 745 reports per node > (149k total), which is beyond the scale we would expect to be > performant on hsql, and at current pace it will still be done in seven > hours or so. > > To the extent that you're still having trouble a stack dump may be > helpful in diagnosing the issue. > > Thanks, > Wyatt > > > On Thu Oct 9 03:50:13 2014, JonY wrote: > > Update: 45 hours - ~75% complete > > > > On Tuesday, October 7, 2014 1:32:02 PM UTC-7, JonY wrote: > > > > Running for 7 hours now. Has exported ~15-20% of the data. > > > > I'm intrigued to see what I end up with. > > > > > > On Tuesday, October 7, 2014 1:19:04 PM UTC-7, Wyatt Alt wrote: > > > > Thanks Jony. > > > > I've loaded up an embedded database to comparable capacity and > > while export isn't quick, it's not nearly as slow as you're > > experiencing. > > > > Fro your process list the PDB appears to be running with a max > > heap size (Xmx) of 1024m. Perhaps increasing this could make > > a difference? > > > > The export process appears to be using 192m, but I think that > > should be using the same JAVA_ARGS as PuppetDB itself and so > > could be a bug. > > > > Wyatt > > > > > > On Tue, Oct 7, 2014 at 11:03 AM, JonY <[email protected]> > wrote: > > > > https://gist.github.com/ce60c590a0531c0b09cd.git > > <https://gist.github.com/ce60c590a0531c0b09cd.git> > > > > > > # rpm -qa | grep puppet > > puppet-server-3.7.1-1.el6.noarch > > puppetlabs-release-6-11.noarch > > puppetdb-terminus-2.2.0-1.el6.noarch > > puppet-3.7.1-1.el6.noarch > > vim-puppet-2.7.20-1.el6.rf.noarch > > puppetdb-2.2.0-1.el6.noarch > > > > Am storing 30 days of data. Yes - it's a fair bit more > > then the default. > > > > > > > > > > On Tuesday, October 7, 2014 9:00:17 AM UTC-7, Wyatt Alt > wrote: > > > > Hey JonY, > > > > Sounds interesting. What version of PuppetDB are you > > using? Do you have reports, facts, and catalogs, or > > only some of those? Can you paste your config.ini? > > > > Also can you give the output of > > > > ps aux |grep java > > top -n1 > > free > > > > in a gist maybe? > > > > Wyatt > > > > > > > > > > > > On Tue, Oct 7, 2014 at 8:30 AM, JonY > > <[email protected]> wrote: > > > > (ok - terrribly unprofessional title.. I get it). > > > > I'm trying to 'do the right thing' and move from > > the embedded DB to Postgres. Following the > > instructions I figured I would dump out the > > contents of the embedded DB and import this into > > Postgres. > > > > So I start 'puppetdb export --outfile <someplace>'. > > > > Pour some coffee. > > > > Watch it dump one node. > > > > Have some more coffee. > > > > Wait. > > > > One more node. > > > > Hours pass. Glaciers melt. > > > > 3+ hours into the process and it has dumped 10 > > nodes (of > 100). At this rate I'm looking at > > about 1.5 days to get this done. Really? > > > > System has 16 cores, 32Gb of RAM.. barely running > > at idle. Tell me I missed some critical parameter. > > > > > > -- > > You received this message because you are > > subscribed to the Google Groups "Puppet Users" > group. > > To unsubscribe from this group and stop receiving > > emails from it, send an email to > > puppet-users...@__googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/__msgid/puppet-users/1fbe251a-__2c25-42ea-9529-d01bbf561980%__40googlegroups.com > > > < > https://groups.google.com/d/msgid/puppet-users/1fbe251a-2c25-42ea-9529-d01bbf561980%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit > > https://groups.google.com/d/__optout > > <https://groups.google.com/d/optout>. > > > > > > -- > > You received this message because you are subscribed to > > the Google Groups "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails > > from it, send an email to [email protected]. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/puppet-users/9b735460-1722-48dc-9ad5-f34f554567f4%40googlegroups.com > > > < > https://groups.google.com/d/msgid/puppet-users/9b735460-1722-48dc-9ad5-f34f554567f4%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > > > For more options, visit https://groups.google.com/d/optout > > <https://groups.google.com/d/optout>. > > > > > > -- > > You received this message because you are subscribed to the Google > > Groups "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [email protected] <javascript:> > > <mailto:[email protected] <javascript:>>. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/puppet-users/b0bd8590-0649-4188-a325-fe41168f96bc%40googlegroups.com > > > < > https://groups.google.com/d/msgid/puppet-users/b0bd8590-0649-4188-a325-fe41168f96bc%40googlegroups.com?utm_medium=email&utm_source=footer>. > > > > For more options, visit https://groups.google.com/d/optout. > > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/5b8b1002-ee31-416c-a198-dbe46014a8d9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
