Hi Darragh, SimpleCov is a ruby-only solution and the puppet code (*.pp) coverage from rspec-puppet does not integrate into it. As far as I can see there is currently no way to get at the raw data without hacking rspec-puppet itself <https://github.com/rodjek/rspec-puppet/blob/3a053673a32587e3507597a7527e1a90a8425cc4/lib/rspec-puppet/coverage.rb#L72> .
Cheers, David On Fri, Jun 15, 2018 at 4:04 PM Darragh Bailey <[email protected]> wrote: > Hi, > > > I'm to understand how to get a full coverage for puppet code with spec > tests to be generated so I can get output like the example from simplecov > at https://github.com/colszowka/simplecov#example-output. > > Ideally I would subsequently try to hook into the coverage repo and do > some post processing looking at where most of the code churn has been in > our puppet control code repo and target those areas as important for test > coverage (and likely refactoring). > > However I've stuck on not being able to get output from simplecov that > shows the manifests that were tested by the spec tests, although the > summary triggered from running 'RSpec::Puppet::Coverage.report!' suggests > that something was capable of tracking coverage correctly. > > Although we have a different layout than normal, my testing so far > suggests that the reporting simply doesn't work between puppet and > simplecov. > > Usually when something like this happens, I assume I'm doing something > blindingly stupid, so I've put together some example code that shows my > problem should anyone like to run through the spec tests from it. > > Doing some simple experiments I've pushed up a repository showing the > issue https://github.com/electrofelix/puppet-simplecov_bug containing 3 > commits > - (HEAD commit): convert to match more closely skeleton layout from > https://github.com/garethr/puppet-module-skeleton/blob/master/skeleton/Gemfile > - (HEAD~1 commit): try to follow a more standard layout > - (root commit): Use simple define taken from our codebase and matching > the layout we have as closely as possible > > > Note we're still stuck on Puppet 3.8 and using ruby 2.1.10 via rvm while I > was doing the spec tests, plan to upgrade to 5 in the near future, so if > this problem has been solved via something in the puppet engine (some > suggestions that the multiple loading by puppet was wiping SimpleCov's > tracking over coverage) and we just need to hurry up that's great. > > But no matter what layout/format I seem to try, the simplecov report > prints nothing as having been covered, e.g.: > > > *************************************************************************************************** > Total resources: 9 > Touched resources: 8 > Resource coverage: 88.89% > Untouched resources: > > Ssh::Config_entry[example custom-host] > > Finished in 2.01 seconds (files took 1.05 seconds to load) > 12 examples, 0 failures > Coverage report generated for RSpec to <path>/puppet-simplecov_bug/html. 0 > / 67 LOC (0.0%) covered. > > COVERAGE: 0.00% -- 0/67 lines in 2 files > > > +----------+-------------------------------+-------+--------+-----------------------------------------+ > | coverage | file | lines | missed | > missing | > > +----------+-------------------------------+-------+--------+-----------------------------------------+ > | 0.00% | manifests/server/localuser.pp | 67 | 67 | 5-13, 15-19, > 21-34, 36-44, 46-55, 57-76 | > > +----------+-------------------------------+-------+--------+-----------------------------------------+ > 1 file(s) with 100% coverage not shown > > *************************************************************************************************** > > RSpec::Puppet::Coverage.report causes the first part to be printed and > certainly it shows it's tracking coverage on the files touched, however the > simplecov report at the end does not appear to be able to correlate the > tests in 'spec/defines/server_localuser_spec.rb' as having tested > 'manifests/server/localuser.pp'. > > Additionally what is missing from both is all files that are not touched > at all so far by tests, though I do have some code that appears to be able > to handle that by passing 'track_files("**/*.pp")' to the SimpleCov object > configuration. > > > > Thinking maybe I started from the wrong place, I took > https://github.com/garethr/puppet-module-skeleton and created a fresh > module from that and then dropped in both the spec file ( > https://github.com/electrofelix/puppet-simplecov_bug/blob/master/spec/defines/server_localuser_spec.rb) > and define ( > https://github.com/electrofelix/puppet-simplecov_bug/blob/master/manifests/server/localuser.pp) > that were being tested and the result remained the same. > > > Rspec::Puppet::Coverage.report! gave: > > Total resources: 15 > Touched resources: 14 > Resource coverage: 93.33% > Untouched resources: > > Ssh::Config_entry[example custom-host] > > > But SimpleCov gave: > > Coverage report generated for RSpec to <path>/test-coverage/coverage. 0.0 > / 0.0 LOC (100.0%) covered. > > > So even worse than before... > > > Resources I've looked at: > > https://github.com/garethr/puppet-module-skeleton > https://github.com/rodjek/rspec-puppet > https://github.com/rodjek/rspec-puppet#producing-coverage-reports > https://github.com/rodjek/rspec-puppet/issues/316 > https://github.com/puppetlabs/puppetlabs-rook (seems to have the same > issue) > > https://github.com/colszowka/simplecov#example-output > https://github.com/colszowka/simplecov/issues/682 > https://github.com/colszowka/simplecov/issues/389 > https://github.com/colszowka/simplecov/issues/340 > https://github.com/colszowka/simplecov/issues/77 > > my example of the issue: > https://github.com/electrofelix/puppet-simplecov_bug > > > Hoping that there is something relatively trivial here that I'm missing > that allows the reports generated by simplecov to correctly reference spec > tests with puppet code. > > > > Regards, > Darragh > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/3afe6401-44d9-4bcb-8628-6bc76e4b3491%40googlegroups.com > <https://groups.google.com/d/msgid/puppet-users/3afe6401-44d9-4bcb-8628-6bc76e4b3491%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- Cheers, David https://twitter.com/dev_el_ops -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CALF7fHao7EMwGGA6KLDfuGPtBh%3D3GPbwte%3Dmi7fpaUgpUUWVmg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
