|
When environment_timeout is set to 0, caching of parsed manifests is essentially a no-op as any object put into the cache is immediately marked as expired. However, Puppet still builds a cache when the timeout is set to 0 which consumes memory. This memory usage can be significant, especially when classifier sync hits the resource_types/* endpoint on each environment.
Reproduction Case
-
Install puppet-agent and puppetserver.
-
Edit /opt/puppetlabs/puppet/puppet.conf and set environment_timeout to 0 in the master section.
-
Edit /opt/puppetlabs/puppet/auth.conf and allow unrestricted access to the environments and resource_types endpoints:
path /puppet/v3/environments
|
auth any
|
method find
|
allow *
|
|
path /puppet/v3/resource_type
|
auth any
|
method search
|
allow *
|
-
Create a large number of classes in the production environment, and then a large number of copies of that environment (the number created is a bit excessive since the classes are simple, but real-world environments have shown this behavior with only hundreds of classes):
mkdir -p /etc/puppetlabs/code/environments/production/modules/test_classes/manifests
|
|
cat <<EOF | /opt/puppetlabs/puppet/bin/erb > /etc/puppetlabs/code/environments/production/modules/test_classes/manifests/init.pp
|
<% 100000.times do |i| %>
|
class test_class_<%= i %> () {}
|
<% end %>
|
EOF
|
|
for i in $(seq 100);do cp -r /etc/puppetlabs/code/environments/production /etc/puppetlabs/code/environments/test_${i}; done
|
-
Restart the puppetserver.
-
Simulate classifier syncs using the following script:
#!/bin/bash
|
|
get_env_list() {
|
curl -s -k \
|
https://localhost:8140/puppet/v3/environments |\
|
/opt/puppetlabs/puppet/bin/ruby -rjson -e \
|
'puts JSON.load($stdin.read())["environments"].keys.sort'
|
}
|
|
get_env_classes() {
|
start_time=`date +%s`
|
curl -s -k \
|
"https://localhost:8140/puppet/v3/resource_types/*?environment=$1" > /tmp/${1}_classes.json;
|
echo "Sync time: $(expr `date +%s` - $start_time) seconds";
|
}
|
|
get_all_classes() {
|
for env in $(get_env_list); do
|
echo "Fectching classes for: $env";
|
get_env_classes $env
|
done
|
}
|
|
while :; do get_all_classes; done
|
Outcome
During class sync, each class is persisted to the cache despite environment_timeout being set to zero. This causes the Puppet Server to run out of memory:
![]()
-
-
-
Expected Outcome
When `environment_timeout` is set to 0, no memory is wasted on caching classes.
Applying a quick spike for this change:
diff --git a/puppet/environments.rb b/puppet/environments.rb
|
index fa5a028..0ac12b7 100644
|
--- a/puppet/environments.rb
|
+++ b/puppet/environments.rb
|
@@ -304,7 +304,8 @@ module Puppet::Environments
|
if result = @cache[name]
|
return result.value
|
elsif (result = @loader.get(name))
|
- @cache[name] = entry(result)
|
+ e = entry(result)
|
+ @cache[name] = e unless e.is_a?(NotCachedEntry)
|
result
|
end
|
end
|
Results in the Puppet Server being able to sustain syncing a large number of classes:
![]()
|
