The reqs_per_event setting was causing a client that was doing large
batch-gets (of a few hundred keys) to see some timeouts. Since memcached
will delay responding fully until other connections are serviced and our
client will wait until the batch is done, we see some client-side timeouts
for the users of our client library. Our solution has been to up the
setting during startup, but just as a thought experiment I was asking if we
could have done it dynamically to avoid losing data. At the moment there's
quite a lot of machinery to change the setting (deploy, copy data over with
our cache warmer, flip traffic, tear down old boxes) and I would have
rather left everything as is and adjusted the setting on the fly until our
client's problem was resolved.
I'm interested in patching this specific setting to be settable, but having
it fully dynamic in nature is not something I'd want to tackle. There's a
natural tradeoff of latency for other connections / throughput for the one
that is currently being serviced. I'm not sure it's a good idea to
dynamically change that. It might cause unexpected behavior if one bad
client sends huge requests.
*Scott Mansfield*
Product > Consumer Science Eng > EVCache > Sr. Software Eng
{
M: 352-514-9452
E: [email protected]
K: {M: mobile, E: email, K: key}
}
On Tue, Jan 24, 2017 at 11:53 AM, dormando <[email protected]> wrote:
> Hey,
>
> Would you mind explaining a bit how you determined the setting was causing
> an issue, and what the impact was? The default there is very old and might
> be worth a revisit (or some kind of auto-tuning) as well.
>
> I've been trending as much as possible to online configuration, inlcuding
> the actual memory limit.. You can turn the lru crawler on and off,
> automoving on and off, manually move slab pages, etc. I'm hoping to make
> the LRU algorithm itself modifyable at runtime.
>
> So yeah, I'd take a patch :)
>
> On Mon, 23 Jan 2017, 'Scott Mansfield' via memcached wrote:
>
> > There was a single setting my team was looking at today and wish we
> could have changed dynamically: the
> > reqs_per_event setting. Right now in order to change it we need to shut
> down the process and start it again
> > with a different -R parameter. I don't see a way to change many of the
> settings, though there are some that
> > are ad-hoc changeable through some stats commands. I was going to see if
> I could patch memcached to be able
> > to change the reqs_per_event setting at runtime, but before doing so I
> wanted to check to see if that's
> > something that would be amenable. I also didn't want to do something
> specifically for that setting if it was
> > going to be better to add it as a general feature.
> > I see some pros and cons:
> >
> > One easy pro is that you can easily change things at runtime to save
> performance while not losing all of
> > your data. If client request patterns change, the process can react.
> >
> > A con is that the startup parameters won't necessarily match what the
> process is doing, so they are no
> > longer going to be a useful way to determine the settings of memcached.
> Instead you would need to connect
> > and issue a stats settings command to read them. It also introduces
> change in places that may have
> > previously never seen it, e.g. the reqs_per_event setting is simply read
> at the beginning of the
> > drive_machine loop. It might need some kind of synchronization around it
> now instead. I don't think it
> > necessarily needs it on x86_64 but it might on other platforms which I
> am not familiar with.
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to
> > [email protected].
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "memcached" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/memcached/C6l8aoXQO4A/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
