I guess when I say dynamic I mostly mean runttime-settable. Dynamic is a
little harder so I tend to do those as a second pass.
You're saying your client had head-of-line blocking for unrelated
requests? I'm not 100% sure I follow.
Big multiget comes in, multiget gets processed slightly slower than normal
due to other clients making requests, so requests *behind* the multiget
time out, or the multiget itself?
How long is your timeout? :P
I'll take a look at it as well and see about raising the limit in `-o
modern` after some performance tests. The default is from 2006.
thanks!
On Wed, 25 Jan 2017, 'Scott Mansfield' via memcached wrote:
> The reqs_per_event setting was causing a client that was doing large
> batch-gets (of a few hundred keys) to see some timeouts. Since memcached will
> delay responding fully until
> other connections are serviced and our client will wait until the batch is
> done, we see some client-side timeouts for the users of our client library.
> Our solution has been to
> up the setting during startup, but just as a thought experiment I was asking
> if we could have done it dynamically to avoid losing data. At the moment
> there's quite a lot of
> machinery to change the setting (deploy, copy data over with our cache
> warmer, flip traffic, tear down old boxes) and I would have rather left
> everything as is and adjusted the
> setting on the fly until our client's problem was resolved.
> I'm interested in patching this specific setting to be settable, but having
> it fully dynamic in nature is not something I'd want to tackle. There's a
> natural tradeoff of
> latency for other connections / throughput for the one that is currently
> being serviced. I'm not sure it's a good idea to dynamically change that. It
> might cause unexpected
> behavior if one bad client sends huge requests.
>
>
> Scott Mansfield
> Product > Consumer Science Eng > EVCache > Sr. Software Eng
> {
> M: 352-514-9452
> E: [email protected]
> K: {M: mobile, E: email, K: key}
> }
>
> On Tue, Jan 24, 2017 at 11:53 AM, dormando <[email protected]> wrote:
> Hey,
>
> Would you mind explaining a bit how you determined the setting was
> causing
> an issue, and what the impact was? The default there is very old and
> might
> be worth a revisit (or some kind of auto-tuning) as well.
>
> I've been trending as much as possible to online configuration,
> inlcuding
> the actual memory limit.. You can turn the lru crawler on and off,
> automoving on and off, manually move slab pages, etc. I'm hoping to make
> the LRU algorithm itself modifyable at runtime.
>
> So yeah, I'd take a patch :)
>
> On Mon, 23 Jan 2017, 'Scott Mansfield' via memcached wrote:
>
> > There was a single setting my team was looking at today and wish we
> could have changed dynamically: the
> > reqs_per_event setting. Right now in order to change it we need to
> shut down the process and start it again
> > with a different -R parameter. I don't see a way to change many of
> the settings, though there are some that
> > are ad-hoc changeable through some stats commands. I was going to see
> if I could patch memcached to be able
> > to change the reqs_per_event setting at runtime, but before doing so
> I wanted to check to see if that's
> > something that would be amenable. I also didn't want to do something
> specifically for that setting if it was
> > going to be better to add it as a general feature.
> > I see some pros and cons:
> >
> > One easy pro is that you can easily change things at runtime to save
> performance while not losing all of
> > your data. If client request patterns change, the process can react.
> >
> > A con is that the startup parameters won't necessarily match what the
> process is doing, so they are no
> > longer going to be a useful way to determine the settings of
> memcached. Instead you would need to connect
> > and issue a stats settings command to read them. It also introduces
> change in places that may have
> > previously never seen it, e.g. the reqs_per_event setting is simply
> read at the beginning of the
> > drive_machine loop. It might need some kind of synchronization around
> it now instead. I don't think it
> > necessarily needs it on x86_64 but it might on other platforms which
> I am not familiar with.
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to
> > [email protected].
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
>
> --
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "memcached" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/memcached/C6l8aoXQO4A/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "memcached" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>
>
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
