Nice, thank you very much.
Are you ok with :
Example :
•Key : test
•Data : 90 bytes
•Flags : 0
Size : (4+1) + (90+2) + (1+2+4) + 48 + 8 = 160 Bytes => Slab 4 (192 Bytes)
Key (characters count) + 1
+ Data (characters count) + 2 bytes ( \r\n )
+ Header
+ Chunk Size
+ CAS Size
Header = Flags (characters count) + nData (characters number of DATA count
=> 90 = 2 characters)
+ 4 bytes (2 spaces and \r\n)
Chunk Size = 48 bytes (default)
CAS Size = 8 bytes (platform 64 bits)
If CAS disabled:
Size : (4+1) + (90+2) + (1+2+4) + 48 = 152 Bytes => Slab 3 (152 Bytes)
Sounds good :)
Le lundi 16 novembre 2015 23:54:34 UTC+1, Dormando a écrit :
>
> key + 1b
> val + 2b
> item struct 48b
> [optional CAS] 8b
> then the suffix trailer printf: please read the manpage for snprintf:
>
> snprintf(suffix, 40, " %d %d\r\n", flags, nbytes - 2);
>
> there are four characters in there. two spaces, and then \r and \n. the
> two %d's change into: "0" for flags, and "90" for bytes (ignore the -2 in
> there). that's 7b total.
>
> In your no-cas test: 5 + 92 + 48 + 7 == 152. that's the exact size of slab
> 3. (152b).
>
> On Mon, 16 Nov 2015, Nicolas Martinez wrote:
>
> > The test was :
> > •Key : test
> > •Data : 90 bytes
> > •Flags : 0
> >
> > :(
> >
> > Le lundi 16 novembre 2015 19:01:50 UTC+1, Nicolas Martinez a écrit :
> > Ok... i think i have understood what you say:
> >
> > Key (Characters Number) + 1 + Data (Characters Number) + Header +
> Chunk Size + CAS Size
> >
> > Header = Flags (Characters Number) + Key (Characters Numbers)
> >
> > + 2 bytes ( \r\n )
> >
> > + 4 bytes (2 spaces and 1 \r)
> >
> > Chunk Size = 48 bytes (default)
> >
> > CAS Size = 8 bytes (64 bits platform)
> >
> >
> > Seems to be good:
> >
> > 4 + 1 + 90 + (1+4+2+4) + 48 + 8 = 162 Bytes => Slab 4 (192 Bytes)
> >
> >
> > But, if i start Memcached with -C, it's wrong.
> >
> > 4 + 1 + 90 + (1+4+2+4) + 48 = 154 Bytes => Slab 3 (152 Bytes)
> >
> >
> > It must be in Slab4 (192 Bytes) No?
> >
> >
> > Le lundi 16 novembre 2015 15:37:59 UTC+1, Nicolas Martinez a
> écrit :
> > Thank you very much for yours answers.Ok for CAS... i don't
> use -C so i have to add 8 bytes
> >
> > i still don"t understand these lines :
> > >> 2b appended to data_bytes for an extra "\r\n" +
> > So, \r\n == 2b ?
> >
> > >> + 4 bytes for spaces and \r\n
> > which spaces?
> > What is this \r\n ? Isn't already counted before?
> > There is 1 "\r\n" and 1 "\"
> >
> > echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc ${memcached_server}
> 11211
> >
> >
> > With your example:
> > * Key : 2c
> > * Val : 28 bytes
> > * Flg : 0 (1bytes)
> >
> > turns into:
> > * Key : 3b
> >
> > => key number characters + 1
> > * Val : 30b
> >
> > => 28 bytes + 2 bytes ("\r\n")
> > * Hdr : 4b + 3b == 7b
> >
> > => What are 4b?
> > => 3b are: flags (1b) + ??
> > * Itm : 56b
> >
> > => 48b + 8b (CAS)
> > => 96b. which is the cap for slab 1 in a default setup.
> >
> >
> >
> > Thank you again.
> >
> > Le lundi 16 novembre 2015 00:31:07 UTC+1, Dormando a écrit :
> > Read carefully:
> >
> > item_size_ok(const size_t nkey, const int flags, const int nbytes)
> {
> >
> > passes:
> >
> > size_t ntotal = item_make_header(nkey + 1, flags, nbytes,
> > prefix, &nsuffix);
> >
> > Then conditionally:
> >
> > if (settings.use_cas) {
> > ntotal += sizeof(uint64_t);
> > }
> >
> > item_make_header is doing:
> > nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n", flags,
> nbytes - 2);
> >
> > Then:
> >
> > return sizeof(item) + nkey + *nsuffix + nbytes;
> >
> > It's convoluted but shirt.
> >
> > the lengths are:
> > key +
> > 1 +
> > data_bytes +
> > 2b appended to data_bytes for an extra "\r\n" +
> > stringified rep of the flags + data length
> > + 4 bytes for spaces and \r\n (these are carriage returns, one
> byte each)
> > + 8b for CAS if enabled
> >
> > CAS can be turned off via the -C starttime arg. it takes up 8
> bytes.
> >
> > Example:
> > * Key : 2c
> > * Val : 28b
> > * Flg : 0 (1b)
> >
> > turns into:
> > * Key : 3b
> > * Val : 30b
> > * Hdr : 4b + 3b == 7b
> > * Itm : 56b
> > => 96b. which is the cap for slab 1 in a default setup.
> >
> > It's tough to get it exact for small chunks due to the way the
> header is
> > added. You should ballpark or tune the -f value to align with your
> > observed data.
> >
> > On Sun, 15 Nov 2015, Nicolas Martinez wrote:
> >
> > > Hi,
> > > Is CAS always used?
> > > If yes, we have to always add 56 bytes to the KEY and VALUE ?
> > > you don't count FLAGS characters??
> > >
> > > I've found that Flags's size (number of characters) impact the
> storage.
> > >
> > > Example:
> > > * Key : 2 characters = 2 bytes
> > > * Value : 28 characters = 28 bytes
> > > * FLAGS : 1 characters = 1 bytes
> > > => 31 bytes
> > >
> > > seems to take the same storage as
> > > * Key : 1 characters = 1 bytes
> > > * Value : 28 characters = 28 bytes
> > > * FLAGS : 2 characters = 2 bytes
> > > => 31 bytes ... wich is the limit to be stored in Slab1
> > >
> > > ok for the /r/n ... should take 4 bytes no?
> > >
> > > So, if we count 56 bytes for CAS :
> 56(cas)+31(key+value+flags)+4(/r/n)= 91
> > >
> > > Not good... :(
> > >
> > > where I'm wrong ??
> > >
> > > Le samedi 14 novembre 2015 23:55:12 UTC+1, Dormando a écrit :
> > > The mysql docs don't speak for the main tree... that's
> their own thing.
> > >
> > > the "sizes" binary that comes with the source tree tells
> you how many
> > > bytes an item will use (though I intend to add this output
> to the 'stats'
> > > output somewhere). With CAS this is 56 bytes.
> > >
> > > 56 + 2 + 30 == 88. Class 1 by default (in 1.4.24) is 96
> bytes, but the
> > > item still ends up in class 2.
> > >
> > > Why is this? (unfortunately?) because memcached
> pre-renders part of the
> > > text protocol into the item header:
> > >
> > > *nsuffix = (uint8_t) snprintf(suffix, 40, " %d %d\r\n",
> flags, nbytes -
> > > 2);
> > > return sizeof(item) + nkey + *nsuffix + nbytes;
> > >
> > > so the flags + length are getting flattened + \r\n added
> to the end.
> > > Together that's just enough to push it over the edge. It'd
> also be nice to
> > > add a highly optimized numerics printf so I could twiddle
> options to save
> > > a few bytes of memory in objects, but don't get your hopes
> up for that
> > > happening soon :)
> > >
> > > On Sat, 14 Nov 2015, Nicolas Martinez wrote:
> > >
> > > > Add: Memcached version : 1.4.4 (RedHat)
> > > >
> > > > Le samedi 14 novembre 2015 14:49:37 UTC+1, Nicolas
> Martinez a écrit :
> > > > Hi, few days i'm reading Memcached documentation
> and blogs... and i don't understand how objects are stored.
> > > >
> > > > My test
> > > >
> > > > 3 slabs :
> > > >
> > > > * 96.0 Bytes
> > > > * 120.0 Bytes
> > > > * 240.0 Bytes
> > > > Everywhere, it's told :
> > > > * if data is < 96 Bytes, it will be stored in Slabs1
> (96B)
> > > > * if data > 96B and < 120B, it will be stored in
> Slabs2 (120B)
> > > > * if data > 120B, it will be stored in Slabs3 (240B)
> > > > * etc.
> > > > BUT, for example, when i'm creating an 30B object, it's
> stored in Slab2 (120B), and NOT in Slab1 (96B).
> > > >
> > > > External sources:
> > > > For example, the default size for the smallest
> block is 88 bytes (40 bytes of value, and the default 48 bytes for the key
> and flag data). If the size of the first item you store into the cache is
> less than 40 bytes, then a slab with a block size of 88 bytes is created
> and the value stored.
> > > > =>
> https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-memcached-using-memory.html
>
> > > >
> > > >
> > > > WRONG
> > > >
> > > > A slab class is a collection of pages divided into
> same sized chunks. Each slab class is referenced to by its chunk size. So
> we’ll have Slab class 80kb, Slab class 100kb and so on. When an object
> needs to be stored, its size determines where it gets stored. So if the
> object is larger than 80kb but less than 100kb, it
> > gets stored into
> > > Slab
> > > > class 100kb.
> > > > =>
> http://returnfoo.com/2012/02/memcached-memory-allocation-and-optimization-2/
> > > >
> > > >
> > > > WRONG
> > > >
> > > > How i create an object:
> > > >
> > > > data=$(pwgen 30 -c 1)
> > > > echo -e 'set 30 0 3600 30\r\n'$data'\r'| nc
> ${memcached_server} 11211
> > > >
> > > >
> > > > So, when 30B object is creating :
> > > > * key name : 30 = 2 bytes
> > > > * value: 30 characters = 30 bytes
> > > > * tags : 0 = 1 bytes
> > > > => All = 33 bytes
> > > > If i add the default 48b as explained on Mysql website :
> 33 + 48 = 81B ... so < Slab1 (91B)... but always stored in Slab2 (120B)
> > > >
> > > > So, the size used to store object in the good Slab is
> not:
> > > > * object value size
> > > > * sum of KEY, VALUE and TAGS in bytes
> > > > KEY size : 1 character = 1 B
> > > > VALUE size : 1 character = 1 B
> > > > TAGS size : 1 character = 1 B
> > > >
> > > > ... as read everywhere
> > > >
> > > > So, It seems that: (SUM of KEY+VALUE+TAGS )
> > > > * For slab1 96.0 Bytes, data stored if <= 31 B (SUM of
> 2+28+1 )
> > > > * For slab2 120.0 Bytes, data stored if <= 55 B (SUM
> of 2+52+1 )
> > > > * For slab3 152.0 Bytes, data stored if <= 87 B (SUM
> of 2+84+1 )
> > > > * For slab4 192.0 Bytes, data stored if <= 126 B (SUM
> of 3+122+1 )
> > > > * For slab5 240.0 Bytes, data stored if <= 174 B (SUM
> of 3+170+1 )
> > > > * etc.
> > > >
> > > > My configuration :
> > > > * Chunk Size : 48
> > > > * Chunk Growth Factore: 1,25
> > > > * Max Bytes: 64.0 MBytes
> > > >
> > > > So, someone could explain me how the data is stored in
> the right Slabs???
> > > > How calculate it???
> > > >
> > > > Thank you
> > > >
> > > >
> > > > --
> > > >
> > > > ---
> > > > You received this message because you are subscribed to
> the Google Groups "memcached" group.
> > > > To unsubscribe from this group and stop receiving emails
> from it, send an email to [email protected].
> > > > For more options, visit
> https://groups.google.com/d/optout.
> > > >
> > > >
> > >
> > > --
> > >
> > > ---
> > > You received this message because you are subscribed to the
> Google Groups "memcached" group.
> > > To unsubscribe from this group and stop receiving emails from
> it, send an email to [email protected].
> > > For more options, visit https://groups.google.com/d/optout.
> > >
> > >
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups "memcached" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to [email protected] <javascript:>.
> > For more options, visit https://groups.google.com/d/optout.
> >
> >
--
---
You received this message because you are subscribed to the Google Groups
"memcached" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
