Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Matt McCutchen
On Jun 21, 8:57 pm, "Brian Smith" wrote: > From arcfour.c: > > http://mxr.mozilla.org/mozilla/source/security/nss/lib/freebl/arcfour... > > My guess is that valgrind is considering malloc(5) to allocate 5 bytes, when > really it allocates 8 bytes at least (because of alignment). Strictly speaking

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Matt McCutchen
On Jun 22, 7:10 pm, Mads Kiilerich wrote: > Wouldn't the bulk word processing (sic!) that really matters > perform even better if it didn't have to consider masks because leading > and trailing bytes had been handled byte by byte? That is already done. There is separate code for the leading, bul

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Mads Kiilerich
Nelson Bolyard wrote, On 06/22/2010 07:49 AM: On 2010-06-21 17:57 PDT, Brian Smith wrote: From arcfour.c: http://mxr.mozilla.org/mozilla/source/security/nss/lib/freebl/arcfour.c#390 My guess is that valgrind is considering malloc(5) to allocate 5 bytes, when really it allocates 8 bytes at

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Matt McCutchen
On Jun 22, 1:49 am, Nelson Bolyard wrote: > I presume that there must be some incantation that one can give to valgrind > that will force it to shut up about arcfour. You can write a valgrind suppression, which is essentially a stack trace pattern that will cause matching errors to be ignored: h

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Nelson B Bolyard
On 2010-06-21 17:57 PDT, Brian Smith wrote: > From arcfour.c: > > http://mxr.mozilla.org/mozilla/source/security/nss/lib/freebl/arcfour.c#390 > > My guess is that valgrind is considering malloc(5) to allocate 5 bytes, > when really it allocates 8 bytes at least (because of alignment). See the exp

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Robert Relyea
> > Yes, it works correctly, so I assume that it doesn't use the extra > bytes. It is however not just reading uninitialized memory (which > valgrind usually tracks correctly), but it reads unallocated memory. > It is also interesting that it doesn't read beyond the bounds for > small sizes, so

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-22 Thread Nelson Bolyard
On 2010-06-21 17:57 PDT, Brian Smith wrote: > From arcfour.c: > > http://mxr.mozilla.org/mozilla/source/security/nss/lib/freebl/arcfour.c#390 > > My guess is that valgrind is considering malloc(5) to allocate 5 bytes, when > really it allocates 8 bytes at least (because of alignment). I presume

RE: PK11_CipherOp with RC4 and invalid memory access

2010-06-21 Thread Brian Smith
dev-tech-crypto-bounces+brian=briansmith@lists.mozilla.org > [mailto:dev-tech-crypto-bounces+brian=briansmith@lists.mozilla.org] On > Behalf Of Mads Kiilerich > Sent: Monday, June 21, 2010 7:06 PM > To: Robert Relyea > Cc: mozilla's crypto code discussion list > Subject

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-21 Thread Mads Kiilerich
Robert Relyea wrote, On 06/22/2010 01:54 AM: On 06/19/2010 01:43 PM, Mads Kiilerich wrote: Hi I'm trying to port an application from OpenSSL to NSS. The biggest problem right now is that valgrind reports that NSS accesses invalid memory when using RC4. There is no problem with chunk sizes u

Re: PK11_CipherOp with RC4 and invalid memory access

2010-06-21 Thread Robert Relyea
On 06/19/2010 01:43 PM, Mads Kiilerich wrote: > Hi > > I'm trying to port an application from OpenSSL to NSS. The biggest > problem right now is that valgrind reports that NSS accesses invalid > memory when using RC4. There is no problem with chunk sizes up to 8 > and sizes divisible with 4, but fo

PK11_CipherOp with RC4 and invalid memory access

2010-06-19 Thread Mads Kiilerich
Hi I'm trying to port an application from OpenSSL to NSS. The biggest problem right now is that valgrind reports that NSS accesses invalid memory when using RC4. There is no problem with chunk sizes up to 8 and sizes divisible with 4, but for other sizes it access the source and destination i