Also posted on Stack Overflow:
Why is more memory being used when encoding a slice of structs with smaller
length
https://stackoverflow.com/questions/78136093/why-is-more-memory-being-used-when-encoding-a-slice-of-structs-with-smaller-leng
On Monday, March 11, 2024 at 12:39:56 PM UTC-4 Kwaku Biney wrote:
> Go version
>
> go version devel go1.23-e8b5bc63be linux/amd64
>
> When using binary.Write for encoding a slice of structs, I encountered
> some weird behaviour where memory allocations in a particular path was more
> than I expected.
>
> I wrote some benchmarks in the standard library's encoding/binary package
> to demonstrate this.
> func BenchmarkWriteSlice1000Structs(b *testing.B) { slice :=
> make([]Struct, 1000) buf := new(bytes.Buffer) var w io.Writer = buf
> b.SetBytes(int64(Size(slice))) b.ResetTimer() for i := 0; i < b.N; i++ {
> buf.Reset() Write(w, BigEndian, slice) } b.StopTimer() } func
> BenchmarkWriteSlice10Structs(b *testing.B) { slice := make([]Struct, 10)
> buf := new(bytes.Buffer) var w io.Writer = buf
> b.SetBytes(int64(Size(slice))) b.ResetTimer() for i := 0; i < b.N; i++ {
> buf.Reset() Write(w, BigEndian, slice) } b.StopTimer() }
>
> I ran both benchmarks and these are the results:
>
> - Encoding a slice with 1000 struct elements
>
> root@ubuntu-s-2vcpu-2gb-fra1-01:~/go/src/encoding/binary# ../../../bin/go
> test -run='^$' -memprofile memprofile.out -benchmem -bench
> BenchmarkWriteSlice1000Structs -count=10
> root@ubuntu-s-2vcpu-2gb-fra1-01:~/go/src/encoding/binary# ../../../bin/go
> tool pprof memprofile.out File: binary.test Type: alloc_space Time: Mar 9,
> 2024 at 3:27pm (UTC) Entering interactive mode (type "help" for commands,
> "o" for options) (pprof) top Showing nodes accounting for 1305.40MB, 99.84%
> of 1307.48MB total Dropped 8 nodes (cum <= 6.54MB) flat flat% sum% cum cum%
> 1302.13MB 99.59% 99.59% 1304.21MB 99.75% encoding/binary.Write 3.27MB 0.25%
> 99.84% 1307.48MB 100% encoding/binary.BenchmarkWriteSlice1000Structs 0 0%
> 99.84% 1305.31MB 99.83% testing.(*B).launch 0 0% 99.84% 1307.48MB 100%
> testing.(*B).runN
>
> - Encoding a slice with 10 struct elements
>
> root@ubuntu-s-2vcpu-2gb-fra1-01:~/go/src/encoding/binary# >
> ../../../bin/go test -run='^$' -memprofile memprofile.out -benchmem -bench
> BenchmarkWriteSlice10Structs -count=10 warning: GOPATH set to GOROOT
> (/root/go) has no effect
> root@ubuntu-s-2vcpu-2gb-fra1-01:~/go/src/encoding/binary# ../../../bin/go
> tool pprof memprofile.out warning: GOPATH set to GOROOT (/root/go) has no
> effect File: binary.test Type: alloc_space Time: Mar 9, 2024 at 4:24pm
> (UTC) Entering interactive mode (type "help" for commands, "o" for options)
> (pprof) top Showing nodes accounting for 905.58MB, 100% of 905.58MB total
> flat flat% sum% cum cum% 792.58MB 87.52% 87.52% 905.58MB 100%
> encoding/binary.Write 113MB 12.48% 100% 113MB 12.48%
> reflect.(*structType).Field 0 0% 100% 905.58MB 100%
> encoding/binary.BenchmarkWriteSlice10Structs 0 0% 100% 113MB 12.48%
> encoding/binary.dataSize 0 0% 100% 113MB 12.48% encoding/binary.sizeof 0 0%
> 100% 113MB 12.48% reflect.(*rtype).Field 0 0% 100% 905.58MB 100%
> testing.(*B).launch 0 0% 100% 905.58MB 100% testing.(*B).runN (pprof)
>
> Per the benchmarks, there is a rise in total memory allocated incurred at
> reflect.(*structType).Field when encoding a slice of 10 struct elements
> compared to a slice of 1000 struct elements. I expected to see the memory
> incurred to be at worst, the same if not less when encoding a slice of
> structs with lesser length. I draw my conclusion from here since we are
> calling sizeof on the same struct type regardless of the length of the
> slice.
>
>
> https://github.com/golang/go/blob/74726defe99bb1e19cee35e27db697085f06fda1/src/encoding/binary/binary.go#L483
>
> Also, looking at the primary source of the allocations, per the line
> below, since we are working with the same struct type, I expect the number
> of allocations here to be the same regardless since both benchmarks are
> working with the same struct type and hence have the same fields.(
> https://github.com/golang/go/blob/74726defe99bb1e19cee35e27db697085f06fda1/src/reflect/type.go#L1061
> )
>
> I am just wondering if this change is due to the random sampling used by
> the profiler and hence it is nothing to worry about or if there is some
> sort of logical explanation? I might be misinterpreting these results.
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/e9ea5c9a-d9e3-43f6-b804-2d3f26e775cfn%40googlegroups.com.
