On Sat, May 30, 2020, 6:08 PM Tom Larsen <[email protected]> wrote:
> I've managed to batch the incoming data, so I thought I would provide an
> update:
>
> Batching 10 records at a time reduced runtime to just under 40 seconds
> (beating Python!), so the slowdown I am seeing is overhead. In my case the
> overhead equates to about 5 microseconds per call from C to Go.
>
Thanks for the update.
It's interesting that needm and extram are so high. Are these calls from
threads started by C to Go, as opposed to calls from Go to C to Go? In the
current implementation that is the worst case.
Ian
On Wednesday, May 27, 2020 at 1:08:17 PM UTC-4, Tom Larsen wrote:
>>
>> I am attempting to build a Golang SDK for the Alteryx analytic
>> application. Alteryx provides a C API for interacting with the engine, so
>> I thought I would use cgo to build a bridge between Alteryx and Go.
>>
>> The basic flow-of-control looks something like this:
>>
>> 1. The engine pushes a record of data (a C pointer to a blob of
>> bytes) to my SDK by calling a cgo function (iiPushRecord). So, C is
>> calling
>> Go here. My cgo function looks like this:
>>
>> //export iiPushRecord
>> func iiPushRecord(handle unsafe.Pointer, record unsafe.Pointer) C.long {
>> incomingInterface := pointer.Restore(handle).(IncomingInterface)
>> if incomingInterface.PushRecord(record) {
>> return C.long(1)
>> }
>> return C.long(0)
>> }
>>
>> 2. My SDK calls a method on an interface that does something with the
>> data. For my basic example, I'm just copying the data to some outgoing
>> buffers (theoretically, a best case scenario).
>> 3. The interface object pushes the data back to the engine by calling
>> my SDK's PushRecord function, which in turn calls a similar C function on
>> the engine. The PushRecord function in my SDK looks like this:
>>
>> func PushRecord(connection *ConnectionInterfaceStruct, record
>> unsafe.Pointer) error {
>> result := C.callPushRecord(connection.connection, record)
>> if result == C.long(0) {
>> return fmt.Errorf(`error calling pII_PushRecord`)
>> }
>> return nil
>> }
>>
>>
>> and the callPushRecord function in C looks like this:
>>
>> long callPushRecord(struct IncomingConnectionInterface * connection, void
>> * record) {
>> return connection->pII_PushRecord(connection->handle, record);
>> }
>>
>>
>> When I execute my base code 10 million times (simulating 10 million
>> records) in a unit test, it will execute in 20-30 seconds. This test does
>> not include the cgo calls. However, when I package the tool and execute it
>> in Alteryx with 10 million records, it takes about 1 minute 20 seconds to
>> execute. I benchmarked against an equivalent tool I built using Alteryx's
>> own Python SDK, which takes 1 minute. My goal is to be faster than Python.
>>
>> I ran a CPU profile while Alteryx was running. Of the 1.38 minute
>> runtime, the profile samples covered 42.95 seconds. The profile starts out
>> like this:
>>
>> crosscall2 (0%) -> _cgoexp_89e40a732b6d_iiPushRecord (0%) -> runtime
>> cgoballback (0%) -> runtime cgocallback_gofunc (0.14%)
>>
>> At this point, the profile branches into 3:
>>
>> 1. runtime cgocallback, which eventually calls all of my SDK code.
>> This branch accounts for 17.06 seconds in total
>> 2. runtime needm, which accounts for 8.21 seconds in total
>> 3. runtime dropm, which accounts for 17.43 seconds in total
>>
>> If you want a graphical display of the profile, it's here:
>> https://i.stack.imgur.com/CphbG.png
>>
>> It looks like the C to Go overhead is responsible for ~60% of the total
>> execution time? Is this the correct way to interpret the profile? If so,
>> is it because of something I did wrong, or is this overhead inherent to the
>> runtime? There isn't noticeable overhead when my Go code calls C, so the
>> upfront overhead from C to Go really surprised me. Is there anything I can
>> do here?
>>
>> I am running Go 1.14.3 on windows/amd64. It's actually a Windows 10 VM
>> on my Macbook, if that makes any difference.
>>
>> All of the code is on GitHub: https://github.com/tlarsen7572/goalteryx
>>
>> Note: I asked this on SO a few days ago, but got no answers, so I thought
>> I would try here. I hope that's ok.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/67cf04eb-ea28-4ac9-b341-ee8d33af992a%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-nuts/67cf04eb-ea28-4ac9-b341-ee8d33af992a%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcW5qKQphXusb3LcjSGaPuSRuQB3aCEGYfWd7vgLSZ0REw%40mail.gmail.com.
