And after I re-read your answer, I realize that I probably just restated
very badly what you wrote, Egon. Thanks! I'll have some time this evening
to hack on it.
On Wednesday, May 31, 2017 at 2:51:51 PM UTC-5, Michael Brown wrote:
>
> Ah! I think I have the kernel of a usable idea here. Thanks.
>
> How does this sound: template 1 outputs template 2. Input to template 1
> kicks off the goroutines and places {{ get_results token_xyz }} as the
> output, but also has an output channel that we can wait on for all of the
> answers Then, template 2 is executed with a function that pulls the output
> from the channel. Sort of hard to explain, but seems on the face to be
> workable. I'll post results after I prototype it.
>
> --
> Michael
>
> On Wednesday, May 31, 2017 at 1:23:40 PM UTC-5, Egon wrote:
>>
>> On Wednesday, 31 May 2017 20:34:56 UTC+3, Michael Brown wrote:
>>>
>>> This is almost certainly going to be a case where you have the correct
>>> solution and I don't have the go experience to properly understand it. I
>>> wasn't quite understanding how your code was "patching" the results and how
>>> the template package knows to wait for it.
>>>
>>
>> Let's say you have a template
>>
>> {{ sleep }} xxx {{ sleep }}
>>
>> Once you render with that approach you get something like this...
>>
>> <<token1>> xxx <<token2>>
>>
>> So instead of returning the actual value you replace them with a token.
>> And spawn a go routine to fetch the answers.
>>
>> You wait for the results or just some, doesn't matter.
>>
>> Once you get a results, you replace it in the buffer...
>>
>> token1 -> "alpha"
>> token2 -> "beta"
>>
>> "alpha" xxx "beta"
>>
>> Or whatever the result is.
>>
>> However this approach is pretty limited when you want to range over the
>> results.
>>
>> Doing two passes over the template, might work better, but needs somewhat
>> better logic to handle lookup; but that could work for handling multiple
>> results...
>>
>> {{ sleep }}
>>
>> In the first pass you start the goroutines and each output goes into an
>> array of channels
>>
>> var rpcs = []func() interface{}
>>
>> var funcMap = template.FuncMap { "sleep": func() { rpcs = append(rpcs,
>> sleep) }, }
>>
>> // now you schedule goroutines in whatever way you need for the rpcs and
>> write the results to a channel
>> var results = []chan interface{}
>>
>> the second pass would do it as
>> current := 0
>>
>> var funcMap = template.FuncMap { "sleep": func() interface{} {
>>
>> result := <-results[current]
>>
>> current++
>>
>> return result
>>
>>
>> // ...
>>
>>
>> Without knowing the actual template, it is hard to recommend something
>> better simpler or easier.
>>
>> Let me describe my problem.
>>>
>>> I have a REST interface running on an embedded system. The REST api is
>>> mandated by another entity and I have zero control over the output, I have
>>> to produce specific conformant output. The data that I need to build the
>>> interface is on the system in a variety of other processes, and I can get
>>> that data via a couple of different RPC mechanisms, however sometimes these
>>> RPC mechanisms can be slow. (I'm in progress building go bindings for them).
>>>
>>> The current code which creates the REST interface is a huge morass of C
>>> code that generates the JSON output, so the exact structure of the JSON
>>> output is hardcoded in the code.
>>>
>>> I have an opportunity here, and that is that it appears to me that, with
>>> a little work, I can completely templatize the rest output. That is, I can
>>> have zero page-specific code and render almost all of the output by
>>> providing a small number of generic data access functions. The issue I ran
>>> into is the serial nature of the substitution kills the performance in my
>>> prototype.
>>>
>>> Yes, I can make a specific go function per page that gathers all the
>>> data I need and provides it to the template. But that would mean that I'd
>>> need to maintain both the templates and the data access functions. This is
>>> still an improvement on the old way of doing things, but I was hoping to
>>> jump straight to a fully templatized system, which according to my initial
>>> analysis, would be considerably less code and would not really be abusing
>>> the template system (most of the function calls are very straightforward
>>> and there is no heavy processing of the output).
>>>
>>> --
>>> Michael
>>>
>>> On Wednesday, May 31, 2017 at 10:56:57 AM UTC-5, Egon wrote:
>>>>
>>>> Both of my described approaches run the funcs serially however it does
>>>> not wait for the response and later patxhes the results.
>>>>
>>>> Can you describe the whole thing you are building? Piecing the
>>>> requirements and purpose together from comments is difficult.
>>>>
>>>> I.e. how much memory, how big is request latency, who gets the output,
>>>> where do the templates come from...
>>>>
>>>> On May 31, 2017 6:40 PM, "Michael Brown" <[email protected]> wrote:
>>>>
>>>> The best thing I can think of is to modify text/template to add futures
>>>> support, and then do multi-pass rendering. The place to add this looks
>>>> relatively simple, however the implementation looks complicated (and I
>>>> just
>>>> wrote my first program in go a couple weeks ago...)
>>>>
>>>> The problem that I see with the solution below is that text/template
>>>> execution tries to instantiate the values of each function serially and
>>>> essentially waits for completion on each, serially. I've spent the last
>>>> couple hours examining the text/template implementation.
>>>> --
>>>> Michael
>>>>
>>>>
>>>> On Wednesday, May 31, 2017 at 8:59:20 AM UTC-5, Egon wrote:
>>>>>
>>>>> The best idea I can think of (without digging and modifying
>>>>> text/template) is to use special tokens and replace them afterwards...
>>>>> Of course this approach has limitations in what it can do.
>>>>>
>>>>> *// note: code untested and incomplete*
>>>>>
>>>>> type Token string
>>>>> type Queries struct {
>>>>> pending sync.WaitGroup
>>>>> mu sync.Mutex
>>>>> responses map[Token]string
>>>>> }
>>>>>
>>>>> func (q *Queries) createToken() Token {
>>>>> return unique token
>>>>> }
>>>>>
>>>>> func (q *Queries) Do(fn func() string) Token {
>>>>> token := q.createToken()
>>>>> q.pending.Add(1)
>>>>> go func(){
>>>>> defer q.pending.Done()
>>>>> result := fn()
>>>>> q.mu.Lock()
>>>>> q.responses[token] = result
>>>>> q.mu.Unlock()
>>>>> }()
>>>>> return token
>>>>> }
>>>>>
>>>>> func (q *Queries) Wait(){ q.pending.Wait() }
>>>>> func (q *Queries) Patch(data []byte) []byte {
>>>>> // replace tokens with responess
>>>>> }
>>>>>
>>>>> func main() {
>>>>>
>>>>> q := NewQueries()
>>>>>
>>>>> var funcMap = template.FuncMap {
>>>>>
>>>>> "sleep": func() Token { return q.Do(func() string {
>>>>> time.Sleep(1 * time.Second); return "slept" }) },
>>>>>
>>>>> }
>>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}}
>>>>> {{sleep}} {{sleep}}")
>>>>> var buf bytes.Buffer
>>>>>
>>>>> tmpl.Execute(&buf, nil)
>>>>>
>>>>> q.Wait()
>>>>>
>>>>> os.Stdout.Write(q.Patch(buf.Bytes))
>>>>>
>>>>> }
>>>>>
>>>>> The other approach would be to do multiple passes:
>>>>>
>>>>> 1. execute template
>>>>> 2. collect funcs that haven't been run yet
>>>>> 2.1. no funcs left --> output
>>>>> 3. execute these funcs, cache the func values
>>>>> 4. goto step 1 using the cache
>>>>>
>>>>> On Wednesday, 31 May 2017 16:26:15 UTC+3, Michael Brown wrote:
>>>>>>
>>>>>> I am designing a system that will heavily use text/template
>>>>>> processing and I've run into one issue that is going to be a show
>>>>>> stopper
>>>>>> for me if I can't figure out a way around it.
>>>>>>
>>>>>> Execute() on a template will run all of the functions in the template
>>>>>> serially.
>>>>>>
>>>>>> For example, when you run the code below, you can see it output
>>>>>> "slept" once every second until it completes after 3 seconds. In this
>>>>>> example, the sleep is simulating an RPC call to another process that may
>>>>>> take some considerable time (few tenths of a second), but there will be
>>>>>> a
>>>>>> large number of these calls that could all theoretically run in parallel
>>>>>> (ie. there are no data dependencies between them). I'd really like to
>>>>>> know
>>>>>> a way that I could have the templating engine run all of the functions
>>>>>> at
>>>>>> once and collect the output, ie. in the example below, the entire
>>>>>> program
>>>>>> should run in 1 second.
>>>>>>
>>>>>> package main
>>>>>>
>>>>>>
>>>>>> import (
>>>>>>
>>>>>> "text/template"
>>>>>>
>>>>>> "os"
>>>>>>
>>>>>> "time"
>>>>>>
>>>>>> )
>>>>>>
>>>>>>
>>>>>> var funcMap = template.FuncMap {
>>>>>>
>>>>>> "sleep": func() string { time.Sleep(1 * time.Second); return
>>>>>> "slept" },
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> func main() {
>>>>>>
>>>>>> tmpl, _ := template.New("test").Funcs(funcMap).Parse("{{sleep}}
>>>>>> {{sleep}} {{sleep}}")
>>>>>>
>>>>>> tmpl.Execute(os.Stdout, nil)
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> --
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "golang-nuts" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/golang-nuts/j0WwjQE11rw/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> [email protected].
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/d/optout.
