Ilya, Realized it is a StackOverflow so increasing the memory size may not help. So maybe increasing the stack size may help:
The attribute to specify the JVM options is CONTAINER_JVM_OPTIONS The setting for stack size is -Xss Chandni On Fri, Mar 11, 2016 at 3:46 PM, Chandni Singh <[email protected]> wrote: > Hey Ilya, > > Can you please remove the duplicate output port from your implementation > of NewLineFileInputOperator as well. > > Thanks, > Chandni > > On Fri, Mar 11, 2016 at 3:42 PM, Ashwin Chandra Putta < > [email protected]> wrote: > >> Why do you want to have thread locality and partitioning together, you >> will >> lose parallel processing. What is the use case? >> >> Regards, >> Ashwin. >> >> On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya < >> [email protected]> >> wrote: >> >> > Now with files: >> > https://gist.github.com/ilganeli/7f770374113b40ffa18a >> > >> > From: "Ganelin, Ilya" <[email protected]<mailto: >> > [email protected]>> >> > Reply-To: "[email protected]<mailto: >> > [email protected]>" <[email protected]<mailto: >> > [email protected]>> >> > Date: Friday, March 11, 2016 at 3:02 PM >> > To: "[email protected]<mailto:[email protected] >> >" >> > <[email protected]<mailto:[email protected]>> >> > Subject: Stack overflow errors when launching job >> > >> > Hi guys – I’m running into a very frustrating issue where certain DAG >> > configurations cause the following error log (attached). When this >> happens, >> > my application even fails to launch. This does not seem to be a YARN >> issue >> > since this occurs even with a relatively small number of >> partitions/memory. >> > >> > I’ve attached the input and output operators in question. >> > I can get this to occur predictable by >> > >> > 1. Increasing the partition count on my input operator (reads from >> > HDFS) - values above 20 cause this error >> > 2. Increase the partition count on my output operator (writes to >> HDFS) >> > - values above 20 cause this error >> > 3. Set stream locality from the default to either thread local, node >> > local, or container_local on the output operator >> > >> > This behavior is very frustrating as it’s preventing me from >> partitioning >> > my HDFS I/O appropriately, thus allowing me to scale to higher >> throughputs. >> > >> > Do you have any thoughts on what’s going wrong? I would love your >> feedback. >> > >> > >> > ________________________________ >> > >> > The information contained in this e-mail is confidential and/or >> > proprietary to Capital One and/or its affiliates and may only be used >> > solely in performance of work or services for Capital One. The >> information >> > transmitted herewith is intended only for use by the individual or >> entity >> > to which it is addressed. If the reader of this message is not the >> intended >> > recipient, you are hereby notified that any review, retransmission, >> > dissemination, distribution, copying or other use of, or taking of any >> > action in reliance upon this information is strictly prohibited. If you >> > have received this communication in error, please contact the sender and >> > delete the material from your computer. >> > ________________________________________________________ >> > >> > The information contained in this e-mail is confidential and/or >> > proprietary to Capital One and/or its affiliates and may only be used >> > solely in performance of work or services for Capital One. The >> information >> > transmitted herewith is intended only for use by the individual or >> entity >> > to which it is addressed. If the reader of this message is not the >> intended >> > recipient, you are hereby notified that any review, retransmission, >> > dissemination, distribution, copying or other use of, or taking of any >> > action in reliance upon this information is strictly prohibited. If you >> > have received this communication in error, please contact the sender and >> > delete the material from your computer. >> > >> >> >> >> -- >> >> Regards, >> Ashwin. >> > >
