Hi Christian, Thank you for all your support. I will debug on this issue and will keep you posted.
*Thank you & Regards,* *Amarnath MB* On Fri, May 3, 2019 at 2:34 PM Christian Mauderer < christian.maude...@embedded-brains.de> wrote: > ----- Ursprüngliche Mail ----- > > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> > > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < > rav...@mistralsolutions.com>, "Shekhar Suman Singh" > > <shekha...@mistralsolutions.com> > > Gesendet: Freitag, 3. Mai 2019 10:37:20 > > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS > > > Hi Christian, > > > > Sorry for the formatting issue, from now onward i will follow like you > > suggested. > > > >>Hello Amarnath, > >> > >>just a note in front: Could you try to keep the '>' signs (some mail > > programs show them as a solid line) > >>at the front of the lines intact and don't use ones on your newly written > > lines? Otherwise it's a little > >>hard to see the answers. There's normally no strong formatting > > requirements on this list (also we theoretically have a policy: > >>https://devel.rtems.org/wiki/RTEMSMailingLists#Policies) but please try > to > > keep the mails readable. > >> > >>I tried to fix the indentation signs in my answer below. I hope that I > > caught all your remarks. > > > > Thanks. > > > >>----- Ursprüngliche Mail ----- > >>> Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > >>> An: "Christian Mauderer" <christian.maude...@embedded-brains.de> > >>> CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < > > rav...@mistralsolutions.com>, "Shekhar Suman Singh" > >>> <shekha...@mistralsolutions.com> > >>> Gesendet: Freitag, 3. Mai 2019 09:35:27 > >>Then it would be best to find the reason for the exception and avoid the > > interrupt lock if possible. > > > > We are discussing this issue with our FPGA team and hope to resolve it > soon. > > > >>> > >>>> > Flash controller is implemented using the AHB to external SRAM > > interface > >>>> in > >>>> > the Cortex-M System Design Kit and there are no pins shared between > > RAM > >>>> and > >>>> > NOR Flash. We can't doubt that flash is blocking bus accesses for > the > >>>> RAM, > >>>> > because the test executes fine in the uboot running from RAM. > >>>> > >>>> So it's a custom FPGA based design? Most likely your RAM and Flash > >>>> controller are both connected to the same system bus (AHB in that > > case). I > >>>> don't know the modules you mentioned but you might want to take a look > > at > >>>> the documentation whether they lock the AHB or not. > >>>> > >>> Yes its a custom FPGA based design. Sure, I will go through the > >>> documentation. > >> > >>If the flash controller really blocks all other bus accesses (which would > > be an odd design), > >>you might can move the RAM and Flash to an independent bus or maybe add > > some kind of bus bridge > >>to separate the Flash. But I'm not that deep into the ARM blocks. > > > > Our FPGA team has to see whether it is feasible. > > > >>> > >>>> U-Boot most likely doesn't use interrupts. I assume you are waiting > for > > a > >>>> flash operation to be finished in some busy wait loop there (polling a > > flag > >>>> or similar). That loop can be fully put into the processor cache (if > you > >>>> have one which is quite likely). So in your U-Boot test case, the > > external > >>>> RAM most likely isn't accessed during the flash operations. If you > have > > a > >>>> problem > >>>> with conflicting bus accesses you maybe only see them there if you > > disable > >>>> all caches. > >>> > >>> You are correct, we are waiting in a loop polling the busy status of > NOR > >>> flash. Currently, the caches are enabled in both uboot and RTEMS. > >>> > >> > >>Maybe you can try to disable them in U-Boot and see whether your behavior > > changes? > >>It could be a bug in disguise if the U-Boot code only works due to > caches. > > These are > >>quite nasty because they sometimes depend on the exact position of the > > code. In most > >>cases it works because the loop is in the cache but if for some reason > the > > processor > >>has to load a new cache line for the loop it doesn't work. Really hard to > > find because > >>you can change something somewhere completely different and that moves > > your code and > >>triggers a bug. > > > > Ya sure, I will test the same test code on uboot with caches disabled. > > > >>>> In RTEMS on the other hand you are running with enabled interrupts and > >>>> task switches. So RAM access are more or less guaranteed during a > longer > >>>> flash operation if you don't lock the interrupts. > >>> > >>> Okay, got it. > >>> > >>>> > > >>>> > You had mentioned there are a lot of possible reasons for a > >>>> > RTEMS_FATAL_SOURCE_EXCEPTION, is it somewhere documented what all > > reasons > >>>> > can cause this exception. Any references would be very much helpful. > >>>> > >>>> Basically for an ARM system, you have that source: > >>>> > >>>> cpukit/score/cpu/arm/arm-exception-default.c:24: rtems_fatal( > >>>> RTEMS_FATAL_SOURCE_EXCEPTION, (rtems_fatal_code) frame ); > >>>> > >>>> That is default handler for all exceptions where no extra handler is > >>>> installed. So basically every exception that is listed in the ARM > manual > >>>> can be the source. > >>>> > >>> Thank you that was helpful. > >>> > >> > >>If you have a debugger connected, you can set a breakpoint to that > handler > > and > >>maybe to _Terminate. Sometimes you can get a source together with the > > reference > >>manual of your processor. > > > > Sure, we will look in to it. > > > >>Maybe take a look at your linker command file or use objdump to analyze > > your elf file. Something seems odd > >>there. > > Below is content of linkcmd > > MEMORY { > > RAM : ORIGIN = 0x40008000, LENGTH = 64M - 512k - 32k > > RAM_MMU : ORIGIN = 0x40008000 + 64M - 512k - 32k, LENGTH = 32k > > } > > REGION_ALIAS ("REGION_START", RAM); > > REGION_ALIAS ("REGION_VECTOR", RAM); > > REGION_ALIAS ("REGION_TEXT", RAM); > > REGION_ALIAS ("REGION_TEXT_LOAD", RAM); > > REGION_ALIAS ("REGION_RODATA", RAM); > > REGION_ALIAS ("REGION_RODATA_LOAD", RAM); > > REGION_ALIAS ("REGION_DATA", RAM); > > REGION_ALIAS ("REGION_DATA_LOAD", RAM); > > REGION_ALIAS ("REGION_FAST_TEXT", RAM); > > REGION_ALIAS ("REGION_FAST_TEXT_LOAD", RAM); > > REGION_ALIAS ("REGION_FAST_DATA", RAM); > > REGION_ALIAS ("REGION_FAST_DATA_LOAD", RAM); > > REGION_ALIAS ("REGION_BSS", RAM); > > REGION_ALIAS ("REGION_WORK", RAM); > > REGION_ALIAS ("REGION_STACK", RAM); > > REGION_ALIAS ("REGION_NOCACHE", RAM); > > REGION_ALIAS ("REGION_NOCACHE_LOAD", RAM); > > > > bsp_stack_irq_size = DEFINED (bsp_stack_irq_size) ? bsp_stack_irq_size : > > 4096; > > bsp_stack_abt_size = DEFINED (bsp_stack_abt_size) ? bsp_stack_abt_size : > > 1024; > > > > bsp_section_rwbarrier_align = DEFINED (bsp_section_rwbarrier_align) ? > > bsp_section_rwbarrier_align : 1M; > > > > Looks OK to me. It's still odd why the PC was on some flash address. I > don't think that I can help you a lot more at the current point. So I'll > just let you investigate on that topic some more. > > > *Thank you & Regards,* > > *Amarnath MB* > > > > On Fri, May 3, 2019 at 1:37 PM Christian Mauderer < > > christian.maude...@embedded-brains.de> wrote: > > > >> Hello Amarnath, > >> > >> just a note in front: Could you try to keep the '>' signs (some mail > >> programs show them as a solid line) at the front of the lines intact and > >> don't use ones on your newly written lines? Otherwise it's a little > hard to > >> see the answers. There's normally no strong formatting requirements on > this > >> list (also we theoretically have a policy: > >> https://devel.rtems.org/wiki/RTEMSMailingLists#Policies) but please try > >> to keep the mails readable. > >> > >> I tried to fix the indentation signs in my answer below. I hope that I > >> caught all your remarks. > >> > >> ----- Ursprüngliche Mail ----- > >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > >> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> > >> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < > >> rav...@mistralsolutions.com>, "Shekhar Suman Singh" > >> > <shekha...@mistralsolutions.com> > >> > Gesendet: Freitag, 3. Mai 2019 09:35:27 > >> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS > >> > >> > On Thu, May 2, 2019 at 3:35 PM Christian Mauderer < > >> > christian.maude...@embedded-brains.de> wrote: > >> > > >> >> ----- Ursprüngliche Mail ----- > >> >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > >> >> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> > >> >> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < > >> >> rav...@mistralsolutions.com>, "Shekhar Suman Singh" > >> >> > <shekha...@mistralsolutions.com> > >> >> > Gesendet: Donnerstag, 2. Mai 2019 11:18:48 > >> >> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS > >> >> > >> >> > Hi Christian, > >> >> > > >> >> > As per your suggestion, i tried disabling global interrupt before > my > >> >> erase > >> >> > call and enabling it afterwards. With this setup the test routine > is > >> >> > passing without giving exception. So i think, interrupt was the > issue > >> >> here. > >> >> > > >> >> > >> >> Hello Amarnath, > >> >> > >> >> good. Than you have at least a hint toward the problem. I wouldn't > see > >> it > >> >> as the solution. Like already discussed, depending on your > application, > >> >> this can lead to missed interrupts. If you have a very specific case > for > >> >> writing the flash that isn't during your normal operation (for > example > >> >> firmware updates) you are most likely fine with that solution. But if > >> you > >> >> write for example log files, you might get a quite unpredictable > >> behavior > >> >> later. > >> >> Hi Christian, > >> >> > >> > Thank you for the detailed explanation. Even though we are using NOR > >> > flash for firmware storage, there is a chance that it may be used for > log > >> > files later. > >> > > >> > >> Then it would be best to find the reason for the exception and avoid the > >> interrupt lock if possible. > >> > >> > > >> >> > Flash controller is implemented using the AHB to external SRAM > >> interface > >> >> in > >> >> > the Cortex-M System Design Kit and there are no pins shared between > >> RAM > >> >> and > >> >> > NOR Flash. We can't doubt that flash is blocking bus accesses for > the > >> >> RAM, > >> >> > because the test executes fine in the uboot running from RAM. > >> >> > >> >> So it's a custom FPGA based design? Most likely your RAM and Flash > >> >> controller are both connected to the same system bus (AHB in that > >> case). I > >> >> don't know the modules you mentioned but you might want to take a > look > >> at > >> >> the documentation whether they lock the AHB or not. > >> >> > >> > Yes its a custom FPGA based design. Sure, I will go through the > >> > documentation. > >> > >> If the flash controller really blocks all other bus accesses (which > would > >> be an odd design), you might can move the RAM and Flash to an > independent > >> bus or maybe add some kind of bus bridge to separate the Flash. But I'm > not > >> that deep into the ARM blocks. > >> > >> > > >> >> U-Boot most likely doesn't use interrupts. I assume you are waiting > for > >> a > >> >> flash operation to be finished in some busy wait loop there (polling > a > >> flag > >> >> or similar). That loop can be fully put into the processor cache (if > you > >> >> have one which is quite likely). So in your U-Boot test case, the > >> external > >> >> RAM most likely isn't accessed during the flash operations. If you > have > >> a > >> >> problem > >> >> with conflicting bus accesses you maybe only see them there if you > >> disable > >> >> all caches. > >> > > >> > You are correct, we are waiting in a loop polling the busy status of > NOR > >> > flash. Currently, the caches are enabled in both uboot and RTEMS. > >> > > >> > >> Maybe you can try to disable them in U-Boot and see whether your > behavior > >> changes? It could be a bug in disguise if the U-Boot code only works > due to > >> caches. These are quite nasty because they sometimes depend on the exact > >> position of the code. In most cases it works because the loop is in the > >> cache but if for some reason the processor has to load a new cache line > for > >> the loop it doesn't work. Really hard to find because you can change > >> something somewhere completely different and that moves your code and > >> triggers a bug. > >> > >> >> In RTEMS on the other hand you are running with enabled interrupts > and > >> >> task switches. So RAM access are more or less guaranteed during a > longer > >> >> flash operation if you don't lock the interrupts. > >> > > >> > Okay, got it. > >> > > >> >> > > >> >> > You had mentioned there are a lot of possible reasons for a > >> >> > RTEMS_FATAL_SOURCE_EXCEPTION, is it somewhere documented what all > >> reasons > >> >> > can cause this exception. Any references would be very much > helpful. > >> >> > >> >> Basically for an ARM system, you have that source: > >> >> > >> >> cpukit/score/cpu/arm/arm-exception-default.c:24: rtems_fatal( > >> >> RTEMS_FATAL_SOURCE_EXCEPTION, (rtems_fatal_code) frame ); > >> >> > >> >> That is default handler for all exceptions where no extra handler is > >> >> installed. So basically every exception that is listed in the ARM > manual > >> >> can be the source. > >> >> > >> > Thank you that was helpful. > >> > > >> > >> If you have a debugger connected, you can set a breakpoint to that > handler > >> and maybe to _Terminate. Sometimes you can get a source together with > the > >> reference manual of your processor. > >> > >> > > >> >> > > >> >> > Thank you once again for your prompt response, it really saved lot > of > >> our > >> >> > time. > >> >> > > >> >> > Below is the exception frame we got, > >> >> > FYI, 0x0 - 0x007FFFFF is NOR Flash and 0x40000000 to 0x7FFFFFFF the > >> RAM. > >> >> > >> >> Yust a note: Putting memory at 0 might hides some NULL-pointer > accesses. > >> >> If you can do that in your design you might want to think about > putting > >> >> some boot ROM there that isn't accessible by the application (for > >> example > >> >> locked via MPU). > >> >> > >> > I got it. We have kept uboot at 0x0 and RTEMS at 0x40000 offset of > flash, > >> > we have restricted access from application to these uboot and RTEMS > >> > sectors. > >> > > >> >> Of course there can be worse types of memory Flash (which should be > >> mostly > >> >> a read only memory). I had a controller with some RAM there once. > >> Finding > >> >> NULL-pointer accesses on such a system is really nasty. > >> >> > >> >> > > >> >> > *** FATAL *** > >> >> > fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION) > >> >> > > >> >> > R0 = 0x00000040 R8 = 0x00100000 > >> >> > R1 = 0x00080000 R9 = 0x00000001 > >> >> > R2 = 0x00000048 R10 = 0x4010a3e8 > >> >> > R3 = 0x00000048 R11 = 0x4011737c > >> >> > R4 = 0x00000a00 R12 = 0x00000000 > >> >> > R5 = 0x00000500 SP = 0x40100c40 > >> >> > R6 = 0x00000055 LR = 0x4000a36c > >> >> > R7 = 0x000000aa PC = 0x003fde80 > >> >> > CPSR = 0x200000d2 VEC = 0x00000001 > >> >> > RTEMS version: 5.0.0. > >> >> > RTEMS tools: 7.4.0 20181206 (RTEMS 5, RSB > >> >> > 40ae056f12e1cbe530f76a3ebd1e2ac745a888ef, Newlib > >> >> > dc6e94551f09d3a983afd571478d63a09d6f66fa) > >> >> > executing thread ID: 0x08a010002 > >> >> > executing thread name: Alpi > >> >> > >> >> Your program counter (PC) points to a Flash address here. Are you > sure > >> >> that your application runs entirely from RAM? > >> >> > >> > We are sure that RTEMS is executing entirely from RAM, we are > debugging > >> on > >> > this issue. > >> > >> Maybe take a look at your linker command file or use objdump to analyze > >> your elf file. Something seems odd there. > >> > >> Best regards > >> > >> Christian > >> > >> >> > >> >> Best regards > >> >> > >> >> Christian > >> >> > >> >> > > >> >> > *Thank you & Regards,* > >> >> > *Amarnath MB* > >> >> > > >> >> > > >> >> > > >> >> > On Thu, May 2, 2019 at 12:49 AM Christian Mauderer < > >> >> > christian.maude...@embedded-brains.de> wrote: > >> >> > > >> >> >> > >> >> >> ----- Ursprüngliche Mail ----- > >> >> >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > >> >> >> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de > > > >> >> >> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < > >> >> >> rav...@mistralsolutions.com>, "Shekhar Suman Singh" > >> >> >> > <shekha...@mistralsolutions.com> > >> >> >> > Gesendet: Mittwoch, 1. Mai 2019 17:07:13 > >> >> >> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS > >> >> >> > >> >> >> > Hi Christian, > >> >> >> > > >> >> >> > Thanks for your quick response. > >> >> >> > Our boot process is like, first u-boot will be loaded from the > NOR > >> >> flash > >> >> >> > and then the uboot copies RTEMS application from NOR flash to > the > >> RAM. > >> >> >> > After that RTEMS executes entirely from RAM. > >> >> >> > > >> >> >> > How can i issue a global interrupt disable? Is it using * > >> >> >> > rtems_interrupt_disable(0)*? > >> >> >> > One more doubt, if i issue a global interrupt disable then will > >> there > >> >> be > >> >> >> > chance that i can miss few clock ticks? > >> >> >> > > >> >> >> > *Thank you & Regards,* > >> >> >> > *Amarnath MB* > >> >> >> > > >> >> >> > >> >> >> Hello Amarnath, > >> >> >> > >> >> >> with a global interrupt lock, all kinds of events can be missed > >> >> including > >> >> >> clock ticks. Most should be processed just a little late but some > >> >> >> interfaces might loose packets or data (depending on the data > rate / > >> >> flash > >> >> >> access times). So if not necessary it's not a good solution. > >> >> >> > >> >> >> The rtems_interrupt_disable() function should be called with a > level > >> >> >> argument. See the example at > >> >> >> > >> >> > >> > https://docs.rtems.org/branches/master/c-user/interrupt_manager.html#rtems-interrupt-disable > >> >> . > >> >> >> Note that for SMP configurations, you might have to use other > >> functions. > >> >> >> > >> >> >> But if your application runs entirely from RAM than the bus access > >> >> >> shouldn't be the problem. So the interrupt might isn't the right > >> guess. > >> >> >> > >> >> >> Do you have any more information on the exception and where it > >> happens? > >> >> >> Some output or a stack trace from a debugger? > >> >> >> > >> >> >> What kind of flash controller is used? Can it block bus accesses > for > >> the > >> >> >> RAM? Does your RAM share pins with the flash? > >> >> >> > >> >> >> With kind regards > >> >> >> > >> >> >> Christian > >> >> >> > >> >> >> > > >> >> >> > > >> >> >> > On Wed, May 1, 2019 at 8:16 PM Christian Mauderer < > >> >> >> > christian.maude...@embedded-brains.de> wrote: > >> >> >> > > >> >> >> >> ----- Ursprüngliche Mail ----- > >> >> >> >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > >> >> >> >> > An: "RTEMS Users" <users@rtems.org> > >> >> >> >> > CC: "Ravi G Patil" <rav...@mistralsolutions.com>, "Shekhar > >> Suman > >> >> >> Singh" > >> >> >> >> <shekha...@mistralsolutions.com> > >> >> >> >> > Gesendet: Mittwoch, 1. Mai 2019 16:28:23 > >> >> >> >> > Betreff: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS > >> >> >> >> > >> >> >> >> > Hi All, > >> >> >> >> > > >> >> >> >> > I'm developing RTEMS 5.00 BSP and device drivers for a custom > >> >> >> ARM926EJ-S > >> >> >> >> > core. I was successful in porting and building BSP for the > ARM > >> >> core. > >> >> >> >> > > >> >> >> >> > We are facing a strange issue with the generic NOR flash > driver > >> we > >> >> >> have > >> >> >> >> > developed. Our device drivers are designed such that it can > be > >> used > >> >> >> with > >> >> >> >> > RTEMS as well as the bare metal programs. > >> >> >> >> > Our driver is working fine in bare metal program (erase, > read, > >> >> write > >> >> >> >> > everything), but when we use the driver with RTEMS > application > >> and > >> >> >> issue > >> >> >> >> an > >> >> >> >> > erase call, then the application gives > >> >> RTEMS_FATAL_SOURCE_EXCEPTION. > >> >> >> >> > > >> >> >> >> > For testing driver in RTEMS, we have added each test routine > as > >> a > >> >> >> custom > >> >> >> >> > shell command using rtems_shell_add_cmd(). > >> >> >> >> > > >> >> >> >> > For your info we also tested the same driver with u-boot and > it > >> >> works > >> >> >> >> > fines. > >> >> >> >> > Can anyone guide me on this issue? > >> >> >> >> > > >> >> >> >> > *Thank you & Regards,* > >> >> >> >> > *Amarnath MB* > >> >> >> >> > > >> >> >> >> > >> >> >> >> Hello Amarnath, > >> >> >> >> > >> >> >> >> it's a little hard with the given information to tell you a > >> concrete > >> >> >> >> problem. There are a lot of possible reasons for a > >> >> >> >> RTEMS_FATAL_SOURCE_EXCEPTION. > >> >> >> >> > >> >> >> >> From your description, my guess would be some problem with an > >> >> interrupt. > >> >> >> >> Most U-Boot code avoids interrupts. I don't know about your > bare > >> >> metal > >> >> >> >> application but maybe in some test application, you don't have > too > >> >> much > >> >> >> >> interrupts too. In RTEMS you have most likely at least a > periodic > >> >> tick > >> >> >> >> interrupt. > >> >> >> >> > >> >> >> >> Do you execute your code from the same flash or do you keep > some > >> >> data in > >> >> >> >> the flash? In that case it could be an access error during a > flash > >> >> >> erase or > >> >> >> >> write (for example caused by an interrupt). In that case you > might > >> >> have > >> >> >> to > >> >> >> >> do a global interrupt disable before you enter certain > routines. > >> >> >> >> > >> >> >> >> Best regards > >> >> >> >> > >> >> >> >> Christian > >> >> >> > >> >> > >> -- > >> -------------------------------------------- > >> embedded brains GmbH > >> Christian Mauderer > >> Dornierstr. 4 > >> D-82178 Puchheim > >> Germany > >> email: christian.maude...@embedded-brains.de > >> Phone: +49-89-18 94 741 - 18 > >> Fax: +49-89-18 94 741 - 08 > >> PGP: Public key available on request. > >> > >> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG. > > -- > -------------------------------------------- > embedded brains GmbH > Christian Mauderer > Dornierstr. 4 > D-82178 Puchheim > Germany > email: christian.maude...@embedded-brains.de > Phone: +49-89-18 94 741 - 18 > Fax: +49-89-18 94 741 - 08 > PGP: Public key available on request. > > Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG. >
_______________________________________________ users mailing list users@rtems.org http://lists.rtems.org/mailman/listinfo/users
