----- Ursprüngliche Mail ----- > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" > <rav...@mistralsolutions.com>, "Shekhar Suman Singh" > <shekha...@mistralsolutions.com> > Gesendet: Freitag, 3. Mai 2019 10:37:20 > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS
> Hi Christian, > > Sorry for the formatting issue, from now onward i will follow like you > suggested. > >>Hello Amarnath, >> >>just a note in front: Could you try to keep the '>' signs (some mail > programs show them as a solid line) >>at the front of the lines intact and don't use ones on your newly written > lines? Otherwise it's a little >>hard to see the answers. There's normally no strong formatting > requirements on this list (also we theoretically have a policy: >>https://devel.rtems.org/wiki/RTEMSMailingLists#Policies) but please try to > keep the mails readable. >> >>I tried to fix the indentation signs in my answer below. I hope that I > caught all your remarks. > > Thanks. > >>----- Ursprüngliche Mail ----- >>> Von: "Amarnath MB" <amarnath...@mistralsolutions.com> >>> An: "Christian Mauderer" <christian.maude...@embedded-brains.de> >>> CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < > rav...@mistralsolutions.com>, "Shekhar Suman Singh" >>> <shekha...@mistralsolutions.com> >>> Gesendet: Freitag, 3. Mai 2019 09:35:27 >>Then it would be best to find the reason for the exception and avoid the > interrupt lock if possible. > > We are discussing this issue with our FPGA team and hope to resolve it soon. > >>> >>>> > Flash controller is implemented using the AHB to external SRAM > interface >>>> in >>>> > the Cortex-M System Design Kit and there are no pins shared between > RAM >>>> and >>>> > NOR Flash. We can't doubt that flash is blocking bus accesses for the >>>> RAM, >>>> > because the test executes fine in the uboot running from RAM. >>>> >>>> So it's a custom FPGA based design? Most likely your RAM and Flash >>>> controller are both connected to the same system bus (AHB in that > case). I >>>> don't know the modules you mentioned but you might want to take a look > at >>>> the documentation whether they lock the AHB or not. >>>> >>> Yes its a custom FPGA based design. Sure, I will go through the >>> documentation. >> >>If the flash controller really blocks all other bus accesses (which would > be an odd design), >>you might can move the RAM and Flash to an independent bus or maybe add > some kind of bus bridge >>to separate the Flash. But I'm not that deep into the ARM blocks. > > Our FPGA team has to see whether it is feasible. > >>> >>>> U-Boot most likely doesn't use interrupts. I assume you are waiting for > a >>>> flash operation to be finished in some busy wait loop there (polling a > flag >>>> or similar). That loop can be fully put into the processor cache (if you >>>> have one which is quite likely). So in your U-Boot test case, the > external >>>> RAM most likely isn't accessed during the flash operations. If you have > a >>>> problem >>>> with conflicting bus accesses you maybe only see them there if you > disable >>>> all caches. >>> >>> You are correct, we are waiting in a loop polling the busy status of NOR >>> flash. Currently, the caches are enabled in both uboot and RTEMS. >>> >> >>Maybe you can try to disable them in U-Boot and see whether your behavior > changes? >>It could be a bug in disguise if the U-Boot code only works due to caches. > These are >>quite nasty because they sometimes depend on the exact position of the > code. In most >>cases it works because the loop is in the cache but if for some reason the > processor >>has to load a new cache line for the loop it doesn't work. Really hard to > find because >>you can change something somewhere completely different and that moves > your code and >>triggers a bug. > > Ya sure, I will test the same test code on uboot with caches disabled. > >>>> In RTEMS on the other hand you are running with enabled interrupts and >>>> task switches. So RAM access are more or less guaranteed during a longer >>>> flash operation if you don't lock the interrupts. >>> >>> Okay, got it. >>> >>>> > >>>> > You had mentioned there are a lot of possible reasons for a >>>> > RTEMS_FATAL_SOURCE_EXCEPTION, is it somewhere documented what all > reasons >>>> > can cause this exception. Any references would be very much helpful. >>>> >>>> Basically for an ARM system, you have that source: >>>> >>>> cpukit/score/cpu/arm/arm-exception-default.c:24: rtems_fatal( >>>> RTEMS_FATAL_SOURCE_EXCEPTION, (rtems_fatal_code) frame ); >>>> >>>> That is default handler for all exceptions where no extra handler is >>>> installed. So basically every exception that is listed in the ARM manual >>>> can be the source. >>>> >>> Thank you that was helpful. >>> >> >>If you have a debugger connected, you can set a breakpoint to that handler > and >>maybe to _Terminate. Sometimes you can get a source together with the > reference >>manual of your processor. > > Sure, we will look in to it. > >>Maybe take a look at your linker command file or use objdump to analyze > your elf file. Something seems odd >>there. > Below is content of linkcmd > MEMORY { > RAM : ORIGIN = 0x40008000, LENGTH = 64M - 512k - 32k > RAM_MMU : ORIGIN = 0x40008000 + 64M - 512k - 32k, LENGTH = 32k > } > REGION_ALIAS ("REGION_START", RAM); > REGION_ALIAS ("REGION_VECTOR", RAM); > REGION_ALIAS ("REGION_TEXT", RAM); > REGION_ALIAS ("REGION_TEXT_LOAD", RAM); > REGION_ALIAS ("REGION_RODATA", RAM); > REGION_ALIAS ("REGION_RODATA_LOAD", RAM); > REGION_ALIAS ("REGION_DATA", RAM); > REGION_ALIAS ("REGION_DATA_LOAD", RAM); > REGION_ALIAS ("REGION_FAST_TEXT", RAM); > REGION_ALIAS ("REGION_FAST_TEXT_LOAD", RAM); > REGION_ALIAS ("REGION_FAST_DATA", RAM); > REGION_ALIAS ("REGION_FAST_DATA_LOAD", RAM); > REGION_ALIAS ("REGION_BSS", RAM); > REGION_ALIAS ("REGION_WORK", RAM); > REGION_ALIAS ("REGION_STACK", RAM); > REGION_ALIAS ("REGION_NOCACHE", RAM); > REGION_ALIAS ("REGION_NOCACHE_LOAD", RAM); > > bsp_stack_irq_size = DEFINED (bsp_stack_irq_size) ? bsp_stack_irq_size : > 4096; > bsp_stack_abt_size = DEFINED (bsp_stack_abt_size) ? bsp_stack_abt_size : > 1024; > > bsp_section_rwbarrier_align = DEFINED (bsp_section_rwbarrier_align) ? > bsp_section_rwbarrier_align : 1M; > Looks OK to me. It's still odd why the PC was on some flash address. I don't think that I can help you a lot more at the current point. So I'll just let you investigate on that topic some more. > *Thank you & Regards,* > *Amarnath MB* > > On Fri, May 3, 2019 at 1:37 PM Christian Mauderer < > christian.maude...@embedded-brains.de> wrote: > >> Hello Amarnath, >> >> just a note in front: Could you try to keep the '>' signs (some mail >> programs show them as a solid line) at the front of the lines intact and >> don't use ones on your newly written lines? Otherwise it's a little hard to >> see the answers. There's normally no strong formatting requirements on this >> list (also we theoretically have a policy: >> https://devel.rtems.org/wiki/RTEMSMailingLists#Policies) but please try >> to keep the mails readable. >> >> I tried to fix the indentation signs in my answer below. I hope that I >> caught all your remarks. >> >> ----- Ursprüngliche Mail ----- >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> >> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> >> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < >> rav...@mistralsolutions.com>, "Shekhar Suman Singh" >> > <shekha...@mistralsolutions.com> >> > Gesendet: Freitag, 3. Mai 2019 09:35:27 >> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS >> >> > On Thu, May 2, 2019 at 3:35 PM Christian Mauderer < >> > christian.maude...@embedded-brains.de> wrote: >> > >> >> ----- Ursprüngliche Mail ----- >> >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> >> >> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> >> >> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < >> >> rav...@mistralsolutions.com>, "Shekhar Suman Singh" >> >> > <shekha...@mistralsolutions.com> >> >> > Gesendet: Donnerstag, 2. Mai 2019 11:18:48 >> >> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS >> >> >> >> > Hi Christian, >> >> > >> >> > As per your suggestion, i tried disabling global interrupt before my >> >> erase >> >> > call and enabling it afterwards. With this setup the test routine is >> >> > passing without giving exception. So i think, interrupt was the issue >> >> here. >> >> > >> >> >> >> Hello Amarnath, >> >> >> >> good. Than you have at least a hint toward the problem. I wouldn't see >> it >> >> as the solution. Like already discussed, depending on your application, >> >> this can lead to missed interrupts. If you have a very specific case for >> >> writing the flash that isn't during your normal operation (for example >> >> firmware updates) you are most likely fine with that solution. But if >> you >> >> write for example log files, you might get a quite unpredictable >> behavior >> >> later. >> >> Hi Christian, >> >> >> > Thank you for the detailed explanation. Even though we are using NOR >> > flash for firmware storage, there is a chance that it may be used for log >> > files later. >> > >> >> Then it would be best to find the reason for the exception and avoid the >> interrupt lock if possible. >> >> > >> >> > Flash controller is implemented using the AHB to external SRAM >> interface >> >> in >> >> > the Cortex-M System Design Kit and there are no pins shared between >> RAM >> >> and >> >> > NOR Flash. We can't doubt that flash is blocking bus accesses for the >> >> RAM, >> >> > because the test executes fine in the uboot running from RAM. >> >> >> >> So it's a custom FPGA based design? Most likely your RAM and Flash >> >> controller are both connected to the same system bus (AHB in that >> case). I >> >> don't know the modules you mentioned but you might want to take a look >> at >> >> the documentation whether they lock the AHB or not. >> >> >> > Yes its a custom FPGA based design. Sure, I will go through the >> > documentation. >> >> If the flash controller really blocks all other bus accesses (which would >> be an odd design), you might can move the RAM and Flash to an independent >> bus or maybe add some kind of bus bridge to separate the Flash. But I'm not >> that deep into the ARM blocks. >> >> > >> >> U-Boot most likely doesn't use interrupts. I assume you are waiting for >> a >> >> flash operation to be finished in some busy wait loop there (polling a >> flag >> >> or similar). That loop can be fully put into the processor cache (if you >> >> have one which is quite likely). So in your U-Boot test case, the >> external >> >> RAM most likely isn't accessed during the flash operations. If you have >> a >> >> problem >> >> with conflicting bus accesses you maybe only see them there if you >> disable >> >> all caches. >> > >> > You are correct, we are waiting in a loop polling the busy status of NOR >> > flash. Currently, the caches are enabled in both uboot and RTEMS. >> > >> >> Maybe you can try to disable them in U-Boot and see whether your behavior >> changes? It could be a bug in disguise if the U-Boot code only works due to >> caches. These are quite nasty because they sometimes depend on the exact >> position of the code. In most cases it works because the loop is in the >> cache but if for some reason the processor has to load a new cache line for >> the loop it doesn't work. Really hard to find because you can change >> something somewhere completely different and that moves your code and >> triggers a bug. >> >> >> In RTEMS on the other hand you are running with enabled interrupts and >> >> task switches. So RAM access are more or less guaranteed during a longer >> >> flash operation if you don't lock the interrupts. >> > >> > Okay, got it. >> > >> >> > >> >> > You had mentioned there are a lot of possible reasons for a >> >> > RTEMS_FATAL_SOURCE_EXCEPTION, is it somewhere documented what all >> reasons >> >> > can cause this exception. Any references would be very much helpful. >> >> >> >> Basically for an ARM system, you have that source: >> >> >> >> cpukit/score/cpu/arm/arm-exception-default.c:24: rtems_fatal( >> >> RTEMS_FATAL_SOURCE_EXCEPTION, (rtems_fatal_code) frame ); >> >> >> >> That is default handler for all exceptions where no extra handler is >> >> installed. So basically every exception that is listed in the ARM manual >> >> can be the source. >> >> >> > Thank you that was helpful. >> > >> >> If you have a debugger connected, you can set a breakpoint to that handler >> and maybe to _Terminate. Sometimes you can get a source together with the >> reference manual of your processor. >> >> > >> >> > >> >> > Thank you once again for your prompt response, it really saved lot of >> our >> >> > time. >> >> > >> >> > Below is the exception frame we got, >> >> > FYI, 0x0 - 0x007FFFFF is NOR Flash and 0x40000000 to 0x7FFFFFFF the >> RAM. >> >> >> >> Yust a note: Putting memory at 0 might hides some NULL-pointer accesses. >> >> If you can do that in your design you might want to think about putting >> >> some boot ROM there that isn't accessible by the application (for >> example >> >> locked via MPU). >> >> >> > I got it. We have kept uboot at 0x0 and RTEMS at 0x40000 offset of flash, >> > we have restricted access from application to these uboot and RTEMS >> > sectors. >> > >> >> Of course there can be worse types of memory Flash (which should be >> mostly >> >> a read only memory). I had a controller with some RAM there once. >> Finding >> >> NULL-pointer accesses on such a system is really nasty. >> >> >> >> > >> >> > *** FATAL *** >> >> > fatal source: 9 (RTEMS_FATAL_SOURCE_EXCEPTION) >> >> > >> >> > R0 = 0x00000040 R8 = 0x00100000 >> >> > R1 = 0x00080000 R9 = 0x00000001 >> >> > R2 = 0x00000048 R10 = 0x4010a3e8 >> >> > R3 = 0x00000048 R11 = 0x4011737c >> >> > R4 = 0x00000a00 R12 = 0x00000000 >> >> > R5 = 0x00000500 SP = 0x40100c40 >> >> > R6 = 0x00000055 LR = 0x4000a36c >> >> > R7 = 0x000000aa PC = 0x003fde80 >> >> > CPSR = 0x200000d2 VEC = 0x00000001 >> >> > RTEMS version: 5.0.0. >> >> > RTEMS tools: 7.4.0 20181206 (RTEMS 5, RSB >> >> > 40ae056f12e1cbe530f76a3ebd1e2ac745a888ef, Newlib >> >> > dc6e94551f09d3a983afd571478d63a09d6f66fa) >> >> > executing thread ID: 0x08a010002 >> >> > executing thread name: Alpi >> >> >> >> Your program counter (PC) points to a Flash address here. Are you sure >> >> that your application runs entirely from RAM? >> >> >> > We are sure that RTEMS is executing entirely from RAM, we are debugging >> on >> > this issue. >> >> Maybe take a look at your linker command file or use objdump to analyze >> your elf file. Something seems odd there. >> >> Best regards >> >> Christian >> >> >> >> >> Best regards >> >> >> >> Christian >> >> >> >> > >> >> > *Thank you & Regards,* >> >> > *Amarnath MB* >> >> > >> >> > >> >> > >> >> > On Thu, May 2, 2019 at 12:49 AM Christian Mauderer < >> >> > christian.maude...@embedded-brains.de> wrote: >> >> > >> >> >> >> >> >> ----- Ursprüngliche Mail ----- >> >> >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> >> >> >> > An: "Christian Mauderer" <christian.maude...@embedded-brains.de> >> >> >> > CC: "RTEMS Users" <users@rtems.org>, "Ravi G Patil" < >> >> >> rav...@mistralsolutions.com>, "Shekhar Suman Singh" >> >> >> > <shekha...@mistralsolutions.com> >> >> >> > Gesendet: Mittwoch, 1. Mai 2019 17:07:13 >> >> >> > Betreff: Re: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS >> >> >> >> >> >> > Hi Christian, >> >> >> > >> >> >> > Thanks for your quick response. >> >> >> > Our boot process is like, first u-boot will be loaded from the NOR >> >> flash >> >> >> > and then the uboot copies RTEMS application from NOR flash to the >> RAM. >> >> >> > After that RTEMS executes entirely from RAM. >> >> >> > >> >> >> > How can i issue a global interrupt disable? Is it using * >> >> >> > rtems_interrupt_disable(0)*? >> >> >> > One more doubt, if i issue a global interrupt disable then will >> there >> >> be >> >> >> > chance that i can miss few clock ticks? >> >> >> > >> >> >> > *Thank you & Regards,* >> >> >> > *Amarnath MB* >> >> >> > >> >> >> >> >> >> Hello Amarnath, >> >> >> >> >> >> with a global interrupt lock, all kinds of events can be missed >> >> including >> >> >> clock ticks. Most should be processed just a little late but some >> >> >> interfaces might loose packets or data (depending on the data rate / >> >> flash >> >> >> access times). So if not necessary it's not a good solution. >> >> >> >> >> >> The rtems_interrupt_disable() function should be called with a level >> >> >> argument. See the example at >> >> >> >> >> >> https://docs.rtems.org/branches/master/c-user/interrupt_manager.html#rtems-interrupt-disable >> >> . >> >> >> Note that for SMP configurations, you might have to use other >> functions. >> >> >> >> >> >> But if your application runs entirely from RAM than the bus access >> >> >> shouldn't be the problem. So the interrupt might isn't the right >> guess. >> >> >> >> >> >> Do you have any more information on the exception and where it >> happens? >> >> >> Some output or a stack trace from a debugger? >> >> >> >> >> >> What kind of flash controller is used? Can it block bus accesses for >> the >> >> >> RAM? Does your RAM share pins with the flash? >> >> >> >> >> >> With kind regards >> >> >> >> >> >> Christian >> >> >> >> >> >> > >> >> >> > >> >> >> > On Wed, May 1, 2019 at 8:16 PM Christian Mauderer < >> >> >> > christian.maude...@embedded-brains.de> wrote: >> >> >> > >> >> >> >> ----- Ursprüngliche Mail ----- >> >> >> >> > Von: "Amarnath MB" <amarnath...@mistralsolutions.com> >> >> >> >> > An: "RTEMS Users" <users@rtems.org> >> >> >> >> > CC: "Ravi G Patil" <rav...@mistralsolutions.com>, "Shekhar >> Suman >> >> >> Singh" >> >> >> >> <shekha...@mistralsolutions.com> >> >> >> >> > Gesendet: Mittwoch, 1. Mai 2019 16:28:23 >> >> >> >> > Betreff: RTEMS_FATAL_SOURCE_EXCEPTION in RTEMS >> >> >> >> >> >> >> >> > Hi All, >> >> >> >> > >> >> >> >> > I'm developing RTEMS 5.00 BSP and device drivers for a custom >> >> >> ARM926EJ-S >> >> >> >> > core. I was successful in porting and building BSP for the ARM >> >> core. >> >> >> >> > >> >> >> >> > We are facing a strange issue with the generic NOR flash driver >> we >> >> >> have >> >> >> >> > developed. Our device drivers are designed such that it can be >> used >> >> >> with >> >> >> >> > RTEMS as well as the bare metal programs. >> >> >> >> > Our driver is working fine in bare metal program (erase, read, >> >> write >> >> >> >> > everything), but when we use the driver with RTEMS application >> and >> >> >> issue >> >> >> >> an >> >> >> >> > erase call, then the application gives >> >> RTEMS_FATAL_SOURCE_EXCEPTION. >> >> >> >> > >> >> >> >> > For testing driver in RTEMS, we have added each test routine as >> a >> >> >> custom >> >> >> >> > shell command using rtems_shell_add_cmd(). >> >> >> >> > >> >> >> >> > For your info we also tested the same driver with u-boot and it >> >> works >> >> >> >> > fines. >> >> >> >> > Can anyone guide me on this issue? >> >> >> >> > >> >> >> >> > *Thank you & Regards,* >> >> >> >> > *Amarnath MB* >> >> >> >> > >> >> >> >> >> >> >> >> Hello Amarnath, >> >> >> >> >> >> >> >> it's a little hard with the given information to tell you a >> concrete >> >> >> >> problem. There are a lot of possible reasons for a >> >> >> >> RTEMS_FATAL_SOURCE_EXCEPTION. >> >> >> >> >> >> >> >> From your description, my guess would be some problem with an >> >> interrupt. >> >> >> >> Most U-Boot code avoids interrupts. I don't know about your bare >> >> metal >> >> >> >> application but maybe in some test application, you don't have too >> >> much >> >> >> >> interrupts too. In RTEMS you have most likely at least a periodic >> >> tick >> >> >> >> interrupt. >> >> >> >> >> >> >> >> Do you execute your code from the same flash or do you keep some >> >> data in >> >> >> >> the flash? In that case it could be an access error during a flash >> >> >> erase or >> >> >> >> write (for example caused by an interrupt). In that case you might >> >> have >> >> >> to >> >> >> >> do a global interrupt disable before you enter certain routines. >> >> >> >> >> >> >> >> Best regards >> >> >> >> >> >> >> >> Christian >> >> >> >> >> >> -- >> -------------------------------------------- >> embedded brains GmbH >> Christian Mauderer >> Dornierstr. 4 >> D-82178 Puchheim >> Germany >> email: christian.maude...@embedded-brains.de >> Phone: +49-89-18 94 741 - 18 >> Fax: +49-89-18 94 741 - 08 >> PGP: Public key available on request. >> >> Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG. -- -------------------------------------------- embedded brains GmbH Christian Mauderer Dornierstr. 4 D-82178 Puchheim Germany email: christian.maude...@embedded-brains.de Phone: +49-89-18 94 741 - 18 Fax: +49-89-18 94 741 - 08 PGP: Public key available on request. Diese Nachricht ist keine geschäftliche Mitteilung im Sinne des EHUG. _______________________________________________ users mailing list users@rtems.org http://lists.rtems.org/mailman/listinfo/users
