Preloading Dialogs on Application Start

Question

Hello,

My question is fairly specific to our use case so I will try to explain it in detail but please feel free to ask any follow-up questions.

At present, most of our application runs through a single dialog (will be referred to as the Main Dialog) which presents and dismisses sub dialogs as needed. To improve run-time performance, I had created two variables in this Main Dialog to hold two of our larger Sub-dialogs. The current implementation of our GUI will populate these variables as part of the boot sequence, which does create a noticeable increase in the boot time, but calls these dialogs essentially instantly when the user requests them during runtime. I had assumed this to be clever way to circumvent long loading times for larger dialogs at runtime, however we have now run into an issue. Our GUI is running on an iMX RT1062 Cortex-M7 in a bare-metal configuration, and after long running sessions the system will seem to fault. This causes what is effectively a crash, the display goes black and the MCU is unresponsive. Removing these Pre-loaded dialogs/variables seems to cause the issue go away. Even more bizarre is that this crash can happen after 6 hours of runtime, or even after 3-4 days. We recently had a system crash after 10 minutes. This crash occurs even if the system is turned on and not interacted with. We have booted and system and left it untouched for 3 days and seen a crash. These pre-loaded dialogs are never presented or dismissed. Is it possible there is a memory issue? Is this type of implementation known to cause a problem? We have handlers for the various IMXRT faults as well as a watchdog peripheral all of which will software reset when an interrupt is called, however a software reset does not occur and no debug information is sent out of our console. Based on our research, this would imply some kind of CPU fault, which occurs before these interrupts can be serviced. Is there anyway the GUI code could cause such a fault?

 

I would appreciate any support on this, I know this is a very specific use case.

Thank you,

Liam

Answer 1

Hi Liam,

thanks for the comprehensive description of the issue. Nevertheless, let me ask some additional things:

1.) Which version of Embedded Wizard Studio are you using?

2.) Which color format are you using?

3.) Which compiler and optimization level are you using?

4.) Can you please post the entire messages on the console from startup until the hang-up occurs?

Thanks and best regards,
Manfred.

Comments

Hi Manfred, 

 

Thank you for getting back to me. Currently we are transitioning between EW 9.30 and EW 12.03, but the build we are referring to is on 9.30. 

The color format is RGB565. The compiler is arm-none-eabi-gcc. The optimization level is Optimize for Size (-Os). 

I am attaching a link to a text file containing our debug output as it was too long to post here:

https://ask.embedded-wizard.de/?qa=blob&qa_blobid=14378338284807403117

Thank you again Manfred,

Liam

---

Hi Liam,

thanks for the additional infos. The startup message of the Embedded Wizard GUI application seems to be incorrect (Memory Pool, Framebuffer Adress,...) - there must be something wrong.

You can send me the file ewconfig.h instead - so that I can have a look on the system configuration.

Thanks and regards,

Manfred.

---

Hi Manfred, 

Apologies, I had modified the EwPrint function to work with our Console Buffer and output file, but it had a known bug in this version. Below is the correct EwPrintSystemInfo() output from the build:

 

[EW EVENT] ---------------------------------------------
[EW EVENT] Target system                                CIRAS4-HMI_PC143 at 528MHz
[EW EVENT] Color format                                 RGB565
[EW EVENT] MemoryPool address                           0x80977000
[EW EVENT] MemoryPool size                              21532671 bytes
[EW EVENT] Framebuffer address                          0x80800000
[EW EVENT] Doublebuffer address                         0x808BB800
[EW EVENT] Framebuffer size                             800 x 480
[EW EVENT] EwScreeenSize                                800 x 480
[EW EVENT] Graphics accelerator                         PXP
[EW EVENT] Vector graphics support                      enabled
[EW EVENT] Warp function support                        enabled
[EW EVENT] Index8 bitmap resource format                enabled
[EW EVENT] RGB565 bitmap resource format                enabled
[EW EVENT] Bidirectional text support                   enabled
[EW EVENT] Operating system                             none
[EW EVENT] External flash device                        Hyper Flash
[EW EVENT] Toolchain                                    GCC
[EW EVENT] C-Compiler version                           10.3.1
[EW EVENT] Build date and time                          May  3 2023, 07:55:11
[EW EVENT] Runtime Environment (RTE) version            9.30
[EW EVENT] Graphics Engine (GFX) version                9.30
[EW EVENT] Max surface cache size                       8388608 bytes
[EW EVENT] Glyph cache size                             256 x 256
[EW EVENT] Max issue tasks                              1024
[EW EVENT] Surface rotation                             0
[EW EVENT] ---------------------------------------------
===================[SYSTEM BOOT COMPLETED]===================

Here is the ewconfig.h file:

https://ask.embedded-wizard.de/?qa=blob&qa_blobid=3125010506917380727

 

Thank you,

Liam

---

Hi Liam,

first of all, I assume that the issue is not related to the dialogs that you have pre-loaded and kept into the memory. As long as there is enough memory in the system, this is a possible and legal approach.

I think that - due to different memory layout - a sporadic hangup occurs that is always in your system, but it only appears sometimes in this constellation.

Please check the following:

1.) Stack size - ensure that the stack size (defined in your linker script) is large enough (4...8 kByte).

2.) MPU Settings - please make sure that the MPU settings are matching exactly to the memory layout of your target system. Make sure that there are no speculative prefetches into address areas without physical memory.

3.) With version 9.30.02 a workaround for targets with write-back data cache configuration was added: The SEMC queue allows for reordering of SDRAM operations. In some conditions, the reodering can cause problems that appear as reading incorrect/stale data from the SDRAM. Changing the BMCRn values to increase the WAGE weighting prevents the reordering of operations and avoids the issue.

Please make sure that your EwBspSystemInit() function contains the following initialization:

  /* Workaround for SEMC queue:
     The SEMC queue allows for reordering of SDRAM operations. In some conditions,
     the reodering can cause problems that appear as reading incorrect/stale data
     from the SDRAM. Changing the BMCRn values to increase the WAGE weighting
     prevents the reordering of operations and avoids the issue. */
  {
    uint32_t* SEMC_BMCR0 = (uint32_t*)0x402F0008;
    *SEMC_BMCR0 = 0x80;
    uint32_t* SEMC_BMCR1 = (uint32_t*)0x402F000C;
    *SEMC_BMCR1 = 0x80;
  }

Let me know if one of these suggestions helps to solve the issue...

Best regards,
Manfred.