Compiler/TMDSLCDK6748: RTS library causing issue with EMIF?

Much appreciated!

Hi Yordan,

Just wanted to confirm that we are still seeing this problem. Has the design team been able to take a look? Again, I'm happy to provide additional information and context as needed.

Best regards,

David

Hi Rahul,

Thank you very much for offering your assistance with this matter. I have made marginal progress since my last post, but I will answer your questions before getting into that.

Have you checked the EMIFA configuration registers? Are there any differences?

I have compared the EMIFA configuration registers and ensured that they are identical.

Are you using the same driver as that used by the NAND Writer? Look at the function NAND_open and LOCAL_flashGetDetails for how the NAND writer reads the ONFI string from the ID bytes.

In trying to debug this issue I have familiarized myself with the NAND Writer driver and its functions, including those you mention. I am using almost entirely the same code as the driver, and where it is not exactly the same it is functionally identical.

I did stumble upon a way to "fix" the problem by adjusting the link order of the build, but that fix causes another problem. Let me explain:

When I link "lrts64plus_elf.lib" before the "Generated Linker Command files" produced by (I assume) XDC for SYS/BIOS, then I do get the expected "ONFI" response shifted out of the NAND cache register. Success! ...Or so I thought.

Unfortunately, this change also seems to disable TI-RTOS. Tasks fail to run and the system seems to idle and do nothing -- stuck in the SYS/BIOS function Task_allBlockedFunction(). I have attached a couple screenshots for clarity.

When all the entries are removed from the Link Order list, the behavior is identical to the "OOOO" case (second screenshot).

As I am unfamiliar with the inner workings of XDC and TI_RTOS, I must admit that I have very little clue as to what is happening here. Any thoughts or insight you can offer would be greatly appreciated.

Thank you,

David

Rahul,

I excluded several sections from the platform and used my own linker command to define those sections. That really hasn't been an issue thus far.

I have attached a stripped-down version of the project along with a couple additional resources you might need. In the current configuration, if you place a breakpoint on line 171 of nandwriter.c you will see the correct "ONFI" response in devID16[], but the clock module and semaphore will not execute. Try switching the link order of -lrts64plus_elf.lib and Generated Linker Command files, and you will find that the clock and semaphore works, but the DSP reads in "OOOO" for the ONFI response. (Removing all link order specifications result in the same behavior.)

Please let me know if I can provide any clarification, or if you need any additional include files. I appreciate your help with this.

Many thanks,

David Askew

1050.PDI_Core_NAND_Example.zip1526.PDI_Additional_Resources.zip

Attached is the custom platform I am using.

-David

1588.Platform_evm6748custom.zip

If you provide me with a list of the files and include paths you require then I would be happy to upload them here.

In addition to some other files you might be missing, I have attached the custom file ”tistdtypes2.h" which is essentially the standard "tistdtypes.h" with certain typedefs removed. This was a consequence of importing the NANDWriter drivers into a TI-RTOS project. I removed those specific typedefs because they were causing errors about "invalid redeclaration of type name". For example, the "Int8" typedef is defined as "char" in tistdtypes.h but also defined as "xdc_Int8" in \ti\xdctools_3_30_06_67_core\packages\xdc\std.h

This brings me to the more general point, which is that I suspect this whole problem is a result of importing the NANDWriter drivers into a TI-RTOS project. (The NANDWriter_DSP project was not created within a TI-RTOS project). Importing was not a simple process as it required modifying a few files (such as tistdtypes2.h) in order to avoid compatibility issues resulting in compiler and linker errors.

"Also note RTOS is including the required RTS libraries in the linker command file, so you don`t need to explicitly link to the rts64xplus_elf.lib"

Yes, the auto-generated linker command file includes the library "ti.targets.rts6000.ae674", but not rts64xplus_elf.lib. As mentioned in my last post: When the explicit link to rts64xplus_elf.lib is removed, the response to READ ID commands become "OOOO" instead of "ONFI", and I am thus stuck with the original problem.

To your knowledge, has anyone successfully implemented the NANDWriter drivers in a TI-RTOS project? A basic example project that demonstrates this task may be all I need to solve the issue. Is there some other NAND chip driver I should be using for TI-RTOS projects? Your advice and assistance are very much appreciated, as always.

Thank you,

David Askew

8831.PDI_Additional_Files_tistdypes2.zip

Hi Rahul, have you had a chance to look into this yet?

I tried single-stepping to find out what is causing SYS/BIOS to hang in Task_allBlockedFunction(). The infinite loop occurs at line 255 in Task.c:

Void Task_startCore(UInt coreId)
{
   //<omitted code>
    /* stall until a task is ready */
    while (Task_module->curSet == 0) {
        Task_allBlockedFunction();
    }

 //<omitted code>

}

I should reiterate that this problem occurs ONLY when explicitly linking rts64xplus_elf.lib before "Generated Linker Command files". Yet if I don't explicitly add rts64xplus_elf.lib (and link it first) then the reads from the NAND cache register fail to shift out properly, as I've described.

My question really boils down to this:

If the disassembly is identical in both cases, and the EMIFA control registers are also the exactly same in both cases, then how could the RTS libraries and their link order change the behavior of the NAND chip?

This is the phenomenon that I am seeing. The RTS libraries linked by TI-RTOS are somehow causing dysfunction in the interface between the NAND device and the DSP. Has anyone successfully implemented the NANDWriter drivers in a TI-RTOS project?

Thanks,

David Askew

RandyP,

Thank you so much for your thoughtful advice. One thing I was not clear about in the last post was that I am encountering this problem even with disassembly that is literally identical, from the same project but with a different link order. So perhaps we can knock #2 off the list. (The original post was assembly from two different but similar projects – one of which does define a very large stack size. This was before I realized that link order was the determining factor.)

As for #1, I'm hopeful that this might lead to the source of the issue. Comparing the .map files of both versions, i.e. same project with different link orders, it's clear there are quite a few differences. Although I am still looking into it, I thought it best to provide this status update. I have also attached the two .map files for reference (renamed as .txt files in order to upload them)

I do wonder if the cached/non-cached thing is what is causing issues when writing to and reading from the NAND memory. However I'm not sure what the best way to test for this might be.

Thanks again for your assistance and insight. There are definitely some leads to follow now!

Best regards,
David Askew

NAND_Broken__RTOS_Working.txt   NAND_Working__RTOS_Dead.txt 

RandyP,

I stepped through that loop in both versions and recorded the values of relevant registers. I found that the only real difference was the value being read in from the NAND cache register. I have attached an excel sheet that shows the recorded values.

In the working version you can see where the NAND cache register (0x62000000) shifts out the next 16 bits of data after a read operation. In the non-working version the shift never happens; the memory browser can see 'O' at 0x62000000 and 'N' at 0x62000002, etc., but because the register doesn't shift, the NANDWriter driver just reads a string of 'O's instead of "ONFI".

It's strange because the EMIF configuration registers look the same in both cases. I will have to think more on this tomorrow.

Thanks,
David

PDI_Assembly_Stepthrough_Regs.xlsx

I carefully looked over and compared .map files of both builds and found no significant differences in data and or program memory locations. They merely differ in the number of modules. (Modules in the version with working NAND but broken SYS/BIOS almost look like a subset of the other version).

Other than that, the only notable difference is at the beginning of the .stack section. In the version where NAND data fails to shift, the beginning of .stack is allocated to boot.ae674 : boot.oe674 (.stack), whereas in the version where SYS/BIOS fails to call Tasks, the beginning of .stack is allocated to rts64plus_elf.lib : boot.obj (.stack).

Stepping through the loop where the NAND data is read, I find no differences in assembly or even register values, except for the data read from the NAND chip's cache register by the EMIF (address 0x62000000). My previous post includes a record of this. I have triple-checked that all of the EMIF registers are identical in both cases. It really looks like the SYS/BIOS libaries make the EMIF fail to interface properly, but I'm not sure how this is possible if the EMIF registers are all the same.

On a side note:

Over the past two months I have repeatedly but respectfully asked Rahul Prabhu for something – anything – in the form of guidance or insight on the issue of SYS/BIOS disrupting the EMIF asynchronous controller. Very little has been offered in response. I have also tried reaching out to Rahul in a private message, which remains unanswered as I type this post.

I am not qualified enough (or conceited enough) to claim to have discovered a flaw in SYS/BIOS. There is, very possibly, something that I am missing; Unfortunately, I have gleaned very few reasons to think so ...at least until this past week. I feel that Rahul's determined silence on the matter can only be interpreted in a couple ways: either as a combination of discourtesy and negligence, or as a sign that there really is a problem with SYS/BIOS.

I appreciate your generosity, RandyP, in reaching across to another area and providing assistance, which has been really helpful. If you or Rahul have anything to add in light my recent posts, I will certainly welcome it. Thanks for your time.

Best Regards,
David Askew

RandyP,

I understand that some of this is not your primary domain, and I appreciate the help regardless.

Do any errors show up in NAND EMIF NANDFSR?

No, not in either case. All the EMIF registers look identical in both cases.

What physical addresses are the two .stack sections placed in and what size are they?

In both builds the stack section is located at the beginning of L2 RAM, which is 0x11800000 on the TMS320C6748. It's the same size as the L2 RAM: 256kB in both cases. I would think that would be sufficient for a project that is nearly empty save for SYS/BIOS and some NAND drivers, but I'll try playing with the size a bit and see what happens.

Please send a link to the BIOS disrupting EMIF thread.

I was actually referring to this thread. You will notice that it started two months ago. During that period I discovered it was the RTS libraries causing the NAND problem. (Explicitly linking another library first, rts64plus_elf.lib, actually fixes the NAND issue, but causes problems with SYS/BIOS operations.)

Because the DSP assembly code and register data are essentially identical in both cases, I assume that there must be something wrong with the way SYS/BIOS handles the EMIF when interfaced with NAND memory.

Rahul is one of the best, which also means one of the busiest on our excellent E2E DSP support team. It is good that he is aware of that issue.

Given the company he works for, I have no doubt that Rahul is both competent and knowledgeable. I have no doubt that he's busy too, but it's hard to make the case that this issue has been simply overlooked. Weeks go by between responses, and yet Rahul continues to be active elsewhere on these forums. I would love to be wrong on this, truly. And I do hope he is aware of the SYS/BIOS-EMIF issue, as you say, though it would be nice to have some indication!

Anyway, thanks again for your continued interest. I will investigate the stack size idea.

Best regards,
David Askew

David,

I apologize for the delayed response on this issue. I have been involved in debugging a release blocking issue for Processor SDK RTOS which is why I have not been able to reply to e2e posts that requires greater involvement than pointing the users to the right resource to unblock them in their development.

As I mentioned earlier, I have not been able to reproduce this issue due to several missing files in your earlier zip file that you had shared. Also, I was able to add the NANDWriter code to a Task and run it using TI RTOS and SYSBIOS without any issues but it appears that you are also not having this issue based on your earlier analysis. 

At the moment, the best way to proceed forward would be if you can provide me a self contained project with all the required files and document to help reproduce this issue on the LCDK. From the analysis so far it is unclear if the issue is caused due to TI RTOS configuration/Compiler or some integration issue in your project for me to provide specific guidance. 

Besides the guidance provided by Randy, it may also be useful to look at the map file, DSP cache configuration. TI RTOS enables the cache for this platform and also uses platform setting from BIOS kernel instead of linker command file. Other thing to try is to down grade the compiler to CGT 7.4.16 and see if this makes a difference. As you can see the NANDWriter in the Flash utility was provided with CCS project and older compiler so I am not certain if this is making a difference.

Regards,

Rahul

PS: I have also asked a C6000 Compiler expert to take a look at the link order and provide comments as to why this would make a difference in this case.

Rahul,

Thank you for your reply.

Rahul Prabhu said:

I have not been able to reproduce this issue due to several missing files in your earlier zip file that you had shared. [...]the best way to proceed forward would be if you can provide me a self contained project with all the required files[...]

Regarding "a self contained project": I have not included every single source file, header file, and library that the project needs because such a thing would, of course, be enormous in size – and because I assume you already have standard packages such as the MCSDK, CSL, XDCtools, and TI-RTOS. But again, I believe all the other additional files that are needed are included in the zip files. In the interest of portability, I have tried to use path variables in the project wherever possible. Please let me know if you are missing any other files; I promise I will get them to you promptly.

In the current configuration, you will notice that rts64plus_elf.lib is linked in last, resulting in a "OOOO" response to the READ ID operation. Note that this behavior is the same when rts64plus_elf.lib is removed from the project entirely. If instead you link rts64plus_elf.lib first, a proper "ONFI" response is observed, but SYS/BIOS fails to call any tasks, etc. Downgrading to TI v7.4.16 does not seem to have an effect.

I understand you are busy with important tasks, and I appreciate you taking the time to look into this issue.

Best regards,
David Askew

PS: Not sure if it matters, but I have a very basic AIS boot image flashed to my LCDK (via NANDWriter_DSP) that just blinks an LED. I have included the image, "boot2.ais", in the zip file in case this could be relevant.

PDI_Project.zip

I can shed some light on ...

David_Askew said:

In the current configuration, you will notice that rts64plus_elf.lib is linked in last, resulting in a "OOOO" response to the READ ID operation. Note that this behavior is the same when rts64plus_elf.lib is removed from the project entirely. If instead you link rts64plus_elf.lib first, a proper "ONFI" response is observed, but SYS/BIOS fails to call any tasks, etc.

If you do not tell the linker which RTS library to use, it automatically chooses one for you.

I can think of two possible reasons why you see different behavior based on where the linker sees the RTS library on the command line.

One, the same function is present in multiple libraries, and the wrong one is getting linked in.  To see if this might be occurring, add the option --scan_libraries to the link.  If the same function is present in multiple libraries, you will get a diagnostic that shows you which libraries have the function, and which one gets used.  Read more about --scan_libraries in the C6000 assembly tools manual.

Two, causing the linker to see inputs in a different order causes things to be allocated to system memory in a different order.  And this difference in order either masks or unmasks some other problem.  Let's hope this is not the case, because it is much harder to track down.

Thanks and regards,

-George

George,

Thanks for the advice about --scan_libraries, it's a handy tool. I noticed a few key differences between the two builds.

There are three symbols – "_stack", "_c_int00", and "_auto_init_elf" – that are pulled from different sources depending on link order. In the version where rts64plus_elf.lib is linked last, these symbols are pulled from "boot.ae674", but when it's linked first those symbols are pulled from "rts64plus_elf.lib".

There are also numerous symbols which are pulled from "rts64plus_elf.lib" instead of "rts6740_elf.lib " and vice versa. Not sure if that matters as I imagine those two libraries are mostly identical.

In both cases "_stack" is located at the same address and "TI_STACK_SIZE" and "TI_STACKEND" are the same as well. The entry point symbol _c_int00 has a similar address in both cases: c0055b80 and c0056d20.

That said, it's not clear to me how any of those symbols could affect the way the EMIF and it's data are handled.

As for #2, I've compared the .map files of both builds. The modules common between them are allocated in the same order. Also the version with a working SYS/BIOS seems to contains all the modules and symbols of the other .map plus several additional modules pertaining to SYS/BIOS. This fact might explain why tasks and timers fail to work in the one case, but it's hard to see how it might relate to changes in EMIF behavior. Any thoughts?

Thanks,

David

Rahul,

Have you had a chance to look into this yet? Do you need any additional files?

Rahul Prabhu said:

I was able to add the NANDWriter code to a Task and run it using TI RTOS and SYSBIOS without any issues

Does your NANDwriter code read from and write to the NAND chip without any problems? If it does, could you upload your project here so I can compare with my projects?

Thanks,

David

Randy,

It is SYS/BIOS that includes "rts6740_elf.lib" automatically. I discovered (while trying to debug the NANDwriter w/ TI-RTOS project) that I could get the NAND operation to work properly if I explicitly linked "rts64plus_elf.lib" – the same library used in the original NANDWriter project.

It's probably easy to miss now, as this thread has become lengthy, but I have stated several times that the result is exactly the same whether I link "rts64plus_elf.lib" last, or if I don't link it at all. The following truth table should sum it up.

 I agree with you that it's almost certainly an issue of different libraries causing different results. My question is why does build #3 cause NAND-related operational problems with the EMIF? Also, how would I even go about debugging such a problem if I don't have access to the libraries' source code?

RandyP said:

...the best way to get a solution from Rahul, if possible, will be for you to create a zip'd archive project of the failing file sets (and possibly working file sets, separately) including library files and header files and all of your local source files.

It's unclear, though it sounds like Rahul claims to have created a TI-RTOS project with NANDWriter code that works properly. If so, then a copy of his example project might be all I need to get out of his hair and out of yours! Thanks again for your time and assistance.

Best regards,

David

I managed to isolate the source of the problem, though I'm hesitant to make any claims about why it caused the bug. Perhaps you might be able to shed more light on that.

The problem seemed to stem from where I defined the flash memory regions for CS2 and CS3. Your project defined them in the linker command file, "NANDWriter_DSP.cmd", whereas I specified them in a custom platform which ultimately meant they were defined in the TI-RTOS auto-generated linker file "linker.cmd". Removing those entries from the custom platform and then manually adding the memory definitions to my linker script seems to be the solution.

Perhaps those definitions need to be located in the same linker file that defines the ".aemif_mem" section (rather than in linker.cmd)? Though it's not entirely clear to me why that would be. I'm hoping you might have some thoughts on the matter.

Thanks,
David