DCAN FIFO reading while receiving

Bob,

have you been able to reproduce the out-of-order reception with the updated code on your side?

Regards,
Michael

Hi Bob,

Thank you for your confirmation.
Yes this race condition issue is what we tried to communicate over the past replies.

My observations so far:
a) TI project, extended ID: it is running fine
b) TI project, standard ID: issue reproducable but only with optimization
c) TI project, standard ID, new FIFO entry registers (arb, mctl, data0-1) are memcopied to a buffer and later processed -> this works even with opt=off

Although it seems that optimization solves the issue, actually in our production code it does not.
Even with the same functionality (I copied the "canGetData" line by line) the result is the missplaced message in the buffer,
the difference between the TI project and our production code is that we use DMA in the background so that the bus may be used by other peripheries as well when the CAN registers are access in the atomic iteration.

I even tried out to implement obs c) in our case. To just iterate over the FIFO and copy the registers in an atomic way, and process them after the atomic block, hoping that this fast access will solve our issues (in production code we use optimization level=2 AFAIK). Unfortunately it only increased the robustness of the message processing, but didnt solve it.

From my point of view the optimization just accidentally organized the instruction so that the timings of readouts and arriving new can messages somehow was always timed correctly, but I think this is just a matter of time or setup (temperature, bus length, ecu oscillator, etc varying parameters) that will sooner or later cause the same issue. Note that your setup is 2 identical boards, used on room temperature with probably a short can bus wire.

Tamas

PS: I have no idea why the extended/standard ID influences the outcome of the execution but as written in a) and b) I had no problems when using extended format (although it does not necessarly mean that it is working fine, I just didn't have any problems, maybe you could have a look at this)

Hi Bob,

It's been over a week now, do you have any update on this issue?

Tamas

Bob,

I hope you didn't forget about us.
The topic is getting more and more urgent in our project. We need a workaround for this supposed silicon glitch very soon.

Also a code piece that is guaranteed to work under all circumstances would be great.
Under all circumstances means from my point of view:

• independent of the FIFO's size
• independent of any compiler optimization level or toolchain version
• independent on the length and type of a CAN frame
• independent of the internal SoC bus load (given by any other master on the bus, like DMA)

Thank you very much in advance.

Regards,
Michael

Hi Bob,

On our side it is quite the opposite, whenever we extend the code - of the TI project example - we run into the error.
Using our production code it is inevitable to run into, even with taking most purified example it happens.

We have already considered using the DMA but there is no guarantee that it wont happen again since TI has not yet released the root-cause of this issue and a statement that DMA approach is a known workaround. From our side this would require a complete redesign of our base library so we either need a workaround for polling the FIFO or an errata like release that assures of using DMA.

Please if you that you can not proceed further with the analysis, escalate the CAN bug to higher levels.

Best Regards,
Tamas

Any update on this would be very much appritiated.

Thank you for your answer.

Unfortunately we have a system running using the IRQ method and there we see that it is not the first mb of the FIFO that triggers the interrupt but sometimes the 2nd or the 3rd(!), of course this is at least a deterministic way of handling the incoming messages and keeping them ordered.

We can confirm that the IRQ method is also not a solution but a more or less working workaround.

Still awaiting a solution or errata for the polling method. If errata is going to be released please attach the root cause and a statement that using the proposed workaround (other sequence of reading, IRQ, DMA) is design proven and not experimental proven.

Thank you,
Tamas

Any update?

Again a month has passed since the last update request. Any update, information, is anybody working on a solution or the issue and the processor is not supported anymore?

Any progress here?

Hi,

As mentioned before neither the polling nor the IRQ approach works..

I tried using the DMA which seemed to work fine but we are developing API to our customers and we have a certified code which would need complete redesign to use the DMA instead. Meaning that we need an official statement (with a supporting document perhaps) that the DMA can work as expected and it is proven by tests, and those tests reveals the problem with the polling and interrupt mode. Otherwise I can not just say to our development team to rewrite our codebase based on a suggestion read on a forum chat.

If I remember right you were able to reproduce the issue using the TI Hercules board, isn't it possible to attach to this (or an opened) processor and check if it is working as in your simulation?

Yes, we were able to reproduce on our boards. We actually use the same testcase to run on the simulation and it took days non-stop on simulation and we were unable to reproduce the problem. Bear in mind that even on the board it took several real time seconds after many CAN frames have been received to produce the problem. To simulate real time in seconds in simulation takes a lot of time. 

 How did you use the DMA to read the DCAN FIFO? Do you use the RTI to generate a periodic DMA request for the DMA module to read the DCAN FIFO? 

DMA method setup:

 - using the DCAN module's IF3 as source (this will point to a FIFO element, and will jump automatically to the next non-read)
 - 2 buffers[0..1] allocated, DMA points to one of them meanwhile we read the other one
 - DMA is periodically checked if it transferred a FIFO element
 - if DMA transferred something it gets pointed to the other buffer once when finished the current and gets restarted, we process the other buffer

I used this implementation along with a software delay between checking the DMA status. It seemes to work right (didnt get any 'lost' or missplaced messages).

Any official update regarding the issue? Errata?

Dear TI Support,

We have detected the problem originally and sent you exact description of the problem (see posts above with screenshots and data extract from the Lauterbach debugger you have plenty of data available),
We also recreated the issue using DCAN example from HalCogen generated code, Hercules prototype board and the TI CCS v6.0, we sent you the projects at least via emails (not sure if this pops up here).
You have the example codes and proof that it is not just the code as we communicated it several times - when detected the problem first - we used our own written HAL layer (similar to HALcogen).

Also with the support of your experts we narrowed it down that it is not the configuration neither the timing of the code that causes the problem.
Also on this forum you can find similar reports, I at least found problems regarding other processors - using the same DCAN module - the exact problem was reported, I believe that was also admitted long time ago.

Question is: why do we see a similar statement from TI that you are still unsure if this is a code timing issue or silicon bug? Almost 2 years passed and the only answer we have is: we admit that the problem exists we just don't know what it is because the problem _can_not_be_recreated_with____simulations____.

Hello user4642853,

Apologies for any confusion. For sure, I have just joined the discussion so I am not 100% aware of the effort or verification of the issue that had been done at the time of the last inquiry that I responded to. Also, I am not aware without further detail if each poster is identifying the same issue or has the same code set. In short, my comments were relative to my awareness and response to the poster that made the last post. For the overall issue of the FIFO and ordering of the received messages, I am aware of some other devices that have indicated that the ordering of the information within the FIFO cannot be guaranteed. IN the case of the Hercules implementation, we will be including an errata in our next release of the errata documentation. For the time being, you can reference the below early copy of the specific erratum applying to this issue:

Hopefully, this will satisfy your needs for documentation of the issue and the proposed work around.

Hi,

Sorry for the late reply I just noticed a key word in the "Expected Behavior" description.
Your errata says that the DCAN might missplace message with "same arbitration and mask IDs" (at least the expected is that it is not missplaced when same arbitr and mask is received) but this is just a subset of the actual truth.
We have proven and indicated that the arbitration or mask has nothing to do with the missbehavior of the DCAN.
When used in FIFO the DCAN missplaces the messages regardless of mask,id or data size!

Please confirm that you have analysed the general case (issue is valid for any configuration) and not just the case of "messages with same arbitration and mask ID" case.

Thank you.