CCS/PROCESSOR-SDK-AM437X: New version of PRU-ICSS-HSR-PRP-DAN issues

Mare,

1) You can re-build PDK with 'gmake BUILD_PROFILE=DEBUG' to generate "debug" PDK libraries.

2) It's interesting that the priority of the same task changes as well. I don't think there should be so much priority inversion happening here. With the ROV snapshot, do you see any problem when running the application on IDK?

Regards,

Garrett

2.)

Could be packet rec. issue also...look error counter in the statistic.
The application doesn't fail with hard fault.

I mean... I don't understand what I'm doing wrong if demo app work normally to other users...does it?

P.S.
What I also found is that if PTP master is on the network, communication is not stable as it should be. Sometimes ping is not successful. This was seen also in previous releases of PRU_ICSS_HSR_PRP_DAN (but with the custom board and we are suspicious that DP83822 has "bad" moments on our board). About PRU-ICSS-HSR-PRP-DAN_01.00.02.00 speaking: Sometimes if I want reproduce issue...it help that I make a flood of UDP packets, and when I normalize network traffic, pru_icss can't start work properly. On RED-RX packets are refused because of wrong (too big) length. 


Regards, Mare

Hi Garrett!

Is also any good explanation or implementation-specific behaviour for the phenomenon that can PTP sync from port1 (it can calculate peer delay), but port2 is barely able to see master after some time (~20s)... but can't sync (can't calc peer delay). This was run from IDK AM437 and SD img.

Best regards, Mare

1.)

gmake[4]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware/examples/qspi/read_write'
gmake[3]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware'
C:/ti/xdctools_3_55_02_22_core/gmake  qspi_app_read_write PROFILE=release  -s KW_BUILD=no
gmake[3]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware'
gmake[4]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware/examples/qspi/read_write'
# Making am43xx-evm:a9host:debug:example_utils...
# Making am43xx-evm:a9host:debug:qspi_lib...
# Making am43xx-evm:a9host:debug:board...
# Making am43xx-evm:a9host:debug:utils...
# Making am43xx-evm:a9host:debug:soc...
# Making am43xx-evm:a9host:debug:dal...
# Making am43xx-evm:a9host:debug:device...
gmake[4]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware/examples/qspi/read_write'
gmake[3]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware'
gmake[2]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/starterware'
C:/ti/xdctools_3_55_02_22_core/gmake -f C:/ti/pdk_am437x_1_0_17/packages/ti/build/makefile_non-buildinfra pdk_examples_core
gmake[2]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/build'
C:/ti/xdctools_3_55_02_22_core/gmake -C C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag all ALL_BOARDS="evmAM437x idkAM437x skAM437x"
gmake[3]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag'
C:/ti/xdctools_3_55_02_22_core/gmake buildtarget bboard=evmAM437x btests=evmAM437x_DIAG
gmake[4]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag'
C:/ti/xdctools_3_55_02_22_core/gmake -C buzzer BOARD=evmAM437x TARGET=armv7 BOOTMODE=
gmake[5]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag/buzzer'
C:/ti/xdctools_3_55_02_22_core/gmake -f ./build/evmAM437x/armv7/makefile
gmake[6]: Entering directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag/buzzer'
linking C:/ti/pdk_am437x_1_0_17/packages/ti/board/obj/evmAM437x/armv7/buzzer_test.o C:/ti/pdk_am437x_1_0_17/packages/ti/board/obj/evmAM437x/armv7/GPIO_Buzzer_config.o C:/ti/pdk_am437x_1_0_17/packages/ti/board/obj/evmAM437x/armv7/diag_entry.ao C:/ti/pdk_am437x_1_0_17/packages/ti/board/obj/evmAM437x/armv7/gic.o C:/ti/pdk_am437x_1_0_17/packages/ti/board/obj/evmAM437x/armv7/cpu.o C:/ti/pdk_am437x_1_0_17/packages/ti/board/obj/evmAM437x/armv7/diag_common_cfg.o into C:/ti/pdk_am437x_1_0_17/packages/ti/board/bin/evmAM437x/armv7/buzzer_diagExample_evmAM437x_armv7.out ...
arm-none-eabi-gcc.exe: error: C:/ti/pdk_am437x_1_0_17/packages/ti/board/lib/evmAM437x/a9/release/ti.board.aa9fg: No such file or directory
arm-none-eabi-gcc.exe: error: C:/ti/pdk_am437x_1_0_17/packages/ti/osal/lib/nonos/am437x/a9/release/ti.osal.aa9fg: No such file or directory
arm-none-eabi-gcc.exe: error: C:/ti/pdk_am437x_1_0_17/packages/ti/drv/uart/lib/am437x/a9/release/ti.drv.uart.aa9fg: No such file or directory
arm-none-eabi-gcc.exe: error: C:/ti/pdk_am437x_1_0_17/packages/ti/drv/gpio/lib/am437x/a9/release/ti.drv.gpio.aa9fg: No such file or directory
arm-none-eabi-gcc.exe: error: C:/ti/pdk_am437x_1_0_17/packages/ti/drv/i2c/lib/am437x/a9/release/ti.drv.i2c.aa9fg: No such file or directory
arm-none-eabi-gcc.exe: error: C:/ti/pdk_am437x_1_0_17/packages/ti/csl/lib/am437x/a9/release/ti.csl.aa9fg: No such file or directory
gmake[6]: *** [build/evmAM437x/armv7/makefile:125: C:/ti/pdk_am437x_1_0_17/packages/ti/board/bin/evmAM437x/armv7/buzzer_diagExample_evmAM437x_armv7.out] Error 1
gmake[6]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag/buzzer'
gmake[5]: *** [makefile:103: build_example] Error 2
gmake[5]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag/buzzer'
gmake[4]: *** [makefile:139: buzzer_build] Error 2
gmake[4]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag'
gmake[3]: *** [makefile:120: evmAM437x] Error 2
gmake[3]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/board/diag'
gmake[2]: *** [C:/ti/pdk_am437x_1_0_17/packages/ti/build/makefile_non-buildinfra:174: board-diag] Error 2
gmake[2]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/build'
gmake[1]: *** [makefile_non-buildinfra:126: pdk_examples] Error 2
gmake[1]: Leaving directory 'C:/ti/pdk_am437x_1_0_17/packages/ti/build'
gmake: *** [makefile:97: non-buildinfra_all] Error 2

This it try of PDK re-build with DEBUG profile
Starterware make file was also corrected: 

ifeq ($(OS),Windows_NT)
    # Object clean on Windows
    #BUG:CLEAN_RECURSIVE_OBJ=cmd /C del /s ti\\starterware\\*.o ti\\starterware\\*.a > NUL
	CLEAN_RECURSIVE_OBJ=cmd /C del /s .\\*.o .\\*.a > NUL
else

If I previously build with the release profile, then debug build is successful. Need to have release pdk lib. present before making debug??
Also if release libraries are present, application build in debug profile take release libraries!?

Strange... where I made a mistake???

Hi,

I'm stuck because it is difficult to upgrade the firmware if the initial image shows strange behaviour.
Are my post so wrong that problems seem totally irrelevant or I'm not on the priority list on your full schedule?
We start upgrading our product on new PDK/PRP cuse previous NDK, PDK and PTP were unstable and on the new version, I'm full of hope, but things starting falling...

Kind Regards, Mare

Hi Mare,

Sorry I was sidetracked, will be trying to replicate the debug build and multiple tasks issue you have seen first...

Regards,

Garrett

Hi,

what I found today that after RedLifeCheckTaskCreate() a ROV view get corruptet. 

    hsrPrphandle->redLifeCheckSemaphore = RedSemaphoreCreate();
    hsrPrphandle->nodesTableSemaphore = RedSemaphoreCreate();

It seems that things go wrong after these two lines. If this is just coincidence, I don't know. I was track PC with HW Trace Analyser if some ISR was called, but code execution seems linear.

P.S. when I made a project I had a problem with projectCreate.bat, cause path for CCS was wrong ( I get CCS930 with TI RTOS), so I manually correct path. I found with a trace that functions from Core-smem.c (C:\ti\ccs930\xdctools_3_60_02_34_core\packages\xdc\runtime\Core-smem.c) are used when objects (semaphore for example) are created.... so I wandering If upgraded CCS version can cause such problems ??

Regards, Mare

Hi Garrett!

It's nice to hear from you those words! What I also found is that project building is likely broken or something similar. Please explain to me purpose of ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\lib\prp\am437x\a9\libprp_lib_AM437x_arm.a ?!? Is this lib made from PRU-ICSS-HSR-PRP-DAN ( include protocol drivers and sync functions)?
What I try was direct linking files from C:\ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\drivers to project, rebuild it a project and after that, RedProtocolStart() did not corrupt memory!
Now memory corruption si made after this function in taskPruss task. It seems that when I rebuild project, files in PRU-ICSS-HSR-PRP-DAN and some unknown (old) build/libs are used. I also saw that PRU-ICSS-HSR-PRP-DAN sources (timesync) are not built as debug (I run debug build in CCS). What is wrong with the project or my approach.

Regards, Mare

Hi Garrett,

When I create project prp_app, this msg. is printed in console:

C:\ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\projects>projectCreate.bat AM437x arm prp_app
**************************************************************************
Environment Configuration Summary:
    CCS_INSTALL_DIR                : C:\ti\ccs930\ccs
    CCS_WORKSPACE_LOC              : C:\ti\workspace
    IA_SDK_HOME                    : C:\ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01
    PDK_INSTALL_PATH               : C:\ti\pdk_am437x_1_0_17\packages
    PDK_ECLIPSE_ID                 : com.ti.pdk.am437x
    PDK_VERSION                    : 1.0.17
    PROJECT_CREATE_DIR             : C:\ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\projects
    PROJECT_CREATE_OPTIONS_FILE_DIR: C:\ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\projects\ccsproject_args
    SYS_BIOS_VERSION               : 6.76.03.01
    NDK_VERSION                    : 3.61.01.01
    XDC_TOOLS_VERSION              : 3.55.02.22_core
    EDMA_VERSION                   : 2.12.05.30
    CGT_VERSION                    : GNU_7.3.1:Linaro
    CCS_DEVICE                     : "Cortex A.AM4379"
    RTSC_TARGET                    : gnu.targets.arm.A9F
    RTSC_PLATFORM                  : ti.platforms.evmAM437X
**************************************************************************

Creating project 'prp_app_AM437x_arm' for 'AM437x' platform in directory 'C:\ti\PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\projects\prp_app_AM437x_arm' by overwriting the project if it exists already...

--------------------------------------------------------------------------------
Creating project 'prp_app_AM437x_arm'...

    NOTE: Compiler version 'GNU_7.3.1:Linaro' is not currently installed! - defaulting to 'GNU_7.2.1:Linaro'.

Done!

Is the problem in any prebuild lib because of compiler version?
Regards, Mare

Mare,

PDK v1.0.17 was tested with GNU 7.2.1, see software-dl.ti.com/.../index_release_specific.html

You can update the 'CGT_VERSION=GNU_7.3.1:Linaro' to 'CGT_VERSION=GNU_7.2.1:Linaro' in the projectCreate.bat script.

I was able to reproduce the BUILD_PROFILE=debug issue, will be submitting a JIRA ticket to get it fixed in future release.

For libprp_lib_AM437x_arm.a, please refer to prp_lib_AM437x_arm.txt at:PRU-ICSS-HSR-PRP-DAN_01.00.05.01\protocols\hsr_prp\projects\ccsproject_args\AM437x, which includes the source files list for the library.

Regards,

Garrett

Hi Garrett!

It is built issue, but don't know why.
Why I think is build problem is that I see address changes of global variables ( for example timeSyncHandle) in Expression view, depend on stack-deep/function call. I'm also getting memory index shift - I see msg. in expression view where the value of the variable is written " {lastSeen_good_drift_index=???".
So something is wrong with the project?

Regards, Mare

Hi,

I made a totally new project, a new workspace... I start from beginning.. No luck! ROV shows a total mess...

I also have problems with memory overwrite... also in program space...but let's just start on the beginning! Can someone confirm that simple/initial project (prp_app) has issues with ROV?

Regards, Mare

Hi Mare,

Can you please try the attached out file I built to see if it has the same RoV issue?

I carried only one AM437x IDK home but found out it’s a broken one that can’t even connected to JTAG...https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/prp_5F00_app_5F00_AM437x_5F00_arm.out

Regards,

Garrett

Hi,

I probably need your am437x_app_pa9fg.rov.xs...?

Regards, Mare

https://e2e.ti.com/cfs-file/__key/communityserver-discussions-components-files/791/AM437x_5F00_release.7z

Mare,

I zipped the entire AM437x_release folder in the attachment.

Regards,

Garrett

Hi Garrett,

let me thank you first for your respond and attachment!
As you can see in the picture, ROV shows the same issue with your build.

Is that ROV or my CCS issue or it is prp_app or something else... I don't know, but the founding solution is our number one priority. We stop OS and PRP/HSR update on our product until basic software is clear of issues.

Hope you get IDK board soon, so we can start investigating...

Kind Regards,

Mare

Hi Garrett,

I can also confirm that the same issue is on other computer (version CCS930), so it seem that is not problem in my CCS instalation...

Regards, Mare

Hi Garett,

We also try your build with old CCS (Version: 7.2.0.00013):

Fxn seems ok on queek look. So is that CCS or XDS problem nor project? Some priorities of tasks are still strange, or not? NDK task on priority 2?

Hi Mare,

I finally got a working IDK and able to see the repeated task issue from ROV with CCS v9.3. Let me check with CCS team to see if there is any known ROV issue with the CCS version.

With the CCSv7.2, does the project work as expected or still behaves as CCSv9.3 except ROV? 

Regards,

Garrett

Hi Garrett,
thanks for the feedback!
I will be very happy if you can give me also feedback on the points below.

1.) CCS7.2 - Other problems such a memory strange behaviour suddenly disappeared (ROV remain)... Not sure how and why... maybe I can't see it at this moment and problems still remain?! I will report about that if I found anything suspicious.

2.) CCS7.2 & CCS930 - What I can confirm is that obviously PTP on port2 doesn't work! Some times also port1 fail to sync (maybe traffic with sampled values help to crash synchronization - 16.8Mbps). Is this issue known? Do you have option to plug your IDK on PTP source? I'm not sure if I will able to, but should I try to debug what is missing for successful sync? (my settings: IEEE802.3/P2P) 

3.) Also when I debug "2.)" issue I pause debug execution for five seconds and after I press run in IDE I was realised that IDK freeze. What I found is that program looping on HWI (void lisrEdma3CCErrHandler0(uint32_t edma3InstanceId)) and it can't recover from Edma error handler. I agree that this isn't something surprising behaviour if I stop CPU with a debugger, but that also happens many times in our product application which is based on older PRU-ISCC-HSR-PRP-DAN release (application is based on prp_app example) without "pausing" code execution. This application has many interrupts and some critical parts where I disable HWI for short time... Are you have any recommendations or guides, what application shouldn't use for PTP driver stability? When I remove lines for PTP handling from "processHighPrioFrames" this issue was solved (but lose PTP functionality), so I'm sure that EDMA problem comes from PTP driver (EDMA set a timer on ICSS?) . Have you ever see this kind of issue? Could "signal traffic" inside CPU or busy CPU cause such problem?
Edit: When I was writing this, Edma loop in error handler happen again... without stopping the code executing.

4.) I look around but didn't found a description of statistic from USART menu:

S T A T I S T I C S

Lookup error A : 00001179
Lookup error B : 03040845

      Tx  WrongLan        Rx    Errors     Nodes     Proxy    Unique   Duplic.     Multi     OwnRx
--------  --------  --------  --------  --------  --------  --------  --------  --------  --------
00002139  00000000  00000000  00007534  00000000  00000000  00000000  00000000  00000000  00000000
00000306  00000093  00000000  05455215                      00000000  00000000  00000000  00000000
03041912  00000000  00000156  00000000                      00000000  00000000  00000000

Show Statistics...completed

What I'm trying to understand from statistic is:
4.a)What are Lookup errors and how different to Errors in table?
4.b)What represents first, the second, third line in the table? 1. LRE-port1? 2. LRE-port2? DAN port?
What kind of errors are this? Wrong size? Packet CRC fail?

Best regards, Mare

Mare,

For ROV issue, I have asked CCS team to provide a know issue list of CCS v9 and v10.

I don't really have a PTP source, will ask our testing team to try reproduce the issue you observed.

For statics (lookup/Rx error), some info in the wiki may help: 

The first two lines corresponds to port 0 and 1. And the third line corresponds to application/host port.

Rx error is logged if a packet shows following properties:

• A packet received with source address same as the DUT MAC address.
• A packet with CRC error.
• An oversized packet.
• A PTP packet that needs to be dropped.

Regards,

Garrett

Hi Garrett!

1.)Anything new from CCS team and testing team for PTP?

2.)Honestly, I can't find a good explanation of what it's supposed to mean "lookup error". 

3.) I also mention that we have in the previous release of ICSS_PRP_HSR_DAN problems with TX and RX. Also, I mention to you that we observed EDMA error handler looping and memory corruption issues when we had PTP frame handling in processHighPrioFrames function. When we remove TimeSync_processPTPFrame() from processHighPrioFrames() things become stable. Because I can't see TI bug tickets, I'm kindly asking you if you can confirm that there are some issues somehow related to my observes?

Regards, Mare

Mare,

I was told there is no ROV in CCSv9.3+ and PTP/EDMA issue in PRP. But our RTOS kernel team is helping to look into more details on the ROV issue.

Regards,

Garrett

Garrett,

Are in the testing team also activities for an explanation why on one port PRP is syncing and on other port can't?

Regards,

Mare

Mare,

Let me follow up again for the 2nd port syncing issue.

I had a debugging session with RTOS kernel team and found that the 'Runtime Object View' tool under 'ROV Classic' in CCS->Tools lists the tasks properly, please give it a try and see if it helps you debug the PTP issue. We will look into the 'ROV Classic' issue sepearately.

Regards,

Garrett

Hi Garrett,

Names look OK, but priority on NDK stack -> 2 seems to be too low? And PTP sync handling on priority 15?

Anything new from the testing group?

Regards, Mare

Mare,

I didn't check NDK stack priority which should align with the settings in .cfg file. Yes, PTP sync handling on Priority 15 per the design:

/**
* @def PTP_SYNC_TASK_PRIORITY
* Priority of task which processes sync frame
*/
#define PTP_SYNC_TASK_PRIORITY 15

I should have update for you from the testing group by end of this week.

Regards,

Garrett

Mare,

We use Hirschmann to test the PTP and the result is shown below. Can you please describe your PTP test setup as well?

•  Two AM335x-ice board + one AM437x board

PLC<=> AM335x-ice <=> AM437x <=> AM335x-ice

Regards,

Garrett

Hi Garrett,

I decided to start debugging. Hope I can found something.

I test AM437-IDK with: Trimble PTP GM200 and Mainberg PTP (can't get in room with device, to check model number).
What I found: Direct PTP master to slave -> If CAT-5 is up to 2m length, FW has a problem on sync.
If I put ethernet switch (P2P configuration) CISCO (need to check model) also never get in lock state. It continuously trying to lock, but never get there. 
If direct (without switch) connection is made, port 1 can sync, port 2 can't. I also try if something is wrong with TRIMBLE PTP, soo I put Mainberg PTP in slave mode and Mainberg PTP get sync ( clock locked ) in mater of seconds.... So what can I say on your post?! I don't know how you do it! I'm running a prebuild example on IDK.
I will start debugging by my own, but also it will be nice if you can do a test with other master sources - sources who don't have TI sync stack.
What is PLC in your schema?

Best regards, Mare

Mare,

We actually tested PTP sync with Oregano master and Hischmann PLC supporting PTP master. As you reported the issue with pre-built example on TI IDK, I have created an internal ticket to track the issue. From the testing log I can see, it seemed we always test PTP with either single AM335x ICEv2, or multiple AM335x ICE v2 with a AM437x IDK. We may not test a single AM437x IDK alone with the PRP application.

Regards,
Garrett

Hi Garrett!

1.) I found that in demo application has  #define TIMESYNC_TEST_PORT_NUM fixed, soo requests for Pdelay are sent just on one port. Any idea how to set the driver to be able to sync on one or another port? Should I modify Pdelay request task to send a request on the port where it sees master?

2.) We finally sync IDK board over switch. The main problem was in switch and setting on it, so sorry for this.

3.) After day or soo, IDK freeze. It seems some issue remain in code. I think it also starts producing sync missed events, but I'm not 100% shure.

4.) I also see that pDelayReq was not sent. I put a breakpoint on TXenque function which send TX data on PRP ports and it returns an error (TX buffer full). It seems that PRU sometimes dies. (what I add to code is USART print) Is possible that code who make CPU busy can result in such problems?

5.) We had IDK locked on PTP (we had state machine in the state "LOCKED" ), then we opened a window in office, which (i think) resolute in the unstable frequency of OSC on IDK. State of PTP slave show creasy numbers, but state machine doesn't want to restart and make a rough adjustment! So PPS and sync values stay far away from synchronized clock forever (because state machine stay in fine-tuning) .

********************************************

Peer Delay on P1 :                      18 ns
Peer Delay on P2 :                      0 ns

--------DUT configured as Slave---------
*********PTP/1588 Params********
Clock Drift :                           99 ns
Curr offset :                           -39773 ns
Min offset :                            -79368 ns
Max offset :                            64235 ns
Num Sync Missed :                       128
UTC Offset (Seconds field) :            1603460949 seconds
Master connected on Port :              1
*******************************

Currently, I can't be in the office for further debugging. If you can confirm or react on uper info let me know.
Thanks for your support Gerrett!

Best Regards, Mare

Hi Mare,

Good to see the progress from your side. Yes, the TIMESYNC_TEST_PORT_NUM is fixed in the build time in the example :-(  If you are able to identify the port that sees master, yes you can modify Pdelay request task to send a request on the port.

Let me sync up with our team with regard to your new findings.

Regards,
Garrett