TMDXIDK5718: TMDXIDK5718: SOF/SyncManager Jitter - PRU-ICSS-ETHERCAT-SLAVE for AM571x_idk

Hi
Can you try probing the Sync0/1 out signal and PDI IRQ signal (mapped to SOC pin via EDIO) from the board ? 

Regards
Dhaval

Hi Dhaval,

Thank you for the support. I have invited the customer to join the discussion.

Regards,

Marvin

Dhaval Khandla said:

Can you try probing the Sync0/1 out signal and PDI IRQ signal (mapped to SOC pin via EDIO) from the board ? 

For this, you will also have to configure PDI ISR DIGIO pin selection register (0x0E0A). As mentioned in the register description for 0x0E0A in PRU ICSS EtherCAT Slave Controller Register List, "PDI ISR DIGIO pin selection register, selects one of pr1_edio_data_outN pins as PDI ISR hw pin, configure 255 to disable. Set corresponding bitmask to enable. Application needs to configure pinmux correctly for this to work"

Regards
Dhaval

Hi Dhaval,

Thanks for the support.  I'll work on probing those signals.  We did do some debugging since the original query and found that the SOF signal does have about a 20us delay to the actual interrupt occurring in software.  For this we measured via pin D7(vout1_d10) and by setting the pinmux in software initialization.  So the SOF signal is arriving at the time we expect, but the delay until the hardware interrupt is about 20us.

Hi Dhaval,

Just checking up to see if you had any input on the SOF signal measurement I provided last week.  I'm still waiting on some hardware support from our team to get those other measurements.

One thing I did also want to ask is that you noted using the signal pr1_edio_data_outN for measurement.  From my understanding of the TMDXidk5718 board PRU1 is not populated for EtherCAT functionality. So in this case we're using PRU2 for our development purposes.  Just wanted to check if this is correct.

Hi

Ryan Hodgman said:

One thing I did also want to ask is that you noted using the signal pr1_edio_data_outN for measurement.  From my understanding of the TMDXidk5718 board PRU1 is not populated for EtherCAT functionality. So in this case we're using PRU2 for our development purposes.  Just wanted to check if this is correct.

As mentioned in https://software-dl.ti.com/processor-industrial-sw/esd/docs/indsw/EtherCAT_Slave/01_00_10/PRU_ICSS_EtherCAT.html#running-ethercat-slave-application, "To use PRU-ICSS1 on the AM571x IDK, the #define PRUICSS_INSTANCE need to set to PRUICSS_INSTANCE_ONE in the [INSTALL-DIR]/protocols/ethercat_slave/include/tiesc_soc.h file".

Regards
Dhaval

Hi Dhaval,

Yes I have tried that before but it did not work for me.  There was a forum post some time back where another customer had a similar issue.
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1089006/am5718-idkam571x-ethercat-slave-on-pru-icss1

Also checking the SDK source code it's noted hat PRU1 ETH0 is not wired for the default IDK.  The PINMUX options are commented out in the SDK source.

Hi

Ryan Hodgman said:

Yes I have tried that before but it did not work for me.  There was a forum post some time back where another customer had a similar issue.
https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1089006/am5718-idkam571x-ethercat-slave-on-pru-icss1

If you are using PRU-ICSS-EtherCAT_Slave_01.00.10.00, did you try the patch provided in https://e2e.ti.com/support/processors-group/processors/f/processors-forum/1047677/tmdxidk5718-pru-icss-ethercat-slave-for-am571x_idk---lines-crossed-error ?

Regards
Dhaval

Yes we have been using the provided patch for the Lines Crossed Error.

Hi Dhaval,

Still working on getting the Sync0/Sync1/PDI measurements.  Not sure if you have any insight on why PRU1 does not work for me even with the provided patch?

Additionally I wanted to bring up another measurement observation we think is related to the Start of Frame Timing.

First some background information.  We're running EtherCAT using the Subordinated Cycles mode.  Sync0 runs at a rate of 55us, Sync1 runs at 220us.  So we have a ratio of 4 Sync0 events to every one Sync1 event.

When measuring events we noticed a discrepancy during controller startup.

At the very start of the Slave receiving Sync interrupts we see the following. The first full cycle(Red) contains 5 Sync0 pulses and 1 Sync1 Pulse. Subsequent cycles(Yellow) are behaving as we expect with only 4 Sync0 pulses to 1 Sync1 pulse.

I set up a read on some of the EtherCAT registers and saw the following. From left to right I'm reading 0x910 System Time, 0x990 Sync0 Next Time, and 0x998 Sync1 Next Time.

 

In Pink I performed 8 reads of these registers at a 55us period prior to receiving any Sync0 Pulse. Note that the Sync0 next time is set here, but Sync1 next time is not.

 

The Red/Blue/Green below corresponds to the places marked in the waveform above. Blue is our first Sync0 pulse, at this point the Sync1 next time is still not updated.

 

It's not until when our first Sync1 pulse occurs in Green that we see this register updated. We're wondering if you might know where the Sync1 next time(0x998) register is updated from, and why it's not done until this point. I know the PRU EtherCAT firmware from Texas Instruments if provided a black box.  Would it be possible to have access to the source code to understand why this register is not updated during startup?

Hi Dhaval,

Just wanted to check in on my previous inquires.  Thanks!

Hi
We are looking into this. I will provide further response by end of this week.

Regards
Dhaval

Hi

Ryan Hodgman said:

Still working on getting the Sync0/Sync1/PDI measurements.  Not sure if you have any insight on why PRU1 does not work for me even with the provided patch?

I verified this again. By changing the macro in tiesc_soc.h file, I am able to get it working on PRU-ICSS1 instance also.

Please note that J51 on the IDK has to be open for this to work. See following excerpt from IDK User Guide:

"The AM571x IDK EVM supports up to four 100Mb Industrial Ethernet ports attached to the PRU-ICSS
subsystems and up to two Gigabit (1000Mb) Ethernet ports connected to the integrated Ethernet switch.
The final number of available ports depends on the configuration selection:
• 4-port Ethernet mode: provides two 100Mb Industrial Ethernet ports and two Gigabit (1000Mb)
Ethernet ports.
• 6-port Ethernet mode: provides four 100Mb Industrial Ethernet ports and two Gigabit (1000Mb)
Ethernet ports.
Selection between these Ethernet modes is controlled by header J51. Installing a shunt on header J51
that shorts these pins together enables 4-port Ethernet mode. Removing the shunt from these pins
enables 6-port Ethernet mode. The LCD output is only available when 4-port Ethernet mode is selected."

Regards
Dhaval

Hi

Marvn Arcangel said:

We're currently only tracing timings using a GPIO toggle in the interrupt handlers for Sync0/Sync1/SyncManager.  We're currently measuring time from SyncManager to Sync0 as ~36us when the expected value is 55us resulting in about 20us jitter from SOF to SyncManaer. When Acontis has measured this timing on an EL9800 board, they do not see such a delay, although they are directly measuring the SOF signal, and not the SyncManager interrupt.

Can you explain this a bit more? If SOF to Sync0 is expected to be 55 us from the master side, doesn't it point to a problem on master side?

Ryan Hodgman said:

When measuring events we noticed a discrepancy during controller startup.

At the very start of the Slave receiving Sync interrupts we see the following. The first full cycle(Red) contains 5 Sync0 pulses and 1 Sync1 Pulse. Subsequent cycles(Yellow) are behaving as we expect with only 4 Sync0 pulses to 1 Sync1 pulse.

Can you confirm if you are seeing this only once during startup and not seeing this at later point?

Regards
Dhaval

Hi Dhaval,

Thanks for the response we really appreciate the assistance.  I've outlined some answers below.

1. PRUICSS Instance One

Okay I've removed the J51 header and changed the DEFAULT_PRUICCS_INSTANCE to PRUICSS_INSTANCE_ONE.  The Master reports Bus scan error 'Bus configuration mismatch (0x9811001e)', 0 slaves found.  Is it possible I would also have to modify the SDK to enable PRU1 eth0 Pinmux as mentioned in my previous post?



2. SOF Jitter
We're measuring the SOF signal directly from the board on pin D7 vout1_d10.  The SOF signal arrives at the time as expected from the master, 55us ahead of the next Sync0.  However, we don't see a Hardware Interrupt for the PDI_ISR/SyncManager until 20us after the SOF signal is seen.  So despite SOF arriving at the time expected, we don't receive the interrupt in software until 20us later.  Measurement for this on pin D7 is below.



3.  Sync0 Timing at Startup
Yes we only see this happening in the first cycle, after which it is corrected.  Notably this is an issue because of the LatchInputSync0Counter variable.  This variable is incremented at each Sync0 interrupt, and used to indicate when PDO Input Mapping should occur.  The Sync1 interrupt resets this counter, so we only see the issue at startup because of the unexpected extra Sync0 interrupt, thus causing Input Mapping to occur early.  It does however correct itself for all subsequent cycles. 

Hi

Ryan Hodgman said:

1. PRUICSS Instance One

Okay I've removed the J51 header and changed the DEFAULT_PRUICCS_INSTANCE to PRUICSS_INSTANCE_ONE.  The Master reports Bus scan error 'Bus configuration mismatch (0x9811001e)', 0 slaves found.  Is it possible I would also have to modify the SDK to enable PRU1 eth0 Pinmux as mentioned in my previous post?

I did not modify anything in PDK for this. Which PDK version are you using?

Regards
Dhaval

processor_sdk_rtos_am57xx_06_03_02_08
pdk_am57xx_1_0_18

Dhaval,

Also one thought to the startup issue with Sync0/Sync1 timing.  While I've only observed this issue at startup, I can't say for sure it's not occurring at later points during runtime as our logs include thousands of cycles.  We're just hoping to get an exact answer as to what in the PRU firmware is setting these registers.

Hi

Ryan Hodgman said:

It's not until when our first Sync1 pulse occurs in Green that we see this register updated. We're wondering if you might know where the Sync1 next time(0x998) register is updated from, and why it's not done until this point. I know the PRU EtherCAT firmware from Texas Instruments if provided a black box.  Would it be possible to have access to the source code to understand why this register is not updated during startup?

It looks like a issue. We will look into this issue and provide more details.

Ryan Hodgman said:

2. SOF Jitter
We're measuring the SOF signal directly from the board on pin D7 vout1_d10.  The SOF signal arrives at the time as expected from the master, 55us ahead of the next Sync0.  However, we don't see a Hardware Interrupt for the PDI_ISR/SyncManager until 20us after the SOF signal is seen.  So despite SOF arriving at the time expected, we don't receive the interrupt in software until 20us later.

According to us, we need to profile the Sync0/Sync1/PDI using HW pins, instead of profiling using GPIOs from ISR. We understand that there are challenges in performing profiling using sync and edio pins. We will try it on our side and provide more details.

Regards
Dhaval

Hi Dhaval,

Thanks for the update.  Could you provide additional information on the correct way to measure the Sync0/Sync1/PDI interrupts using HW pins?  Is D7 vout1_d10 not the correct way to perform this kind of measurement for the PDI?

Also did you confirm which version of the PDK you are using to get PRUICSS1 to work?

Thanks,
Ryan

Hi

Ryan Hodgman said:

Thanks for the update.  Could you provide additional information on the correct way to measure the Sync0/Sync1/PDI interrupts using HW pins?  Is D7 vout1_d10 not the correct way to perform this kind of measurement for the PDI?

We will get back on this.

Ryan Hodgman said:

Also did you confirm which version of the PDK you are using to get PRUICSS1 to work?

I used PDK version 1.0.18.

Regards
Dhaval

Hi

Ryan Hodgman said:

Thanks for the update.  Could you provide additional information on the correct way to measure the Sync0/Sync1/PDI interrupts using HW pins?  Is D7 vout1_d10 not the correct way to perform this kind of measurement for the PDI?

It will be more accurate to measure the Sync0/Sync1/PDI events using HW pins, rather than the corresponding ISRs. In the default example from the EtherCAT package, we also do not see the events in HW pins. This issue is under debug, and we will post a response when he have updates on what changes are needed to get it working.

Dhaval Khandla said:

Ryan Hodgman said:

Also did you confirm which version of the PDK you are using to get PRUICSS1 to work?

I used PDK version 1.0.18.

Were you able to test the alternate PRU-ICSS instance with this version?

Regards
Dhaval

Hi

Ryan Hodgman said:

Also checking the SDK source code it's noted hat PRU1 ETH0 is not wired for the default IDK.  The PINMUX options are commented out in the SDK source.

The code you mentioned is not used for IDK Revision 1.3. Which version are you using?

As mentioned earlier, please note that J51 on the IDK has to be open for PRU-ICSS1 to work. See following excerpt from IDK User Guide:

"The AM571x IDK EVM supports up to four 100Mb Industrial Ethernet ports attached to the PRU-ICSS
subsystems and up to two Gigabit (1000Mb) Ethernet ports connected to the integrated Ethernet switch.
The final number of available ports depends on the configuration selection:
• 4-port Ethernet mode: provides two 100Mb Industrial Ethernet ports and two Gigabit (1000Mb)
Ethernet ports.
• 6-port Ethernet mode: provides four 100Mb Industrial Ethernet ports and two Gigabit (1000Mb)
Ethernet ports.
Selection between these Ethernet modes is controlled by header J51. Installing a shunt on header J51
that shorts these pins together enables 4-port Ethernet mode. Removing the shunt from these pins
enables 6-port Ethernet mode. The LCD output is only available when 4-port Ethernet mode is selected."

Regarding the SYNC/PDI HW pins, it is under debug and I will provide details once we get it working.

Regards
Dhaval

Hi

Dhaval Khandla said:

Regarding the SYNC/PDI HW pins, it is under debug and I will provide details once we get it working.

Here are the details on how to enable the output of SYNC and PDI on AM571x IDK:

1. You need to add following code for pinmux
   

   // Pinmux for PRU-ICSS1 SYNC0 output (Pin D2, Signal, VIN2A_D4, Address 0x4A003578, Mux mode 0xB)
   ((CSL_padRegsOvly) CSL_MPU_CORE_PAD_IO_REGISTERS_REGS)->PAD_VIN2A_D4 = 0x2000B;

   //Pinmux for SOF (Pin F4, Signal VIN2A_D5, Address 0x4A00357C, Mux mode 0xB)
   ((CSL_padRegsOvly) CSL_MPU_CORE_PAD_IO_REGISTERS_REGS)->PAD_VIN2A_D5 = 0x2000B;

   //Pinmux for PRU-ICSS1 EDIO OUT1 (Pin G2, Signal VIN2A_DE0, Address 0x4A003558, Mux mode 0xD)
   ((CSL_padRegsOvly) CSL_MPU_CORE_PAD_IO_REGISTERS_REGS)->PAD_VIN2A_DE0 = 0x2000D;

2. Change PDI_ISR_EDIO_NUM to 1 in tiescbsp.h file, as we are using edio1 output here.

After this, you can probe following pins on IDK

• SYNC0 : Pin 54 in J21
• PDI ISR : Pin 3 in J4
• SOF : Pin 56 in J21

We also had some follow-up questions:

Marvn Arcangel said:

• Expected time from SOF to Sync0 is set to 55us

(Q1) Can you share more details on how is this expected time calculated here?

Marvn Arcangel said:

 We're currently measuring time from SyncManager to Sync0 as ~36us when the expected value is 55us resulting in about 20us jitter from SOF to SyncManaer.

(Q2) Do you mean SOF to SYNC0 or SM to SYNC0, as in the original message, you used 55 us for SM to SYNC0?

(Q3) Have you tried a similar measurement with TwinCAT? As you know, the SOF depends on the timing of packets sent by master, and therefore master can induce a jitter with respect to SOF and PDI/SM updates.

Regards
Dhaval

Hi Dhaval,

Thanks for the update, hopefully I can help answer your questions and clarify below.  If anything is unclear I can attempt to elaborate further.

Regarding the IDK Pinmux for PRU1.  Okay I understand now, our board revision is 1.3.  I'm still unable to get PRU1 to work however, J51 is open and I have set the PRUICSS_DEFAULT_INSTANCE to PRUICSS_INSTANCE_ONE in tiesc_soc.h.  Can you think of anything else I could look at?

Thank you for the PINMUX options, I will try these for signal measurement.

Q1: Expected time is calculated and set by the Master Stack(we're using Acontis as previously mentioned).  The Master is set to BusShift mode using the Slaves as a reference clock.  So the Master controls the send timing of in reference to the Slave to get an expected value of 55us.

Measurement is done with a probe trace.  SOF signal is measured on pin D7(located on U103 on the board).  The SyncManager and Sync0 timings here are measured from the interrupt in software. 

Q2: The measurement I'm referring to is the SyncManager Interrupt(PDI_ISR) to the SYNC0 interrupt.  The timing measured here is 36us.  However, when we directly measure the SOF signal to SYNC0 the timing is 55us as expected.  My understanding is SOF signals the SyncManager Interrupt, so there's a jitter between SOF received and seeing the SyncManager interrupt in software. 

So to summarize:
SOF to Sync0 Interrupt(in software) = 55us
Syncmanager(in software) to Sync0 Interrupt(in software) = 36us

Q3:  We have not tried with TwinCAT.  We have tried with a different ESC device(EL9800) and we also measure SOF to Sync0 as 55us on this device.

Hi

Ryan Hodgman said:

Regarding the IDK Pinmux for PRU1.  Okay I understand now, our board revision is 1.3.  I'm still unable to get PRU1 to work however, J51 is open and I have set the PRUICSS_DEFAULT_INSTANCE to PRUICSS_INSTANCE_ONE in tiesc_soc.h.  Can you think of anything else I could look at?

We will check this. Are you using https://www.ti.com/tool/TMDXIDK5718? I am using it, and these changes were enough for me.

Ryan Hodgman said:

Q2: The measurement I'm referring to is the SyncManager Interrupt(PDI_ISR) to the SYNC0 interrupt.  The timing measured here is 36us.  However, when we directly measure the SOF signal to SYNC0 the timing is 55us as expected.  My understanding is SOF signals the SyncManager Interrupt, so there's a jitter between SOF received and seeing the SyncManager interrupt in software. 

We can not guarantee a fixed interval between SOF and PDI ISR (raised due to SM updates). Depending on the packet length, it will change.  Also, the buffer switching for SM will take place post EOF after CRC validation only. It will be better to measure the timings from HW signals in any case.

Ryan Hodgman said:

Q3:  We have not tried with TwinCAT.  We have tried with a different ESC device(EL9800) and we also measure SOF to Sync0 as 55us on this device

Is it possible to try this with TwinCAT?


Regards
Dhaval

Hi Dhaval,

Thank you this certainly helps clarify the SM timing we're seeing.  I'm working on debugging PRU1 to get it working so I can measure via the provided pinmux options.

We were also curious if there was any update on the Sync1 Next Time 0x998 register at startup.

Hi Dhaval,

I was able to get PRU1 to work for my board.  There was a PINMUX conflict in our use application that was causing the issue.  Thanks for the patience on helping me figure out the root cause.

I used the pinmux setting provided for SOF/Sync0/EDIO OUT1.  I've also changed PDI_ISR_EDIO_NUM to 1 in tiescbsp.h file.

SOF/Sync0 are working fine, however the measurement for EDIO OUT1 is reading constant High.  I'm looking into the documentation, but maybe you might be able to help me understand a root cause for this?

Dhaval,

I checked the schematic and it looks like EDIO_DATA1 for pin G2 is actually on J4 Pin 4, not Pin 3.  Assuming I'm now measuring the correct pin for the PDI my results can be found below.

For this test I was using an expected delay from SOF to Sync0 of 10us.

SOF Pin F4 to Sync0 Pin D2 is as expected around 10us.  

SOF Pin F4 to EDIO Out Pin G2 is at about 15us.  So this is the Jitter which we had previously been concerned about which still appears using the direct pin measurements.  I understand as previously stated that process data buffer could impact this delay in our previous measurements using GPIO toggles in software.  Is that statement still true for this measurement using the hardware signals?

Hi

Ryan Hodgman said:

SOF Pin F4 to EDIO Out Pin G2 is at about 15us.  So this is the Jitter which we had previously been concerned about which still appears using the direct pin measurements.

What is the packet size when you are performing these tests?

SOF and PDI ISR will not occur at the same time. So we can not consider it as a jitter, some delay is expected.

Regards
Dhaval

Hi Dhaval,

The delay is fine given that we have a explainable cause for it.  If you're saying it correlates with packet size or process data size I think we can work with that. 

From my understanding of packet size the maximum size is 1514 bytes: 14 bytes Ethernet header, 2 bytes E88A4 header, 10 bytes EtherCAT header, 1486 bytes EtherCAT data and 2 bytes EtherCAT Working Counter.

In my recent test we have 1 Master 1 Slave. 72 bytes of Output Process Data to the Slave and 72 Bytes of Input Process Data to the Master.  So that would put our packet size at 172 bytes assuming my understanding of this calculation is correct. 

Hi Dhaval,

Wanted to check if there was any update on the Sync1 Start Time Register issue at startup?

Ryan

Ryan Hodgman said:

Wanted to check if there was any update on the Sync1 Start Time Register issue at startup?

This may need a change in PRU-ICSS Firmware. Once the fix is available, I will share it with you.

Regards
Dhaval

Ryan
Sorry for the delayed response here.

Ryan Hodgman said:

Wanted to check if there was any update on the Sync1 Start Time Register issue at startup?

In this case, can you check when does the master update SYNC1 Cycle Time (0x09A4-0x09A7) register? Is it already done when you are reading 0x0 in 0x998 register?

Regards
Dhaval

Hi Dhaval,

No worries on the time between responses I understand fixing these issues is not a simple task.

Yes 0x09A4 SYNC1 Cycle Time register is updated sometime before I'm reading the 0x0 in the 0x998 register.  See attached image, 0x09A0 and 0x09A4 have been initialized with cycle times.  0x0910 System Time is being updated, and 0x0990 Sync0 Next Time is set to occur at a specified future time, however 0x0998 is still reading 0.