DirectIO: System Reset required to re-establish link

Hi Eddie,

I asked about resetting the SRIO peripheral before and we had a conversation about the DIO Library but no one offered any advice on the reset (http://e2e.ti.com/support/dsp/tms320c6000_high_performance_dsps/f/112/p/28628/126831.aspx)

I don't have any problems restarting our system because I always follow the same procedure which is to cycle power i.e. I know the SRIO link will come up if both devices have the ultimate reset. I'm hoping that no one will ever ask me to do a warm reboot!

I can reset the CPU and reload the DSP applications from the debugger but I guess this is because the debugger reload procedure isn't interfering with the SRIO peripheral.

Matt

Hi Matt

I too was also using the debugger to reset the DSP and thus get SRIO to work 100%. Now that I'm on a multicore DSP, the debugger system reset leaves the other cores stuck in reset. Arg.

Hopefully TI can help us sort this out.

(the lack of thread posts on this makes you think that only u and i are the only ones doing SRIO doesn't it)

Cheers

We're using C6474 DSPs with the CPS16 from IDT. I ran into a similar issues where to bring up the system reliably, there has to a "magic" order in which the various components are brought up. I did find a way to do a  warm reset by using register "Port x Control CSR" (eg for port 0: 0x115C)  which allows you to enable and re-enable the ports and thus forces the 2 sides to re-linkup. I also enabled the "DROP_PACKET_ENABLE" bit to make sure the port would not continuously retry to send a packet and then set the ports to "OUTPUT_ERROR_STP" and "INPUT_ERROR_STP" (see  "Port Error and Status CSR").  Results were still intermittent though and in the end I settled with the magic bring up "solution" as my system is running continously anyway. 

I haven't tried this, but maybe if you disable the ports on both sides first and the bring them back up one at a time this will be a reliable way to form a connection. 

What does work but is a bit more dramatic is to disable the port on one side and then use the SRIO_Disconnect() function from this forum:

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/439/p/58721/228796.aspx#228796

After this you reinitialize your interface completely. This way you don't have to do a hard-reset every time. 

Edit: Also check this port on error states: http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/439/t/11735.aspx

Regards,

-Dirk

Thanks Dirk.

In SPn_Ctl (ox115) I tried setting b23 PORT_DISABLE before running the RIO_init() code (DIO Lib 1.1.0). The SRIO port still only works 50% of the time after reloading the DSP and I get this error when transfers fail

SPn_ERR_STAT: b4 - required to initiate Maint Port write operation.

I also tried Travis's srio_disconnect() in the link you posted. When I run that routine nothing works. I can see it clearing the SRIO regs at 0x02d0_0000, but after that, the DIO lib routines fail when they try to access any DIO regs. I can't even view them in the CCS watch window.

And I also tried clearing the GBL_EN b0 global enable bit. No luck.

w.r.t. the "Edit" you added above, I also looked at the "input or output error stopped states" bits in SPn_ERR_STAT and they are not being set. The Port OK bit (b1) is also set. Consequently, I don't imagine writing 0x40fc_8000 to SPn_CS_TX will help as specified by the "Software Assisted Error Recovery" section of C6472 SRIO sprue13j.pdf.

But I tried writting 0x40fc_8000 to SPn_CS_TX and then the DSP goes into RIO_init() and doesn't return. When you hit the emulator halt button you get:

   C64XP_A: Trouble Halting Target CPU: (Error -1060 @ 0x0) Device is not responding to the request. Reset the device, and retry the operation. If error persists, confirm configuration, power-cycle the board, and/or try more reliable JTAG settings (e.g. lower TCLK). (Release 5.0.429.0)

I also get the error below after writing 0x40fc_8000. I can't understand why the CCS gui can read 0x02d0_0000 to 0x02d_00fff but nothing after 0x02d0_1000.

C64XP_A: Trouble Reading Memory Block at 0x2d01000 on Page 0 of Length 0x4: (Error -1202 @ 0x2D01000) Device pipeline is 'not ready'. Choose 'Force' to retry the operation after tying to force the 'ready' state, but that will corrupt your program, and you may need to reset the device. (Release 5.0.429.0)
C64XP_A: Error: (Error -1 @ 0x0) This '-1' value should be returned only during code development. It is a temporary value used when a more appropriate error value has not yet been assigned. This is the most negative of the three trivial error numbers that are used by USCIF adapters, drivers and utilities. (Release 5.0.429.0)

So far the only method to get SRIO to work when reloading the DSP is to

1) do Alt-s, (system reset)

2) reload the DSP

3) disable and re-enable the Altera SRIO block.

I still feel that there is something in the SRIO init missing because when the SRIO fails, I can reload the DSP and then SRIO may or may not work. But the fact that it does work occasionally by simply reloading the DSP and nothing else, it tells me that we're missing something in the code to re-init the SRIO ACK counters.

Cheers

Eddie,

If you use the srio_disconnect function, you should be starting SRIO from scratch, so it will have to be re-initialized/programmed.  Are you resetting the rest of the device too, or just SRIO?  If you are not resetting the rest of the device, you must make sure all traffic is turned off to/from the SRIO peripheral internal to the chip.  Can you run your DSP program in SerDes loopback successfully, reset it with the srio_disconnect and continue on successfully?  Make sure you turn off all port-writes and error detection until you are up and running with a stable system.   If you can do this in loopback, then it leads me to believe that it is an ACKID issue between the FPGA and DSP.

Are you using one 1x port, or two?  My guess is that your error reading the registers above offset 0x1000h is clock related.  Those registers have a dependency on a SerDes generated clock, where the ones below 0x1000 are only dependent on the chip VBUS clock.  So if the SerDes are not programmed you won't be able to access those registers.  SerDes clock from port 0 is used inside the SRIO peripheral, even for other ports.

Hope that helps,

Travis

Hi Travis

I'll try to get responses to the other questions shortly. But for now, here is what I've got

tscheck said:

Are you using one 1x port, or two? 

Ans: 1x port

tscheck said:

If you use the srio_disconnect function, you should be starting SRIO from scratch, so it will have to be re-initialized/programmed.  Are you resetting the rest of the device too, or just SRIO?

In the edma streaming example of the DIO library, I am running it just before RIO_init(). I don't think there are other significant SRIO functions before that, but I'll check and get back to you.

Cheers

Eddie

Hi Eddie, Apologies for the late answer, but I hope I can be of some more help. In the meanwhile I wrapped up the last remaining issues I had with SRIo and can now successfully communicate with other DSPs through the IDT switch.  My main test suite was the edmaStreamingLsu where is goes through 4 different transfer types and reads back the result. While probably unrelated to any of the issues you're experiencing, my main problem was that the switch could not handle the speed at which the DSP was outputting packets and I had to increase the INT pacing value for the LSU Tx complete operation such that subsequent transfers are sent with some dead time in between. Otherwise the buffer of the switch would overflow.

Another issue I had with the DIO lib was that I directed the transfers and reads to other destinations (than the standard), but the doorbell confirmations which are built into DIO are hardcoded to dest 0..... That gave me some headaches (I felt pretty silly once I actually found the problem which was glaringly obvious).

Back to your issues: I found that when a DSPs had established a link to the IDT switch (FPGA in your case), in otherwords, RIO_Init() was called successfully, then I could reset the link by disabling the port at the switch side AND at the DSP side and the bring them both up again. So no reason to call RIO_Init(), which does a whole lot more (at the physical level, reset PLLs etc.).   It is very important that you do it on both sides, such that new links get requested both ways. 

Do you have a way to read/write SRIO regs in your IP-core on the FPGA? It'd be interesting to compare the results of your device with the IDT switch We're using.

I was also wondering: how much work did it take you to implement the FPGA solution? I am curious, as initially we planned on taking this route, but decided to use an intermediate device (IDT FIC, due to the more expensive FPGA required (?) for Xilinx with GTX ports).

I hope there's something here that will help you, let me know how things work out for you. 

Regards,

Dirk

Edit: At some times we had a lot of problems with the EVMC6474 (cannot read mem locations, target disconnected, etc) and at other times it would be stable for many days. We're using CCS3.3 and resetting and reconnecting to the DSPs is something we have done many many times....  I guess this comes with development, but it is frustrating when you're debugging something complex like SRIO.

Hi Dirk

Dirk Buijsman said:

my main problem was that the switch could not handle the speed at which the DSP was outputting packets and I had to increase the INT pacing value for the LSU Tx complete operation such that subsequent transfers are sent with some dead time in between.

We have the same problem!!!!! I'll have to investigate the INT pacing to understand how its implemented. We're trying to fix the problem by cutting the EDMA's LSU ready sync event and tying the sync to our receiving FPGA's FIFO empty signal. Its not the greatest approach because it ignores the LSU BSY bit.

Dirk Buijsman said:

I could reset the link by disabling the port at the switch side AND at the DSP side

Disabling both sides seems to be the only way we can get it to work. When you disable the DSP side, is that with the SPn_Ctl (ox115) b23 PORT_DISABL?

Dirk Buijsman said:

Do you have a way to read/write SRIO regs in your IP-core on the FPGA

Yes, what are you looking for in particular so that I can get you the information?

Dirk Buijsman said:

I was also wondering: how much work did it take you to implement the FPGA solution?

About 2 mo. We're using Altera. We took their reference design and modified it rather quickly. That took about 1 mo. One more mo was due to updating the Altera toolset, dealing with drive issues, and a backflow bug in their IP (which they immediately fixed).

https://www.altera.com/support/software/download/refdesigns/ip/interface/dnl-ref-serial-rapidio-to-ti-6488-dsp.jsp

FYI: the C6474 is the same as the C6488

Cheers

Using the pacing to space out outgoing transactions is a great solution, Dirk.  Eddie, it is simple to use, just check out the RIO_INTDSTn_RATE_CNTL for details. 

Regards,

Travis

Hi Travis/Dirk

It seems that interrupt pacing only works if you are not using the edma streaming. According to sprue13j.pdf Sec 4.7 Interrupt Pacing,

"The counter reloads and immediately starts down-counting each time the CPU writes these registers."

Since the above states the the CPU has to write this register, it would mean that instead of using the EDMA to stream a frame thats greater than 4k bytes, I'd have to interrupt the CPU to write to the INTDSTn_RATE_CNTL register each time 4kB are sent.

It would be great if the RATE_CNTL register was part of the LSUnREGS so that the EDMA could write to the reg.

"When the single pulse is generated, the logic will not generate another pulse, regardless
of interrupt status changes, until the rate control counter register is written again."

Or maybe the reg gets reloaded automatically when I do edma streaming?

Cheers

Eddie

Eddie,

The edma streaming example is writing the INTDSTn_RATE_CNTL register.  I don't recall the value it is using, but am guessing it is 0s.  After reprogamming the LSU regs, it clears the interrupt and programs the rate register by chaining to another edma channel. 

regards, travis

yes, the DIO lib uses 3 chained channels: 1 to write to the pacing register, 1 which writes to the iccr reg (reset IRQs) and the final write is to the LSU regs directly. A new transfer is initiated as soon as LSU reg 5 is written to. The actual values used to write to the pacing registers is programmed by the user in the code which sets up the structures passed to DIO_Init(). This is what I have in my main init function:

#define EDMALSUISRPACINGVAL        ( (Uint32)((float)0500000.0/((float)(3*CLKIN1_INPUT_MHZ*PLLC_PLLM_CLK/PLLC_PREDIV_CLK)/(float)1000.0)) )

// Core 0 LSU & Doorbells
    srioObj.intrPacing[0]    = (Uint32)DBELLISRPACINGVAL;      // For INTDST0: doorbells+lsus' count down in DMA clock cycles (CPU/3 = 333 MHz)
// Core 0 LSU & EDMA
    srioObj.intrPacing[1]    = (Uint32)EDMALSUISRPACINGVAL;    // For INTDST1: edmastreaminglsu's count down in DMA clock cycles (CPU/3 = 333 MHz)
// Core 1 LSU
    srioObj.intrPacing[2]    = (Uint32)EDMALSUISRPACINGVAL;
// Core 1 LSU & EDMA
    srioObj.intrPacing[3]    = (Uint32)EDMALSUISRPACINGVAL;
    srioObj.intrPacing[4]    = 0;           // Not used
    srioObj.intrPacing[5]    = 0;           // Not used
    srioObj.intrPacing[6]    = 0;           // Not used: RIO_INT_DST_6 = Reset vector for booting
// Error & MQT isr
    srioObj.intrPacing[7]    = (Uint32)ERRORISRPACINGVAL;      // ...etc.....: errors+mqtisrs' count down in DMA clock cycles (CPU/3 = 333 MHz)

The int value is very high for testing, but it fixed my problems. This value is copied to the pace regs every time the LSU Tx complete irq triggers (via paceImage[]).

I found the DIO lib interrupt routing very hard to follow, because of the many different ways you can run (3 cores, streaming/ no streaming), quite confusing... It's a nice lib, but man it's complex.

Disabling both sides seems to be the only way we can get it to work. When you disable the DSP side, it that with the SPn_Ctl (ox115) b23 PORT_DISABL?

Thanks for the info on the FPGA side of things, 2 months sound pretty good to me.

Dirk Buijsman said:

the DIO lib uses 3 chained channels: 1 to write to the pacing register, 1 which writes to the iccr reg (reset IRQs) and the final write is to the LSU regs directly

Ah, enlightenment! Now I understand......finally. I'll give that a try to see how it works.

Dirk Buijsman said:

When you disable the DSP side, it that with the SPn_Ctl (ox115) b23 PORT_DISABL?

Yes. I also tried GBL_EN. Here is the code

    // disable SRIO ports (added to see if we can restart without having to to alt-s)
    {
        CSL_Status              status;
        CSL_InstNum             srioNum = 0;  /* Instance number of the SRIO */
        CSL_SrioParam           srioParam;
        //SRIO_disconnect( 0, 0, 0, 0, 0);
        status = CSL_srioInit (&srioContext);
        srioObj.hCslObj = CSL_srioOpen (&srioCslObj, srioNum, &srioParam, &status);
        tmp = hSrioDirectIO->hCslObj->regs->PORT[0].SP_CTL;
        tmp = 0x00800001;  // setting PORT_DISABLE 
        hSrioDirectIO->hCslObj->regs->PORT[0].SP_CTL = tmp;
        //tmp = hSrioDirectIO->hCslObj->regs->GBL_EN;
        //tmp = 0x00000000;  // clear b0 - Global enable
        //hSrioDirectIO->hCslObj->regs->GBL_EN = tmp;
        //hSrioDirectIO->hCslObj->regs->PORT_OPTION[0].SP_CS_TX = 0x40FC8000;
    }

Dirk Buijsman said:

I found the DIO lib interrupt routing very hard to follow,

"Hard to follow" is an understatement. Fortunately, Piyush from TI give me some help. His understanding was amazing. Have you seen the wiring diagram in the DIOLib_1_1_0\doc\DIOLibraryAPIReferenceGuide.html->Modules->TST->testDio_extLoopback? Thats Piyush's creation and its darn good.

Also, I found this tool very good for navigating the DIO libraries and CSL lib.

http://www.scitools.com/

It used to be fairly inexpensive, but now they sell it with all the languages enabled instead of just the language you want.

Cheers

Hi Dirk

Interrupt pacing worked! Thanks a bunch. Now to focus on the SRIO reset issue.

Cheers
Eddie

Nice! This issue cost me quite a bit of time myself. For the reset It looks like you're disabling the port properly. Are you disabling the FPGA port in a similar fashion? After both are disabled, you should bring them backup and I believe it should work again. I should mention I never tried this with 2 DSPs, but only with 2 IDT devices (Switch & FIC).

In your code I did notice you're assigning tmp twice in a row. Unless this is for debugging purposes, are meaning to use tmp |= 00800001; the second time around?

I used to have to reset the switch port which connected to the DSP when the DSP got disconnected from the SRIO chain (after SRIO_disconnect() got called). It never happens the other way around in my system though, as the switch is always online. that's why I never read the SPn_ERR_STAT in the DSP, but in the SRIO switch we have a "realtime" monitor running which would show the bits input and/or output error stopped bits would get set. Is this the case in your DSP as well?

Thanks for the tip on scitools, never heard of it, but it looks like a very handy tool. As for the diagrams from TI: more documentation like the diagram you pointed out would be very welcome. Especially in the area of resource usage: DIO uses IDMA, EDMA, SRIO, timer, INTC and CIC peripherals. Nowhere is there a list with what exactly is used let alone maps for IRQ routing. I am very happy they wrote the lib though, but I guess it would have saved even more time with more docs (or just some diagrams).

Cheers,

Dirk

Hi Guys

I think I found the solution. No resets required. See the link below

C6472 SRIO error/status registers - bits don't indicate error condition

http://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/439/p/135246/490520.aspx#490520

Cheers