DM8148 + EZSDK 5.05.02.00: DVO2 Display "glitching"

This may be related to a known issue. What version of silicon do you have on your system?

Please check advisory 2.1.68 in the silicon errata on the product folder (http://www.ti.com/product/tms320dm8148) for more information,

BR,

Steve

Hi Steve,

I really appreciate your quick response!

I believe we're on Silicon Rev. 2.1 currently, given:

• Package number is TMS320DM8148BCYE0
• Contents of register  0x48140600: 0x3B8F202F

With that said, please let us you what your recommendation is.  I've talked to our HW team, and they're willing to obtain and repopulate a unit with Rev 3 silicon, so that's 1 viable test for us. 

In the meantime, I'll further review the recommendations for that errata item.  Any advice on applying the L2 cache workaround would be appreciated... I see that the L2 cache aux ctrl register can't be programmed from the OS, so I'll have to look into doing that from U-Boot min I assume...

It's interesting that you mention a bandwidth-related errata item.... I've witnessed a few other anomalies that "felt" like bandwidth issues, such as the live video momentarily becoming corrupted in a way that looked a buffer being interpreted as luma data when it's chroma. (The symptom being the image momentarily splits and becomes purple and green.)  It seems like these anomalies appear more frequently with increased RAM usage.

For our intended product, we'll be running the DDR3 at 400 MHz, as this is what is specified in the TRM as the 8148's max supported value. (Oddly, I've seen recommendations from TI about running up at 533 MHz to support 1080P...) For what it's worth, after having tuned some DDR3 timing parameters, I had tested running the DDR3 up to 666MHz, and still saw this issue. 

(Note the "seem" and "felt" -- as these items are hard for me to reproduce in a consistent manner and prove with absolute certainty, take this information with a grain of salt.)

Thank you,

Jon

Jon,

I am not a software guy, so will be of limited help there. I would think that the Kernel should be able to make these changes, but you might need to add them to u-boot and re-build u-boot.

I should be able to supply a few PG3 devices to you for re-population. Private message me with your contact details and I will start the process.

It did take us a while ourselves to be able to somewhat reliably re-produce the issue so I can understand your difficulty catching it.

BR,

Steve

Steve -- thank you! Will be sending you the requested info shortly!

I finally received a board with the PG3 device populated.

Unfortunately, I'm still seeing the same exact behavior with this "glitch" on the RGB display.  Again, it only seems to occur once I start pushing video data (as opposed to just the GUI frame buffers). See the end of this post for a description of what the "glitch" looks like.

However, I'm operating under the assumption that no changes to the EZSDK 5.05.02.00 kernel or bootloader are required for the Rev 3 CPU. Please let me know if this assumption is incorrect.

It looks like I'll be diving back into debugging on the OpenMax side of things.  I'll certainly do my best to provide additional information as needed, and try to identify the simplest possible way to reproduce this, so that I can post some code up here.

If I hear back that someone can look into this, I can put together a diagram of our OpenMax pipeline, and will be happy to dump registers as needed.  I do *not* have access to the M3 firmware, so I can't enable any sort of debug logs there. However, I'll do what I can to provide sufficient info from the ARM side of things.

Just for a quick idea of the video path:

• YUV422 (400x400 or less) into VIP0
• VIP0 output goes to the following components. Given that I have one output and 3 inputs,  I have to do a little bit of manual buffer tracking here to ensure buffer coherency.  Basically, I just ensure each of these components is "done" with a buffer before allowing VIP0 to use the buffer again:
  
  • DEI Scalar
  • SC5 Scalar
  • H.264 Encode
• The DEI scalar scales up the video (to 600x600 or less), an drives the following displasy, both of which center the video:
  • HDMI
  • VGA
• SC5 scales up the video (to 480x480) and centers it on:
  
  • The SD0 display (NTSC)
• The H.264 encode is fairly straight forward...just writing the buffer contents to a file

All the while this is going on, I've got a 1280x720 GUI cloned to /dev/fb0 and /dev/fb1.  Note that since My VGA display is running at 1280x1024, /dev/fb1 is set to 1280x720 and offset, which effectively letterboxes the GUI output on the VGA display; the HDMI and VGA display are clones of one another, but the VGA has "black bars" on the top and bottom.

Correction: After reviewing my notes, I remembered I had reproduced this with a simpler use case-- please ignore the crossed out portion above and refer to the configuration in my original post. 

As you'll see from my first post, I had simplified my use case to only drive the GUI to /dev/fb1, and made the dimensions of /dev/fb1 the full VGA display resolution. I did this to rule out any concerns that I was doing something strange from the GUI & Linux Framebuffer device side of things.

The video output is the only thing on the SD2 display.

To reiterate, I *only* see an intermittent "glitch" in the VGA display output when all of this is running at once. It does *not* occur:

• When running only the GUIs on the framebuffer devices
• When only running the OMX pipeline

The "glitch" appears as maybe 20-40 rows of torn VGA display data, and it always occurs in the same spot, at the bottom of the GUI (which is overlaid atop the OpenMax video.) 

Thank you Steve and to anyone else that might be able to assist on this one; your assistance is greatly appreciated.

Jon

Jon,

When the glitch occurs are you seeing a totally different frame? Or is the glitch visible during moving video? If you are seeing the first case, then its a possible bug in HDVPSS driver. Sometime back we've solved a similar issue for one of our customers. 

As a immediate workaround, you may be able to reduce this behavior by queueing in more buffers in the beginning. Based on our analysis what we found is that it caused by higher interrupt latency between M3<->A8 communication under higher system load.

Hi Renjith,

The symptoms of the glitch aren't appearing in the video from OMX pipeline, but on the GUI layer that's overlaid atop of the video. As there's not really any movement in the GUI, I can't say with certainty just yet if I'm seeing data from a previous frame.

As described in my earlier post, our configuration looks like this:

The video is from our OMX pipeline, and the GUI is atop the video layer, using color keying transparency to make the lower video layer visible.

When the glitch occurs, it only corrupts a region of the GUI, as shown above by the dark green. It's hard to tell, as it happens so quickly, but I think a row is becoming shifted and wrapping around -- I see some text wrap and the position of some GUI widgets shift.

However, when this glitch appears, I don't see any noticeable change in the OMX video stream.

I do not believe it's an issue with the GUI data being corrupted, as I'm able to clone the data to both /dev/fb0 and /dev/fb1 -- the data on dev/fb0 looks fine, while /dev/fb1 glitches.   If I remember correctly, I've also tried switching them; the problem follows the VGA output.

As mentioned in my above posts, this seems to occur when I have both my GUI and OMX stream running. I don't see anything anything out of the ordinary on the HDMI, VGA, or SD outputs when running just one.

Any chance you could share a patch of the HDVPSS driver that I could take a look at?  It seems to me that something's not keeping up when compositing the video and GUI layers...that's just a hunch though.  However, my suspicion seems to somewhat plausible, given your mention of having seen interrupt latency issues.

I'd be more than happy to try to collect more diagnostic info or try to answer your original questions a bit better.

With regard to the workaround associated queuing more buffers -- where would you suggest adding more buffers?  Per the description of my pipelines, I could allocate more buffers for:
  - The VIP Capture output (and scaler inputs)
  - The scaler outputs and display inputs

I'll look to experiment with allocating additional buffers in either or both cases and look to post back by the end of the day tomorrow.

For what it's worth, I appear to have VPSS FW version 0x1000145. I'm using the kernel from the linux-omap3 repo, with the last commit in the drivers/video/ti81xx/video/vpss being 458b09a73526dec5cb4757adcfa02dd8db87b08d.


Thank you very much for your time and help,
Jon

Hi Jon,

Jon S. said:

Any chance you could share a patch of the HDVPSS driver that I could take a look at?  It seems to me that something's not keeping up when compositing the video and GUI layers...that's just a hunch though.  However, my suspicion seems to somewhat plausible, given your mention of having seen interrupt latency issues.

I cannot share the patch as there are license agreement issues to be sorted out. Also we've fixed the same of another customer. We can discuss in detail offline. Please send me an email. 

Jon S. said:

With regard to the workaround associated queuing more buffers -- where would you suggest adding more buffers?  Per the description of my pipelines, I could allocate more buffers for:
  - The VIP Capture output (and scaler inputs)
  - The scaler outputs and display inputs

Here I recommend to queue in scalar/display inputs. One more question, are you using M2M scalar?

Hello Renjith,

Thanks you, I'll follow up with you shortly.

With regard to the scalars we're using -- we're using the DEI SC (both outputs) and SC5, as they're provided in the EZSDK OMX implementation.  I seem  to recall the term "M2M" being used with the RDK McFW....just wanted to note that we hopped on this processor before that was available, and are somewhat stuck with the OMX.

Not sure if that answer suffices... If not, let me know and I'll try to provided a more. For what it's worth, I'll repost the OMX pipeline configuration we implemented via the OpenMax IL API:

• YUV422 (400x400 or less) into VIP0
• VIP0 output goes to the following components. Given that I have one output and 3 inputs,  I have to do a little bit of manual buffer tracking here to ensure buffer coherency.  Basically, I just ensure each of these components is "done" with a buffer before allowing VIP0 to use the buffer again:
  
  • DEI Scalar
  • SC5 Scalar
  • H.264 Encode
• The DEI scalar scales up the video (to 600x600 or less), an drives the following display, both of which center the video:
  • HDMI
  • VGA
• SC5 scales up the video (to 480x480) and centers it on:
  
  • The SD0 display (NTSC)
• The H.264 encode is fairly straight forward...just writing the buffer contents to a file

Thank you,

Jon