Video processing on DM643x: glitches when transferring data between IRAM and DDR in a double-buffering scheme.

This certainly sounds like a cache coherency issue, if you are not running any BCACHE commands to deal with coherency you will likely end up with cache artifacts. Essentially the video ports and the DMA operate outside of the caching structure of the CPU core, so if you have cached data that has been updated by a video port or DMA transfer than the CPU will still see the old data, conversely if the CPU updates some data in its cache that does not get the opportunity to write back to external memory than the video ports and DMA will not see the updated data. This being said when the DMA or video port is bringing in data that will be operated on by the CPU you must invalidate the cache for that region such that the CPU will see all the new data, and when the CPU is done working on data that will be given to a video port or DMA than you will need to write back the cache for that region, so the data the CPU updated gets reflected in external memory.

There is an article on cache coherence on the wiki here, and the accessing pixels in a frame document here uses a cache operation to maintain coherency.

I assume the glitches take the form of flickering horizontal lines? This is typically what I have seen when cache coherency was not properly maintained in a video system.

It could be a memory bandwidth issue, seeing PRV_WBL_O set would indicate such and broken real time deadlines in the video subsystem can appear as artifacts that look similar to cache coherency issues. The VPSS has the highest priority already by default, however so do the EDMA channels, so it may be worth trying to lower the priority of the channels that are not as time critical with the appropriate EDMA QUEPRI register fields. The system priority scheme is discussed in section 3.6.1 of the datasheet.

There is also a register in the DDR controller that may have some impact here, in particular the PBBPR register has the ability to adjust how long DDR transfers stay in the FIFO before yielding to older transfers, this register is discussed in section 4.6 of SPRU986b.