What triggers ISP Previewer start on DM3730?

I'm using a DM3730 with multiple cameras.  I have 4 cameras connected to the parallel interface (CPI).  The cameras provide RGB output, 320x240 at approx 180fps, so I'm using CCDC->PREV->memory->RESZ as the ISP chain.  The cameras provide HS, VS, and PCLK.  I select between the cameras by tri-stating all but one off the bus.  This works well, except when I'm switching cameras.  Since the cameras aren't perfectly in sync with each other, switching causes the HS/VS/PCLK signals to not be exactly right.  When I switch quickly, as when trying to get a single frame from each camera in a round robin fashion, the number of PRV_DONE interrupts I see is sometimes much, much less than the number of HS_VS interrupts.  For example, I may see 190 HS_VS interrupts / second, 180 VD0 interrupts/second but only 10 PRV_DONE interrupts / second.  My current theory is that the nonstandard HS/VS timing is confusing the Previewer.  Can anyone provide more details about the Previewer and what conditions trigger a Previewer start or restart?