Hi Asher,
1). Loopthru_EN pin 2:
It seems you are not using TXOUT pins(pine # 21 & 22). Please make sure when you are running your test Loopthru_en is disabled.
2). Could you please put your scope probe on LOCK/ to make sure it is always low when…
I don't think that there is a SMPTE or other video spec for Broadcast quality H-sync jitter but there are specs for pixel clock jitter and pixel clocks are PLL derived from H-sync. See Fig. 1 and Table 1 in the attached.