TMS320DM8168: Can dm8168 support 1080i50 signal input and 1080i50 signal output?

yes, both VIP and HDVPSS can support this resolution..

 test found that if the 1080i50 signal is connected, the picture will become flat. For example, a square becomes a rectangle. How to modify the video is normal

can you please explain your chains? 

Since it is interlaced input, vip will capture one field at a time, how are you displaying it?

For example, the 23 inch display in the picture becomes flat, and the whole picture becomes flat

I am still not getting your dataflow. You are using capture, SWMS, DEI and display, but how are they connected? 

Please note that since input interlaced, capture will capture field by field. You either have to convert to frame using DEI or upscale it to frame using scalar and then you could display field as frame and can display it correctly. 

Rgds,

Brijesh