TFP410-EP: How to decide the 12 bit or 24 bit on BSEL pin?

It depends on how many parallel lines you are looking to serialize. If you have 24 parallel line you can send all 24 for every rising edge. If you only have 12, you can update them twice per clock cycles and send that data out on the rising and falling edge of the clock to send out 24 bits per clock cycle.