Issue with Power MOSFET thermal issue in power supply uses TPS40060

- Is there any way to reduce power lost on these power mosfet? Could we reduce switching frequency to do that? if frequency reduced, which component(s) should be changed along with?

Switching Frequency directly affacts several factors of MOSFET power loss.  The most obvious is switching losses that dissipate power in the MOSFETs are the switches transition from ON to OFF and OFF to ON.

Switching frequency also affects body-diode conduction and reverse recovery losses, which dissipate power in the low-side and high-side FETs respectively as the body diode of the LOW-side FET carries the inductor current during the OFF time.

Finally, there is the charging and dischaging of the MOSFET output capacitances, which can also dissipate a lot of power in the MOSFETs as the switch node capacitance is charged and discharged each switching cycle.

- Is the compensation circuit affect to mosfet power loss (Both switching and conduction losses)?

Aside from becoming unstable and resulting in an oscillation in the ouptut voltage, compensation should not impact MOSFET power losses.  An unstable ouptut will require the ouptut capacitors to be charged and discharged with that oscillation and that can result in very high power dissipation.

- Could we reduce the input voltage to reduce the switching loss? in this case, which component(s) should be changed along with?

Yes, in several ways.

First, the switching losses experienced by the high-side FET is proportional to the input voltage.

Next, the size of the MOSFETs gate charge is typically proportional to the 2.5 power of the breakdown voltage of the MOSFET.  Lower break-down voltages result in lower gate charge (for the same Rdson) and higher switching speeds for less switching, reverse recovery and output capacitance losses.

To get the full benefit of the reduced input voltage, you'll want to select new MOSFETs with a lower break-down voltage, a new inductor and possibly new capactors.  The KFF resistor will need to be resized for the new UVLO level and the current limit might need to be adjusted for the newly selected MOSFET.  With any inductor change, the compensation loop should be checked to see if any changes are appropriate there.

- In case we must change the mosfet device for next revision, what is the recommendation for our case? Which package is recommended? Current package is PowerP AK SO-8 and PowerPAK 1212-8.

PowerPAK SO-8 MOSFETs have the highest power handling capabiltiies and often offer the best thermal results unless heat sinks can be used.  D-PAK, I-PAK and D^2-PAK also offer very good thermal performance.

- Is it better in term of reducing power loss on these Mosfet if we step down the input voltage to 24V, then from 24V step it down to +12V?

Purely from a power dissipation and MOSFET stress point of view, a 2-step conversion is almost definately better, especially if the higher voltage converter and rely on the point of load converter to support transient, allowing a slower loop and lower switching frequency.  Using 60V MOSFETs at 150kHz to generate 24V and then 30V MOSFETs at 300khz to generate 12V counters some of the increased parasitics of the higher voltage MOSFETs with a lower switching frequency for higher efficiency.

these mosfet PN  got from reference design from TI with the same 48VDC input, the differences are on reference design, the output voltage/current is +3.3V/5A, in our design, the output voltage and current is +12V/4A.

The reference design MOSFETs were selected to support 4A of current with the high-side FET conducting less than 7% of the time and the low-side FET conducting the remaining 93%.  Moving to 12V output with a 25% and 75% increases the conduction losses in the high-side FET by nearly 4x.  Likely this should be increased to a PowerPAK SO-8 device with approximately 1/2 the Rdson of the current PowerPAK 1212 device.

1. The lower Power MOSFET: NTB6412ANG from Onsemi - 100V, 58A, 18.2mOhm - D2PAK package
2. The low-side FET will carry about 3.5Arms current.  This gives a conduction loss factor of 12mW/mOhms.  The switching loss factor is 25mW/nC, though the vast majority of that is in the controller rather than the FET.  The optimized FET will have a Rdson / Qsw ratio of 2mOhm / nC.  That is, 2mOhm per nC of Switch Charge (Qgd + Qgs/2)
3. The NTB6412ANG has a ratio of 44mOhms / nC.  Also, this MOSFET has a horrible Qgd / Qgs ratio of almost 3:1, making it highly succeptible to dV/dt turn-on and inappropriate as a synchronous rectifier MOSFET.

1. The upper MOSFET: SUM110P08-11: 80V, 110A, 110mOhm, TO263 package 
2. This is actually a 11.1mOhm MOSFST with 113nC of gate charge.  At 250kHz, this will draw 28mA of current for 1.36W of power dissipation in the IC just turning the high-side FET on and off.  Switching losses would also be huge with about 65nC of switching charge.
3. The high-side FET carries 2Arms current for a conduction loss factor of 4mW / mOhm and a switching loss factor of 68mW / nC.  An optimzed MOSFET will have an Rdson / Qgsw ratio of 17:1  That is, 17mOhms per nC of switch charge.  Switch charge being approximately (Qgd + Qgs/2).  The SUM110P08-11 at 11.1mOhm and 65nC has a ratio of 0.16 mOhm / nC.  If you can find a MOSFET with an Rdson of 110mOhms and 6.5nC switch charge, that would appear to be a much more optimized MOSFET for this application, though total power dissipation would be approximately 1W.