How to deactivate RST/NMI/SBWTDIO on MSP5359? They are not GPIO pins.

You cannot deactivate them.
What is the problem with those pins?

You can switch the RST pin to NMI mode (which will disable RST functionality) and not enable the NMI. However, this will make it hard to ever attach to the MSP again, as the RST pin will only go back to the /(required) RST functionality after a power cycle - and usually, while the debugger prepares to attach, your firmware is executed and disable sit again, so neither the BSL entry sequence nor SBW will work, and even 4-wire JTAG will have a hard time.
If you want to deactivate RST, you can tie it to VCC with a jumper, and connect it to anything else (like JTAG) through a resistor (to limit any shortcut currents). This effectively sets RST out of function when the jumper is closed.