I've been working with this reference design myself, at first with my own board, but more recently with the exact board files that TI gives in their reference design page. I've found that I can run the CAN just fine at 800 khz but when I bump up the communication rate to 1 mhz I start to get strange behavior on the bus. The most obvious symptom is the bus usage immediately jumps to near 100%. When probing with a scope, I can see that after a message ACK bit, the bus will return briefly to the recessive state then get pulled low again, remaining locked low until the transceiver times out and resets.
In the above image, the bright yellow trace is the can line logic, red trace is TX1, orange trace is RX1, green trace is TX2, and the purple trace is RX2 . About two-thirds of the way through he shot, You can see a message is ended with an ACK bit (red line going low the first time). This locks the bus in a dominant state until the transceiver times out, only to be pulled low again. Eventually the bus will recover and send the next message, but at the end of that message it will repeat this cycle. I've tried both increasing and decreasing the time delay on the circuit from dominant to recessive, it either makes no change or completely breaks the communication. Does this sound familiar to anyone working on this circuit at TI?