PLLM and PLLD

I would suggest leave one of the values in its default state and see, whether you get integer solution of the equation for the other value. If yes, you're done.

Clause 8.6.2 tells, that output divider is is always applied, so your equation is effectively CLK = CLKIN × ((PLLM+1) ÷ (2 × (PLLD+1))). Clause 8.6.1 tells that default (power on) value of PLLD is 0, so CLK = CLKIN × ((PLLM+1) ÷ 2, and to get 666 MHz out of 66.6 one needs to multiply by 20. Thus one have to set PLLM=19.

As you noted, there are two configuration variables, so there is some freedom in selecting them. You'll get similar result if program PLLD=1 (one more divide by 2), but PLLM=39 (multiplication by 40), i.e. CLK = 66.6*(39+1)/(1+1)/2=666.

I cannot judge, which one is better, but would stick to default values to reduce extra programming. Please don't forget, that BWADJ field need to be adjusted as well. In case you look into some TI's platform code, do not trust it and make your mind yourself.