Can DM8168 encode 1080P30 H.264 bitstream,and bitrate down to 768Kbps with good quality?

Young,

In general for 1080p30, 768Kbps is bit less and also the quality of the video encoding is totally content dependent. By the way what is your application? 

Few more Suggestions

1.  Instead of 30 fps go for 24 fps, this is also reasonably okay.
2. All the default settings of encoder are for best quality, try not to modify them.
3. If you application is okay to enable 'B' pictures, you enable them.
4. If your application is video-conferencing or any other in which you are interested only in particular part of video content, then enable 'Region of Interest Encoding' . The over-head in a system would be to detect objects(face) and pass them to encoder to encode with better quality.
5. One more way, but I am not sure if your system handles it: --> Resize the video to 720P --> pass it to encoder --> streaming --> Decoder outputs 720P --> resize to 1080P.

Thanks,

Santosh