SGX X11 with alpha blending

If you use the IOCTL's as described in the VPSS User's guide to configure the primary framebuffer you are using for X for either Pixel or Transparency Key blending, you can accomplish your goal of seeing video through X11.

You are right in that the X server doesn't really support alpha blending, however.  What you'll find is that some windows (like terminal windows) won't bother setting the alpha bits on the frame buffer (Xorg.log should inform you that there's a 32 bit framebuffer and 24 bit color is in use), and thus show video through them.  You can accomplish this same goal by writing your own application and using XRender to write to the frame buffer, which seems to also work.  This works not really by design, but if you can control the entire application stack, you can maybe take advantage of the behavior.

If a color key would be sufficient, you could easily set your X11 background to that color, configure the color keying on your X11 frame buffer, and have video show through this way.

I have all of this working on a DM8148 board.

I don't believe the PVR/DRI driver that we have is capable of running two XServer instances and connecting to two different frame buffers.  You can connect a frame buffer or OpenGL ES application to another frame buffer and blend them, but I would be surprised if two simultaneous X servers worked.

This doesn't sound too bad although I would really like to use X on two framebuffers.  I know that X servers using the fbdev driver look at the FRAMEBUFFER environment variable to determine what framebuffer to use.  Could I modify the X server for SGX too use that environment variable?

Regarding the IOCTL's.  It sounds like I would need to start the X server, then open the /dev/fb0 framebuffer in another application, then use the TIFB_SET_PARAMS custom ioctl call to set blendtype in the ti81xxfb_region_params variable to TI81XXFB_BLENDING_PIXEL (since I want per pixel alpha blending).  Does this sound right?

I'm actually planning on controlling the entire application stack, so I think your idea of using XRender should work for me.  I pretty much just need X for the drawing capabilities and the ability to handle fonts.  I wasn't planning on using X originally, but we wanted to expand the type of fonts we're using and X seems to handle a large number of them.

By the way, is there any way of changing which framebuffer the X server is using?

There are a few undocumented module parameters to the PVR driver, but I am not aware of any which allow you to specify the frame buffer to use.  My impression is that the whole X/DRI implementation wasn't designed in such a way to support multiple instances, but I could be off base on that.

You may be able to use Cairo and/or Clutter to render directly to another one of the framebuffers and get a lot of the font capabilities you get from X.  Qt will also run directly to a frame buffer.

I believe you should be able to do what you want using X, but the X support on the platform is less mature than direct use of the frame buffer in general.

You are right on regarding the IOCTL and the framebuffer.