Scaling 24 bit RGB from input through grpx to output

I have a question about scaling 24 bit RGB from video input through grpx to output.

We need to take 24 bit RGB video (probably coming from a computer source), pipe it through the framebuffer and display it on HDMI (for example).  At some point the video needs to be scaled to some other resolution.  An example would be to take 1024x768@60Hz computer video through the TVP7002 (I understand that the driver for the TVP7002 needs to be changed to support sending RGB as well as receiving separate horizontal and vertical sync) and send it through the framebuffer to the HDMI output.  The video would need to be scaled to 1920x1080@60Hz progressive at some point.  Is it possible to do this and how would it be done?  We also need to be able to downscale (e.g. from 1920x1080@60Hz progressive to 1024x768@60Hz).  I understand that the video input path has a scaler, but it's my understanding that scaler will only downscale not upscale.

How would this be done with RGB 24 bit?

Can the 3D accelerator do this?  If we have to use the 3D accelerator what kind of delay can we expect?

In looking at the HDVPSS block diagram I don't really see a path by which this can be accomplished.

This needs to be pretty real time since the user will expect that when he moves his mouse or performs some mouse function that the computer responds with minimal delay.  Even 100ms delay can be pretty noticeable to a user.

Help?

Carl