Compiler/TDA2SX: How to use samplerExternalOES and sampler2D in the same fragment shader

user6426230

Part Number: TDA2SX

Tool/software: TI C/C++ Compiler

SDK version：PROCESSOR_SDK_VISION_03_08_00_00

The current TI frame camera data is due to the special NV12 format，therefore, the texture target in TI uses GL_TEXTURE_EXTERNAL_OES instead of the usual TEXTURE_2D.

And when GL_TEXTURE_EXTERNAL_OES and TEXTURE_2D are mixed in the same rendering pipeline shader program, they cannot currently render normally.

For TI, TI’s own process of rendering a single camera image is as follows:

1、Bound texture target GL_TEXTURE_EXTERNAL_OES

... ...

glGenTextures(1, &pObj->texYuv[texIndex]);

System_eglCheckGlError("glGenTextures ");

printf("andy debug: gen texture ID %d texIndex %d dmaBufFd %d \n", pObj->texYuv[texIndex],texIndex,dmaBufFd);

glBindTexture(GL_TEXTURE_EXTERNAL_OES, pObj->texYuv[texIndex]);

System_eglCheckGlError("glBindTexture");

glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR);

glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

System_eglCheckGlError("glTexParameteri");

2、

In shader
"" uniform samplerExternalOES fsSampler3; \n" use samplerExternalOES instead of sampler2D

... ...

3、Load rendering

... ...
#ifndef STANDALONE 
    glBindTexture(GL_TEXTURE_EXTERNAL_OES, texYuv[tex2]); 
    glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR); 
    glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR); 
#else 
    glBindTexture(GL_TEXTURE_2D, texYuv[tex2]); 
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); 
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); 
#endif

... ...

But for our own AVM module:

We need to stitch together 4 camera images in real time。

Our shader is as follows：

1. Shader
... ...
uniform mediump sampler2D img1; \n\
uniform mediump sampler2D img2; \n\
uniform mediump sampler2D img3; \n\
uniform mediump sampler2D img4; \n\
uniform mediump sampler2D lut1 \n\
uniform mediump sampler2D lut2; \n\
uniform mediump sampler2D lut3; \n\
uniform mediump sampler2D lut4; \n\

... ...

2.Our image data is UYVY, and the LUT table data format is RGBA

3.Our source code uses GL_TEXTURE_2D for binding：

... ...

for (i = 0; i < 4; i++)

{

if (0 == i)
  glActiveTexture(GL_TEXTURE0);
else if (1 == i)
  glActiveTexture(GL_TEXTURE1);
else if (2 == i)
  glActiveTexture(GL_TEXTURE2);
else
  glActiveTexture(GL_TEXTURE3);
glBindTexture( GL_TEXTURE_2D, texYuv[i]);

}

... ...

However, the current TI frame camera data uses samplerExternalOES due to the special NV12 format. We can also use samplerExternalOES for our own images.
But my own model and configuration table need to be loaded into the fragment shader using sampler2D.

If we modify our shader as follows

... ...
uniform mediump samplerExternalOES img1; \n\
uniform mediump samplerExternalOES img2; \n\
uniform mediump samplerExternalOES img3; \n\
uniform mediump samplerExternalOES img4; \n\
uniform mediump sampler2D lut1 \n\
uniform mediump sampler2D lut2; \n\
uniform mediump sampler2D lut3; \n\
uniform mediump sampler2D lut4; \n\

... ...

glBindTexture(GL_TEXTURE_EXTERNAL_OES, pTextureId[nCamId]);
glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR);

problem:
1、When GL_TEXTURE_EXTERNAL_OES and TEXTURE_2D are mixed in the same rendering pipeline shader program, they cannot currently render normally

2、How to load our RGBA format table also use GL_TEXTURE_EXTERNAL_OES？

over 5 years ago

0 user6426230 over 5 years ago

Prodigy 80 points

Or TI give up using eglCreateImageKHR to create and bind texture unit GL_TEXTURE_EXTERNAL_OES, give up its own NV12 (YUV420, flat format), use the following method

3. TI's camera data also uses TEXTURE_2D texture targets, and the TI camera data format has also become the universal UYVY (YUV422, packed format)
Then load and bind the texture unit by:
GLchar *pImageDataRGB;
GLuint nPhysical = ~0U;

glGenTextures(1, &nTxtUnit);
glBindTexture(GL_TEXTURE_2D, nTxtUnit);
#if 1
glTexDirectVIVMap(GL_TEXTURE_2D, nWidth, nHeight, GL_VIV_UYVY, (GLvoid**) &pImgData, &physical);
#else
glTexDirectVIV(GL_TEXTURE_2D, nWidth, nHeight, GL_VIV_UYVY, &pImageDataRGB);//, &nPhysical);
memcpy(pImageDataRGB, pImgData, nWidth*nHeight*2);
#endif
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
glTexDirectInvalidateVIV(GL_TEXTURE_2D);

How to modify in this way?

0 Hemant Hariyani over 5 years ago

TI__Expert 8385 points

Hello,

Can you please let us know what problem are you facing? Are you able to narrow the problem down in the shader by using one texture at a time? Without changing the openGL application code , if you just change the shader code to output the value read from each texture, you should be able to narrow it down.

You can use RGB with GL_TEXTURE_EXTERNAL_OES. Instead of specifying NV12 as the format, you can use ARGB8888 or any other Rgb drm format.

Will it be possible for you to provide us with a very simple application that highlights the issue?

Regards

Hemant

0 user6426230 over 5 years ago in reply to Hemant Hariyani

Prodigy 80 points

Thanks for reply.

4 textures are used for stitching.
The four image textures come from TI’s GL_TEXTURE_EXTERNAL_OES, and the stitching model or table comes from our own TEXTURE_2D

The issue is:
when GL_TEXTURE_EXTERNAL_OES and TEXTURE_2D are mixed in the same rendering pipeline shader program, they cannot currently render normally.
How to use samplerExternalOES and sampler2D in the same fragment shader

Processors

Processors forum

Compiler/TDA2SX: How to use samplerExternalOES and sampler2D in the same fragment shader