This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: GPU performance degradation and black screen issues

Part Number: TDA4VM

Hello Expert

LINUX

gpu version is 1.13.5776728 

SDK 7.2

I found that a segment of shader code can cause performance degradation and black screen issues

// vertex shader
static std::string getVS(){
    CREATE()
    SS("#version 320 es")
    SS("uniform mat4 aMatrixV;")   
    SS("uniform vec4 std0;")      
    SS("uniform vec4 std2;")     
    SS("uniform vec4 offset;")    

    SS("layout(location = 0) in vec3 aPosition;")
 
    SS("void main(){ ")
    SS("precision highp float;")
    SS("vec4 pos  = vec4(aPosition,1.0);")
    // SS("vec4 offset = vec4(2.00776f, 1.06246924f, 1.07718694f, 0.0f);")
    SS("pos = pos + offset;")
    SS("pos = aMatrixV * pos;") 

#if ORIGINAL
    //The following calculations have been found to cause two problems
    //1. Compared to not executing the following code, 100% increases the calculation time by more than twice
    //2. In actual shaders, there is a probability of causing 'black screen'
    SS("  pos = pos / pos.w;")
#endif

#if CURRENT_TEST_OK
    //We found that having a large range of variables can cause performance issues, 
    //so we have excluded some "invalid points" when the pos range is known
    //This modification did not cause a "black screen" issue in the actual shader
    SS(" if(abs(pos.w) < 0.01f){pos.z = 10.0f; pos.w = 1.0f;}")
    SS("  pos = pos / pos.w;")
#endif

    SS("  float sx = (2.0 / (std0[2] - std0[0]));")
    SS("  float sy = (2.0 / (std0[1] - std0[3]));")
    SS("  pos.x = (pos.x - std0[0]) * sx - 1.0;")
    SS("  pos.y = (pos.y - std0[3]) * sy - 1.0;")
        
    SS("  float disp_scale_x = (std2[2] / 2.0);")
    SS("  float disp_scale_y = (std2[3] / 2.0);")
    SS("  pos.x = pos.x * disp_scale_x + (std2[0] - (-1.0 * disp_scale_x));")
    SS("  pos.y = pos.y * disp_scale_y + (std2[1] - (1.0 * disp_scale_y));")
    SS("  gl_Position = pos;}")
    return TOShaderStr()
}

There are some differences between the above code and the actual code. Currently, it has been found that the most relevant issue is  " pos = pos / pos. w "

"performance degradation" : Performing this division will double the calculation time, 

We have found that the range of pos values becomes larger after division. May I ask if this will cause such a poor performance problem and what is the reason?

 

 black screen” : Following the general process, there is a probability that the output will be completely black out

glBindFrameBuffer -> glClear() "black color" ->  draw 1 ... Draw Problem Shader  ...  draw N -> glFinish() -> glReadPix  -> output is all black

Draw Problem Shader :please refer demo/shader.h

8372.demo.zip

We have collected logs of the problem time through  : pvrdebug -loggroups main,mts,hwr

pvrlogdump_2201010044.txt.gz

Please help us find out what is the cause of the black screen. Is it effective to avoid "excessive range calculation"?

Regards

  • Add:
    The black screen problem cannot be reproduced in the demo code , but the performance problem can be reproduced.

    The pvr log is collected when problems occur in actual projects.

  • Hello Li,

    I'll get this demo setup and get back to you on the renderer active spike. It seems like a basic calculation, floating point division, that should not be causing such a large load.

    As for the black screen problem, let me send the logs for analysis. Did you get any logs in your console? Do you have those logs saved as well?

    Thanks,

    Erick

  • Hello Erick

    I'll get this demo setup and get back to you on the renderer active spike. It seems like a basic calculation, floating point division, that should not be causing such a large load.

    Yes, please take a look at our further experimental results. We still find that the value range of ”pos“ has a very large impact on subsequent computing performance. Can you please refer to the demo to see if this result can be reproduced ?

    draw 20 times per frame , run 2000 frames:

    demo code1 , Average 30ms :

    SS("  pos = pos / pos.w;")   

    ... subsequent calculations

     demo code2 , Average 130ms :

    SS("  if( abs(pos.w < 1e-2)) { pos.w = 1e-6;}")    //To enlarge the range of results

    SS("  pos = pos / pos.w;")   

    ... subsequent calculations

     demo code3 , Average 13ms :

    SS("  if( abs(pos.w < 1e-2)) { pos.w = 1.0;}")    //To narrow the value range of the result, even less more than code1

    SS("  pos = pos / pos.w;")   

    ... subsequent calculations

    As for the black screen problem, let me send the logs for analysis. Did you get any logs in your console? Do you have those logs saved as well?

    Checked stdout, syslog, glerror, there is no error output, and there is no "PVR" output.

    In addition, it was found that when collecting logs through PVRCarbon, the "black screen" problem will not occur when running 200,000 frames.

    (run 100,000  with:    export PVRCARBON_align_memory=false)

    Normally, the problem will recur when running about  10,000-20,000 frames.

    Is there a “programming guide” document similar to that for TI chips that I can refer to for this GPU?

    Regards

  • Li,

    Checked stdout, syslog, glerror, there is no error output, and there is no "PVR" output.

    In addition, it was found that when collecting logs through PVRCarbon, the "black screen" problem will not occur when running 200,000 frames.

    (run 100,000  with:    export PVRCARBON_align_memory=false)

    Normally, the problem will recur when running about  10,000-20,000 frames.

    Thank you, this is valuable information. PVRCarbon shows what the GPU sees, perhaps this can prove whether or not the GPU is involved in this issue. If there are no OpenGL errors, and no GPU driver errors, the application seems to be running fine.

    Did you test the PVRCarbon playback on the target device and see no issues with the playback?

    Is there a “programming guide” document similar to that for TI chips that I can refer to for this GPU?

    There is not since the GPU is programmed through OpenGL or Vulkan, and these are standards that have plentiful online documentation available. We are developing more documentation around the specifics of the GPU, including debugging, performance analysis and extra features that are available. But at the moment we do not have this ready.

    Regards,

    Erick

  • Hello Erick 

    The error problems displayed by playback are mainly as follows:

    I swap the eglBuffer every frame, The actual output is to shared memory, so this operation is invalid.

    Please give me an email and I will send it to you .prvcbn

    Is there any feedback from the graphics card supplier?

    Regards

  • Hello,

    One issue here is if you are using the output to shared memory, PVRCarbon cannot record the output for playback. This happens when Vision Apps / OpenVX framework is used, for example. We are looking to see if there are any ways to record from the shared memory, but currently it is not supported.

    Currently still looking into the high loading from this "pos".

    Regards,

    Erick

  • Thanks, Looking forward to it

  • Thanks, I'll keep you updated.

  • Hello

    Any progress on load issues caused by division? We pay special attention to issues that cause black screens.

  • Hello Li,

    No updates so far from the analysis team. I'll ping them to see if there's any progress.

    Thanks,

    Erick

  • Hello Li,

    Can you please run the application with the following apphint: "ShaderRecompileEnable=0"

    You can add it in the file /etc/powervr.ini:

    [default]
    ShaderRecompileEnable=0

    And see if you still get the black_screen error?

    Also, would you be able to share the PVRCarbon trace that you took?

    Regards,

    Erick

  • Hello Erick

    The environment and configuration I tried before have not been "easily" reproduced. I will try again. Should I need reboot after update file ? What is the principle of this modification?
    The trace file is as follows:

    main.out.pvrcbn.zip

    Regards

  • Hello Erick

    It's seems worked,  I'm a bit confused. Does this file configuration have a persistence effect? I changed the configuration to 0, deleting the file will not cause black screen issues to recur. Keeping the file and changing the configuration to 1 will cause black screen issues to recur very frequently

    The testing steps are as follows:

    1 reboot  
    2 ShaderRecompileEnable=1  
    3 2000 frames, Number of occurrences 1086 normal 26ms , black output 60ms
    4   1083
    5   1057
    6 ShaderRecompileEnable=0  
    7 2000 frames, Number of occurrences 0
    8   0
    9   0
    10 ShaderRecompileEnable=0  
    11 50'000 frames, Number of occurrences 0
    12 reboot  
    13 50'000 frames, Number of occurrences 0
    14 ShaderRecompileEnable=1  
    15 2000 frames, Number of occurrences 1111

    What is the specific reason for this modification? I need to use it in the production version. Please provide the details. Thanks

    Regards

  • Li,

    The option is not persistent, the driver will pick up the option when you run the program, it can't be changed while the program is running.

    Actually, the default value is 0x3FFF, this is a bitfield. Each bit corresponds to different shader types being enabled for recompilation. Setting to 0x0 masks all of the shader types for recompilation. I will check with our team what the consequences are for production, if it is needed.

    Regards,

    Erick

  • Hello Erick

    This issue is of great concern to our clients. Please explain the solution and cause of the problem as much as possible to avoid risks in the production and manufacturing process

    Regards

  • Hello Li,

    We only got the results of the experiment recently. Now we are looking on why this fixes the issue. The explanation will come soon, but we are waiting for our vendor, IMG, to review the results and the PVRCarbon trace and come back with an analysis.

    Regards,

    Erick

  • Thanks alot, looking forward

    Regards

  • Li,

    The IMG team provided me some patches, but it looks like they need to be re-worked. I'll be back soon with the updated patches that will hopefully fix the driver without the need for this app hint.

    Thanks,

    Erick

  • Thanks

    Our system configuration is as follows. Please check if the patch is correctly installed

    LINUX

    gpu version is 1.13.5776728 

    SDK 7.2

    Regards

  • Li,

    Please find attached the libraries that are updated with the patch from IMG to fix the issue you are seeing without the Apphints we used. Please apply them to your filesystem to overwrite the existing ones.

    ddk_1.13_sdk_7.2_patched_shader_failure_and_black_screen.tar.gz

    Let me know if this fixes your issue and if you run into any issues.

    Thanks,

    Erick

  • Hello Erick 

    Please check is any problem of  my test, i found ifno powervr.ini file or ShaderRecompileEnable=0 the black output problem will not appear (The problem may recur, but the probability is very low. This phenomenon has occurred since the last test.), 

    so i set  ShaderRecompileEnable=1 try again ,Problems will still occur with high probability

    reboot  
    patch on, no powervr.ini file  
    reboot  
    20‘000 frames, Number of occurrences 0
    add powervr.ini,  ShaderRecompileEnable=1  
    2000 frames, Number of occurrences 1059
    add powervr.ini,  ShaderRecompileEnable=0 0

    Regards

  • Hello Li,

    i found ifno powervr.ini file

    You are not suppose to use the powervr.ini file anymore. Please don't set any values. It looks like the issue has gone away if you do not touch the powervr.ini file, correct?

    Also, recall my message that you should not set the ShaderRecompileEnable=1 because it is a bitfield, it needs to be either "0" or "0x3FFF":

    Actually, the default value is 0x3FFF, this is a bitfield. Each bit corresponds to different shader types being enabled for recompilation. Setting to 0x0 masks all of the shader types for recompilation.

    Regards,

    Erick

  • You are not suppose to use the powervr.ini file anymore. Please don't set any values. It looks like the issue has gone away if you do not touch the powervr.ini file, correct?

    No patch on,can be replicated, with a relatively low probability

    I will retest it in the actual product environment

    Regards

  • Li,

    Ok, if it can still be recreated than something is still missing. Can you share a similar PVRCarbon as last time?

    Regards,

    Erick

  • Hello Erick 

    1 reboot  
    2 patch on  
    3 reboot  
    4 2000 frames, Number of occurrences >1000 Repeated many times before reappearing
    7

    Preload PVRCabon related so

    LD_PRELOAD="/opt/cache/libPVRCarbon.so:....

     
    8 2000 frames, Number of occurrences 1044 Repeated many times before reappearing
    9 get main.out.pvrcbn  

    1385.main.out.pvrcbn.zip

    please check new pvrcbn

    Regards

  • Thank you Li,

    I don't understand the table clearly, does this mean that the PVRCarbon trace was captured once the issue re-appeared?

    I'll forward it for analysis.

    Thanks,

    Erick

  • Hello Erick

    Yes, the problem is reproduced, both after ”installing the patch”  and  “ installing the patch  then recording using PVRCabon”.

    Let me describe it:
    After installing the patch, I tested it several times and the problem recurred. Use PRE_LOAD mode to run PVRCarbon and test it many times. The problem reappears. The trace file is recorded when it recurs.

    I enabled PVRCarbon through the PRELOAD method before, and the problem did not recur. Maybe it is the impact of the patch this time?

    In addition, it was found that when collecting logs through PVRCarbon, the "black screen" problem will not occur when running 200,000 frames.

    Regards

  • Hello,

    Ok, the problem still occurring, the team is looking into what else could be causing the issue.

    Can you clarify what you mean by this:

    Use PRE_LOAD mode to run PVRCarbon and test it many times.

    What is the PRE_LOAD? Is it a setting in PVRCarbon?

    Thanks,

    Erick

  • Hello

    I follow the setup :

    The patch contains some "so" such as libegl.so , whitch are the same "so" required to run PVRCarbon , it's OK?

    Regrads

  • The patch contains some "so" such as libegl.so , whitch are the same "so" required to run PVRCarbon , it's OK?

    Yes, that is fine because PVRCarbon will pick up it's own libraries first, and these libraries will point to the system libraries to load the actual functions. It just wraps around them, so you need both the PVRCarbon .so files and the system .so files. The patch I provided only modifies them.

    Ok, thanks for the info.

    Regards,

    Erick

  • Hello Erick

    Can this solution 'add powervr. ini, ShaderRecompileEnable=0' be used for mass production?

    Regarss

  • Li,

    Can this solution 'add powervr. ini, ShaderRecompileEnable=0' be used for mass production?

    It would not be recommended since it's avoiding an issue within the driver.

    I have received an updated patch, it includes the previous patch but adds one more patch. Can you please give these new libraries a try and let me know if it resolves the issue or if it is still present?

    3386.ddk_1.13_sdk_7.2_patched_shader_failure_and_black_screen.tar.gz

    Thanks,

    Erick

  • Hello 

    It's not work,

    In the code I provided, changing “offset” to below replicates more frequently

    8372.demo.zip

    A: 1.99219108,1.05947781,1.0851475, 0,0

    B: 1.9898001, -1.02998471, 1.09057605,0.0

    In actual code, A input will be rendered once per frame, and B input will be rendered again

    In addition I reproduce the method of repeatedly starting and stopping unit testing programs, which has a relatively high probability

    Regards

  • Li,

    Thanks for the results. Can you explain what the columns mean? I'm not sure what the four 20,000 frame columns with zero mean? And what the two numbers in the last two columns mean 0 and >5000.

    A: 1.99219108,1.05947781,1.0851475, 0,0

    B: 1.9898001, -1.02998471, 1.09057605,0.0

    In actual code, A input will be rendered once per frame, and B input will be rendered again

    In addition I reproduce the method of repeatedly starting and stopping unit testing programs, which has a relatively high probability

    Understood, you have methods to reproduce it more quickly. Do you have a video of this failure, just as extra information for our team?

    And did you capture a PVRCarbon this time when you ran it?

    Regards,

    Erick

  • Hello Erick

    powervr.ini  This configuration still has a recurring issue today

  • Li,

    Earlier, you mentioned the fix was working, has further testing showed that it is not working with the default drivers?

    We've confirmed with the GPU vendor that the powervr.ini change is an optimization, so disabling it by setting ShaderRecompileEnable=0 is simply going to reduce the performance and can be used in production.

    Can you please confirm that using the "ShaderRecompileEnable=0" will not work in production for you with the default GPU drivers (not the new libraries I provided)?

    There's a more specific setting that you can use:

    ShaderRecompileEnable=0xFFDFFFFF

    This will only disable the particular shader that you are probably seeing the issue with. But if ShaderRecompileEnable=0 does not work, then this setting won't work either.

    Regards,

    Erick

  • Hi Erick

    They are using the clear UM with appHints but failed at customer vehicle.

    Regards

    Zekun

  • Zekun,

    Ok, so originally, it looked like it fixed the issue, but now it does not?

    Can you please try using the following:

    1) The latest patched UM libs

    3386.ddk_1.13_sdk_7.2_patched_shader_failure_and_black_screen.tar.gz

    2) The powervr.ini AppHint as follows:

    [default]
    ShaderRecompileEnable=0xFFDFFFFF

    This should fix the issue, but if it does not, then we need to get feedback from IMG.

    Thanks,

    Erick

  • Hello Erick 

    Sorry for late

    Can you explain what the columns mean? I'm not sure what the four 20,000 frame columns with zero mean?

    These columns represent the number of times x frames are executed and a black screen appears under different inputs

    I will provide a binary program for testing videos and additional reproducible issues to Zekun

    Can you please try using the following:

    OK,I will start testing today

    Regards

  • Thanks Li, Zekun has provided these and we are working on using them for our debug.

    Thanks,

    Erick

  • Li,

    For completeness I'm attaching the latest patched libraries for your testing:

    ddk_1.13_sdk_7.2_patched_shader_failure_and_black_screen_2.tar.gz

    Thanks,

    Erick