DRA756: Declink_h264DecodeFrame takes more then 100ms(top) sometime

RamPrasad, thanks for your advise which is very valuable.

result of "omapconf show opp" command shows:

OMAPCONF (rev v1.73-17-g578778b built Thu Dec 28 05:15:12 IST 2017)

HW Platform:
Generic DRA74X (Flattened Device Tree)
DRA75X ES2.0 GP Device (STANDARD performance (1.0GHz))
TPS659038 ES2.2

SW Build Details:
Build:
release_details_get(): could not open /etc/issue.net file?!
Version: UNKNOWN
Kernel:
Version: 4.4.84+
Author: ******
Toolchain: gcc version 5.3.1 20160113 (Linaro GCC 5.3-2016.02)
Type: #1 SMP PREEMPT
Date: Fri Nov 29 13:17:40 CST 2019

                                                | Temperature | Voltage    |   Frequency      | OPerating Point

VDD_CORE / VDD_CORE0   | 71C / 159F  | 1.060 V     |                          | NOM             |

| L3                                           |                   |                   |  266 MHz          |                           |
| DMM                                      |                   |                     | 266 MHz          |                   |
| EMIF1                                     |                   |                   | 266 MHz           |                  |
| EMIF2                                     |                   |                   | 266 MHz           |                  |
| LP-DDR2                               |                   |                   | 532 MHz             |                   |
| L4                                          |                   |                   | 266 MHz             |                   |
| IPU1                                      |                   |                   | 425 MHz             |                   |
| Cortex-M4 Cores                   |                   |                   | 212 MHz             |                   |
| IPU2                                      |                   |                   | 425 MHz              |                   |
| Cortex-M4 Cores                   |                   |                   | 212 MHz              |                   |
| DSS                                       |                   |                   | 192 MHz              |                   |
| BB2D                                     |                   |                   | (2128 MHz) (1)    |                   |

| VDD_MPU / VDD_CORE1   | 72C / 161F | 1.100 V        |                          |        NOM |
| MPU (CPU1 ON)                   |                     |                     | 1000 MHz       |           |

| VDD_GPU / VDD_CORE2    | 72C / 161F  |        1.060 V |                      |           NOM |
| GPU                                       |                  |                        |           425 MHz |           |

| VDD_DSPEVE / VDD_CORE3 | 70C / 158F | 1.060 V     |                              | OVERDRIVE |
| DSP1                                   |                    |                     | 700 MHz           |           |
| DSP2                                   |                     |                     | 700 MHz           |           |
| EVE1                                   |                     |                     | (0 MHz) (1)        |           |
| EVE2                                   |                      |                     | (0 MHz) (1)        |           |
 
| VDD_IVA / VDD_CORE4 | 72C / 161F | 1.060 V           |                            | NOM    |
| IVA                                      |                       |                     | 388 MHz        |                         |

does IVA 388 MHz  mean IVAHD here? if so, how can i change it to 532MHz?

the EMIF1/2 is 266 MHz which is half of 533Mhz. can i trust this result? 

based on our usecase, please advance how can i optimize our system, both in clocks and prioritys of the basic modules of each link/chain.

thanks in advance.

Hi Wen,

Which version of visionSDK is used here?

I am seeing all Operating points at NOM and it should have been HIGH. 

omapconf shows the device is DRA74x which means IVA's OPP HIGH is 532MHz .

Thanks

RamPrasad

hi, RamPrasad

we are using vision_sdk_3.2.

a quick search of IVA NOM in the fourm, i found

https://e2e.ti.com/support/processors/f/791/p/549323/2006872?tisearch=e2e-sitesearch&keymatch=IVA%25252525252525252520nom#2006872

I looked into u-boot implementation in PSDKLA 3.0 and found following configuration in file “/u-boot/configs/dra7xx_evm_defconfig”

CONFIG_DRA7_IVA_OPP_HIGH=y

after modification the config file, the Operating points of IVA is HIGH now:

| VDD_IVA / VDD_CORE4            |    80C / 176F           |        1.060 V       |                   | HIGH |
| IVA                                               |                                 |                            | 532 MHz  |          |

the result of decode time tasks about 70ms(top) but i think it maybe a tiny difference between the old one.

any ideas?

btw, the EMIF1/2 is 266 MHz which is half of 533Mhz. can i trust this result? 

thanks in advance.

hi, RamPrasad

any update?

Hi Wen,

I have VisionSDK 3.06 setup and I am seeing attached opp information. 

In your case no IP is set for HIGH .

Can you check what this command shows from u-boot source code?

cd ti_components/os_tools/linux/u-boot/u-boot

cat .config | grep IVA

My observation is 

# CONFIG_DRA7_IVA_OPP_NOM is not set
# CONFIG_DRA7_IVA_OPP_OD is not set
CONFIG_DRA7_IVA_OPP_HIGH=y

Thanks

RamPrasad

hi, RamPrasad

i have modified the uboot config  file in the previous post:

boot/configs/dra7xx_evm_defconfig

CONFIG_DRA7_IVA_OPP_HIGH=y

and  the  result  of iva is high.

but  this doesnot affect decode performance.

here is what i got from .config:

cat .config| grep IVA
# CONFIG_DRA7_IVA_OPP_NOM is not set
# CONFIG_DRA7_IVA_OPP_OD is not set
CONFIG_DRA7_IVA_OPP_HIGH=y
CONFIG_HAVE_PRIVATE_LIBGCC=y
# CONFIG_USE_PRIVATE_LIBGCC is not set

Regards, wen

hi, RamPrasad

any update about this issue?

Hi Wen,

I would recommend to use OPP_HIGH for DSP and GPU also and then check the behavior. Default u-boot configuration doesn't set the OPP to NOM, so not sure why your configuration sets to NOM or OD for all IPs. 

or  please try bto use u-boot from visionSDK 3.07, it is the latest visionSDK 

Please try with these commands one by one and check the behavior. These commands set the priorities of IPU2 and IVAHD to 1.

omapconf write 0x4e000624 0x09000000
omapconf write 0x4e00062C 0x00000090

Please refer section 4.1.1 of http://www.ti.com/lit/an/sprabx1a/sprabx1a.pdf on setting priorities 

Please also refer section 4.2 of this doc 

Thanks

RamPrasad

hi, RamPrasad

i have GPU DSP1&2 IVA running at HIGH, the priorities of IPU2&1 and IVAHD set to 1, but the result is a tiny difference between the very old one.

65119us is observed after the above modifactions.

~ # omapconf show opp
OMAPCONF (rev v1.73-17-g578778b built Thu Dec 28 05:15:12 IST 2017)

HW Platform:
  Generic DRA74X (Flattened Device Tree)
  DRA75X ES2.0 GP Device (STANDARD performance (1.0GHz))
  TPS659038  ES2.2 

SW Build Details:
  Build:
release_details_get(): could not open /etc/issue.net file?!
    Version: UNKNOWN
  Kernel:
    Version: 4.4.84+
    Author: yang_yong@ubuntu
    Toolchain: gcc version 5.3.1 20160113 (Linaro GCC 5.3-2016.02)
    Type: #1 SMP PREEMPT
    Date: Fri Nov 29 13:17:40 CST 2019

|-----------------------------------------------------------------------------------|
|                        | Temperature | Voltage | Frequency      | OPerating Point |
|-----------------------------------------------------------------------------------|
| VDD_CORE / VDD_CORE0   | 69C / 156F  | 1.060 V |                | NOM             |
|   L3                   |             |         |  266  MHz      |                 |
|   DMM                  |             |         |  266  MHz      |                 |
|   EMIF1                |             |         |  266  MHz      |                 |
|   EMIF2                |             |         |  266  MHz      |                 |
|     LP-DDR2            |             |         |  532  MHz      |                 |
|   L4                   |             |         |  266  MHz      |                 |
|   IPU1                 |             |         |  425  MHz      |                 |
|     Cortex-M4 Cores    |             |         |  212  MHz      |                 |
|   IPU2                 |             |         |  425  MHz      |                 |
|     Cortex-M4 Cores    |             |         |  212  MHz      |                 |
|   DSS                  |             |         |  192  MHz      |                 |
|   BB2D                 |             |         | (2128 MHz) (1) |                 |
|                        |             |         |                |                 |
| VDD_MPU / VDD_CORE1    | 71C / 159F  | 1.100 V |                | NOM             |
|   MPU (CPU1 ON)        |             |         |  1000 MHz      |                 |
|                        |             |         |                |                 |
| VDD_GPU / VDD_CORE2    | 70C / 158F  | 1.060 V |                | HIGH            |
|   GPU                  |             |         |  532  MHz      |                 |
|                        |             |         |                |                 |
| VDD_DSPEVE / VDD_CORE3 | 69C / 156F  | 1.060 V |                | HIGH            |
|   DSP1                 |             |         |  750  MHz      |                 |
|   DSP2                 |             |         |  750  MHz      |                 |
|   EVE1                 |             |         | (0    MHz) (1) |                 |
|   EVE2                 |             |         | (0    MHz) (1) |                 |
|                        |             |         |                |                 |
| VDD_IVA / VDD_CORE4    | 71C / 159F  | 1.060 V |                | HIGH            |
|   IVA                  |             |         |  532  MHz      |                 |
|                        |             |         |                |                 |
|-----------------------------------------------------------------------------------|

Notes:
  (1) Module is disabled, rate may not be relevant.

~ # omapconf write 0x4e000624 0x09000000
~ # omapconf write 0x4e00062C 0x00000090
~ # omapconf write 0x4e00062C 0x00000099

i will try to modify the Bandwidth Regulator and let you know the result.

hi, RamPrasad

i have add BWRegulator for iva, but the system just failed to start, the log read as:

[HOST] [IPU1-0] 72634.231867 s: CHAINS: Application Started !!!
[HOST] [IPU1-0] 72634.235161 s: CHAINS: Sv + Adas Sysbios Started !!!
[HOST] [IPU1-0] 72634.235283 s: AppCtrl_setSystemL3DmmPri!
[HOST] [IPU1-0] 72634.235679 s:
[HOST] [IPU1-0] 72634.235710 s: ### XDC ASSERT - ERROR CALLBACK START ###
[HOST] [IPU1-0] 72634.235801 s:
[HOST] [IPU1-0] 72634.235923 s: E_hardFault: FORCED
[HOST] [IPU1-0] 72634.235984 s:
[HOST] [IPU1-0] 72634.236045 s: ### XDC ASSERT - ERROR CALLBACK END ###
[HOST] [IPU1-0] 72634.236106 s:
[HOST] [IPU1-0] 72634.236808 s:
[HOST] [IPU1-0] 72634.236838 s: ### XDC ASSERT - ERROR CALLBACK START ###
[HOST] [IPU1-0] 72634.236899 s:
[HOST] [IPU1-0] 72634.237052 s: E_busFault: IMPRECISERR: Delayed Bus Fault, exact addr unknown, address: e000ed38
[HOST] [IPU1-0] 72634.237143 s:
[HOST] [IPU1-0] 72634.237204 s: ### XDC ASSERT - ERROR CALLBACK END ###
[HOST] [IPU1-0] 72634.237265 s:
[HOST] [HOST ] 72637.616665 s: SYSTEM: System A15 Init in progress !!!
[HOST] [HOST ] 72637.616726 s: SYSTEM: Waiting DSP1 Init SUCCESS !!!
[HOST] [HOST ] 72637.616726 s: SYSTEM: Waiting DSP2 Init SUCCESS !!!
[HOST] [HOST ] 72637.616757 s: SYSTEM: Waiting IPU2 Init SUCCESS !!!
[HOST] [HOST ] 72639.812450 s: SYSTEM: Waiting to [IPU1-0] ... state = 65537
[HOST] [HOST ] 72642.007442 s: SYSTEM: Waiting to [IPU1-0] ... state = 65537
[HOST] [HOST ] 72644.173793 s: SYSTEM: Waiting to [IPU1-0] ... state = 65537
[HOST] [HOST ] 72646.358903 s: SYSTEM: Waiting to [IPU1-0] ... state = 65537

what i have done in thet souce code:

Chains_main:

Vps_printf(" CHAINS: Sv + Adas Sysbios Started !!!");

gChains_usecaseCfg.numLvdsCh = 7;

Vps_printf(" AppCtrl_setSystemL3DmmPri!");
AppCtrl_setSystemL3DmmPri();                                     //add here

//...

Void AppCtrl_setSystemL3DmmPri()
{
/* Assert Mflag of DSS to give DSS highest priority */
Utils_setDssMflagMode(UTILS_DSS_MFLAG_MODE_FORCE_ENABLE);

/* enable usage of Mflag at DMM */
Utils_setDmmMflagEmergencyEnable(TRUE);

/* Set DMM as higest priority at DMM and EMIF */
Utils_setDmmPri(UTILS_DMM_INITIATOR_ID_DSS, 0);

Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P1, 1000);
Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P2, 1000);

Utils_setBWRegulator(UTILS_DMM_INITIATOR_ID_IVA, 1000);                                  //add here
}

please advise me how to do it.

Hi Wen,

I suggest you to apply these changes using omapconf write command before strating the usecase

hi, RamPrasad

it works after a little modification: 

Chains_main：

Utils_setAppInitState(SYSTEM_IPU_PROC_PRIMARY, CORE_APP_INITSTATUS_IPU_PRIMARY_ALL_DONE);

status = System_linkControl(SYSTEM_LINK_ID_APP_CTRL,
APP_CTRL_LINK_CMD_SET_DMM_PRIORITIES,
NULL,
0,
TRUE);

Vps_printf("SYSTEM_LINK_ID_APP_CTRL %d|", status);

#ifdef IPU_PRIMARY_CORE_IPU1
IPU1_0_main(NULL, NULL);

status comes out 0 so it's sucecssed, there are no differrence about  performance compare to the very old one.

regards, wen.

hi, RamPrasad

in the first post, the SCI_MA_MPU_P2 is using to much emif bw:

SCI_MA_MPU_P2 | 220.883907 1117.839188

but i dont think we can use dmm to control the mpu with emif request(the device datasheet has a Memory Subsystem Overview which tells mpu is request emif directly with out DMM)

it decode performance is related to the peek data of SCI_MA_MPU_P2 in Statistics Collector, the very last thing we can do is set the priority, am i right?

regards, wen

hi, RamPrasad.

i need to say sorry that i have send the wrong data at the first post(the decode is not working when the Statistics Collector is running), 

please analyze this result instead:

Statistics Collector,

STATISTIC Avg Data Peak Data
COLLECTOR MB/s MB/s
--------------------------------------------------
SCI_EMIF1 RD+WR | 566.365281 3090.439422
SCI_EMIF2 RD+WR | 823.858270 2155.900998
SCI_EMIF1 RD ONLY | 309.901695 1671.054464
SCI_EMIF1 WR ONLY | 260.515843 1420.081511
SCI_EMIF2 RD ONLY | 527.971667 3128.853649
SCI_EMIF2 WR ONLY | 296.546599 1424.560207
SCI_MA_MPU_P1 | 71.570719 302.704705
SCI_MA_MPU_P2 | 183.343419 850.119303
SCI_DSS | 401.901217 437.338080
SCI_IPU1 | 33.107404 42.318894
SCI_VIP1_P1 | 39.070339 51.558000
SCI_VIP1_P2 | 57.687818 65.021202
SCI_VPE_P1 | 136.401427 436.873507
SCI_VPE_P2 | 136.411739 437.003035
SCI_DSP1_MDMA | 5.430534 5.737301
SCI_DSP1_EDMA | 0.000000 0.000000
SCI_DSP2_MDMA | 5.564167 5.989887
SCI_DSP2_EDMA | 0.000000 0.000000
SCI_EVE1_TC0 | 0.000000 0.000000
SCI_EVE1_TC1 | 0.000000 0.000000
SCI_EVE2_TC0 | 0.000000 0.000000
SCI_EVE2_TC1 | 0.000000 0.000000
SCI_EDMA_TC0_RD | 0.003822 0.065946
SCI_EDMA_TC0_WR | 0.003823 0.065946
SCI_EDMA_TC1_RD | 0.000000 0.000000
SCI_EDMA_TC1_WR | 0.000000 0.000000
SCI_VIP2_P1 | 9.483616 43.096384
SCI_VIP2_P2 | 21.902342 98.974537
SCI_VIP3_P1 | 27.107120 153.236135
SCI_VIP3_P2 | 34.161385 211.268457
SCI_EVE3_TC0 | 0.000000 0.000000
SCI_EVE3_TC1 | 0.000000 0.000000
SCI_EVE4_TC0 | 0.000000 0.000000
SCI_EVE4_TC1 | 0.000000 0.000000
SCI_IVA | 200.005473 1374.099988
SCI_GPU_P1 | 118.212548 1725.597023
SCI_GPU_P2 | 118.672237 1732.792698
SCI_GMAC_SW | 0.002630 0.174879
SCI_OCMC_RAM1 | 0.000000 0.000000
SCI_OCMC_RAM2 | 0.000000 0.000000
SCI_OCMC_RAM3 | 0.000000 0.000000

 i cannot simply understand why GPU_P1&2 are so much high: 1725MB/S. after a little dig in the forum, i found:

GPU is being limited to 1.3GBps with these two instructions.

Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P1, 1000);
Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P2, 1000);

WHY?

attachment for the details infomactions.Statistics Collector.txt

hi, RamPrasad

any update here ?

Hi Wen,

Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P1, 1000);
Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P2, 1000);

were added to limit GPU BW for a specific usecase. You can comment these two and allow GPU to go beyond 1GBPS.

Can you try applying following settings one by one?

This is to increase EMIF priority. This has worked for similar issue.

1. No change of EMIF system port priority, increase priorities of IPU2, IVA and VPE
omapconf write 0x4e000624 0x0A000000
omapconf write 0x4e00062C 0x000000A0
omapconf write 0x4e000630 0xA0000000

2. No change of EMIF system port priority, increase priorities of IPU2, IVA, VPE, GPU and BB2D
omapconf write 0x4e000624 0x0A000000
omapconf write 0x4e00062C 0x000000A0
omapconf write 0x4e000630 0xA0000000
omapconf write 0x4e000634 0x00000BBB

3. Increase EMIF system port priority, increase priorities of IPU2, IVA and VPE
omapconf write 0x40D00040 0xC0000000
omapconf write 0x4e000624 0x0A000000
omapconf write 0x4e00062C 0x000000A0
omapconf write 0x4e000630 0xA0000000

4. Increase EMIF system port priority, increase priorities of IPU2, IVA, VPE, GPU and BB2D
omapconf write 0x40D00040 0xC0000000
omapconf write 0x4e000624 0x0A000000
omapconf write 0x4e00062C 0x000000A0
omapconf write 0x4e000630 0xA0000000
omapconf write 0x4e000634 0x00000BBB

Thanks

RamPrasad

hi, RamPrasad

i have done this by a debug command:

Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P1, 1000);
Utils_setBWLimiter(UTILS_DMM_INITIATOR_ID_GPU_P2, 1000);

Utils_setBWRegulator(UTILS_DMM_INITIATOR_ID_IVA, 800);

but i can still observe that the IVA GPU go up to ~2GiB

SCI_IVA | 204.034240 2003.259853
SCI_GPU_P1 | 140.557736 2242.502267
SCI_GPU_P2 | 141.078969 2248.949424

i will update you later with above commands.

hi, RamPrasad

this is what i was done when test your script:

1. start the system, the usercase is running

2. run the script you post(on-line)

3. check the log and summarized, reboot the system

this is what i got:

1. 66065us was observed after a short test.

2. 66065us was observed after a short test.

3. 73385us was observed after a short test.

3. 66065us was observed after a short test.

please tell me if anything is wrong.

regards, wen

hi, RamPrasad

any ideas ?

regards, wen

hi, RamPrasad

it seems that the cpu loading is too high:

LOAD: CPU: 95.4% HWI: 12.3%, SWI:26.1%

LOAD: TSK: SYSTEM_MSGQ : 4.3%
LOAD: TSK: DUP0 : 3.1%
LOAD: TSK: DUP1 : 1.8%
LOAD: TSK: DUP2 : 0.4%
LOAD: TSK: DUP3 : 0.3%
LOAD: TSK: IPC_OUT_0 : 3.0%
LOAD: TSK: IPC_OUT_2 : 2.6%
LOAD: TSK: IPC_OUT_3 : 4.7%
LOAD: TSK: IPC_OUT_4 : 0.9%
LOAD: TSK: IPC_OUT_5 : 1.0%
LOAD: TSK: IPC_OUT_6 : 0.9%
LOAD: TSK: MERGE0 : 2.2%
LOAD: TSK: MERGE2 : 2.0%
LOAD: TSK: MERGE3 : 1.6%
LOAD: TSK: NULL_SRC2 : 0.2%
LOAD: TSK: SELECT0 : 1.6%
LOAD: TSK: SYNC0 : 1.8%
LOAD: TSK: SYNC1 : 1.5%
LOAD: TSK: DISPLAY0 : 1.1%
LOAD: TSK: ENC0 : 2.9%
LOAD: TSK: DEC0 : 0.7%
LOAD: TSK: CAPTURE : 2.1%
LOAD: TSK: VPE0 : 4.9%
LOAD: TSK: SWOSD_0 : 5.8%
LOAD: TSK: DEC_PROCESS_TSK_0 : 1.5%
LOAD: TSK: ENC_PROCESS_TSK_0 : 1.9%
LOAD: TSK: STAT_COLL : 0.6%
LOAD: TSK: MISC : 1.6%

TSK:

4.3+3.1+1.8+0.4+0.3+3.0+2.6+4.7 +0.9 +1.0 +0.9 +2.2 +2.0 +1.6 +0.1 +1.6 +1.8 +1.5 +1.1 +2.9 +0.7 +2.1 +4.9 +5.8 +1.5 +1.9 +0.6 +1.7 = 57

HWI+SWI+TSK:

12.3 + 26.0 + 57 = 95.3

any ideas? can we cut done the SWI?

hi, RamPrasad,

i have optimize the system and the LOAD is : CPU: 78.3% HWI: 10.9%, SWI:20.3%

but i can still observe 40ms sometime.

please advise!!

regards, wen

hi, RamPrasad,

hi, RamPrasad,

Hi Wen,

Do you see this issue of decoder taking >40ms with default use case of NullSrc->Decode->VPE->Display usecase? or only with your complex chain-link usecase?

I haven't seen the issue for 1920x1080 streams also. 

Is there a way to bypass encode part in your link and test the usecase and get the decode time?

As there is only IVAHD, encoding and decoding will happen in sequence, so decoding time may be involving waiting time for IVAHD. To confirm this please check if you can bypass encode.

Thanks

RamPrasad

hi, RamPrasad,

it's good to hear from you.

decoder taking >40ms is observed only with my complex chain-link usecas.

by the way, i have optimized the cpu load to 70% and the decoder take about 10-30ms most of frames(not every frame).

i have confimed that the encoder is disabled when decode h264 which takes 10-40ms each frame.

this is what i have tried:

1. trid decode h264 frm which cames from j6 h264enc, but nothing changed.

hi, RamPrasad,

sorry for the long-delay. i have successfully created a simple use case that you meantioned in the previous post:

UseCase: chains_decVpeDisplay(the VPE link drops all the data from the previous link)

NullSource_dec -> Decode -> VPE -> IPCOut_Capture

after a little while, i found that the cose is <8113us which is quite good.

what should i do to shot the reason why i got big cose with the complex use case ?

this post has also told about DDR bandwidth when vpe is in de-interlacing mode,should i check it?