HEVC encoder hangs randomly

Andrey, please confirm us the Encoder version you are using (you posted C66x_h265vdec_01_00_00_29) thanks

Paula

Hi Kuladeepak,

Problem can be not only in HEVC encoder but in our code (callbacks for example) or any kind of memory corruption that leads to unexpected results.

I don't know how to access "*pLCU_completed" array. Can you give more details about that?

Regards,
Andrey Lisnevich

Hi shashikantha srinivas,

The memory dump: https://drive.google.com/folderview?id=0B7aam1m6VieebDNCMkNaSkhrbnc&usp=sharing

(Allocated under "IVIDMC3_SHMEM_ATTRS_RMT_UNCACHED_LOC_SL2" when it has hung)

Hi Andrey,

We were able to get the cause for the hang issue. It seems two cores are executing a critical code which is inside the lock simultaneously. ie two cores are getting a single lock. 

Can you please review your lock acquire and release codes and fix the issue? Unless the critical section is executed exclusively by a single core, this hang issue cannot be fixed.

Thanks and Regards,

Shashikantha

Hi Shashikantha,

I do not see problems in lock routines.

This is the test:

#pragma DATA_SECTION(lock, ".dd3uncached")
#pragma DATA_ALIGN(lock, 8)
static int lock = -1;

#pragma DATA_SECTION(lockCounter1, ".dd3uncached")
#pragma DATA_ALIGN(lockCounter1, 8)
static int lockCounter1 = 0;

#pragma DATA_SECTION(lockCounter2, ".dd3uncached")
#pragma DATA_ALIGN(lockCounter2, 8)
static int lockCounter2 = 0;

static void lockCheckTask(UArg arg0, UArg arg1)
{
    int core = DNUM;
    System_printf("LOCK CHECK TASK\n");
    System_flush();

    int i;
    for (i = 0; i < 10000; i++) {
        TranscodeComponent_lockAcquire(core, &lock);
        lockCounter1++;
        lockCounter2++;
        TranscodeComponent_lockRelease(core, &lock);
    }

    System_printf("core %d lockCounter1=%d lockCounter2=%d\n", core, lockCounter1, lockCounter2);
    System_flush();
}

And this is results from all 8 cores:

[2015-01-22 14:30:51] core 0 lockCounter1=44147 lockCounter2=44147
[2015-01-22 14:30:51] core 1 lockCounter1=77499 lockCounter2=77499
[2015-01-22 14:30:51] core 2 lockCounter1=77857 lockCounter2=77857
[2015-01-22 14:30:51] core 3 lockCounter1=78400 lockCounter2=78400
[2015-01-22 14:30:51] core 4 lockCounter1=79118 lockCounter2=79118
[2015-01-22 14:30:51] core 5 lockCounter1=79587 lockCounter2=79587
[2015-01-22 14:30:51] core 6 lockCounter1=79778 lockCounter2=79778
[2015-01-22 14:30:51] core 7 lockCounter1=80000 lockCounter2=80000

From other hand I see problems in the way HEVC codec calls lock callbacks. I added additional asserts to lock release function:

int32_t TranscodeComponent_lockRelease(int32_t user_id, void* lockHandle)
{
    assert(lockHandle != NULL);

    int_fast32_t* lockUserId = (int_fast32_t*) lockHandle;

    _nassert((int) lockUserId % 8 == 0);

    GateMP_Handle gate = GlobalGates_getTranscodeComponentGate();
    IArg gateKey = GateMP_enter(gate);

    if (*lockUserId == user_id) {
        *lockUserId = -1;
        GateMP_leave(gate, gateKey);
        return IVIDMC3_LOCK_ERROR_NONE;
    } else if (*lockUserId == -1) {
       GateMP_leave(gate, gateKey);
       assert(0);
       return IVIDMC3_LOCK_ERROR_DBL_UNLOCK;
    } else {
       GateMP_leave(gate, gateKey);
       assert(0);
       return IVIDMC3_LOCK_ERROR_USER_ID;
    }
}

It fails on assert near IVIDMC3_LOCK_ERROR_USER_ID. It means that lock release was called on lock acquired by different user.

Regards,

Andrey Lisnevich

Hi Andrey,

Is the hang issue resolved after your lock implementation change?

For the mismatch issue, can you remove the threshold completely and always do cache writeback or invalidate only for the region passed to the API, and see whether the mismatch issue is resolved?

Thanks and Regards,

Shashikantha

Hi Andrey,

Andrey Lisnevich said:

I get different output each execution while input is the same. I believe it is not Ok

Yes the output should always bitmatch for the same input for each execution.

To narrow down the issue to cache sync, can you please turn off the cache altogether and verify if the output is same for concurrent runs?

Thanks and Regards,
Shashikantha

Andrey,

By disabling cache, I meant disabling the caching itself. ie in function initializeCaching() set size of L1P, L1D, and L2 to 0. Also use Cache_Mar_DISABLE when calling the Cache_setMar(). Restart the board before running with these changes.

This ensures that no data is cached. If the outputs match after these change, then the problem is related to cache coherency, otherwise the problem lies somewhere else.

Thanks and Regards,
Shashikantha