This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

DM6446 Preview Engine hangs the whole device.

Hi TI team,

 

We are facing a problem running Preview Engine for long time. We developed a video phone with H264 codec. This H264 codec comes with TI DVSDK2.0.

The DM6446 device hangs(There is no response in UART terminal, no response to external ping to DM6446) after running for a while. Sometime the process goes to D-state and it never recovers. The only solution is to reboot the device at that point.


We have discovered a quick way to reproduce this problem. This bug is 100% reproducible.

How to reproduce

1. Keep running preview engine with a input and output buffer continuously on DM6446

2. Keep sending UDP packet from any PC to that DM6446 board as fast as possible.

3. The DM6446 board will hang after a while. The duration to get frozen may takes less than 10 minutes.


Linux kernel code we took from http://software-dl.ti.com/dsps/dsps_public_sw/sdo_sb/targetcontent/dvsdk/DVSDK_2_00/latest//exports/mvl_5_0_0_demo_lsp_setuplinux_02_00_00_140.bin

And the kernel config file is uploaded at http://www.4shared.com/file/KlccP9Qt/config.html

The only change we did in kernel code of davinci_previewer_hw.c file is following because we are feeding input and output buffer from DDR RAM and running is single shot mode.

    /* Set previewer source to DDRAM */
    SETBIT(pcr, PREV_SOURCE_BIT);
//    RESETBIT(pcr, PREV_SOURCE_BIT);

    /* Set one shot mode */
    SETBIT(pcr, PREV_ONESHOT_BIT);
//    RESETBIT(pcr, PREV_ONESHOT_BIT);


Our preview application test code is uploaded in the following link in case if you are interested.

Preview application code: http://www.4shared.com/file/HfVRh8f7/preview_example.html

After downloading the file it needs to be extracted with following command.

tar zxvf preview_example.tgz


Also the code for sending UDP from PC to DM6446 is uploaded at http://www.4shared.com/file/MTX_rBQT/udp_send.html in case if you are interested.

After downloading the file it needs to be extracted with following command.
tar zxvf udp_send.tgz

Let me know if you need any other information to narrow down this bug.

Thank you

--Manjur

  • Could you provide some more details, about the hang. You mention that no repsonse on UART. Can you enable logs to see the last known state and also some clues as to which software module causes the hang.

    Regards
    Varada

  • Hi Varada,

    Thanks for the response. I'm working on this case now.  The test utility for previewer is mainly just an endless loop to call ioctl PREV_PREVIEW function:

    while(1) {
            /* Call PREV_PREVIEW ioctl to do previewing */
            ret = ioctl(prevfd, PREV_PREVIEW, &convert);
    }

    while the udp_send test utility is just a generic program to generate UDP traffic.  When it happens, we lost control to the unit.  Our initial through is there might be some resource conflict in the kernel,  as it happens only when both test utilities are running at high speed.

    Please let us know if you need more information.

    David

  • I put some printk into kernel davinci_previewer.c previewer_ioctl() function:

        case PREV_PREVIEW:
            /* call preview function to do preview */
            if (copy_from_user(&conv, (struct prev_convert *)arg,
                       sizeof(struct prev_convert)))
                return -EFAULT;

            printk("before down\n");
            down_interruptible(&(device->sem));
            printk("after down\n");
            ret = preview(device, &conv);
            printk("before up\n");
            up(&(device->sem));
            printk("after up\n");
            break;

    When the problem happens, it randomly stops at any position before or after down_interruptible()/up() functions and any position inside the preview() function.  Do you have any chance to reproduce the problem?  Thanks.

  • David - Thanks for the info. 

    I have looped in other experts at TI.  They will get back with suggestions.

    Regards

    Varada

  • Varada,

    Could you please update us on where this is sitting in the TI Expert's queue.

    thanks,

    Doug

  • I will follow up and let you know in a couple of days. Thanks

    Varada

  • Hello All,

     I am actively looking into this issue and will inform about the findings/fix shortly.

     

    -Manju

  • Would you please try adding the following in your preview_example.c  code?

    ..........

            if (ret < 0) {
                    printf("\n error in setting parameters");
                    goto out;
            }


            int prev_expand = 0x2c0;  <--//or any other value ( from 0 to 0x3FF) for tuning the preview data transfer throttling
            ret = ioctl(prevfd,PREV_SET_EXP,&prev_expand);
            if (ret < 0) {
                   printf("\n error in setting expand parameters");
                   goto out;
            }


    .......
            /* request one input buffer of size height*width */
            reqbufs.buf_type = PREV_BUF_IN;
            reqbufs.size = HEIGHT * WIDTH;

     

    Background:
    Reg VPSS_SDR_REQ_EXP

    Table 203. SDRAM Non-Real-Time Read Request Expand Register (SDR_REQ_EXP)
    Field Descriptions
    Bit
    Field
    Value
    Description
    31-30 Reserved 0 29-20 PRV_EXP 0-3FFh Preview read request expand. Delay to allow between consecutive read requests from the preview
                                         module. Units are in VPSS clock cycles (153/198 MHz in Normal/Turbo modes). Since the VPSS
                                        DMA priority is typically set to the highest for real-time requirements, this is for spreading any
                                       non-real-time reads with respect to the other traffic in the system. This minimizes the potential of
    locking out other requests for the duration of a frame being read from DDR/SDR.

     

    Although I still see the crash, yesterday night it appeared to take a longer time. Please try on your end.

     

    -Manju

  • Issue still under debug.

    Will keep the forum posted.

    -Manju

  • Thanks for the update.  I tested the patch with different values and unfortunately it could still be reproduced less than one minute.

  • I might have found the solution. Will update with the fix after due testing.

     

    -Manju

  • Please try the changes in the attachment and let me know the results7356.preview_fix.txt.

     

    The file is the diff of the default driver file deletions(marked <) and new changes. (>)

     

    Mainly:

    1. Remove:

     struct prev_params params;
     struct prev_convert conv;

    from prev_ioctl and directly use the arg.

    2. Do not do copy_from_user

    Thanks and Regards,

    -Manju

     

     

     

     

  • Thanks a lot for the patch.  The initial test looks good and we are integrating it with production code and will do more load tests.  By the way, there is a small change need to be done in the patch:

    <
    <     /* indicate the completion ofr frame processing */
    <     if (prevdevice)
    <         complete(&(prevdevice->wfc));
    <                              ^^- should be &(prevdevice.wfc)