VIDIOC_DQBUF ioctl fails after a while

I have one thread that captures then processes buffers of video data, and another thread that displays the processed buffers.

The capture thread uses the V4L2 framework to capture buffers however the longer my processing step the more likely I find the ioctl call to dequeue the captured video frame is to fail. I list a simplified version of my code below:

v4l2_format fmt;
fmt.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
ret = ioctl(captureFd, VIDIOC_G_FMT, &fmt);
if(ret < 0)
{
    PRINT("VideoPreview: Get format failed\n");
}

fmt.fmt.pix.width = CAPTUREX;
fmt.fmt.pix.height = CAPTUREY;
fmt.fmt.pix.bytesperline = fmt.fmt.pix.width*PIXELBYTES_CAPT;
fmt.fmt.pix.sizeimage = fmt.fmt.pix.bytesperline * fmt.fmt.pix.height;
fmt.fmt.pix.pixelformat = V4L2_PIX_FMT_RGB24;
ret = ioctl(captureFd, VIDIOC_S_FMT, &fmt);
if(ret < 0)
{
    PRINT("VideoPreview: Set capture format failed\n");
}

// request buffer (we are operating in userPtr mode)
v4l2_requestbuffers reqbuf;
reqbuf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
reqbuf.count = NUMCAPTBUFS;
reqbuf.memory = V4L2_MEMORY_USERPTR;
ret = ioctl(captureFd, VIDIOC_REQBUFS, &reqbuf);
if (ret < 0)
{
    PRINT("VideoPreview: Request buffer failed\n");
}

buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
buf.memory = V4L2_MEMORY_USERPTR;
buf.length = CAPTUREX*CAPTUREY*PIXELBYTES_CAPT;
for(i = 0; i < NUMCAPTBUFS; i++)
{
    buf.index = i;
    buf.m.userptr = (unsigned long)captureBuffer[i];
    ret = ioctl(captureFd, VIDIOC_QBUF, &buf);
    if(ret < 0)
    {
        PRINT("VideoPreview: Queue captureBuffer %d failed\n", i);
        return 0;
    }
}

int a = V4L2_BUF_TYPE_VIDEO_CAPTURE;
ret = ioctl(captureFd, VIDIOC_STREAMON, &a);
if(ret < 0)
{
    PRINT("VideoPreview: Start capture failed\n");
}

while(!stopThread)
{
    // Take filled capture buffer back from the V4L2 driver.
    ret = ioctl(captureFd, VIDIOC_DQBUF, &buf);
    if(ret < 0)
    {
        PRINT("VideoPreview: Dequeue frame failed. Trying again.\n");
        // Re-trying the ioctl here continually results in ret < 0.
    }
    TRACE(VideoPreview, "VIDIOC_DQBUF %d\n", buf.index);

    // Process the captured buffer given by pointer buf.m.userptr.
    dsp_process(buf.m.userptr, destBuffer);
    
    // check for lockup of HDVPSS VIP
    if(buf.flags & V4L2_BUF_FLAG_ERROR)
    {
        // There is code here but I've never seen this get called.
    }

    buf.m.userptr = (unsigned long)captureBuffer[buf.index];
    buf.length = CAPTUREX*CAPTUREY*PIXELBYTES_CAPT;
    ret = ioctl(captureFd, VIDIOC_QBUF, &buf);
    if(ret < 0)
    {
        PRINT("VideoPreview: Queue captureBuffer failed\n");
        return 0;
    }
}

Can anyone tell me why the dequeue call is failing so often? It usually runs fine if the dsp_process function doesn't spend much time blocking, but if the dsp_process function blocks for "too long", the dequeue ioctl fails and never recovers no matter how many times I call it.

Thanks,

Ralph