We've been trying to figure out why we're dropping a large number of frames when encoding H.264 video on a custom DM6446 board. Setting CE_DEBUG=3, we noticed that a call to MSGQ_get() from within VIDENC_process() appears to be taking ~50ms to complete. If the order of the trace output is to be believed, though, this occurs before the DSP algorithm executes its process function. We certainly can't fathom a reason for the frame processing to take this long... its a D1 NTSC frame being encoded at around 1Mb/s, and our vendor specs say this should only take around 25ms.
We are running a 2.6.28 git kernel and compiling against a "fat" CodecEngine 2.23. I don't believe we saw this problem under 2.6.10MV/CodecEngine 1.10, but I'm in the process of dragging out that filesystem/kernel combination to verify my sanity ;).
Anybody have any insight as to what's going on? Much appreciated in advance. Trace snippet follows:
@1,454,227us: [+0 T:0x4001f000 S:0x03874944] ti.sdo.ce.video.VIDENC - VIDENC_process> Enter (handle=0x44c90, inBufs=0x3874a74, outBufs=0x3874a68, inArgs=0x443c8, outArgs=0x443e0)
@1,454,428us: [+5 T:0x4001f000 S:0x038748dc] CV - VISA_allocMsg> Allocating message for messageId=0x00026700
@1,454,600us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> Enter(virtAddr=0x411d9000, size=307200)
@1,454,768us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> Enter(virtAddr=0x411d9000, size=307200)
@1,454,929us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> found in cb(Sc=0x411d9000, Ec=0x41249800, Ss=0x411d9000, Es=0x41224000, PSc=0x89bc6000)
@1,455,096us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> returning physAddr=0x89bc6000
@1,455,255us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> return (0x89bc6000)
@1,455,407us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> Enter(virtAddr=0x41224000, size=76800)
@1,455,567us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> Enter(virtAddr=0x41224000, size=76800)
@1,455,728us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> found in cb(Sc=0x411d9000, Ec=0x41249800, Ss=0x41224000, Es=0x41236c00, PSc=0x89bc6000)
@1,455,894us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> returning physAddr=0x89c11000
@1,456,048us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> return (0x89c11000)
@1,456,201us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> Enter(virtAddr=0x41236c00, size=76800)
@1,456,360us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> Enter(virtAddr=0x41236c00, size=76800)
@1,456,575us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> found in cb(Sc=0x411d9000, Ec=0x41249800, Ss=0x41236c00, Es=0x41249800, PSc=0x89bc6000)
@1,456,757us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> returning physAddr=0x89c23c00
@1,456,915us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> return (0x89c23c00)
@1,457,071us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> Enter(virtAddr=0x412a4000, size=307200)
@1,457,230us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> Enter(virtAddr=0x412a4000, size=307200)
@1,457,386us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> found in cb(Sc=0x412a4000, Ec=0x412ef180, Ss=0x412a4000, Es=0x412ef000, PSc=0x89afb000)
@1,457,552us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> returning physAddr=0x89afb000
@1,457,704us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> return (0x89afb000)
@1,457,854us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> Enter(virtAddr=0x412ef000, size=384)
@1,458,012us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> Enter(virtAddr=0x412ef000, size=384)
@1,458,170us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> found in cb(Sc=0x412a4000, Ec=0x412ef180, Ss=0x412ef000, Es=0x412ef180, PSc=0x89afb000)
@1,458,335us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getPhysicalAddress> returning physAddr=0x89b46000
@1,458,484us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferPhysicalAddress> return (0x89b46000)
@1,458,639us: [+0 T:0x4001f000 S:0x038748e4] CV - VISA_call(visa=0x44c90, msg=0x41169c80): messageId=0x00026700, command=0x0
@1,458,807us: [+0 T:0x4001f000 S:0x038748ac] OC - Comm_put> Enter(queue=0x2, msg=0x41169c80)
@1,459,039us: [+0 T:0x4001f000 S:0x038748ac] OC - Comm_put> return (0)
@1,459,221us: [+0 T:0x4001f000 S:0x038748a4] OC - Comm_get> Enter(queue=0x10001, msg=0x387495c, timeout=-1)
@1,507,703us: [+0 T:0x4001f000 S:0x038748a4] OC - Comm_get> MSGQ_get() status=0x8000, return (0)
@1,507,938us: [+0 T:0x4001f000 S:0x03874834] OC - Comm_put> Enter(queue=0x0, msg=0x41168c80)
@1,508,185us: [+0 T:0x4001f000 S:0x03874834] OC - Comm_put> return (0)
@1,508,422us: [+0 T:0x4001f000 S:0x0387482c] OC - Comm_get> Enter(queue=0x10000, msg=0x38748cc, timeout=-1)
@1,508,791us: [+0 T:0x4001f000 S:0x0387482c] OC - Comm_get> MSGQ_get() status=0x8000, return (0)
[DSP] @0,511,501tk: [+5 T:0x8b9fb21c S:0x8b9fd1e4] CN - NODE> 0x8fa82398(xavsc_ateme#0) call(algHandle=0x8fa82450, msg=0x8fe06c80); messageId=0x00026700
[DSP] @0,511,588tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> Enter(addr=0x89bc6000, sizeInBytes=307200)
[DSP] @0,511,982tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> return
[DSP] @0,512,024tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> Enter(addr=0x89c11000, sizeInBytes=76800)
[DSP] @0,512,167tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> return
[DSP] @0,512,209tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> Enter(addr=0x89c23c00, sizeInBytes=76800)
[DSP] @0,512,352tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> return
[DSP] @0,512,395tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> Enter(addr=0x89afb000, sizeInBytes=307200)
[DSP] @0,512,788tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> return
[DSP] @0,512,830tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> Enter(addr=0x89b46000, sizeInBytes=384)
[DSP] @0,512,890tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheInv> return
[DSP] @0,512,933tk: [+0 T:0x8b9fb21c S:0x8b9fd14c] ti.sdo.ce.video.VIDENC - VIDENC_process> Enter (handle=0x8fa82450, inBufs=0x8b9fd1ec, outBufs=0x8b9fd1f8, inArgs=0x8fe06db8, outArgs=0x8fe06dc8)
[DSP] @0,513,035tk: [+5 T:0x8b9fb21c S:0x8b9fd12c] CV - VISA_enter(visa=0x8fa82450): algHandle = 0x8fa82480
[DSP] @0,513,095tk: [+0 T:0x8b9fb21c S:0x8b9fd10c] ti.sdo.ce.alg.Algorithm - Algorithm_activate> Enter(alg=0x8fa82480)
[DSP] @0,513,157tk: [+0 T:0x8b9fb21c S:0x8b9fd10c] ti.sdo.ce.alg.Algorithm - Algorithm_activate> Exit
[DSP] @0,622,199tk: [+5 T:0x8b9fb21c S:0x8b9fd12c] CV - VISA_exit(visa=0x8fa82450): algHandle = 0x8fa82480
[DSP] @0,622,282tk: [+0 T:0x8b9fb21c S:0x8b9fd10c] ti.sdo.ce.alg.Algorithm - Algorithm_deactivate> Enter(alg=0x8fa82480)
[DSP] @0,622,371tk: [+0 T:0x8b9fb21c S:0x8b9fd10c] ti.sdo.ce.alg.Algorithm - Algorithm_deactivate> Exit
[DSP] @0,622,424tk: [+0 T:0x8b9fb21c S:0x8b9fd14c] ti.sdo.ce.video.VIDENC - VIDENC_process> Exit (handle=0x8fa82450, retVal=0x0)
[DSP] @0,622,493tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWb> Enter(addr=0x89afb000, sizeInBytes=307200)
[DSP] @0,622,892tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWb> return
[DSP] @0,622,934tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWb> Enter(addr=0x89b46000, sizeInBytes=384)
[DSP] @0,622,994tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWb> return
[DSP] @0,623,037tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWbInv> Enter(addr=0x8b80f218, sizeInBytes=330240)
[DSP] @0,623,456tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWbInv> return
[DSP] @0,623,499tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWbInv> Enter(addr=0x8b864cac, sizeInBytes=82560)
[DSP] @0,623,649tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWbInv> return
[DSP] @0,623,692tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWbInv> Enter(addr=0x8b87af6c, sizeInBytes=82560)
[DSP] @0,623,842tk: [+0 T:0x8b9fb21c S:0x8b9fd184] OM - Memory_cacheWbInv> return
[DSP] @0,623,888tk: [+5 T:0x8b9fb21c S:0x8b9fd1e4] CN - NODE> returned from call(algHandle=0x8fa82450, msg=0x8fe06c80); messageId=0x00026700
[DSP] @0,625,554tk: [+0 T:0x8b80006c S:0x8b80402c] CR - processRmsCmd(0x8fe05ca8, 4056): cmd = 5
[DSP] @0,625,611tk: [+0 T:0x8b80006c S:0x8b80402c] CR - remote time = 0x0, trace buffer size = 4032
@1,518,781us: [+0 T:0x4001f000 S:0x03874864] CE - Engine_fwriteTrace> returning count [3435]
@1,518,966us: [+0 T:0x4001f000 S:0x038748e4] CV - VISA_call Completed: messageId=0x00026700, command=0x0, return(status=0)
@1,519,144us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> Enter(physAddr=0x89afb000, size=307200)
@1,519,310us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> Enter(physAddr=0x89afb000, size=307200)
@1,519,472us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> found in cb(Sc=0x89afb000, Ec=0x89b46180, Ss=0x89afb000, Es=0x89b46000)
@1,519,636us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> returning virtAddr=0x412a4000
@1,519,788us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> return (0x412a4000)
@1,519,941us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> Enter(physAddr=0x89b46000, size=384)
@1,520,096us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> Enter(physAddr=0x89b46000, size=384)
@1,520,252us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> found in cb(Sc=0x89afb000, Ec=0x89b46180, Ss=0x89b46000, Es=0x89b46180)
@1,520,414us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> returning virtAddr=0x412ef000
@1,520,567us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> return (0x412ef000)
@1,520,722us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> Enter(physAddr=0x8b80f218, size=330240)
@1,520,881us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> Enter(physAddr=0x8b80f218, size=330240)
@1,521,035us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> returning virtAddr=0x0
@1,521,189us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> return (0x0)
@1,521,343us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> Enter(physAddr=0x8b864cac, size=82560)
@1,521,502us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> Enter(physAddr=0x8b864cac, size=82560)
@1,521,657us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> returning virtAddr=0x0
@1,521,810us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> return (0x0)
@1,522,039us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> Enter(physAddr=0x8b87af6c, size=82560)
@1,522,216us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> Enter(physAddr=0x8b87af6c, size=82560)
@1,522,381us: [+1 T:0x4001f000 S:0x038748ac] OM - Memory__getVirtualAddress> returning virtAddr=0x0
@1,522,535us: [+0 T:0x4001f000 S:0x038748ac] OM - Memory_getBufferVirtualAddress> return (0x0)
@1,522,687us: [+5 T:0x4001f000 S:0x038748bc] CV - VISA_freeMsg(0x44c90, 0x41169c80): Freeing message with messageId=0x00026700
@1,522,855us: [+0 T:0x4001f000 S:0x03874944] ti.sdo.ce.video.VIDENC - VIDENC_process> Exit (handle=0x44c90, retVal=0x0)