Clock coupling between video and audio??

mmm, interesting.  I wonder if this has something to do with ARM scheduler not being able to allocate enough time for audio processing.  Is there a way for you to measure the ARM load when both audio and video are running together?

Juan Gonzales said:

Is there a way for you to measure the ARM load

Running "top" shows about 50% CPU utilization.

Another thing that comes to mind is the effect that running the video thread may have.  Is it adding a few usec delay in between each audio frame hence not all expected frames are capture before we run out of time but none were dropped in the sequence, or is it cpaturing many audio frames at the expected rate and missing one every now and then.

Juan Gonzales said:

... the effect that running the video thread may have

There is no indication that running the thread itself is having an impact.  TIme deltas between audio buffers are very consistent.  With video and audio running at different clock/sample rates it would be very difficult for the threads, themselves, to generate a uniform delay.  Original testing was performed at 720x480 resolution.  Current testing is being done at 352x240 to intentionally unload the CPU.  And, switching between HIGH profile, HIGH quality and BASELINE profile, STANDARD quality have no effect.  If this were a loading/timing issue due to CPU resource contention I would expect to see some impact from one or more of these changes.

There is no indication that entire buffers/frames are being lost.  There IS however some indication that individual samples are possibly being lost.  When this delay occurs, there is audible noise that sounds like static.  Supposition is that this is due to dropped samples AND/OR an artifact of the audio data being sampled at 48K but actually being read/transferred out at a lower clock rate (more testing has pinned it down to more like 47971.5 +/- 1.0 kHz)

Tom Hanson said:

If I run PCM audio, without video, at 48kHz sampling, with blocking reads of 1024 samples, the data is read at 48,000 samples/sec

If I run the same exact configuration while running video (EDIT: H.264 ONLY) in parallel, the audio data is read  at approx (EDIT: 47971 samples/sec.)

OK, it looks highly probable that there's a bug in the latest H.264 encoder.

We've done a lot of testing: PCM audio vs AAC audio; OSS vs ALSA; block reads vs. non-blocking reads; MPEG4 vs H.264;  etc., etc.  What it comes down to is: if the H.264 encoder is running (i.e. acutally encoding frames) the audio data reads out of the capture device at +/- 47971 Hz.  If the H.264 encoder is not running, or if MPEG4 is used instead, the audio data reads out of the capture device at +/- 48000 Hz. 

The above occurs running the H.264 codec from dvsdk_2_10_01_18/dm365_codecs_01_00_06.  If that codec (and nothing else) is replaced with the version from /dvsdk_2_10_00_13/dm365_codecs_01_00_03 the problem goes away.  Audio comes out at 48kHz no matter what the video configuration is.

There is some indication that this may be a DMA issue, but I'll leave that to the experts!

Happy (bug) hunting...