AM68A: Model performance lags under Optiflow versus Python

Hi,

We have assigned this thread to correct expert, please expect response from her.

Thanks

I haven't seen any updates on this, but I have a few of my own and figured I'd update this post:

- I've updated post_process.py to honor the dataset.yaml colormap settings for the segmentation overlay.  Overall processing time from this consumes around 26ms for my model.

- I've updated infer_pipe.py to be multi-threaded: one thread for pulling the frame, one for doing inference, and one for producing the frame + inference result  output image.  Total time for these parallelized thread operations is ~104ms (model inference consumes about 90ms of that!)  So, I'm a little sitting at a little less than 10fps which is pretty close to the Optiflow approach.

- With my segmentation model, the gstreamer side of things takes roughly 72ms.  Totaling this with the Python code, I'm looking at about a 104ms+72ms=176ms latency measurement.  This is better than using the OptiFlow approach (as noted above, Optiflow appears to have around 500ms of latency).

The long and short of all this is that I don't believe I'll proceed with using Optiflow.  It'd still be nice to know why latency appears to be so high with it, but it's now less of a concern as I'll be using Python as my baseline for future development.

Hello,

Could you please share the commands you are running for these tests? Are you directly running the GStreamer pipeline that is generated by the Optiflow application?

Thank you,

Fabiana

Fabiana,

Here is a list of comprehensive instructions for replicating the issue:

Dev kit requires imx219 camera attached on CAM0 connector, monitor connected via HDMI, and ethernet connection

Program SD card w/09.02.00.05 (available here: https://www.ti.com/tool/download/PROCESSOR-SDK-LINUX-AM68A)

Modify SD card's "uEnv.txt" to enable imx219 cam by adding "k3-am68-sk-bb-rpi-cam-imx219.dtbo" to "name_overlays" overlay definition.

Insert SD card into dev kit and apply power

Wait for "Edge AI gallery" screen to appear on monitor

Login to dev kit via ssh

k3conf set clock 8 0 500000000 # derates dev kit to behave as TDA4VL

vi /opt/edgeai-gst-apps/configs/imx219_cam_example.yaml
-> change 'flow0' to reference model2: "flow0: [input0,model2,output0,[320,150,1280,720]]"

export GST_DEBUG_FILE=/run/trace-optiflow.log
export GST_DEBUG_NO_COLOR=1
export GST_DEBUG="GST_TRACER:7"
export GST_TRACERS="latency(flags=pipeline+element)"

./optiflow/optiflow.py ./configs/imx219_cam_example.yaml
-> Note framerate reported on screen at top-right corner: 30
-> With yourself in view, simply observe lag by moving your arm up and down. Lag is noticeable.
-> hit Ctrl-C to terminate after ~60 seconds of operation

export GST_DEBUG_FILE=/run/trace-python.log

./apps_python/app_edgeai.py ./configs/imx219_cam_example.yaml
-> Note framerate reported on screen at top-right corner: 25
-> With yourself in view, simply observe lag by moving your arm up and down. Visible lag is less than with Optiflow.
-> Observe reported "total time" from app_edgeai.py is ~40ms.
-> hit Ctrl-C to terminate after ~60 seconds of operation

Do the following from a Linux machine:

Clone this repo: github.com/.../GStreamerLatencyPlotter.git and follow install instructions

cd GStreamerLatencyPlotter

mkdir captures

Copy /run/trace-optiflow.log and /run/trace-python.log from dev kit to captures/ dir

node main.js captures/trace-optiflow.log
-> Observe "TOTAL" latency mean is ~210ms

node main.js captures/trace-python.log
-> Observe "TOTAL" latency mean is 33ms. Overall latency measure is 33ms + 40ms ("total time" reported by app_edgeai.py) = ~73ms.

Python app's 73ms latency is a lot less than the 210ms of Optiflow. Further, you can visibly see
the lag difference between the Optiflow and Python approach.

Thank you for providing detailed steps. I have ran this and observed similar results on my end. I am currently looking into what is causing this latency with my team and running a few tests.

Thanks,

Fabiana

Any updates on this?  I'm curious what your team has found out.

Please allow me until the end of the week to get back to about this.

Thank you for your patience.

- Fabiana