• State
  • Locked Locked
  • Replies 6 replies
  • Subscribers 99 subscribers
  • Views 230 views
  • Users 0 members are here
Support feedback
Options
Options
Related

This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4AP-Q1: Profiling memory stats in edgeai-tidl-tools during model inference

MadhuBala E
Intellectual 275 points
Part Number: TDA4AP-Q1

Tool/software:

Hi

I am running a custom model inference on j784s4_evm (TDA4AP) using sd boot mode (on top of HLOS). Above is the summary logged while running the model inference and perf_stats tool application in parallel for getting the DDR performance. I could see device_config.cfg file inside the edgeai-tidl-tools folder which mentions the utilization of L2SRAM , MSMC , DDR ram space during TIDL running 

I have couple of questions here 

1.Is there a way to view the memory utilization of L2, MSMC, and DDR for a given model—specifically, how much memory each layer is consuming? Can we obtain detailed logs during model compilation, alongside the model artifacts, that display memory usage per layer across the different memory levels? 

2. Can we get the memory usage logs during model inference running on TI SOC ?

3. Also in above attached screenshot , can you elaborate/interpret more on DDR performance statictics

Thanks,

Madhu

  • 0
    TI__Genius 14420 points

    Hi Madhu,

    I will look into options for this.   

    Regards,

    Chris

  • 0
    TI__Genius 14420 points

    Hi Madhu,

    If you can please send me your model, I can run a profile for you and send the results.

    Regards,
    Chris

  • 0 in reply to Chris Tsongas
    Intellectual 275 points

    Hi chris

    Here is the custom model that I am using now , I have shared you the drive link for that model https://drive.google.com/file/d/1zvn8BI8h23JvTiRqfyYXeY3joOXC5GgA/view?usp=drive_link

    Also please let me know the steps required for profiling that you are doing

    Thanks,

    Madhu

  • 0 in reply to MadhuBala E
    TI__Genius 14420 points

    Hi Madhu,

    The model you sent me will not import.  Can you please send an importable version of the model?  Preferably in ONNX format.  Profiling takes some time, so please send me a model that you know is working before I profile it.   Basically, make sure it imports with TIDL (osrt_python) first.  

    ['mcw_yolo']
    Available execution providers : ['TIDLExecutionProvider', 'TIDLCompilationProvider', 'CPUExecutionProvider']

    Running 1 Models - ['mcw_yolo']


    Running_Model : mcw_yolo


    Running shape inference on model /home/root/yolov8.tflite

    Process Process-1:
    Traceback (most recent call last):
    File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
    File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
    File "/home/root/examples/osrt_python/advanced_examples/unit_tests_validation/ort/./onnxrt_ep.py", line 218, in run_model
    sess = rt.InferenceSession(config['model_path'] ,providers=EP_list, provider_options=[delegate_options, {}], sess_options=so)
    File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 387, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
    File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 428, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
    onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from /home/root/yolov8.tflite failed:/root/onnxruntime/onnxruntime/core/graph/model.cc:138 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&, const onnxruntime::ModelOptions&) Missing opset in the model. All ModelProtos MUST have at least one entry that specifies which version of the ONNX OperatorSet is being imported.

    ^CTraceback (most recent call last):
    File "/home/root/examples/osrt_python/advanced_examples/unit_tests_validation/ort/./onnxrt_ep.py", line 287, in <module>
    nthreads = join_one(nthreads)
    File "/home/root/examples/osrt_python/advanced_examples/unit_tests_validation/ort/./onnxrt_ep.py", line 269, in join_one
    sem.acquire()
    KeyboardInterrupt

    Regards,

    Chris

  • 0 in reply to Chris Tsongas
    Intellectual 275 points

    Hi chris

    Thanks for the response and sorry for the delay

    Here I am sending you the model (model.tflite) in the google drive that imports with TIDL (osrt_python). Also Can you please provide me the steps that I can take from my side to perform profiling for getting the memory utilization details with more information as mentioned above, 

    I have couple of questions here 

    1.Is there a way to view the memory utilization of L2, MSMC, and DDR for a given model—specifically, how much memory each layer is consuming? Can we obtain detailed logs during model compilation, alongside the model artifacts, that display memory usage per layer across the different memory levels? 

    2. Can we get the memory usage logs during model inference running on TI SOC ?

    3. Also in above attached screenshot , can you elaborate/interpret more on DDR performance statictics? 

    https://drive.google.com/file/d/1Q-v8zpahuzwQKWT2e28TVSFNgAnXbgow/view?usp=drive_link 

    Thanks,

    Madhu

  • 0 in reply to MadhuBala E
    TI__Genius 14420 points

    Hi Madhu,

    I dowmloaded your new model and will try it out.

    1.Is there a way to view the memory utilization of L2, MSMC, and DDR for a given model—specifically, how much memory each layer is consuming? Can we obtain detailed logs during model compilation, alongside the model artifacts, that display memory usage per layer across the different memory levels? 

    Yes, but at this time I would have to run it for you and send you the results.  If you want to run it on your own, it will require an NDA.

    2. Can we get the memory usage logs during model inference running on TI SOC ?

    Assuming the model imports, you will get everything in a CSV file.

    3. Also in above attached screenshot , can you elaborate/interpret more on DDR performance statictics? 

    Not really, I cannot tell you much from a screenshot. 

    Regards,

    Chris