RTOS: AM5728 Jailhouse running sysbios Interrupt Latency too long

gpio interrupt is same， sysbios&jailhouse Interrupt Latency is for a long times

Hello Valvalvalval,

Can you please share more details on how you get above results? What is your goal in terms of latency numbers?

best regards,
David Zhou

how you get above results? ：

ti-processor-sdk-linux-am57xx-evm-05.01.00.11/board-support/extra-drivers/jailhouse-0.8/inmates/ti_app/ti-app.c

we had try make a fpga i/o 1ms interrupt the am5728 ，but the sometimes latency is same for a long times.

What is your goal in terms of latency numbers?

we want more communication bus on sysbios，max latency > 1ms is dont work, or  max latency should max latency < 10us.

Hi,

The ti-app.c uses dedicated timer to generate interrupts and uses the same timer to measure interrupt latency. By reading the time at entrance of the interrupt handler the app knows the latency.

To make it work as a jailhouse inmate the Linux dts file explicitly says that this timer and this interrupt will not be used by Linux, but by the inmate.

The cell configuration also need to take care about the timer's memory map and the interrupts usage by the inmate

The RTOS configuration file also should take care about interrupt configuration and times that are allowed for the RTOS inmate.

You didn't provide any specific information about your application, how you configure Linux DTS, cell configuration and all inmate resources allocation. There is no any specific details about your inmate. it is not clear what the FPGA is and what generates the interrupts. If that is not a timer, than how do you measure the interrupt latency. Please could you provide details of you application and the setup.

Thanks

We will not discuss FPGA-related test methods now.

I am using sysbios in the jailhouse to use timer to get the interrupt latency. The worst case is more than 2ms, which is an impossible delay.

The first step to debug the issue is to run your RTOS application w/o the Jailhouse. Just make the app as an RTOS application and run it via CCS.
Prove that it works with desired latency as a standalone RTOS app.

Please look at my data carefully!!!!!!!!!!

There is interrupt latency data for running RTOS application w/o the Jailhouse !!!

Prove that it can run as a standalone RTOS application with a worst interrupt latency of 10us.

But when I put it in the jailhouse, the worst case for interrupt latency is up to 2ms.

Hello Customer,

Sorry for delayed reply on this E2E thread, we will need to take a close look and reproduce your results. Due to the holidays in USA, this is going to be delayed but we will be working on this and get back to you soon.

best regards,
David Zhou

Hi, could you please send your "ti-app" TI-RTOS version project (or makefile), binaries, etc. So we can reproduce your results?

thank you,

Paula

thanks，i had move my project to idkAM5728，so you can test on idkAM5728 board.

zip include CCS project、jailhouse cell files、dts and launch.sh

idkAM5728-TEST.zip

This is just a friendly reminder.

Hi sorry for the delay, I have been out of office. I will reply soon (next few days).

thank you,
Paula

Hi, thanks a lot for sharing your Jailhouse inmate RTOS project, binary and script. I was able to run it.
A quick question, for running RTOS Int latency test w/o Jailhouse. Did you use the same tools versions (XDC, Sysbios, PDK, etc) and same compiler/linker project options as the CCS project you shared here?. (just want to confirm).

On the other hand, let me check with our developers w.r.t your jitter results and come back to you.

thank you,
Paula

Ok, thank you for your work.

This is a CCS project which you need.

The same tools versions (XDC, Sysbios, PDK, etc) and same compiler/linker project options as the CCS project  which running RTOS Int latency test w/o Jailhouse.

idkAM5728_test_without_jailhouse_CCS_project.zip

Hi, thanks for sharing non-jailhouse TI-RTOS project.

After running for few minutes I see

TEST1: JAILHOUSE TI-RTOS:
jitter: 19969 ns, min: 18722 ns, max: 568618 ns
jitter: 21409 ns, min: 18722 ns, max: 568618 ns
jitter: 19500 ns, min: 18722 ns, max: 568618 ns
jitter: 19202 ns, min: 18722 ns, max: 568618 ns
jitter: 19118 ns, min: 18722 ns, max: 568618 ns


TEST2: TI-RTOS (non-jailhouse)
jitter: 14776 ns, min: 14124 ns, max: 40320 ns
jitter: 14972 ns, min: 14124 ns, max: 40320 ns
jitter: 14960 ns, min: 14124 ns, max: 40320 ns
jitter: 14628 ns, min: 14124 ns, max: 40320 ns
jitter: 14644 ns, min: 14124 ns, max: 40320 ns

Then TEST1 max jiiter ~570us, TEST2 max jitter ~40us. Let me know if this agree with your results. Or if in order to get max jitter ~2ms in TEST1 I need to leave the test running longer.

Thank you,
Paula

Ok, thank you for your work.

（Let me know if this agree with your results）YES.It is my results.

Because I optimized the program, so the TEST1 max jiiter ~570us, and there will be not  get max jitter ~2ms.TEST1 NO need to leave the test running longer. 

We want to know why TEST1 is so much bigger than TEST2.

And how do I get TEST1 results closer to TEST2.

Thanks for confirming. We will check and come back to you.
Paula

Hi,  after checking your code, I did few modifications which helped a little bit to reduce jitter. Summary list of changes below:

• I think you had timer2 by mistake enabled in *.cfg. As a test, I built RTOS application using “TimerSupport.availMask = 0x0080;”.  It worked OK.

• I compared your app.cfg  and the one from TI_RTOS_PSKD/../jailhouse/rtos/icss-emac . I merged differences that I considered applied, such as MMU settings, CacheEnable, and timer number. Attached new project for your reference.

With changes, max jitter is ~500us, however, I only saw once the jitter close to 500us, most of the time is ~47us, and once in a while ~180us

Summary results:

New results Jailhouse-RTOS:

jitter: 4783 ns, min: 2720 ns, max: 509300 ns

jitter: 4702 ns, min: 2720 ns, max: 509300 ns

jitter: 4726 ns, min: 2720 ns, max: 509300 ns

jitter: 4726 ns, min: 2720 ns, max: 509300 ns

Previous results Jailhouse-RTOS:

jitter: 19969 ns, min: 18722 ns, max: 568618 ns

jitter: 21409 ns, min: 18722 ns, max: 568618 ns

jitter: 19500 ns, min: 18722 ns, max: 568618 ns

jitter: 19202 ns, min: 18722 ns, max: 568618 ns

I am checking with our MMU experts if there is any additional optimization with can do for TIMER8 memory region. However, please keep in mind that getting same results as baremetal might not be possible, as SYSBIOS might some times runs other tasks in the background.

idkAM5728_inmate_opt.zip

thank you,

Paula

Yes, I have enable timer2 and timer8 on inmate.

Our application scenario is industry.

We want the maximum interrupt delay to be less than 50us.

Best regards.

No sure if you are aware, but we have an EtherCAT example running with Jailhouse. Just to let you know, in case you want to check an RTOS industrial application example running with Jailhouse

processors.wiki.ti.com/.../Processor_SDK_Jailhouse_Hypervisor

thank you,
Paula

Also, for helping us understand better, and possibly give you ideas/tips/alternatives, could you please elaborate your use-case? what would be running on Linux? what would be running on RTOS, what intercommunication is required?

thank you,
Paula

Hey, is there a new way to reduce the interruption delay of jailhouse-sysbios?

Hi, we are working on getting a better understanding on what could cause max int latency to go up once in a while. We would need more time to analyze it. I will come back here when we have more information.

thank you,

Paula

Hey,

Early testing has such a phenomenon, I don't know if it can help your work:

Jailhouse-sysbios interrupt latency test in custom board, when Linux printk "Net: Registered protocol family 15" and "Initializing XFRM netlink socket", get max int latency instantaneously.

thank you.

Hi a quick update, we are working on getting a debug setup ready using Lauterbach (TRACE32) in order to be able to have information from both cores with Hypervisor in a sync way.

I typically debug only RTOS, so this is a little bit more complex scenario. I will let you know when we have some insights to share.

thank you,
Paula

For comparison test, I ported the jailhouse to orange pi lite (Allwinner H3 CortexA7 4Core 1.08GHz).

The inmate is running freertos to test interrupt latency.

MMU is on.

Test Results：

We observed the curve of the interrupt latency generated by sysbios running the timer in the jailhouse.

The red curve is the current interrupt delay.

We found that the larger interrupt latency is accurate for 30s.

Hi, thanks a lot for these data points.

With respect to your Jailhouse porting to orange pi lite, could you elaborate how did you do it? just porting hypervisor/jailhouse?

For your information, I have been pulled to another activity but I will try to come back to this debug soon.

thank you,
Paula

Tanks for your work.

The test code for orangepi lite needs to be sorted out before I can publish it.

Now there is a new test phenomenon.

When sysbios is used as the inmate test interrupt delay for jailhouse, core0 enters the tab button via serial or runs for(;;);, which causes a large interrupt latency.

YES! IT WORKS!

Jailhouse-sysbios interrupt latency is less than 55us.

Seems to be dynamically adjusted by the CPU frequency：

cd /sys/devices/system/cpu/cpu0/cpufreq
echo performance > scaling_governor
echo 1000000 > scaling_max_freq