66AK2H06: Problem with QMSS

Ilya Mazurenko

Part Number: 66AK2H06

Hello Team,

Could you help with our customer's question please.

They cant launch qmss on Linux.

Linux use keystone-uio.dtsi, where:

uio_qmss: qmss {

compatible = "ti,uio-module-drv";

mem = <0x02a00000 0x00100000

0x23a00000 0x00200000>;

};

In k2hk-uio.dtsi file this section is not redefined

In documentation: Documentation/devicetree/bindings/soc/ti/keystone-navigator-qmss.txt

qmss: qmss@2a40000 {

compatible = "ti,keystone-qmss";

dma-coherent;

This section is pulled through keystone-k2hk-netcp.dtsi

In documentation on QMSS module in Keystone 2 the base adress 0x02a00000 is changed to 0x02a40000

The question:
Is described above the reason of the following message(when the try to open QMSS transport):

TransportQmss Version : 0x02000001

Version String: Linux IPC Transports Revision: 2.0.0.01:Jan 23 2018:19:13:09

Process 1 : Initialized RM_Client0

Process 1 : Opening RM client socket /var/run/rm/rm_client0

fw_memMap: Failed to find fd to map 0x02a00000.

Process 1 : Creating TransportQmss instance

fw_memMap: Failed to find fd to map 0x02a00000.

TransportQmss_create : mpm_transport_open failed

name = "arm-qmss-generic", msg_size_bytes:4096, rm_service_h=0x830f68

ERROR Process 1 : Failed to create TransportQmss handle

Edit: Even after manual change of adresses the /dev/qmss still not appears.
The question 2:
When the /dev/qmss should appear? Which driver should be loaded?

“rmServer.out /usr/bin/device/k2h/global-resource-list.dtb /usr/bin/device/k2h/policy_dsp_arm.dtb” - was launched without errors.

Thank you in advance!

Ilya.

over 7 years ago

0 Rex Chang over 7 years ago

TI__Guru 50170 points

Hi, Ilya,

The address difference is they point to different location of registers. Please refer to the device memory map in the datasheet, and section 5.3 Memory Map of Multicore Navigator User's Guide.

Are you able to run any of the QMSS_LLD ARM examples, qmInfraDmaMC_k2h.out, qmInfraDmaSC_k2h.out, or qmQAllocTest_k2h.out? They are in tisdk-server-extra-rootfs-image-k2hk-evm.tar.xz filesystem. I used the images from procSDK 4.1 release instead of downloading the latest 4.2. I am able to run the multicore DMA example on ARM.

root@k2hk-evm:/usr/include/ti/drv/rm/device/k2h# rmServer.out global-resource-list.dtb policy_dsp_arm.dtb
Starting rmServer.out
Daemon returned 0 as return value.
root@k2hk-evm:/usr/include/ti/drv/rm/device/k2h# qmInfraDmaMC_k2h.out
No command line given. Forking off 4 tasks
coreNum: 1; pid=926
coreNum: 0; pid=925
coreNum: 2; pid=927
coreNum: 3; pid=928

Initialized RM_Client0

Initialized RM_Client1

Initialized RM_Client2

Initialized RM_Client3

core 0: got CMEM CMA phys 0x87cc00000; virt 0xb6d16000
core 0: Mapping 1048576 bytes from /dev/qmss at 0x02a00000
core 0: Mapping 1048576 bytes from /dev/qmss at 0x23a00000
thread priority set to 99
core 1: got CMEM CMA phys 0x87cc00000; virt 0xb6d16000
core 1: Mapping 1048576 bytes from /dev/qmss at 0x02a00000
core 1: Mapping 1048576 bytes from /dev/qmss at 0x23a00000
thread priority set to 99
core 2: got CMEM CMA phys 0x87cc00000; virt 0xb6d16000
core 2: Mapping 1048576 bytes from /dev/qmss at 0x02a00000
core 2: Mapping 1048576 bytes from /dev/qmss at 0x23a00000
thread priority set to 99
core 3: got CMEM CMA phys 0x87cc00000; virt 0xb6d16000
core 3: Mapping 1048576 bytes from /dev/qmss at 0x02a00000
core 3: Mapping 1048576 bytes from /dev/qmss at 0x23a00000
thread priority set to 99
Successfully opened RX QPEND queue 665
Successfully opened UIO dev /dev/uio3
core 0: my rx queue: 665

-----------------------Initializing---------------------------
Successfully opened RX QPEND queue 663
Successfully opened UIO dev /dev/uio1
core 1: my rx queue: 663
Core 0 : Memory region 0 inserted
Successfully opened RX QPEND queue 664
Successfully opened UIO dev /dev/uio2
core 2: my rx queue: 664
Successfully opened RX QPEND queue 662
Successfully opened UIO dev /dev/uio0
core 3: my rx queue: 662
Core 0 : Memory region 1 inserted
Core 0 : Tx Completion Queue Number : 9025 opened
Core 0 : Number of Tx descriptors requested : 32. Number of descriptors allocated : 32
Core 0 : Number of Rx descriptors requested : 32. Number of descriptors allocated : 32
Core 0 : Number of Sync free descriptors requested : 32. Number of descriptors allocated : 32
Core 0 : Sync Queue Number : 9026 opened
Core 0 : Sync Cfg Queue Number : 898 opened
Core 0 : Sync Free Queue Number : 897 opened
Core 0 : Receive Free Queue Number : 8928 opened
Core 0 : Transmit Free Queue Number : 736 opened
Core 0 : System initialization completed: 736
Core 0 : Publishing RM nameserver names for shared queues
Core 0: Getting receive queues for other cores
Core 1 : Rx Free Queue Number : 8928 opened
Core 2 : Rx Free Queue Number : 8928 opened
Core 3 : Rx Free Queue Number : 8928 opened
Core 1 : Tx Free Queue Number : 736 opened
Core 2 : Tx Free Queue Number : 736 opened
Core 3 : Tx Free Queue Number : 736 opened
Core 1 : Tx Completion Queue Number : 9025 opened
Core 2 : Tx Completion Queue Number : 9025 opened
Core 3 : Tx Completion Queue Number : 9025 opened
Core 1 : Sync Queue Number : 9026 opened
Core 2 : Sync Queue Number : 9026 opened
Core 3 : Sync Queue Number : 9026 opened
Core 1 : Sync Free Queue Number : 897 opened
Core 2 : Sync Free Queue Number : 897 opened
Core 3 : Sync Free Queue Number : 897 opened
Core 1 : Sync Cfg Queue Number : 898 opened
Core 2 : Sync Cfg Queue Number : 898 opened
Core 3 : Sync Cfg Queue Number : 898 opened
core 1: waiting for packets on 663
core 2: waiting for packets on 664
core 3: waiting for packets on 662
core 0: Got core 1's RX QID: 663
core 0: Got core 2's RX QID: 664
core 0: Got core 3's RX QID: 662

--------------------Queue status CORE 0----------------------
After Initialization

Tx Free Queue 736 Entry Count : 32
Rx Free Queue 8928 Entry Count : 32
Tx completion Queue 9025 Entry Count : 0
Sync Queue 9026 Entry Count : 0
Sync free Queue 897 Entry Count : 29
Sync Cfg Queue 898 Entry Count : 3
-------------------------------------------------------------

Core 0 : Opened Tx channel : 16
Core 0 : Opened Rx channel : 16
Core 0 : Transmit Queue Number : 816
Core 0 : Receive Queue Number : 665
Core 0 : Opened Rx flow : 16

Core 0 : Transmitting 8 packets..........

*************************************************************
Core 0 : Opened Tx channel : 16
Core 0 : Opened Rx channel : 16
Core 0 : Transmit Queue Number : 816
Core 0 : Receive Queue Number : 663
Core 0 : Opened Rx flow : 16

Core 0 : Transmitting 8 packets..........

*************************************************************
core 1: got interrupt - process 8
core 1: got 8 packets
Core 1 : Waiting for sync signal
Core 0 : Opened Tx channel : 16
Core 0 : Opened Rx channel : 16
Core 0 : Transmit Queue Number : 816
Core 0 : Receive Queue Number : 664
Core 0 : Opened Rx flow : 16

Core 0 : Transmitting 8 packets..........

*************************************************************
core 2: got interrupt - process 8
core 2: got 8 packets
Core 2 : Waiting for sync signal
Core 0 : Opened Tx channel : 16
Core 0 : Opened Rx channel : 16
Core 0 : Transmit Queue Number : 816
Core 0 : Receive Queue Number : 662
Core 0 : Opened Rx flow : 16

Core 0 : Transmitting 8 packets..........

*************************************************************
core 3: got interrupt - process 8
core 3: got 8 packets
Core 3 : Waiting for sync signal
core 0: waiting for packets on 665
core 0: got interrupt - process 8
core 0: got 8 packets
Core 1 : Got sync signal
Core 3 : Got sync signal
*************************************************************
*************************************************************

Core 2 : Got sync signal
*************************************************************

Core 0 : Waiting for sync signal
Core 0 : Got sync signal
*************************************************************

--------------------Queue status CORE 0----------------------
After packet processing

Tx Free Queue 736 Entry Count : 0
Rx Free Queue 8928 Entry Count : 32
Tx completion Queue 9025 Entry Count : 32
Sync Queue 9026 Entry Count : 4
Sync free Queue 897 Entry Count : 24
Sync Cfg Queue 898 Entry Count : 4
-------------------------------------------------------------

Core 0 : Waiting for other cores to ack sync signal
Core 0 : acks found
Core 1 : Receive free queue closed successfully. Ref count : 3
Core 3 : Receive free queue closed successfully. Ref count : 2
Core 2 : Receive free queue closed successfully. Ref count : 1
Core 0 : Receive free queue closed successfully. Ref count : 0
Core 1 : Transmit completion queue closed successfully. Ref count : 3
Core 3 : Transmit completion queue closed successfully. Ref count : 2
Core 2 : Transmit completion queue closed successfully. Ref count : 1
Core 0 : Transmit completion queue closed successfully. Ref count : 0
Core 1 : Transmit free queue closed successfully. Ref count : 3
Core 3 : Transmit free queue closed successfully. Ref count : 2
Core 2 : Transmit free queue closed successfully. Ref count : 1
Core 0 : Transmit free queue closed successfully. Ref count : 0
Core 1 : Sync queue closed successfully. Ref count : 3
Core 3 : Sync queue closed successfully. Ref count : 2
Core 2 : Sync queue closed successfully. Ref count : 1
Core 0 : Sync queue closed successfully. Ref count : 0
Core 1 : Sync free queue closed successfully. Ref count : 3
Core 3 : Sync free queue closed successfully. Ref count : 2
Core 2 : Sync free queue closed successfully. Ref count : 1
Core 0 : Sync free queue closed successfully. Ref count : 0
Core 1 : Sync queue closed successfully. Ref count : 3
Core 3 : Sync queue closed successfully. Ref count : 2
Core 2 : Sync queue closed successfully. Ref count : 1
Core 0 : Sync queue closed successfully. Ref count : 0
Core 0 : CPPI CPDMA closed successfully
Core 0 : CPPI exit successful
Core 0: Cleaning regions
Core 1 : CPPI CPDMA closed successfully
Core 1 : CPPI exit successful
Core 1: exit QMSS
Core 3 : CPPI CPDMA closed successfully
Core 3 : CPPI exit successful
Core 3: exit QMSS
Core 2 : CPPI CPDMA closed successfully
Core 2 : CPPI exit successful
Core 2: exit QMSS
*******************************************************
******** QMSS Multicore (1) Example Done (PASS) *******
*******************************************************
*******************************************************
******** QMSS Multicore (3) Example Done (PASS) *******
*******************************************************
*******************************************************
******** QMSS Multicore (2) Example Done (PASS) *******
*******************************************************
Core 0: exit QMSS
Core 0 : Deleting RM nameserver names for shared queues
*******************************************************
******** QMSS Multicore (0) Example Done (PASS) *******
*******************************************************
Waiting for children to exit
All children passed
root@k2hk-evm:/usr/include/ti/drv/rm/device/k2h#

You may want to check what were done differently from the example.

Rex

0 Ilya Mazurenko over 7 years ago in reply to Rex Chang

Expert 1300 points

Hello Rex,

Thank you a lot for the reply!

With your help the customer solved the issue with registers.

But there real problem: They can't start interacting between Linux and DSP cores.

Description:

The customer managed to run mpm-transort through shared memory, but this method is not the best, because OS should constantly poll the mailbox. It seems like there should be a qmss mechanism with interrupts for normal operation.

The customer took DSP_EP and ARM_EP examples and tried to adapt it for their own project.

Under Linux it seems like it's working, "/ dev / qmss" has appeared, but we are spinning in a loop:

do {

status = MessageQ_open(rem_msg_q_name, &msg_q_id_arr[i]);

sleep(1);

} while ((status == MessageQ_E_NOTFOUND) ||

(status == MessageQ_E_TIMEOUT));

Where rem_msg_q_name = "RM_Client_DSP_1”, status = MessageQ_E_NOTFOUND.

After loading program to DSP0, then it also goes in loop:

* ======== VirtQueue_startup ========

Void VirtQueue_startup(UInt16 procId, Bool isHost)

{

/* Wait for first kick from host, which happens to coincide with the

* priming of host's receive buffers, indicating host is ready to send.

* Since interrupt is cleared, we throw away this first kick, which is

* OK since we don't process this in the ISR anyway.

Log_print0(Diags_USER1, "VirtQueue_startup: Polling for host int...");

while (!VirtQueue_InterruptProxy_intClear(procId, NULL));

It is waiting something from HOST. What is DSP waiting for?

The Linux part:

Ipc_transportConfig(&TransportRpmsg_Factory);

status = Ipc_start();

rm_msg_hub(MAX_REMOTE_DSPS); -> looped

The DSP part in stack order:

/* Setup TransportRpmsg for host communication */

IpcMgr_ipcStartup(); ->

status = TransportRpmsgSetup_attach(procId, 0); ->

handle = TransportRpmsg_create(procId, &params, &eb);

RPMessage_init(remoteProcId); ->

/* Plug Vring Interrupts, and wait for host ready to recv kick: */

VirtQueue_startup(remoteProcId, isHost);

Also the process stops on “Process 1 : Attempting to open DSP 1 queue: TEST_MsgQ_Proc_1”. Logs in attachment - Log1 with error, Log2 after reboot with check inside ARM

Fullscreen putty_2.log Download

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.01.26 14:07:24 =~=~=~=~=~=~=~=~=~=~=~=                      
root@k2hk-user:/media/ram# rmServer.out /usr/bin/device/k2h/global-resource-list.dtb /usr/bin/device/k2h/policy_dsp_arm.dtb
Starting rmServer.out
Daemon returned 0 as return value.

root@k2hk-user:/media/ram# ./multiProcessTest_k2h.out
*********************************************************
******** TransportQmss Linux Multi-Process Test *********
*********************************************************
TransportQmss Version : 0x02000001
Version String: Linux IPC Transports Revision: 2.0.0.01:Jan 24 2018:19:40:03
Process 1 : Initialized RM_Client1
Process 0 : Initialized RM_Client0
Process 1 : Creating TransportQmss instance
Process 0 : Creating TransportQmss instance
Process 2 : Initialized RM_Client2
Process 2 : Creating TransportQmss instance
Process 3 : Initialized RM_Client3
Process 3 : Creating TransportQmss instance
Process 1 : Local MessageQ: Process_1_MsgQ, QId: 0x80
Process 1 : Attempting to open remote queue: Process_2_MsgQ
Process 3 : Local MessageQ: Process_3_MsgQ, QId: 0x81
Process 3 : Attempting to open remote queue: Process_0_MsgQ
Process 2 : Local MessageQ: Process_2_MsgQ, QId: 0x82
Process 2 : Attempting to open remote queue: Process_3_MsgQ
Process 0 : Local MessageQ: Process_0_MsgQ, QId: 0x83
Process 0 : Attempting to open remote queue: Process_1_MsgQ
Process 2 : Opened Remote queue: Process_3_MsgQ, QId: 0x81
Process 0 : Opened Remote queue: Process_1_MsgQ, QId: 0x80
Process 0 : Allocating round trip test MessageQ msg
Round Trip -    1
Process 0 : Sending msg to Process 1
Process 1 : Opened Remote queue: Process_2_MsgQ, QId: 0x82
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Opened Remote queue: Process_0_MsgQ, QId: 0x83
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    2
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    3
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    4
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    5
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    6
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    7
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    8
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -    9
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   10
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   11
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   12
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   13
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   14
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   15
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   16
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   17
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   18
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   19
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   20
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   21
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   22
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   23
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   24
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   25
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   26
Process 0 : Flushing transport's dst cache of dst MessageQ queue ID 0x80
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Flushing transport's dst cache of dst MessageQ queue ID 0x82
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Flushing transport's dst cache of dst MessageQ queue ID 0x81
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Flushing transport's dst cache of dst MessageQ queue ID 0x83
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   27
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   28
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   29
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   30
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   31
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   32
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   33
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   34
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   35
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   36
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   37
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   38
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   39
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   40
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   41
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   42
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   43
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   44
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   45
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   46
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   47
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   48
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   49
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   50
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   51
Process 0 : Flushing transport's entire dst cache
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Flushing transport's entire dst cache
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Flushing transport's entire dst cache
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Flushing transport's entire dst cache
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   52
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   53
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   54
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   55
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   56
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   57
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   58
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   59
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   60
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   61
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   62
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   63
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   64
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   65
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   66
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   67
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   68
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   69
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   70
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   71
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   72
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   73
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   74
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   75
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   76
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   77
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   78
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   79
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   80
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   81
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   82
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   83
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   84
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   85
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   86
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   87
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   88
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   89
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   90
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   91
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   92
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   93
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   94
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   95
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   96
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   97
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   98
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -   99
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Sending msg to Process 0
Process 0 : Received msg with good data from Process 3
Round Trip -  100
Process 0 : Sending msg to Process 1
Process 1 : Received msg with good data from Process 0
Process 1 : Sending msg to Process 2
Process 2 : Received msg with good data from Process 1
Process 2 : Sending msg to Process 3
Process 3 : Received msg with good data from Process 2
Process 3 : Freeing round trip test MessageQ msg
Test PASSED
Cleaning up
Test Complete!
root@k2hk-user:/media/ram#

Fullscreen putty_1.log Download

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.01.26 13:58:57 =~=~=~=~=~=~=~=~=~=~=~=
rmServer.out /usr/bin/device/k2h/global-resource-list.dtb /usr/bin/device/k2h/policy_dsp_arm.dtb
Starting rmServer.out
Daemon returned 0 as return value.
root@k2hk-user:/media/ram# mpmcl reset dsp0
reset succeeded
root@k2hk-user:/media/ram# mpmcl reset dsp0 1
reset succeeded
root@k2hk-user:/media/ram# mpmcl load dsp0 transportQmssDspEpK2HC66TestProject.out
load succeeded
root@k2hk-user:/media/ram# mpmcl load dsp1 transportQmssDspEpK2HC66TestProject.out
load succeeded
root@k2hk-user:/media/ram# mpmcl run dsp0
run succeeded
root@k2hk-user:/media/ram# mpmcl run dsp0
run succeeded

root@k2hk-user:/media/ram# ./armEpTest_k2h.out
*********************************************************
* ARMv7 Linux TransportQmss Heterogeneous Test (ARM EP) *
*********************************************************
TransportQmss Version : 0x02000001
Version String: Linux IPC Transports Revision: 2.0.0.01:Jan 24 2018:19:40:03
Process 0 : Starting RM Message Hub
Process 1 : Initialized RM_Client0
Process 1 : Opening RM client socket /var/run/rm/rm_client0
Process 1 : Creating TransportQmss instance
Process 0 : Created RM hub queue: RM_Message_Hub, Qid: 0x80
Process 0 : Opening RM_Client_DSP_1
Process 1 : Local MessageQ: TEST_MsgQ_Proc_0, QId: 0x81
Process 1 : Attempting to open DSP 1 queue: TEST_MsgQ_Proc_1
^Z
[1]+  Stopped(SIGTSTP)        ./armEpTest_k2h.out

root@k2hk-user:/media/ram# ./multiProcessTest_k2h.out
*********************************************************
******** TransportQmss Linux Multi-Process Test *********
*********************************************************
TransportQmss Version : 0x02000001
Version String: Linux IPC Transports Revision: 2.0.0.01:Jan 24 2018:19:40:03
Process 1 : Initialized RM_Client1
Process 1 : Creating TransportQmss instance
Process 0 : Initialized RM_Client0
Process 0 : Creating TransportQmss instance
Process 3 : Initialized RM_Client3
Process 3 : Creating TransportQmss instance
Process 2 : Initialized RM_Client2
Process 2 : Creating TransportQmss instance
Process 1 : Local MessageQ: Process_1_MsgQ, QId: 0x82
Process 1 : Attempting to open remote queue: Process_2_MsgQ
TransportQmss_create : mpm_transport_open failed
ERROR Process 0 : Failed to create TransportQmss handle
Process 2 : Local MessageQ: Process_2_MsgQ, QId: 0x83
Process 2 : Attempting to open remote queue: Process_3_MsgQ
Process 3 : Local MessageQ: Process_3_MsgQ, QId: 0x84
Process 3 : Attempting to open remote queue: Process_0_MsgQ
Process 1 : Opened Remote queue: Process_2_MsgQ, QId: 0x83
Process 2 : Opened Remote queue: Process_3_MsgQ, QId: 0x84
^Z
[2]+  Stopped(SIGTSTP)        ./multiProcessTest_k2h.out
root@k2hk-user:/media/ram#

Edit 1: They collect their own Linux image with the connection of the necessary modules:

ARAGO_IMAGE_EXTRA_INSTALL ?= ""

IMAGE_INSTALL += " \

packagegroup-core-boot \

${ARAGO_IMAGE_EXTRA_INSTALL} \

IMAGE_INSTALL += "\

packagegroup-arago-base \

packagegroup-arago-console \

packagegroup-arago-base-tisdk \

packagegroup-arago-test \

${VIRTUAL-RUNTIME_initramfs} \

IMAGE_INSTALL += "openssh openssh-sftp openssh-sftp-server gdbserver multiprocmgr ipc-transport-qmss uio-module-drv qmss-lld qmss-pdsp-fw rm-lld"

In attachment there are logs wth various options (logs.zip).5707.logs.zip

Edit 2: I should add, as i understand, almost similar problem described in:

https://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/p/541046/1983711

https://e2e.ti.com/support/dsp/c6000_multi-core_dsps/f/639/t/544167

But i have not found there any complete answer from TI's team. Or may be we are missing something...

0 Rex Chang over 7 years ago in reply to Ilya Mazurenko

TI__Guru 50170 points

Hi, Ilya,

From the logs, the benchmark and multi-process werefine, but armEP test failed due to IPC_start failed. In the links you provided, customer resolved it by himself and found that " it turns out the place where the shared region is defined is really important. If placed in the middle of the configuration file (like the original dsp_ep.cfg from the example), the IPC start blocks. If the shared region is defined at the end of the configuration file, it works! "

Could your customer compare its configuration with the one posted in the thread to see if that is the problem?

Rex

0 Ilya Mazurenko over 7 years ago in reply to Rex Chang

Expert 1300 points

Hello Rex,

Thank you for the reply!

The additional question from the customer:

1)Due to source code of prj «transportQmssDspEpK2HC66TestProject» in «dsp_ep.c» file, the IPC initialization is:

/* Setup TransportRpmsg for host communication */

IpcMgr_ipcStartup();

/* Setup IPC for DSP to DSP communication */

status = Ipc_start();

Then Ipc_start() turns to the endless cyrcle.

But in documentation mentioned that Keystone 2 in Linux-DSP mode should only use the IpcMgr_ipcStartup(). But when doing so, the ti_sdo_ipc_SharedRegion_attach() and SharedRegion_getHeap() wasnt called and the initialization of ti_sdo_ipc_GateMP_Instance_init() returns NULL.

The question:

Should the SharedRegion_module->regions[0].heap be initializated in ARM-Linux?

If it is, then: Why wasnt it initializated by «armEpTest_k2h.out» from «ipc-transport-qmss-test»?

Also the customer adds:

1)If Ipc_start() left in DSP module then it doesnt turn to the cycle. And then, if we try to make Ipc_attach(2) to another DSP, after calling status = Ipc_procSyncStart(remoteProcId, Ipc_module->ipcSharedAddr) it returns status = -11 (Ipc_E_NOTREADY).

Log from DSP2:

2 Resource entries at 0x810000

Core 1 : ******************************************************

Core 1 : SYS/BIOS DSP TransportQmss Heterogeneous Test (DSP EP)

Core 1 : ******************************************************

Core 1 : Device name: TMS320TCI6636

Core 1 : Processor names: HOST,CORE0,CORE1

Core 1 : IPC Core ID: 2

Core 1 : Number of DSPs 2

Core 1 : Number of test iterations: 100

Core 1 : Starting IPC core 2 with name ("CORE1")

TransportRpmsg_Instance_init: remoteProc: 0

registering rpmsg-proto:rpmsg-proto service on 61 with HOST

[t=0x00052e1c] xdc.runtime.Main: NameMap_sendMessage: HOST 53, port=61

2) In the example project (transportQmssDspEpK2HC66TestProject) there are function:

/* The region heap will do the alignment */

regionHeap = SharedRegion_getHeap(obj->regionId);

It returns NULL without Ipc_start() (If we left only IpcMgr_ipcStartup()).

If we turns on the Ipc_start(), when the DSP2 is trying attach to DSP1 the process crashes with reinitialization(because through the RM server we can work only with HOST).

But at this moment we dont need operations between DSP cores.

Stack of calls when NULL is returned:

initTsk >>>

GateMP_create >>>

ti_sdo_ipc_GateMP_create >>>

ti_sdo_ipc_GateMP_Instance_init__E >>>

ti_sdo_ipc_GateMP_Instance_init >>>

SharedRegion_getHeap(0) return NULL!!!

Stack when the error in Ipc_start() CORE0 occurs:

Ipc_start >>>

status = Ipc_attach(baseId = 2); >>>

status = ti_sdo_ipc_MessageQ_SetupTransportProxy_attach(2, 0); return -1; !!!

TransportRpmsgSetup_attach

TransportRpmsg_Instance_init

/* This MessageQ Transport over RPMSG only talks to the "HOST" for now: */ !!!

Could you help us please to reslove our customer's problem. This is very important project.

The customer wants to made data reception on PCIe and distribution to DSP, and then processing the data and data tranfer with SPI in two directions.

0 Rex Chang over 7 years ago in reply to Ilya Mazurenko

TI__Guru 50170 points

Hi, Ilya,

I'll close this thread for QMSS. Could you submit a new thread for this IPC question? I'll have a DSP engineer to help in the new thread.

Rex

0 Ilya Mazurenko over 7 years ago in reply to Rex Chang

Expert 1300 points

Hello Rex,

I have made new thread for the last post:
e2e.ti.com/.../662636

Ilya.

Processors

Processors forum

66AK2H06: Problem with QMSS