[TDA4] Shared memory between A72 <-> c66, A72 <-> r5f, A72 <-> c71

Hi,

  Add a question:

   (5) Is  shared memory in r5f core have cache or not ?

 pls, update. 

Thanks & Regards,

Lei

<1>

See this page for information about memory map and how to change it

${PSDKRA_INSTALL_PATH}/psdk_rtos_auto/docs/user_guide/developer_notes_memory_map.html

<2> <4>

A demo of shared memory including cache ops can be found here

vision_apps/apps/basic_demos/app_linux_arm_ipc

See also vision_apps/utils/ipc

<3> <5>

ION Shared memory, typically used for exchanging large pixel buffers, is cached.

Some other IPC memory used for passing smaller messages is non-cached.

See above developer note for more details.

NOTE: My recommendation is not to change memory map unless you really need it. Most use-cases can be run with default memory map. I recommend you try with this and only later change it. Note, R5F and C6x are 32b so if you spill beyond 32b space things may not work

regards
Kedar

Hi Kedar,

 1. ION Shared memory  is virtual memory. 

     How to do if i want to Physical Shared memory  ?

 2. gen_linker_mem_map.py auto generate file : k3-j721e-vision_apps.dts,

     How to apply this file?  

　 Need I to modify the file manually follow this ?

/*
* IMPORTANT NOTE: Follow below instructions to apply the updated memory map to linux dts/dtso/dtsi files,
*
* 1. Copy the memory sections, from the generated dts file, to the file shown below under reserved_memory: reserved-memory { ... }
* ${LINUX_KERNEL_PATH}/arch/arm64/boot/dts/ti/k3-j721e-som-p0.dtsi
*
* 2. In file ${LINUX_KERNEL_PATH}/arch/arm64/boot/dts/ti/k3-j721e-auto-common.dtso
* - Remove the fragment@xyz { ... } entries for xyz = 101 to 119
*
* 3. In file ${LINUX_KERNEL_PATH}/arch/arm64/boot/dts/ti/k3-j721e-vision-apps.dtso
* - Remove the &reserved_memory { ... } entry
*
* 4. The entries are removed since the same are updated in k3-j721e-som-p0.dtsi, hence we should not have duplicate outdated one's
*
* 5. Rebuild the dtb, dtbo and use the updated dtb, dtbo files
* - In PSDKLA install directory, doing below should build the dtb and dtbo
* make linux-dtbs
* - Copy the below updated dtb, dtbo files to "boot/" folder in your target linux SD card filesystem
* arch/arm64/boot/dts/ti/k3-j721e-common-proc-board.dtb
* arch/arm64/boot/dts/ti/k3-j721e-auto-common.dtbo
* arch/arm64/boot/dts/ti/k3-j721e-vision-apps.dtbo
*/

 3. vision_apps/apps/basic_demos/app_tirtos/tirtos_linux shared memory between in c66x_1, c66x_2, c7x_1, mcu2_0, mcu2_1, mpu1 core, 

  So how to do add mcu1_0, mcu1_1, mcu3_0, mcu3_1 core  ?

Thanks & Regards,

Lei

Hi Kedar,

4. If change DDR memory form 4G to 8G, how to porting it ?

5. Can you tell  the API of synchronous shared memory operations ?

Please update, 

Thanks & Regards,

Lei

Hi Kedar,

6. How to set   shared memory to  memreserve ?

  We  don't want to use ION memory  in A72 side,  but if set shared memory to no-map reserved-memory, 

 It will have bus error when memset

Please update, 

Thanks & Regards,

Lei

1.

See vision_apps/utils/mem/include/app_mem.h

appMemGetVirt2PhyBufPtr()

See vision_apps/utils/mem/src/app_mem_ion.c for implementation of the same.

2.

Applying the dtsi file has to be done manually

3.

One can add mcu1_0, mcu1_1, mcu3_0, mcu3_1 similar to mcu2-1 core. Unfortunately we don't have a ready example for the same. We plan to have a example showing this in future releases.

4.

Firstly we need to check if 8G is supported on the EVM. I think 4G is the DDR size on TI EVM.

Assuming 8G is available on a board one needs to change DDR init sequence in SPL/uboot to support it.

5."API of synchronous shared memory", not sure what you mean by this. If you mean to say cache operations, then again see app_mem.h for cache operation APIs.

6. To reserve a memory you need to set in dts/dtsi file. See ION memory segment for example.

However if you try to simply mmap this via /dev/mem, there few things to note.

a. Any memory mapped to user space via /dev/mem is mapped as strongly ordered, non-cache memory. If you want to say use this for large pixel data sharing between say A72 and some other CPU/HW, this this will be inefficient from A72 point of view since data is not cached.

b. Memory mapped via /dev/mem MUST be accessed in a aligned manner, ex, 32b access MUST be 32b aligned, 64b access MUST be 64b aligned, else one gets a bus error. Optimized functions like memcpy could result in unaligned accesses and hence you will see a bus error.

It is strong recommendation to use a memory allocator like ION or develop a similar kernel module for shared memory access.

Both the above constraints are not present when using ION memory allocator.

regards
Kedar

ps, sorry for delay in responding (it was due to year end holidays)

Hi Kedar,

  Thanks for you help.

  This follow code will lead system hang. Why ?

int lock_mem(void* x){
asm("ldaxr w2,[x0]");
}
int main(int argc,char**argv){
long long addr = 0xbc000000;
long long size = 1*1024*1024;
int fd = open("/dev/mem",O_RDWR|O_SYNC);
if(fd < 0){
perror("open");
return 0;
}
char *ret = mmap(NULL,size,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_LOCKED,fd,addr);
if((intptr_t)ret < 0){
perror("mmap");
return 0;
}
lock_mem(ret);
return 0;

}

Thanks & Regards,

Lei

Hi Kedar,

system hang (A72) log:

ERROR: Unhandled External Abort received on 0x80000001 at EL3!
ERROR: exception reason=0 syndrome=0xbf000002
PANIC in EL3 at x30 = 0x000000007000442c
x0 = 0x0000000000000000
x1 = 0x0000000000000060
x2 = 0x0000000000000060
x3 = 0x000000000000000b
x4 = 0x0000000000000062
x5 = 0x0000000000000008
x6 = 0x000000000000003b
x7 = 0x0000000000000000
x8 = 0x0000000000000062
x9 = 0x0000000000000000
x10 = 0x0101010101010101
x11 = 0x0000000000000028
x12 = 0x00000000000000d1
x13 = 0x0000ffffdd52d92d
x14 = 0x0000ffffb03e0f60
x15 = 0x00000000000000d5
x16 = 0x0000ffffb03e9ad8
x17 = 0x0000ffffb0443308
x18 = 0x0000ffffdd52d9d0
x19 = 0x0000000000000000
x20 = 0x00000000bf000002
x21 = 0x0000600000000048
x22 = 0x0000000000000000
x23 = 0x0000ffffb0443350
x24 = 0x0000000000000000
x25 = 0x0000000000000000
x26 = 0x0000000000000000
x27 = 0x0000000000000000
x28 = 0x0000000000000000
x29 = 0x000000007000a160
scr_el3 = 0x000000000000073d
sctlr_el3 = 0x0000000030cd183f
cptr_el3 = 0x0000000000000000
tcr_el3 = 0x0000000080803520
daif = 0x00000000000002c0
mair_el3 = 0x00000000004404ff
spsr_el3 = 0x0000000080000000
elr_el3 = 0x0000ffffb03e96f4
ttbr0_el3 = 0x000000007000e420
esr_el3 = 0x00000000bf000002
far_el3 = 0x0000000000000000
spsr_el1 = 0x0000000060000000
elr_el1 = 0x0000ffffb03e9450
spsr_abt = 0x0000000000000000
spsr_und = 0x0000000000000000
spsr_irq = 0x0000000000000000
spsr_fiq = 0x0000000000000000
sctlr_el1 = 0x0000000034d5d91d
actlr_el1 = 0x0000000000000000
cpacr_el1 = 0x0000000000300000
csselr_el1 = 0x0000000000000000
sp_el1 = 0xffff000011130000
esr_el1 = 0x0000000056000000
ttbr0_el1 = 0x00000008c16d6c00
ttbr1_el1 = 0x108c000080e20000
mair_el1 = 0x0000bbff440c0400
amair_el1 = 0x0000000000000000
tcr_el1 = 0x00000034f5507510
tpidr_el1 = 0x0000800876e60000
tpidr_el0 = 0x0000ffffb0509430
tpidrro_el0 = 0x0000000000000000
dacr32_el2 = 0x0000000000000000
ifsr32_el2 = 0x0000000000000000
par_el1 = 0x0000000000000000
mpidr_el1 = 0x0000000080000001
afsr0_el1 = 0x0000000000000000
afsr1_el1 = 0x0000000000000000
contextidr_el1 = 0x0000000000000000
vbar_el1 = 0xffff000008081800
cntp_ctl_el0 = 0x0000000000000005
cntp_cval_el0 = 0x0000009bc2b0160c
cntv_ctl_el0 = 0x0000000000000000
cntv_cval_el0 = 0x0000000000000000
cntkctl_el1 = 0x00000000000000e6
sp_el0 = 0x000000007000a160
isr_el1 = 0x0000000000000040
cpuectlr_el1 = 0x0000001b00000040
cpumerrsr_el1 = 0x0000000000000000
l2merrsr_el1 = 0x0000000000000000

Thanks & Regards,

Lei

Lei,

Why are you trying to expressly call "ldaxr" instructions? Why are you not using synchronization tools like mutex etc?

- Subhajit

Subhajit,

 Because we need locks shared by multiple process in linux 

Thanks & Regards,

Lei

If that is the case, you can put your mutex in a shared memory that is accessible by multiple processes. You can make the mutex shared among processes by calling this:

int pthread_mutexattr_setpshared(pthread_mutexattr_t *attr,
    int pshared);

You can use Posix semaphore for multi process mutex.

Subhajit,

There will be the following problems on multi-core shared memory, but there will be no such problems on other memory ranges:
  1. Multithreaded shared locks cannot guarantee atomicity in critical sections
  2. The lock priority reversal function cannot be turned on, otherwise it will fail to unlock：

　　pthread_mutexattr_setprotocol(&attr,PTHREAD_PRIO_INHERIT);

   log:
  Expectations:
  value of atomic add var is 40000000
  abnormal situation:
  value of atomic add var is 37021808

Relate source code，See attachment  

Thanks & Regards,

Lei

#include <sys/mman.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <pthread.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/wait.h>


pthread_mutex_t *lock;;

void* thread_fun(void*arg){
    volatile int *p = arg;
    int i = 0;
    for(i = 0; i < 10000000;i++){
        int ret = pthread_mutex_lock(lock);
        if(ret != 0){
            printf("lock ret = %d,errno = %d\n",ret,errno);
        }
        (*p)++;
       // usleep(1);
        ret = pthread_mutex_unlock(lock);
        if(ret != 0){
            printf("unlock ret = %d,errno = %d\n",ret,errno);
        }
    }
    return NULL;
}

int main(int argc,char**argv){
    long long addr = 0xC0000000;
    long long size = 0x100000;
    int fd = open("/dev/mem",O_RDWR);//|O_SYNC);
    if(fd < 0){
        perror("open");
        return 0;
    }
    char *ret = mmap(NULL,size,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_LOCKED,fd,addr);
    if((intptr_t)ret < 0){
        perror("mmap");
        return 0;
    }
    *(volatile int*)ret = 0;
    lock = (void*)((char*)ret+4);
    pthread_mutexattr_t attr;
    pthread_mutexattr_init(&attr);
    int pret = pthread_mutexattr_setpshared(&attr,PTHREAD_PROCESS_SHARED);
    if(pret != 0){
        printf("pret is %d@%d\n",pret,__LINE__);
        return 0;
    }
    //pret = pthread_mutexattr_setprotocol(&attr,PTHREAD_PRIO_INHERIT);
    if(pret != 0){
        printf("pret is %d@%d\n",pret,__LINE__);
        return 0;
    }
    pret = pthread_mutex_init(lock,&attr);
    if(pret != 0){
        printf("pret is %d@%d\n",pret,__LINE__);
        return 0;
    }

    int fork_ret = fork();
    pthread_t tid[2];
    int pthread_ret = pthread_create(&tid[0],NULL,&thread_fun,ret);
    pthread_ret = pthread_create(&tid[1],NULL,&thread_fun,ret);
    pthread_join(tid[0],NULL);
    pthread_join(tid[1],NULL);
    if(fork_ret){
        printf("value of atomic add var is %d\n",*(volatile int*)ret);
    }
    return 0;
}

/* 在多核共享内存上面会有如下问题，而在其他的内存区间上面不会有这种问题：
 * 1. 多线程共享锁无法保证临界区的原子性
 * 2. 锁优先级反转功能不能打开，否则会解锁失败
 * log:
 * 期望情况:
 *    value of atomic add var is 40000000
 * 异常情况:
 *    value of atomic add var is 37021808
 * */

I can see that you have done two pthread_join calls to make sure the threads exit. But I do not see a waitpid call to ensure that the forked process returning.

- Subhajit

Subhajit,

Thank you for u help.

  if enable this function:

　　pthread_mutexattr_setprotocol(&attr,PTHREAD_PRIO_INHERIT);

 then multithreaded shared locks cannot guarantee atomicity in critical sections

   log:
  Expectations:
  value of atomic add var is 40000000
  abnormal situation:
  value of atomic add var is 36976716

Relate source code，See attachment  

#include <sys/mman.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <pthread.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/wait.h>


pthread_mutex_t *lock;;

void* thread_fun(void*arg){
    volatile int *p = arg;
    int i = 0;
    for(i = 0; i < 10000000;i++){
        int ret = pthread_mutex_lock(lock);
        if(ret != 0){
            printf("lock ret = %d,errno = %d\n",ret,errno);
        }
        (*p)++;
       // usleep(1);
        ret = pthread_mutex_unlock(lock);
        if(ret != 0){
    //        printf("unlock ret = %d,errno = %d\n",ret,errno);
        }
    }
    return NULL;
}

int main(int argc,char**argv){
    long long addr = 0xC0000000;
    long long size = 0x100000;
    int fd = open("/dev/mem",O_RDWR);//|O_SYNC);
    if(fd < 0){
        perror("open");
        return 0;
    }
    char *ret = mmap(NULL,size,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_LOCKED,fd,addr);
    if((intptr_t)ret < 0){
        perror("mmap");
        return 0;
    }
    *(volatile int*)ret = 0;
    lock = (void*)((char*)ret+4);
    pthread_mutexattr_t attr;
    pthread_mutexattr_init(&attr);
    int pret = pthread_mutexattr_setpshared(&attr,PTHREAD_PROCESS_SHARED);
    if(pret != 0){
        printf("pret is %d@%d\n",pret,__LINE__);
        return 0;
    }
    pret = pthread_mutexattr_setprotocol(&attr,PTHREAD_PRIO_INHERIT);
    if(pret != 0){
        printf("pret is %d@%d\n",pret,__LINE__);
        return 0;
    }
    pret = pthread_mutex_init(lock,&attr);
    if(pret != 0){
        printf("pret is %d@%d\n",pret,__LINE__);
        return 0;
    }

    int fork_ret = fork();
    pthread_t tid[2];
    int pthread_ret = pthread_create(&tid[0],NULL,&thread_fun,ret);
    pthread_ret = pthread_create(&tid[1],NULL,&thread_fun,ret);
    pthread_join(tid[0],NULL);
    pthread_join(tid[1],NULL);
    if (fork_ret) {
		int s;
		waitpid(-1, &s,NULL);
        printf("value of atomic add var is %d\n",*(volatile int*)ret);
    }
    return 0;
}

/* 在多核共享内存上面会有如下问题，而在其他的内存区间上面不会有这种问题：
 * 1. 多线程共享锁无法保证临界区的原子性
 * 2. 锁优先级反转功能不能打开，否则会解锁失败
 * log:
 * 期望情况:
 *    value of atomic add var is 40000000
 * 异常情况:
 *    value of atomic add var is 36976716
 * */

Thanks & Regards,

Subhajit,

Thank you for u help.

1. Remember: PTHREAD_PRIO_INHERIT is very, very slow on you machine.

  Why this ?

2.  we want alloc share memory in Ti ION allocer , can not use shm_open("/mutex-mem"), but it is the same issue in open("/dev/mem"), So we used the 

open("/dev/mem") function to be test code.

3. So  please do fix   test_mmap_mt_inherit_2.c  issue.

Thanks & Regards,

1. I do not have a clear answer for this. Unless this is important, we should not spend time on answering this

2 and 3. Please refer to my attached application and make appropriate changes to your code.

- Subhajit

Subhajit,

Unfortunately shm_open("/mutex-mem") does not meet our needs. Becase we want used shard memory between in all cores(A72,c66,c71).

Thanks & Regards,

lei

I did not mention anything about shmem in particular. You can use mmaped memory just like I used shmem.

I was referencing to understanding the logic in my application and modify your source code accordingly.

Unfortunately I do not have a setup to test out devmem based direct memory sharing, and therefore I can only provide you reference codes that adhere to linux / posix standard APIs

- Subhajit

Subhajit,

1727.main.c use shm_open("/mutex-mem")  function . so the phyical addr of alloced shared memory is not in 0xC0000000(multi-core share  phyical addr) area, 

test_mmap_mt_inherit.c use open("/dev/mem") function, just ensure the phyical addr of alloced shared memory is in 0xC0000000(multi-core share  phyical addr) area,

so It is not shm_open("/mutex-mem")  function or open("/dev/mem") function issue. It is used pthread_mutexattr_setpshared(&attr,PTHREAD_PROCESS_SHARED) function in  0xC0000000(multi-core share  phyical addr) area issue

Thanks & Regards,

lei

Hi, experts

 Any update?

Thanks & Regards,

lei