Other Parts Discussed in Thread: TPIC2810
Tool/software: Linux
Hi,
I'm developing over IDK am5728 with PROCESSOR-SDK-LINUX-AM57X version v05.00 and PROCESSOR-SDK-RTOS-AM57X version v05.00.
I'm using the jailhouse hypervisor and I'm able to run linux on core 0 and on core 1 the jailhouse inmate demo processor_sdk_rtos_am57xx_5_00_00_15/demos/jailhouse-inmate/rtos/icss_emac
Now I'm trying to create inter cell communication. I need the cores to be able to send interrupts to one another.
I'm using the uio-ivshmem module from https://github.com/henning-schild-work/ivshmem-guest-code
To simplify things, first I'm trying to send an interrupt from a baremetal inmate to the root-cell Linux. Later I will try with RTOS inmate.
Once I have that working I will need the RTOS inmate to receive an interrupt from the Linux root cell.
In the dts file some GIC inputs are reserved for the inmate:
ocp {
pruss1_eth {
status = "disabled";
};
pruss2_eth {
status = "disabled";
};
ti,irqs-skip = <10 44 127 129 133 134 135 136 137 139 140>;
};
};
};
in the root cell I set
.vpci_irq_base = 100 - 32
and in the inmate cell
.vpci_irq_base = 134 - 32,
pci and uio_ivshmem init seem fine
[ 45.928817] OF: PCI: host bridge /vpci@0 ranges:
[ 45.933482] OF: PCI: MEM 0x30100000..0x30101fff -> 0x30100000
[ 45.939690] pci-host-generic 30000000.vpci: ECAM at [mem 0x30000000-0x300fffff] for [bus 00]
[ 45.948567] pci-host-generic 30000000.vpci: PCI host bridge to bus 0001:00
[ 45.956098] pci_bus 0001:00: root bus resource [bus 00]
[ 45.961539] pci_bus 0001:00: root bus resource [mem 0x30100000-0x30101fff]
[ 45.968480] pci 0001:00:00.0: [1af4:1110] type 00 class 0xff0000
[ 45.968517] pci 0001:00:00.0: reg 0x10: [mem 0x00000000-0x000000ff 64bit]
[ 45.968881] PCI: bus0: Fast back to back transfers disabled
[ 45.974591] pci 0001:00:00.0: BAR 0: assigned [mem 0x30100000-0x301000ff 64bit]
[ 45.982475] virtio-pci 0001:00:00.0: enabling device (0000 -> 0002)
[ 45.988997] uio_ivshmem 0001:00:00.0: using jailhouse mode
[ 45.995644] The Jailhouse is opening.
[ 46.603491] Created Jailhouse cell "AM572X-IDK-ICSS"
root@am57xx-evm:~# lspci -v
0000:00:00.0 PCI bridge: Texas Instruments Multicore DSP+ARM KeyStone II SOC (rev 01) (prog-if 00 [Normal decode])
Memory at 20100000 (64-bit, non-prefetchable) [size=1M]
Bus: primary=00, secondary=01, subordinate=ff, sec-latency=0
I/O behind bridge: None
Memory behind bridge: None
Prefetchable memory behind bridge: None
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [70] Express Root Port (Slot-), MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: pcieport
0001:00:00.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
Subsystem: Red Hat, Inc Inter-VM shared memory
Flags: bus master, fast devsel, latency 0, IRQ 179
Memory at 30100000 (64-bit, non-prefetchable) [size=256]
Kernel driver in use: uio_ivshmem
Still, I can't generate an interrupt from the inmate to the root cell.
cat /proc/interrupts
...
179: 0 GICv2 100 Edge uio_ivshmem
...
If I understand correctly, I need to write the value 1 to the address 0x3010000c. I'm using gic-demo to do so from the interrupt context.
Whenever the inmate writes to this address I get a dump from the kernel
[ 53.306728] ------------[ cut here ]------------
[ 53.311518] WARNING: CPU: 0 PID: 28 at /home/stx-ti/Projects/tisdk/build/arago-tmp-external-linaro-toolchain/work-shared/am57xx-evm/kernel-source/drivers/bus/omap_l3_noc.c:147 l3_interrupt_handler+0x254/0x370
[ 53.330115] 44000000.ocp:L3 Standard Error: MASTER MPU TARGET PCIE2 (Read): At Address: 0x0010000C : Data Access in User mode during Functional access
[ 53.343649] Modules linked in: jailhouse(O) uio_ivshmem(O) uio can_raw can ecatmc r8169 mc_gp_timer ec_master xhci_plat_hcd xhci_hcd usbcore dwc3 udc_core usb_common ti_prueth pru_rproc pruss pruss_intc snd_soc_omap_hdmi_audio omap_aes_driver omap_sham pruss_soc_bus c_can_platform c_can can_dev omap_wdt ahci_platform libahci_platform libahci libata scsi_mod ti_vpe ti_vip ti_sc ti_csc ti_vpdma dwc3_omap rtc_omap extcon_palmas rtc_palmas gpio_pisosr ov2659 omap_des gpio_tpic2810 v4l2_fwnode des_generic crypto_engine omap_crypto omap_remoteproc virtio_rpmsg_bus rpmsg_core remoteproc sch_fq_codel cryptodev(O)
[ 53.397569] CPU: 0 PID: 28 Comm: irq/23-l3-app-i Tainted: G W O 4.14.40-rt29-gd5443cbd3b #8
[ 53.397572] Hardware name: Generic DRA74X (Flattened Device Tree)
[ 53.397574] Backtrace:
[ 53.397590] [<c020b220>] (dump_backtrace) from [<c020b504>] (show_stack+0x18/0x1c)
[ 53.397597] r7:00000009 r6:60000013 r5:00000000 r4:c105773c
[ 53.397606] [<c020b4ec>] (show_stack) from [<c091609c>] (dump_stack+0x90/0xa4)
[ 53.397615] [<c091600c>] (dump_stack) from [<c022a864>] (__warn+0xec/0x104)
[ 53.397620] r7:00000009 r6:c0bc94c8 r5:00000000 r4:d422fe40
[ 53.397628] [<c022a778>] (__warn) from [<c022a8bc>] (warn_slowpath_fmt+0x40/0x48)
[ 53.397634] r9:00000011 r8:d4206850 r7:c0bc9334 r6:00000002 r5:c0bc93e8 r4:c0bc9498
[ 53.397642] [<c022a880>] (warn_slowpath_fmt) from [<c051a5f0>] (l3_interrupt_handler+0x254/0x370)
[ 53.397645] r3:d42066c0 r2:c0bc9498
[ 53.397647] r4:80080001
[ 53.397654] [<c051a39c>] (l3_interrupt_handler) from [<c027fc64>] (irq_forced_thread_fn+0x28/0x7c)
[ 53.397659] r10:c027fc3c r9:d4206bc0 r8:d41df100 r7:00000001 r6:00000000 r5:d41df100
[ 53.397661] r4:d4206bc0
[ 53.397667] [<c027fc3c>] (irq_forced_thread_fn) from [<c027ffc0>] (irq_thread+0x130/0x208)
[ 53.397671] r7:00000001 r6:00000000 r5:ffffe000 r4:d4206be4
[ 53.397678] [<c027fe90>] (irq_thread) from [<c02481e4>] (kthread+0x164/0x16c)
[ 53.397683] r10:d406bb28 r9:c027fe90 r8:d4206bc0 r7:d422e000 r6:00000000 r5:d4206c00
[ 53.397685] r4:d421e040
[ 53.397693] [<c0248080>] (kthread) from [<c02079a0>] (ret_from_fork+0x14/0x34)
[ 53.397698] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0248080
[ 53.397700] r4:d4206c00
[ 53.397706] ---[ end trace 0000000000000004 ]---
and a different interrupt is incremented:
cat /proc/interrupts
...
23: 3 WUGEN 10 Level l3-app-irq
...
Is there anything wrong with the cells configuration?
Is writing to 0x3010000c the right way to trigger the uio_ivshmem interrupt in the Linux side?
Can TI supply a working setup for uio-ivshmem comm between a linux root cell and a rtos inmate as a starting point?
Thanks a lot,
Nir.
/* * Jailhouse, a Linux-based partitioning hypervisor * * Configuration for uart-demo inmate on AM57XX-EVM: * * Copyright (c) Texas Instruments, Inc. * * Authors: * Vitaly Andrianiov <vitalya@ti.com> * * This work is licensed under the terms of the GNU GPL, version 2. See * the COPYING file in the top-level directory. */ #include <jailhouse/types.h> #include <jailhouse/cell-config.h> #define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0]) struct { struct jailhouse_cell_desc cell; __u64 cpus[1]; struct jailhouse_memory mem_regions[10]; struct jailhouse_irqchip irqchips[1]; struct jailhouse_pci_device pci_devices[1]; } __attribute__((packed)) config = { .cell = { .signature = JAILHOUSE_CELL_DESC_SIGNATURE, .revision = JAILHOUSE_CONFIG_REVISION, .name = "AM572X-IDK-ICSS", .flags = JAILHOUSE_CELL_PASSIVE_COMMREG, .num_irqchips = 1, .cpu_set_size = sizeof(config.cpus), .num_memory_regions = ARRAY_SIZE(config.mem_regions), .num_pci_devices = ARRAY_SIZE(config.pci_devices), .vpci_irq_base = 134 - 32, }, .cpus = { 0x2, }, .mem_regions = { /* OCRAM buffer area */ { .phys_start = 0x40340000, .virt_start = 0x40340000, .size = 0xC0000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, /* UART... */ { .phys_start = 0x48020000, .virt_start = 0x48020000, .size = 0x1000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, /* ... */ { .phys_start = 0x48050000, .virt_start = 0x48050000, .size = 0x10000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, { /* TIMER8 */ .phys_start = 0x48826000, .virt_start = 0x48826000, .size = 0x1000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, /* L4_CFG */ { .phys_start = 0x4a000000, .virt_start = 0x4a000000, .size = 0xE10000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, /* PRUSS */ { .phys_start = 0x4b200000, .virt_start = 0x4b200000, .size = 0x100000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, /* RAM loader */ { .phys_start = 0xed000000, .virt_start = 0x0, .size = 0x10000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE, }, /* RAM RTOS 224MB*/ { .phys_start = 0xe0000000, .virt_start = 0x80000000, .size = 0xd000000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE | JAILHOUSE_MEM_LOADABLE, }, /* IVSHMEM registers for PCI 0001:00:00.0 */ { .phys_start = 0x30100000, .virt_start = 0x30100000, .size = 0x1000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO | JAILHOUSE_MEM_ROOTSHARED, }, /* IVSHMEM shared memory region for 0001:00:00.0 */ { .phys_start = 0xEEE00000, .virt_start = 0xEEE00000, .size = 0x100000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED, }, }, .irqchips = { /* GIC */ { .address = 0x48211000, .pin_base = 64, .pin_bitmap = { 0x00001000, 0x0, 0x80000000, 0x000003C2, }, }, }, .pci_devices = { { /* 02:00.0 */ .type = JAILHOUSE_PCI_TYPE_IVSHMEM, //.iommu = 1, .bdf = 0 << 3, .bar_mask = { 0xffffff00, 0xffffffff, 0x00000000, 0x00000000, 0x00000000, 0x00000000, }, .num_msix_vectors = 0, .shmem_region = 9, .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED, }, }, };
/* * Jailhouse, a Linux-based partitioning hypervisor * * Copyright (c) ARM Limited, 2014 * Copyright (c) Siemens AG, 2014-2017 * * Authors: * Jean-Philippe Brucker <jean-philippe.brucker@arm.com> * Jan Kiszka <jan.kiszka@siemens.com> * * This work is licensed under the terms of the GNU GPL, version 2. See * the COPYING file in the top-level directory. */ #include <mach.h> #include <inmate.h> #include "ti-utils.h" #define BEATS_PER_SEC 10 static u64 ticks_per_beat; static volatile u64 expected_ticks; /* * Enables blinking LED * Banana Pi: register 0x1c2090c, pin 24 * Orange Pi Zero: register 0x1c20810, pin 17 */ static void *led_reg; static unsigned int led_pin; unsigned long counter = 0; u32* shmem = (unsigned int*)0xeee00000; u32* regs = (unsigned int*)0x30100000; static void handle_IRQ(unsigned int irqn) { static u64 min_delta = ~0ULL, max_delta = 0; u64 delta; if (irqn != TIMER_IRQ) return; delta = timer_get_ticks() - expected_ticks; if (delta < min_delta) min_delta = delta; if (delta > max_delta) max_delta = delta; ++counter; /*printk("Timer fired, jitter: %6ld ns, min: %6ld ns, max: %6ld ns\n", (long)timer_ticks_to_ns(delta), (long)timer_ticks_to_ns(min_delta), (long)timer_ticks_to_ns(max_delta));*/ ti_putsui("Timer fired, jitter: ", (long)timer_ticks_to_ns(delta), false); ti_putsui(" ns, min: ", (long)timer_ticks_to_ns(min_delta), false); ti_putsui(" ns, max: ", (long)timer_ticks_to_ns(max_delta), false); ti_puts(" ns\n\r"); if(-1 == counter) { *shmem = counter; mmio_write32(regs + 3, 1); counter = 0; } /* if (led_reg) mmio_write32(led_reg, mmio_read32(led_reg) ^ (1 << led_pin));*/ expected_ticks = timer_get_ticks() + ticks_per_beat; timer_start(ticks_per_beat); } void inmate_main(void) { /*printk("Initializing the GIC...\n");*/ ti_puts("Initializing the GIC...\n\r"); gic_setup(handle_IRQ); gic_enable_irq(TIMER_IRQ); /*printk("Initializing the timer...\n");*/ ti_puts("Initializing the timer...\n\r"); ticks_per_beat = timer_get_frequency() / BEATS_PER_SEC; expected_ticks = timer_get_ticks() + ticks_per_beat; timer_start(ticks_per_beat); led_reg = (void *)(unsigned long)cmdline_parse_int("led-reg", 0); led_pin = cmdline_parse_int("led-pin", 0); while (1) asm volatile("wfi" : : : "memory"); }
/* * Jailhouse, a Linux-based partitioning hypervisor * * Copyright (c) Texas Insturments. Inc, 2016 * * Authors: * Vitaly Andrianov <vitalya@ti.com> * * This work is licensed under the terms of the GNU GPL, version 2. See * the COPYING file in the top-level directory. */ #include <jailhouse/types.h> #include <jailhouse/cell-config.h> #define ARRAY_SIZE(a) sizeof(a) / sizeof(a[0]) struct { struct jailhouse_system header; __u64 cpus[1]; struct jailhouse_memory mem_regions[19]; struct jailhouse_irqchip irqchips[2]; struct jailhouse_pci_device pci_devices[1]; } __attribute__((packed)) config = { .header = { .signature = JAILHOUSE_SYSTEM_SIGNATURE, .revision = JAILHOUSE_CONFIG_REVISION, .hypervisor_memory = { .phys_start = 0xef000000, .size = 0x1000000, }, .debug_console = { .address = 0x48020000, .size = 0x1000, /* .divider = 26, */ .flags = JAILHOUSE_CON1_TYPE_8250 | JAILHOUSE_CON1_ACCESS_MMIO | JAILHOUSE_CON1_REGDIST_4 | JAILHOUSE_CON2_TYPE_ROOTPAGE, }, .platform_info = { .pci_mmconfig_base = 0x30000000, .pci_mmconfig_end_bus = 0, .pci_is_virtual = 1, .arm = { .gic_version = 2, .gicd_base = 0x48211000, .gicc_base = 0x48212000, .gich_base = 0x48214000, .gicv_base = 0x48216000, .maintenance_irq = 25, }, }, .root_cell = { .name = "AM57XX-EVM", .cpu_set_size = sizeof(config.cpus), .num_memory_regions = ARRAY_SIZE(config.mem_regions), .num_irqchips = ARRAY_SIZE(config.irqchips), .num_pci_devices = ARRAY_SIZE(config.pci_devices), .vpci_irq_base = 100 - 32, }, }, .cpus = { 0x3, }, .mem_regions = { /* PCI */ { .phys_start = 0x30000000, .virt_start = 0x30000000, .size = 0x001fffff, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* OCMCRAM */ { .phys_start = 0x40300000, .virt_start = 0x40300000, .size = 0x80000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x40380000 - 0x48020000 */ { .phys_start = 0x40380000, .virt_start = 0x40380000, .size = 0x7ca0000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* UART... */ { .phys_start = 0x48020000, .virt_start = 0x48020000, .size = 0xe0000,//0x00001000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x48100000 - 0x48281000 */ { .phys_start = 0x48100000, .virt_start = 0x48100000, .size = 0x110000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* * Leave gap for GIC controller 0x48210000 - 0x41220000 */ /* 0x48220000 - 0x48281000 */ { .phys_start = 0x48220000, .virt_start = 0x48220000, .size = 0x610000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* OMAP WakeupGen */ { .phys_start = 0x48281000, .virt_start = 0x48281000, .size = 0x1000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* PRCM MPU */ { .phys_start = 0x48243000, .virt_start = 0x48243000, .size = 0x1000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x48400000 - 0x48424000 */ { .phys_start = 0x48400000, .virt_start = 0x48400000, .size = 0x24000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* UART... */ { .phys_start = 0x48424000, .virt_start = 0x48424000, .size = 0x2000,//0x00001000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x48426000 - 0x48826000 */ { .phys_start = 0x48426000, .virt_start = 0x48426000, .size = 0x400000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x48826000 - 0x48828000 */ { .phys_start = 0x48826000, .virt_start = 0x48826000, .size = 0x2000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x48828000 - 0x4B300000 */ { .phys_start = 0x48828000, .virt_start = 0x48828000, .size = 0x2ad8000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* 0x4B500000 - 0x58000000 */ { .phys_start = 0x4B500000, .virt_start = 0x4B500000, .size = 0xCB00000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* Display Subsystem & ...*/ { .phys_start = 0x58000000, .virt_start = 0x58000000, .size = 0x8000000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* RAM */ { .phys_start = 0x80000000, .virt_start = 0x80000000, .size = 0x6EE00000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE, }, /* IVSHMEM shared memory region for 0001:00:00.0 */ { .phys_start = 0xEEE00000, .virt_start = 0xEEE00000, .size = 0x100000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_ROOTSHARED, }, /* communication region */ { .phys_start = 0xEEF00000, .virt_start = 0xEEF00000, .size = 0x00100000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_IO, }, /* Leave hole for hypervisor */ /* RAM */ { .phys_start = 0xF0000000, .virt_start = 0xF0000000, .size = 0x10000000, .flags = JAILHOUSE_MEM_READ | JAILHOUSE_MEM_WRITE | JAILHOUSE_MEM_EXECUTE, }, }, .irqchips = { /* GIC */ { .address = 0x48211000, .pin_base = 32, .pin_bitmap = { 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff }, }, /* GIC */ { .address = 0x48211000, .pin_base = 160, .pin_bitmap = { 0xffffffff, 0, 0, 0 }, }, }, .pci_devices = { { /* 02:00.0 */ .type = JAILHOUSE_PCI_TYPE_IVSHMEM, //.iommu = 1, .bdf = 0 << 3, .bar_mask = { 0xffffff00, 0xffffffff, 0x00000000, 0x00000000, 0x00000000, 0x00000000, }, .num_msix_vectors = 0, .shmem_region = 16, .shmem_protocol = JAILHOUSE_SHMEM_PROTO_UNDEFINED, }, }, };