This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM6442: Interrupt routers with Linux boot flow

Part Number: AM6442
Other Parts Discussed in Thread: SYSCONFIG

I'm having some trouble understanding how to configure interrupts for the A53 cores running Linux (I'm using Processor SDK v8.02.00.23 with the GPEVM). The end goal is to route a PRU IEP Compare event to an interrupt that can be serviced by a Linux kernel module.

After reading many E2E threads and the TRM, this is my current understanding (correct me if wrong):

1. U-Boot SPL will request resources on boot from the DMSC based on the board config compiled by the k3-image-gen utility. These resource requests can be modified by editing the files under k3-image-gen-[version]\soc\am64x\evm including board_cfg.c, pm-cfg.c, rm-cfg.c, etc. But it looks like these config files only request resources but don't actually map any interrupts via interrupt routers (?) In other words, I see the default AM64x resource config (rm-cfg.c) allocates 16 interrupts from CMPEVT_INTERTR0 to A53 cores (Linux) via the following code:

	/* This is actually part of .resasg */
	.resasg_entries = {
		/* Compare event Interrupt Router */
		{
			.start_resource = 0,
			.num_resource = 16,
			.type = RESASG_UTYPE (AM64X_DEV_CMP_EVENT_INTROUTER0,
					RESASG_SUBTYPE_IR_OUTPUT),
			.host_id = HOST_ID_A53_2,
		},

But I haven't found a place where any of these interrupts are actually mapped.

2. It looks like the CMPEVT_INTRTR0 interrupt router could be used to map an IEP Compare event to a Linux interrupt. I see that CMPEVT_INTRTR0 has access to IEP Compare inputs and GIC outputs. Presumably the GIC interrupts can be serviced by a custom Linux kernel module. For example input 16 (PRU_ICSSG0_PR1_IEP0_CMP_INTR_REQ_0 from Table 9-91 in TRM rev D page 4694) could be routed to output 0 (GICSS0_spi_IN_48 from Table 10-157 in TRM rev D page 4782) to map the IEP Compare 0 event to the GIC Interrupt 48. This interrupt could be serviced by a kernel module in Linux. But I'm unsure where to set up this mapping (in U-boot? in device tree? an application running in Linux using some API that communicates with DSMC?)

3. If I was running on the R5F or M4 cores, I see that I could use the SciClient API (from MCU+ SDK) to request the DMSC to set up the interrupt routers, including CMPEVT_INTRTR0. For example I see Sciclient_rmIrqSetRaw() is one of the API calls that seems to map interrupts via interrupt routers. This seems like the correct way to set up interrupt routers when NOT using Processor Linux SDK. But again, I'm unsure how this is supposed to be accomplished when using the Processor SDK Linux boot flow.

4. The SciClient API (from MCU+ SDK) appears that it cannot be used from Linux as the DPL drivers only support bare metal and FreeRTOS (?)

Questions

Is my understanding correct?

Is it true that the resource config files compiled using the k3-image-gen utility only allocates interrupts between the various cores, but doesn't actually map any events through the interrupt routers?

What is the correct way to set up the interrupt routers when booting Linux? Is there a way to do this via device tree? (I don't see any interrupt router drivers listed in Processor SDK Linux docs)

Is there a driver similar to SciClient from MCU+ SDK that can be used within Linux to interact with the DMSC?

  • Hello Steven,

    I am going to assume you want to use interrupts that are generated by writing to the PRU R31 register. Example in the PRU Software Support Package (PSSP) packaged in your Linux SDK at example-applications/pru-icss-x.x.x/examples/am64x/PRU_Direct_Connect1/PRU1_Direct_Connect.c.

    PRU Interrupts with MCU+ core 

    See the PRUICSS driver documentation: https://software-dl.ti.com/mcu-plus-sdk/esd/AM64X/08_02_00_31/exports/docs/api_guide_am64x/DRIVERS_PRUICSS_PAGE.html 

    PRU Interrupts with Linux core

    PRU interrupt mapping is done in two places when talking with Linux:

    1) Interrupts going to the PRU are configured in the PRU project INTC map file.

    Reference PSSP example examples/am64x/PRU_RPMsg_Echo_Interrupt0, especially file intc_map_0.h.

    2) Interrupts going to the Linux driver are configured in the Linux devicetree.

    There are two parts of this: Configuring the GIC going to Linux, and configuring the INTC in the PRU.

    Take a look at the Linux devicetree for the AM64x main domain in the Linux SDK at board-support/linux-x.x.x/arch/arm64/boot/dts/ti/k3-am64-main.dtsi

    If you search for icssg0_intc, you will see that the GIC is configured to receive 8 host interrupts from PRU_ICSSG0. Same thing for icssg1_intc.

    If you look at the PRU & RTU cores (e.g., pru0_0), you will see that by default Linux configures the PRU INTC to allow the PRU to send an interrupt named "vring" to the Linux cores. These are intended to enable the PRU RPMsg example to run out of the box. However, this INTC configuration is not needed if loading PRU firmware that does not send any interrupts to Linux (like the PRUx_Direct_Connect example).

    Let me know if there is additional information you need here. I tried to document this in the comments of intc_map_0.h, but this subject should also probably be documented somewhere else.

    Regards,

    Nick

  • Hi Nick, thanks for the info. I've gone through what you provided, although in this case I'm trying to use the Compare Event Interrupt Router (CMPEVT_INTRTR0) rather than the PRU INTC. I believe this requires two additional steps compared to using the PRU INTC:

    1) Allocating an interrupt "resource" in the resource configuration (which gets compiled into an image with the SysFW and applied by DMSC at boot)

    and

    2) Setting up input and output mapping for the desired event using the CMPEVT_INTRTR0 interrupt router.

    But I'm unsure how to set up CMPEVT_INTRTR0 in the device tree. Assuming I map the interrupt router output to the GIC, I should be able to use the gic500 device tree node to make those interrupts visible to Linux. But I've been unable to find an example or documentation showing how to set up any of the AM64x interrupt routers in the device tree.

  • I finally found the driver I was looking for. It is "sci-intr" (see Linux source under drivers/irqchip/irq-ti-sci-intr.c). This is used by the GPIO interrupt routers in the device tree (compatible="ti,sci-intr"), and I assume I can use the Compare Event Interrupt Router in a similar way.

  • Hello Steven,

    Hmm, ok. Let me know how that investigation goes, and I'll check with the developers as well to see if they have any input. You'll need to know the TI-SCI device ID of the Compare event router, I believe it is here https://software-dl.ti.com/tisci/esd/latest/5_soc_doc/am64x/devices.html#enumeration-of-device-ids as AM64X_DEV_CMP_EVENT_INTROUTER0.

    Potential template - Time Sync Router

    The timesync router is very similar to the compare event router. We define the timesync_router in k3-am64-main.dtsi, but it uses a compatible of "pinctrl-single" instead of "ti,sci-intr". At this point, I am not totally sure of why different drivers are used for these two interrupt routers.

    Note that the timesync router example configurations in both k3-am642-evm.dts and k3-am642-sk.dts have a bug as of SDK 8.2. The hardware is physically capable of sending one input signal to multiple output signals. However, the TI-SCI code currently thinks that the timesync router is limited to one output per input. I expect the compare event router to have the same software limitation.

    Regards,

    Nick

  • Without actually running any tests, to configure the compare event router I would probably try something like this:

    k3-am64-main.dtsi:

            cmp_event_router: cmp-event-router@a30000 {
                    compatible = "pinctrl-single";
                    reg = <0x0 0xa30000 0x0 0x800>;
                    #pinctrl-cells = <1>;
                    pinctrl-single,register-width = <32>;
                    pinctrl-single,function-mask = <0x000107ff>;
            };
    

    k3-am642-evm.dts:

    &cmp_event_router {
        pinctrl-names = "default";
        pinctrl-0 = <&pru_iep_interrupt>;
    
        /* Example of the compare event routing */
        pru_iep_interrupt: pru-iep-interrupt {
            pinctrl-single,pins = <
                /*INPUT# and signal name -> OUTPUT# and signal name*/
                TS_OFFSET(OUTPUT#, INPUT#)
                >;
        };
    };

    But again, I have not tested the above code. I am checking with the developers.

    Regards,

    Nick

  • Hi Nick, I was just putting something together to do this. Thanks for pointing me in the right direction, I'll give it a try.

  • Hi Nick,

    I also had to modify k3-am64.dtsi to add the Compare Event router on CBASS0 (change on line 7):

    	cbass_main: bus@f4000 {
    		compatible = "simple-bus";
    		#address-cells = <2>;
    		#size-cells = <2>;
    		ranges = <0x00 0x000f4000 0x00 0x000f4000 0x00 0x000002d0>, /* PINCTRL */
    			 <0x00 0x00600000 0x00 0x00600000 0x00 0x00001100>, /* GPIO */
    			 <0x00 0x00a30000 0x00 0x00a30000 0x00 0x00000800>, /* Compare Event router */
    			 <0x00 0x00a40000 0x00 0x00a40000 0x00 0x00000800>, /* Timesync router */
    			 <0x00 0x01000000 0x00 0x01000000 0x00 0x02330400>, /* First peripheral window */
    			 <0x00 0x08000000 0x00 0x08000000 0x00 0x00200000>, /* Main CPSW */
    			 <0x00 0x0d000000 0x00 0x0d000000 0x00 0x00800000>, /* PCIE_CORE */
    			 <0x00 0x0e000000 0x00 0x0e000000 0x00 0x00000100>, /* Main RTI0 */
    			 <0x00 0x0e010000 0x00 0x0e010000 0x00 0x00000100>, /* Main RTI1 */
    			 <0x00 0x0f000000 0x00 0x0f000000 0x00 0x00c44200>, /* Second peripheral window */
    			 <0x00 0x20000000 0x00 0x20000000 0x00 0x0a008000>, /* Third peripheral window */
    			 <0x00 0x30000000 0x00 0x30000000 0x00 0x000bc100>, /* ICSSG0/1 */
    			 <0x00 0x37000000 0x00 0x37000000 0x00 0x00040000>, /* TIMERMGR0 TIMERS */
    			 <0x00 0x39000000 0x00 0x39000000 0x00 0x00000400>, /* CPTS0 */
    			 <0x00 0x3b000000 0x00 0x3b000000 0x00 0x00000400>, /* GPMC0_CFG */
    			 <0x00 0x3cd00000 0x00 0x3cd00000 0x00 0x00000200>, /* TIMERMGR0_CONFIG */
    			 <0x00 0x3f004000 0x00 0x3f004000 0x00 0x00000400>, /* GICSS0_REGS */
    			 <0x00 0x40900000 0x00 0x40900000 0x00 0x00030000>, /* SA2_UL0 */
    			 <0x00 0x43000000 0x00 0x43000000 0x00 0x00020000>, /* CTRL_MMR0 */
    			 <0x00 0x44043000 0x00 0x44043000 0x00 0x00000fe0>, /* TI SCI DEBUG */
    			 <0x00 0x48000000 0x00 0x48000000 0x00 0x06400000>, /* DMASS */
    			 <0x00 0x50000000 0x00 0x50000000 0x00 0x08000000>, /* GPMC0 DATA */
    			 <0x00 0x60000000 0x00 0x60000000 0x00 0x08000000>, /* FSS0 DAT1 */
    			 <0x00 0x68000000 0x00 0x68000000 0x00 0x08000000>, /* PCIe DAT0 */
    			 <0x00 0x70000000 0x00 0x70000000 0x00 0x00200000>, /* OC SRAM */
    			 <0x00 0x78000000 0x00 0x78000000 0x00 0x00800000>, /* Main R5FSS */
    			 <0x06 0x00000000 0x06 0x00000000 0x01 0x00000000>, /* PCIe DAT1 */
    			 <0x05 0x00000000 0x05 0x00000000 0x01 0x00000000>, /* FSS0 DAT3 */
    
    			 /* MCU Domain Range */
    			 <0x00 0x04000000 0x00 0x04000000 0x00 0x01ff1400>;
    
    		cbass_mcu: bus@4000000 {
    			compatible = "simple-bus";
    			#address-cells = <2>;
    			#size-cells = <2>;
    			ranges = <0x00 0x04000000 0x00 0x04000000 0x00 0x01ff1400>; /* Peripheral window */
    		};
    	};

    With these changes I'm seeing an unhandled external abort when booting. Any ideas why Linux wouldn't be able to access Compare Event Interrupt Router memory during boot? Some kind of firewall issue with DMSC? (running on an unmodified AM64x GP EVM board).

    Boot log snippet below:

    [ 0.672688] pinctrl-single f4000.pinctrl: 180 pins, size 720
    [ 0.679672] pinctrl-single a40000.timesync-router: 512 pins, size 2048
    [ 0.687613] pinctrl-single a30000.cmp-event-router: 512 pins, size 2048
    ERROR: Unhandled External Abort received on 0x80000001 from S-EL1
    ERROR: exception reason=0 syndrome=0xbf000002
    Unhandled Exception from EL1
    x0 = 0x0000000100000001
    x1 = 0xffff0000259a0000
    x2 = 0x0000000000000000
    x3 = 0xffff8000104d5f10
    x4 = 0x00000000000107ff
    x5 = 0x0000000000000000
    x6 = 0x0000000000000000
    x7 = 0x0000000000000000
    x8 = 0xffff000025c1fd60
    x9 = 0xffff000025cec540
    x10 = 0xffff000025cec548
    x11 = 0x0000000000000040
    x12 = 0xffff0000271c9308
    x13 = 0xffff0000271c930a
    x14 = 0xffffffffffffffff
    x15 = 0xffff0000259a0530
    x16 = 0x0000000000000019
    x17 = 0x0000000000000000
    x18 = 0x0000000000000001
    x19 = 0xffff000026e3b080
    x20 = 0x0000000000000001
    x21 = 0x0000000000000000
    x22 = 0xffff000025ceca80
    x23 = 0xffff000026e3b0e0
    x24 = 0x0000000000000000
    x25 = 0xffff000025cec880
    x26 = 0xffff000025cec880
    x27 = 0xffff800010dbc330
    x28 = 0xffff800010dbc308
    x29 = 0xffff8000114ef9b0
    x30 = 0xffff8000104d64e4
    scr_el3 = 0x000000000000073d
    sctlr_el3 = 0x0000000030cd183f
    cptr_el3 = 0x0000000000000000
    tcr_el3 = 0x0000000080803520
    daif = 0x00000000000002c0
    mair_el3 = 0x00000000004404ff
    spsr_el3 = 0x0000000080000005
    elr_el3 = 0xffff800010a8a1bc
    ttbr0_el3 = 0x00000000701cd780
    esr_el3 = 0x00000000bf000002
    far_el3 = 0xfcfffbffc48ee205
    spsr_el1 = 0x0000000060000005
    elr_el1 = 0xffff8000100b2ec0
    spsr_abt = 0x0000000000000000
    spsr_und = 0x0000000000000000
    spsr_irq = 0x0000000000000000
    spsr_fiq = 0x0000000000000000
    sctlr_el1 = 0x0000000034d4d91d
    actlr_el1 = 0x0000000000000000
    cpacr_el1 = 0x0000000000300000
    csselr_el1 = 0x0000000000000000
    sp_el1 = 0xffff8000114ef9b0
    esr_el1 = 0x0000000000000000
    ttbr0_el1 = 0x0000000082f30000
    ttbr1_el1 = 0x0000000082f40000
    mair_el1 = 0x000c0400bb44ffff
    amair_el1 = 0x0000000000000000
    tcr_el1 = 0x00000032f5d07590
    tpidr_el1 = 0xffff80006ed20000
    tpidr_el0 = 0x0000000000000000
    tpidrro_el0 = 0x0000000000000000
    par_el1 = 0x0000000000000000
    mpidr_el1 = 0x0000000080000001
    afsr0_el1 = 0x0000000000000000
    afsr1_el1 = 0x0000000000000000
    contextidr_el1 = 0x0000000000000000
    vbar_el1 = 0xffff800010010800
    cntp_ctl_el0 = 0x0000000000000005
    cntp_cval_el0 = 0x000003a77a04e04a
    cntv_ctl_el0 = 0x0000000000000000
    cntv_cval_el0 = 0x0000000000000000
    cntkctl_el1 = 0x00000000000000d6
    sp_el0 = 0x00000000701cac10
    isr_el1 = 0x0000000000000040
    dacr32_el2 = 0x0000000000000000
    ifsr32_el2 = 0x0000000000000000
    cpuectlr_el1 = 0x0000000000000040
    cpumerrsr_el1 = 0x00000000020c03df
    l2merrsr_el1 = 0x0000000012303d20
    cpuactlr_el1 = 0x00001000090ca000

  • Hello Steven,

    Thanks for the debug logs. I'm signing off for a long weekend, I'll be back on Tuesday. Please ping me if I have not replied by the middle of next week.

    Regards,

    Nick

  • Hi Nick, I think I wrapped my head around how this works. Looks like they are using the pinctrl-single driver as a simple way to write to the interrupt router registers directly. So although this should work for both Time Sync Interrupt Router and Compare Event Interrupt Router, this seems to bypass the DMSC. Not sure if that's desired or not. But it probably gets around the limitation with TI-SCI only allowing one output per input.

    I noticed in the default resource config for the AM64x GP EVM (see k3-image-gen-2022.01\soc\am64x\evm\rm-cfg.c) there are 16 events reserved from the Compare Event router for HOST_ID_A53_2, which is identified in SysConfig as "Cortex A53 context 2 on Main island - VM/OS1". But for the Time Sync router, all 41 events are mapped to HOST_ID_ALL.

    I'm wondering if HOST_ID_A53_2 isn't the correct option to allow Linux to write to the Compare Event router during boot, hence the exception? Is Linux in "context 2" during boot?

    I won't be able to get back to this until the end of next week, but I'll circle back and ping if I haven't heard back by then.

    Thanks for your help!

  • Steven, Nick is OOO today but I just wanted to let you know this is still on our radar screen, thanks.

  • Hi Nick and Andreas,

    Just a note that I tried recompiled U-boot and SysFW after changing the Compare Event interrupts to HOST_ID_ALL in the resource config, but am still seeing the unhandled exception when booting.

  • Hello Steven,

    I am learning with you here. Thank you for continuing to engage and run experiments, I plan to add this information about setting up interrupts in the AM64x Linux Academy once we figure out the steps to get this working: https://dev.ti.com/tirex/explore/node?a=7qm9DIS__LATEST&node=AFT.W2rxrknvOHR6V2clYA__7qm9DIS__LATEST

    I am not sure the details of the definition of context 1 / 2 / 3. However, it looks like HOST_ID_A53_2 defined in board-support/k3-image-gen-2022.01/include/soc/am64x/hosts.h matches A53_2 in board-support/k3-respart-tool/data/am64x/HostNames.json, which is listed as "A53 NonSecure host for Linux / VM1 / RTOS". So I would tentatively expect that to be the right HOST_ID. 

    By the way, we just released a new revision D of the AM64x TRM where we updated the Time Sync chapter to be easier to read (all Time Sync Router inputs and outputs are finally listed in the same place). Unfortunately it looks like the compare event router information is still hidden throughput the TRM. I've filed a bug to get the Compare Event info added in a future revision.

    Just to double-check, could you share the node in your Linux devicetree where you set the input / output mapping?

    Regards,

    Nick

  • Hi Nick,

    I've come to the same conclusion on HOST_ID_A53_2.

    Thanks for your work on the Rev D TRM. I've been using that revision and agree it's an improvement over Rev C. I'm hopeful these improvements will continue as there are sections in the TRM still light on details.

    I couldn't figure out how to attach files, so here's the result of "git diff" showing the 3 device tree files that I edited (using AM64x Processor Linux SDK v8.02.00.23):

    diff --git a/arch/arm64/boot/dts/ti/k3-am64-main.dtsi b/arch/arm64/boot/dts/ti/k3-am64-main.dtsi
    index 9ff5cc05bc..e178c3cbd4 100644
    --- a/arch/arm64/boot/dts/ti/k3-am64-main.dtsi
    +++ b/arch/arm64/boot/dts/ti/k3-am64-main.dtsi
    @@ -577,6 +577,17 @@
     		ti,cpts-periodic-outputs = <6>;
     		ti,cpts-ext-ts-inputs = <8>;
     	};
    +	
    +	cmp_event_router: cmp-event-router@a30000 {
    +		compatible = "pinctrl-single";
    +		/* Compare Event Interrupt Router cfg memory starts at 0x00A30000 and is 2kB long (0x800) */
    +		reg = <0x0 0xa30000 0x0 0x800>;
    +		#pinctrl-cells = <1>;
    +		pinctrl-single,register-width = <32>;
    +		/* Should this be 0x0001007F? Leaving at 0x000107ff to match Timesync Router definition for now */
    +		/* NOTE: see TRM rev D page 4789, MUX_CTRL is bits 0-6, INT_ENABLE is bit 16 */
    +		pinctrl-single,function-mask = <0x000107ff>;
    +	};
     
     	timesync_router: timesync-router@a40000 {
     		compatible = "pinctrl-single";
    diff --git a/arch/arm64/boot/dts/ti/k3-am64.dtsi b/arch/arm64/boot/dts/ti/k3-am64.dtsi
    index 799632383e..343f4ea1f5 100644
    --- a/arch/arm64/boot/dts/ti/k3-am64.dtsi
    +++ b/arch/arm64/boot/dts/ti/k3-am64.dtsi
    @@ -65,6 +65,7 @@
     		#size-cells = <2>;
     		ranges = <0x00 0x000f4000 0x00 0x000f4000 0x00 0x000002d0>, /* PINCTRL */
     			 <0x00 0x00600000 0x00 0x00600000 0x00 0x00001100>, /* GPIO */
    +			 <0x00 0x00a30000 0x00 0x00a30000 0x00 0x00000800>, /* Compare Event router */
     			 <0x00 0x00a40000 0x00 0x00a40000 0x00 0x00000800>, /* Timesync router */
     			 <0x00 0x01000000 0x00 0x01000000 0x00 0x02330400>, /* First peripheral window */
     			 <0x00 0x08000000 0x00 0x08000000 0x00 0x00200000>, /* Main CPSW */
    diff --git a/arch/arm64/boot/dts/ti/k3-am642-evm.dts b/arch/arm64/boot/dts/ti/k3-am642-evm.dts
    index 96f90ebf28..1b5fbc959c 100644
    --- a/arch/arm64/boot/dts/ti/k3-am642-evm.dts
    +++ b/arch/arm64/boot/dts/ti/k3-am642-evm.dts
    @@ -632,6 +632,21 @@
     	};
     };
     
    +/* first value is register offset */
    +/* second is mux ctrl value (="in" w/ bit 16 set to enable event) */
    +#define CMP_OFFSET(out, in)     (0x4+(out)*4) (0x10000 | in)
    +
    +&cmp_event_router {
    +    pinctrl-names = "default";
    +    pinctrl-0 = <&pru_iep_interrupt>;
    +    pru_iep_interrupt: pru-iep-interrupt {
    +        pinctrl-single,pins = <
    +			/* [PRU-ICSSG1 IEP0 CMP0] in48 -> out0 [GICSS0_spi_IN_48] */
    +            CMP_OFFSET(0, 48)
    +            >;
    +    };
    +};
    +
     #define TS_OFFSET(pa, val)     (0x4+(pa)*4) (0x10000 | val)
     
     &timesync_router {
    

  • More info:

    If I boot into Linux using the unmodified device tree and use devmem2 to setup the Compare Event Interrupt Router manually, I get the same crash:

     _____                    _____           _         _
    |  _  |___ ___ ___ ___   |  _  |___ ___  |_|___ ___| |_
    |     |  _| .'| . | . |  |   __|  _| . | | | -_|  _|  _|
    |__|__|_| |__,|_  |___|  |__|  |_| |___|_| |___|___|_|
                  |___|                    |___|
    
    Arago Project http://arago-project.org am64xx-evm ttyS2
    
    Arago 2021.09 am64xx-evm ttyS2
    
    am64xx-evm login: root
    root@am64xx-evm:~# devmem2 0xA30004
    /dev/mem opened.
    Memory mapped at address 0xffffb1e80000.
    Read at address  0x00A30004 (0xffffb1e80004): 0x00000000
    root@am64xx-evm:~# devmem2 0xA30004 w 0x10030
    /dev/mem opened.
    ERROR:   Unhandled External Abort received on 0x80000001 from S-EL1
    ERROR:   exception reason=0 syndrome=0xbf000002
    Unhandled Exception from EL0

    I get similar results if I stop at the U-boot prompt and write manually:

    U-Boot 2021.01-g44a87e3ab8 (May 05 2022 - 17:42:22 +0000)
    
    SoC:   AM64X SR1.0
    Model: Texas Instruments AM642 EVM
    Board: AM64-GPEVM rev A
    DRAM:  2 GiB
    NAND:  0 MiB
    MMC:   mmc@fa10000: 0, mmc@fa00000: 1
    In:    serial@2800000
    Out:   serial@2800000
    Err:   serial@2800000
    Net:   eth0: ethernet@8000000port@1
    Hit any key to stop autoboot:  0
    => md.w 0xa30004
    00a30004: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30014: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30024: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30034: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30044: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30054: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30064: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    00a30074: 0000 0000 0000 0000 0000 0000 0000 0000    ................
    => mw.w 0xa30004 0x10030
    ERROR:   Unhandled External Abort received on 0x80000000 from EL2
    ERROR:   exception reason=0 syndrome=0xbf000002
    Unhandled Exception from EL2

    But, if I switch to OSPI boot (w/ SBL NULL flashed) and connect via JTAG to A53_0, I can read and write to 0xA30004 without crashing:

  • It appears the DMSC has a firewall configured for the Compare Event Interrupt Router but not the Time Sync Interrupt Router. This is why the device tree setup for the Time Sync router gets away with using pinctrl-single to write registers directly (ie no firewall). But this fails for the Compare Event Interrupt Router.

    I'll have to pick this up again next week, but I'd like to try using the "sci-intr" driver instead of "pinctrl-single" (this seems to be how the GPIO interrupt router is configured in the device tree). From skimming the sci-intr code it appears that this driver doesn't bypass the DMSC. If so that will hopefully avoid the crash due to firewall violation.

    Nick, can you double check to see if the DMSC really is firewalling the Compare Event router but not the Time Sync router, and if so, why?

  • Hello Steven,

    Ok, your devicetree looks the way I would expect. This is what I got from the Linux developers over the weekend:

    "CMP EVENT routing configuration is via TiSCI IRQ config APis similar to GPIO IR configuration using res type as described in https://software-dl.ti.com/tisci/esd/latest/5_soc_doc/am64x/resasg_types.html

    I dont think we can do direct reg writes (hence aborts most likely due to firewall  violation)"

    So it looks like your hunch about firewalls is probably correct. I am not sure why firewalls are up for the Compare Event Router but not the Time Sync Router. I'll reach out to the SYSFW developers to see if I can get them to comment on the thought process there.

    Regards,

    Nick

  • Hi Nick, thank you, I'll wait to hear what the SYSFW developers say.

    In general it's been difficult to debug the firewalls the few times I've run up against them. Especially if you don't know what access is causing the crash. It seems this type of violation is caught by the Arm Trusted Firmware which makes it hard to determine where the offending read/write occurs. The TRM has several registers (I think?) that are supposed to log firewall violations like CBASS_FW0_exception_logging_header0/1 (see page 1588 of TRM Rev D). But I tried reading them after the crash and they always read back 0x00000000. I couldn't determine if I was doing something wrong or if the firewall violation wasn't captured in those registers like the TRM indicates. Is there any documentation that discusses debugging firewall violations?

    Also, is there any way to "unlock" the SoC and bypass the DMSC, for example turn off all firewalls and allow all interrupt router / event mappings? This would be a very useful feature for development.

  • Hello Steven,

    I am checking on the "debugging firewall violations" and "turn off firewalls" questions.

    Other information from the SYSFW folks:

    "Firewalls are constrained to a certain host rather than a certain IP. In general, TISCI IRQ APIs will need to be used rather than directly writing to registers. This isn’t AM64x specific, but rather just applies to system firmware as a whole.

     And our firewall information is publicly linked in the linux SDK documentation here (https://software-dl.ti.com/processor-sdk-linux/esd/AM64X/08_02_00_23/exports/docs/linux/Foundational_Components/Sysfw.html?highlight=Guide), which then points to the AM64x firewall settings here (https://software-dl.ti.com/tisci/esd/22_01_02/5_soc_doc/am64x/firewalls.html)."

    I am asking some followup questions about those pages as well, since they still seem to be missing information (e.g., I can see that the compare event router is included in this Firewall region while the time sync router is not. However, there is no explanation for why that is or what that firewall region is supposed to do): Firewall 34, region 6 

    Regards,

    Nick

  • On debugging firewalls in Linux:

    "Firewall exceptions can be asynchronous, so even if Arm Trusted Firmware wasn't catching them, you still wouldn't get an exact location of the offending access instruction.

    The firewall registers are protected and only SYSFW can read them. SYSFW today will print out the details of the exception to the SYSFW UART if tracing is enabled. This is how I usually track down FW issues."

    It looks like there is a requirement to eventually get SYSFW to report the details of the violation to the core that did the offense, but that feature is not scheduled for the next couple of software releases. I will not speculate on a timeline, because features that far out could pretty easily get pushed further back in the schedule or get descoped.

    Regards,

    Nick

  • Hi Nick,

    I finally got the Compare Event Interrupt Router working and can generate an interrupt that can be handled in Linux. Here are the steps I took for anyone else who is following along:

    1. Added the following to k3-am642-main.dtsi:

    	compare_event_intr: interrupt-controller1 {
    		/* compatible with sci-intr driver located at Linux/drivers/irqchips/ti-sci-intr.c */
    		compatible = "ti,sci-intr";
    		/* 1=edge-triggerd interrupts, 4=level-triggered interrupts */
    		/* NOTE: unlike the Time Sync Interrupt Router, the Compare Event Interrupt Router
               only uses edge-triggered interrupts, not level-triggered interrupts, see pg 4778 of RevD TRM */
    		ti,intr-trigger-type = <1>;
    		/* should always be specified per ti,sci-intr.yaml */
    		interrupt-controller;
    		/* parent interrupt controller is the A53 GIC */
    		interrupt-parent = <&gic500>;
    		/* how many cells are used to define each interrupt (ti,sci-intr driver expects 1) */
    		#interrupt-cells = <1>;
    		/* pointer to DMSC registers */
    		ti,sci = <&dmsc>;
    		/* device ID of the interrupt router (see TI-SCI documentation
    		   under SoC Family Specific Documentation » AM64X Devices Descriptions) */
    		/* 1 = AM64X_DEV_CMP_EVENT_INTROUTER0 */
    		ti,sci-dev-id = <1>;
    		/* interrupt-ranges should be specified as <[output] [parent input] [# interrupts]> */
    		/* CMPEVT_INTR Outputs 0-15 ==> GIC SPI IN 48-63 ==> Qty 16 interrupts */
    		ti,interrupt-ranges = <0 48 16>;
    	};
    	
    	isr_handler {
    		compatible = "isr_handler";
    		/* 49 ==> input 49 to CMPEVT_INTR ==> PRU-ICSSG1 IEP0 CMP1 */
    		interrupts = <49>;
    		interrupt-parent = <&compare_event_intr>;
    		test-value = <0x12345678>;
    	};

    2. Wrote a simple kernel driver to handle the interrupt:

    #include <linux/init.h>
    #include <linux/interrupt.h>
    #include <linux/io.h>
    #include <linux/kernel.h>
    #include <linux/module.h>
    #include <linux/of_address.h>
    #include <linux/of_device.h>
    #include <linux/of_irq.h>
    #include <linux/platform_device.h>
    
    static unsigned int irq;
    
    static irqreturn_t irq_handler( int irq, void *dev ) {
        pr_info("isr_handler irq = %d dev = %llx\n", irq, *(unsigned long long *)dev);
        return IRQ_HANDLED;
    }
    
    static int isr_handler_probe( struct platform_device *pdev ) {
        u32 tv;
        struct device *dev = &pdev->dev;
        struct device_node *np = dev->of_node;
    
        dev_info(dev, "probe\n");
    
        if( of_property_read_u32(np, "test-value", &tv) ) {
            dev_err(dev, "of_property_read_u32\n");
            return -EINVAL;
        }
        if( tv != 0x12345678 ) {
            dev_err(dev, "test-value = %llx\n", (unsigned long long)tv);
            return -EINVAL;
        }
        dev_info(dev, "test-value = %lu\n", tv);
    
        irq = irq_of_parse_and_map(dev->of_node, 0);
        if( request_irq(irq, irq_handler, IRQF_SHARED, "isr_handler", dev) < 0 ) {
            dev_err(dev, "request_irq");
            return -EINVAL;
        }
        dev_info(dev, "irq = %u\n", irq);
    
        return 0;
    }
    
    static int isr_handler_remove( struct platform_device *pdev ) {
        dev_info(&pdev->dev, "remove\n");
        free_irq(irq, &pdev->dev);
        return 0;
    }
    
    static const struct of_device_id of_isr_handler_match[] = {
        { .compatible = "isr_handler", },
        {},
    };
    
    MODULE_DEVICE_TABLE(of, of_isr_handler_match);
    
    static struct platform_driver isr_handler_plaform_driver = {
        .probe      = isr_handler_probe,
        .remove     = isr_handler_remove,
        .driver     = {
            .name   = "isr_handler",
            .of_match_table = of_isr_handler_match,
            .owner = THIS_MODULE,
        },
    };
    
    static int isr_handler_init( void ) {
        pr_info("isr_handler_init\n");
        return platform_driver_register(&isr_handler_plaform_driver);
    }
    
    static void isr_handler_exit( void ) {
        pr_info("isr_handler_exit\n");
        platform_driver_unregister(&isr_handler_plaform_driver);
    }
    
    module_init(isr_handler_init)
    module_exit(isr_handler_exit)
    MODULE_LICENSE("GPL");

    I think the remaining question is why the Compare Event Interrupt Router is firewalled when the Time Sync router isn't. I'm also interested if this will still be the case going forward with AM64x SR2.0 and SDK v8.04.

  • Hello Steven,

    Thank you for posting your solution for getting the compare event router working! I am trying to understand your code: where do you conduct the mapping for the compare event router? I would expect that we would need to specify which compare event router output is getting set to receive input 49?

    I have not heard back from the SYSFW team yet on these questions. I have pinged them again (hopefully the email doesn't get lost since it is the end of the workday on a Friday):
    1) Is there an easy-to-understand summary of what is firewalled somewhere? (like https://software-dl.ti.com/tisci/esd/22_01_02/2_tisci_msgs/security/firewall_api.html#initial-configuration, but specific to the part and more helpful)
    2) Region start and end points seem to be defined programmatically. Where do we document WHY the regions are set the way they are? (trying to understand time sync router vs Compare event router in Firewall 34, region 6 of https://software-dl.ti.com/tisci/esd/22_01_02/5_soc_doc/am64x/firewalls.html#list-of-region-based-firewalls)
    3) Is there any way to "unlock" the SoC and bypass the DMSC during development? For example, turn off all firewalls and allow all interrupt router / event mappings?

    FYI, we have filed a ticket for documentation "Need HOWTO for debugging and tracking firewall exceptions" based on your feedback. It has not been accepted yet, so I am not sure on timeframe for when that would be published.

    Regards,

    Nick

  • Hi Nick,

    The mapping is handled entirely by the sci-intr driver. Here is the sequence:

    1. isr_handler driver is loaded which calls the probe() function

    2. probe() calls irq_of_parse_and_map(). This function reads "interrupts=<49>" from the device tree so it knows we have requested interrupt 49. It passes this request to the sci-intr driver since "interrupt-parent = <&compare_event_intr>" and the compare_event_intr node has "compatible = "ti,sci-intr" "

    3. The compare_event_intr node tells the sci-intr driver to route all requests using the Compare Event Interrupt Router (ti,sci-dev-id = <1>) starting with Output 0 to 15, which correspond to GIC interrupts 48-63 (ti,interrupt-ranges = <0 48 16>).

    4. Since interrupt 49 was requested, the sci-intr driver interprets this as a request to map input 49 (IN 49 = ICSSG1 IEP0 Compare 1 Event) using the Compare Event Interrupt Router. It does this using the next available output (Output 0 in this case).

    5. sci-intr passes the mapping request to the DMSC (IN 49 => OUT 0) which avoids any crashes due to firewall violation

    6. sci-intr returns the output interrupt # that was just mapped (OUT 0 = GIC interrupt 48). This irq # is the return value of irq_of_parse_and_map()

    7. isr_handler can then request the correct interrupt using request_irq() using the irq number returned from irq_of_parse_and_map().

    In the long run it would be easier if the Compare Event Interrupt Router were un-firewalled like the Time Sync Router. One use case is routing events to external pins which wouldn't need an interrupt handler. It would be much simpler if we could set up this mapping in the device tree with pinctrl-single.

  • Also thanks for your follow up with the SYSFW team, I'll wait to hear back what they say.

  • Hello Steven,

    Ahh, ok so the GIC output interrupt is auto-assigned at boot time by the driver instead of a static assignment like with the Time Sync Router. That makes sense.

    That is a fair point about the non-interrupt handler usecase.

    I have not heard back anything from the TISCI yet. I've filed some additional tickets requesting updates to the TISCI firewall documentation. Please ping the thread if I don't have another update within a couple of days.

    Regards,

    Nick

  • Hi Nick, just checking to see if you've heard back from the SYSFW team yet.

  • Hi Nick, bumping again

  • Hello Steven,

    Thanks for the ping. I have gotten some partial responses over email, but not enough information to really address your questions about the firewalls (other than during normal operation, directly writing to the registers by bypassing DMSC/SYSFW isn’t allowed, and so TISCI calls need to be used).

    Is there other information you are waiting on me for?

    I have scheduled a meeting on Friday with some of the SYSFW developers so we can discuss verbally. I'll share what I've learned by Monday. Please keep me honest and send another ping if I haven't replied by then.

    Regards,

    Nick

  • Hi Nick,

    OK, that sounds good.

    You're not holding me up. But since we are building new products around the AM64x I'm sure this isn't the last time I'll run up against a firewall issue, so I'd like to learn as much as I can. We also have a need to send Compare events directly to a pin, and the firewall makes this difficult on the Compare Event Router unlike the Time Sync Router. I'd like to understand why one is firewalled and the other is not.

    Thanks for your time helping us with this.

  • Hello Steven,

    Following up from my discussion with the SYSFW folks.

    First off, there was a useful FAQ that we found hidden in another chapter of the TISCI docs: https://software-dl.ti.com/tisci/esd/22_01_02/6_topic_user_guides/firewall_faq.html . I have submitted a request to make the FAQ easier to find.

    Answered questions

    Q1) Is there an easy-to-understand summary of what is firewalled somewhere? (like https://software-dl.ti.com/tisci/esd/22_01_02/2_tisci_msgs/security/firewall_api.html#initial-configuration, but specific to the part and more helpful)

    A1) Not currently. I have filed a request to provide a summary of what is firewalled by default for each processor. Time to update documentation is unknown.

    Q2) Can we just put the processor in a "debug" state to disable firewalls?

    A2) No, it is not possible to disable the firewalls.

    Q3) Why is the compare event router firewalled, but time sync router is not firewalled?

    A3) No real reason. The current SYSFW blocks one-to-many interrupt routing, so the options were to a) rewrite the SYSFW interrupt code and retain the firewall, or b) just remove the firewall from the affected signal routers. We decided to remove the firewall from the Time Sync Router. It looks like there was also discussion about removing the firewall from the Compare Event Router, but it just didn't happen.

    Next steps

    I am still checking on these topics:

    1) Is there a way to configure the compare event router to send an event to a pin with the sci-intr driver?

    2) Can the customer change the start and end address of a firewall region?

    3) Can the customer change the CBA_PERMISSIONs of a firewall region?

    4) Is there any reason NOT to remove the firewall from the Compare Event Router? If not, I'll file a request to remove the firewall.

    Regards,

    Nick

  • Hi Nick, much appreciated.

    Please file a request to remove the firewall from the Compare Event Router as we do require one-to-many routing with the Compare Event Router. This will also make it easier to configure the Compare Event routing in the device tree as previously discussed.

    I also read through the firewall FAQ - very helpful. It did answer my previous question about how to debug firewall violations (the answer is: you can't directly read the registers, but the DMSC writes that info to trace when a violation occurs).

    I think you've answered all of my questions, thanks again. I'll wait to mark this as closed until we get a tentative date on the Compare Event Router firewall removal. This is a feature we need by early next year if possible.

  • Sounds good. I'll respond with updates as they come.

    If I have not replied back within a week or so, please feel free to ping the thread to make sure I haven't dropped your thread on accident.

    Regards,

    Nick

  • Hi Nick, bumping to see if you've been able to get a tentative date for removing the Compare Event Interrupt Router firewall on AM64x.

  • Hi Nick, pinging again.

  • Hi Nick, another bump.

    We're hoping to get confirmation on this before the holiday season. We're almost to the point where the DMSC limitation of 1 output per input for interrupt routers will limit our progress. Unfortunately there isn't any workaround I'm aware of while the Compare Event Interrupt Router is firewalled. Since this firewall update is a small change, we hope this can make it into the next SDK release.

    Thanks!

  • Hello Steven,

    Thanks for the repeated pings! The requirement to remove the firewall from the compare event router has been approved for my team as of this week. I am still waiting to hear back from the SYSFW developers on timeframe. There might not be time to get this into the December 2022 release for AM64x - it sounds like the firewall change would impact all processors with a compare event router, so even if the change itself is just a line or two of code, it might take some time to get signoff from the other processor teams. If it does not make the December release, I would assume it would be in the February/March 2023 SDK release (speculative since the developers haven't committed to anything yet).

    I'm on vacation for the next couple of days, but I'll be back in the office by the middle of next week.

    Regards,

    Nick

  • Hi Nick,

    We can live with Feb/Mar 2023. Thanks for your persistence on this issue, it's much appreciated.

  • Hello Steven,

    We are in the planning phase now. Right now the dev team wants to plan to implement for the SDK release over summer 2023. Does that still work for your schedule? (it sounds like the change itself will be easy to make, but the time to align across all affected processors and business groups, update docs, update Linux devicetrees, etc is anticipated to be more work than I was expecting).

    Regards,

    Nick

  • Let's say "summer" is targeting around July, worst case August.

  • Hi Nick,

    That's later than we'd like, but I've come up with a few alternatives that should keep us going in the meantime (polling instead of interrupts, writing to GPIO pins instead of using interrupt router, etc). If there's any way to release earlier that would be appreciated, but otherwise we'll make it work. Thanks again.

  • Hello Steven,

    Sounds good. I'll see if I can move the schedule forward, and if not I'll let you know.

    Regards,

    Nick

  • Hello Steven,

    Good news! The developers are absolute champs and were able to update the SYSFW for AM64x SDK 8.6 to remove the firewall from the compare event router. So as soon as SDK 8.6 comes out (hopefully within about 3-4 weeks), you should be able to program the compare event router just like you would program the time sync router.

    I have filed requirements to update the documentation and the Linux devicetree cbass_main entry, but the docs and devicetree will lag behind a bit. Not sure if the compare event router will be explicitly documented in SDK 9.0 (summertime) or later. If you have any additional questions, please feel free to create a new thread to continue the discussion.

    Regards,

    Nick