This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

Errors and freezes when loading PRU firmware that now allocates memory to PRU Shared RAM

I am using a BeagleBone Black Wireless system to do some A/D work.   The output of version.sh is at the end of this message.

 

I use PRU0 to read three channels of A/D in a loop that takes about 2.75 ms on average.  I need to store the data from two of these channels in an array so that when an event is detected in the analog input, I stop and analyze the collected data.  Although in production we do not need to send data back to the host program, we are doing it now via RPMSG so we can collect the data and store it in a file for analysis.

 

I use the counter in the IEP to get precise interval timing to associate with each data collection event. 

 

Late last week, as I started to add more functionality step by step, I started to have some issues I have not seen before and appreciate help understanding these.    I am not a kernel or driver guy.  I run what I download on recommendations of others.   The changes I have made have not changed the remoteproc, virtio, etc. setups.   I have established four unsigned int arrays in the PRU's shared memory space.  Using the C code below. 

 

#define SHARE_MEM 0x10000

#define E_RING_BUFFER_SIZE 10

volatile unsigned int *DetTSampleSet = (unsigned int *) SHARE_MEM;

volatile unsigned int *DetBSampleSet = (unsigned int *) (SHARE_MEM + (E_RING_BUFFER_SIZE * sizeof(unsigned int)));

volatile unsigned int *TimeSampleSet = (unsigned int *) (SHARE_MEM + (E_RING_BUFFER_SIZE * sizeof(unsigned int)*2));

volatile unsigned int *PressureSampleSet = (unsigned int *) (SHARE_MEM + (E_RING_BUFFER_SIZE * sizeof(unsigned int)*3));

 

 

I added them one by one and the code worked as each one was added.   Then I began to put my A/D raw data into the array variables and pass it to a routine that right now only adds two of them together and sends the result back as an unsigned int too.

 

I boot from an SD card and the data that is collected is written to that card as well.   As the data is returned from the PRU to the Host via RPMSG, I write one message per cycle to the screen using printf to know where we are in the process and write the data to the file on the SD card in CSV format.   Each record has 10 fields.  So at the end of each record there is output to the screen. I have noticed that this output tends to slow down or 'freeze' for a second or two more often as I have worked with this project more and more.  

 

So I have a couple of primary questions about this and a few more observations are shared below.

 

  1. Could it be that the SD card is going bad or wearing down with use?  I would not have expected this based on its age.
  2. The current rev of the PRU code is freezing the system completely.  Only the PWR, WL and BT leds are on and they are full on.  I have to cycle the power to restart the system.
  3. I am using the IEP.  Is this related to the wi-fi somehow?  I do not think so but wanted to confirm this.

 

I also get a variety of errors in the journal and would appreciate any insights into them as I don't understand them.

 

Here's the first (the last line of this block of messages).  The program has run even when this message is logged.  What does this mean?

 

May12 12:33] remoteproc remoteproc1: powering up 4a334000.pru

[  +0.000679] remoteproc remoteproc1: Booting fw image am335x-pru0-fw, size 122452

[  +0.013186] pruss 4a300000.pruss: configured system_events[63-0] = 00000000,00030000

[  +0.000022] pruss 4a300000.pruss: configured intr_channels = 0x00000005 host_intr = 0x00000005

[  +0.011886] remoteproc remoteproc1: registered virtio0 (type 7)

[  +0.000022] remoteproc remoteproc1: remote processor 4a334000.pru is now up

[  +0.088779] virtio_rpmsg_bus virtio0: rpmsg host is online

[  +0.001103] pm33xx pm33xx: PM: Unable to allocate memory from ocmcram

[  +0.040430] pm33xx: probe of pm33xx failed with error -12

 

Here's the second…  I think this has to do with the wi-fi trying to establish something it can't.  Is there a way to kill this? 

My thought is that it's taking cycles away on the Linux host side that I need to avoid.

 

[May12 12:50] wlcore: WARNING no fw rx ba on tid 2

 

 

I got this once yesterday and it forced a hard reset. I have received is this one in the terminal window.  I have never seen it before.

 

Message from syslogd@beaglebone at May 11 16:35:05 ...

 systemd[1]: Caught <ABRT>, dumped core as pid 32667.

 

Broadcast message from systemd-journald@beaglebone (Tue 2021-05-11 16:35:05 EDT):

 

systemd[1]: Caught <ABRT>, dumped core as pid 32667.

 

 

Message from syslogd@beaglebone at May 11 16:35:05 ...

 systemd[1]: Freezing execution.

 

Broadcast message from systemd-journald@beaglebone (Tue 2021-05-11 16:35:05 EDT):

 

systemd[1]: Freezing execution.

 

 

debian@beaglebone:/var/lib/cloud9$ sudo /opt/scripts/tools/version.sh

[sudo] password for debian:

git:/opt/scripts/:[b39ec679648a6be8f25f48bd1c9784c1fc5a0c46]

eeprom:[A335BNLTBWA52027BBWG0227]

model:[TI_AM335x_BeagleBone_Black_Wireless]

dogtag:[BeagleBoard.org Debian Buster IoT Image 2020-04-06]

bootloader:[microSD-(push-button)]:[/dev/mmcblk0]:[U-Boot 2019.04-00002-g07d5700e21]:[location: dd MBR]

bootloader:[eMMC-(default)]:[/dev/mmcblk1]:[U-Boot 2018.03-00002-gac9cce7c6a]:[location: dd MBR]

UBOOT: Booted Device-Tree:[am335x-boneblack-uboot-univ.dts]

UBOOT: Loaded Overlay:[AM335X-PRU-RPROC-4-19-TI-00A0]

UBOOT: Loaded Overlay:[BB-ADC-00A0]

UBOOT: Loaded Overlay:[BB-BBBW-WL1835-00A0]

UBOOT: Loaded Overlay:[BB-BONE-eMMC1-01-00A0]

UBOOT: Loaded Overlay:[BB-I2C2-RTC-DS3231]

UBOOT: Loaded Overlay:[BB-W1-P9.12-00A2]

kernel:[4.19.94-ti-r61]

nodejs:[v10.15.2]

/boot/uEnv.txt Settings:

uboot_overlay_options:[enable_uboot_overlays=1]

uboot_overlay_options:[uboot_overlay_addr4=/lib/firmware/BB-W1-P9.12-00A0.dtbo]

uboot_overlay_options:[disable_uboot_overlay_video=1]

uboot_overlay_options:[disable_uboot_overlay_audio=1]

uboot_overlay_options:[uboot_overlay_pru=/lib/firmware/AM335X-PRU-RPROC-4-19-TI-00A0.dtbo]

uboot_overlay_options:[enable_uboot_cape_universal=1]

uboot_overlay_options:[dtb_overlay=/lib/firmware/BB-I2C2-RTC-DS3231.dtbo]

pkg check: to individually upgrade run: [sudo apt install --only-upgrade <pkg>]

pkg:[bb-cape-overlays]:[4.14.20210401.0-0~buster+20210401]

pkg:[bb-wl18xx-firmware]:[1.20200322.0-0rcnee0~buster+20200322]

pkg:[kmod]:[26-1]

pkg:[librobotcontrol]:[1.0.4-git20190227.1-0rcnee0~buster+20190327]

pkg:[firmware-ti-connectivity]:[20190717-2rcnee1~buster+20200305]

groups:[debian : debian adm kmem dialout cdrom floppy audio dip video plugdev users systemd-journal bluetooth netdev i2c gpio pwm eqep remoteproc admin spi iio docker tisdk weston-launch xenomai cloud9ide]

cmdline:[console=ttyO0,115200n8 bone_capemgr.uboot_capemgr_enabled=1 root=/dev/mmcblk0p1 ro rootfstype=ext4 rootwait coherent_pool=1M net.ifnames=0 lpj=1990656 rng_core.default_quality=100 quiet]

dmesg | grep remote

[   69.444537] remoteproc remoteproc0: wkup_m3 is available

[   69.731914] remoteproc remoteproc0: powering up wkup_m3

[   69.731947] remoteproc remoteproc0: Booting fw image am335x-pm-firmware.elf, size 217148

[   69.732222] remoteproc remoteproc0: remote processor wkup_m3 is now up

[   71.971917] remoteproc remoteproc1: 4a334000.pru is available

[   71.982645] remoteproc remoteproc2: 4a338000.pru is available

dmesg | grep pru

[   71.971917] remoteproc remoteproc1: 4a334000.pru is available

[   71.972101] pru-rproc 4a334000.pru: PRU rproc node pru@4a334000 probed successfully

[   71.982645] remoteproc remoteproc2: 4a338000.pru is available

[   71.982788] pru-rproc 4a338000.pru: PRU rproc node pru@4a338000 probed successfully

dmesg | grep pinctrl-single

[    0.951277] pinctrl-single 44e10800.pinmux: 142 pins, size 568

dmesg | grep gpio-of-helper

[    0.964973] gpio-of-helper ocp:cape-universal: ready

lsusb

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

END

  • Hi Walter,

    This forum only supports TI Processor SDK, not any other software which is not released by TI.

    It seems you are using Debian, which I guess is provided by the Beagle community. So Please ask the community for help.

  • This is the first response of this kind I've received from TI.  Frankly, it's a bit puzzling because I'd think TI would want to support any project that might result in the sale of more chips.   But I know having expertise on the team for every nuance is not cost effective too.

    My other questions have been answered even though I'm running the same OS (yes, Debian).  I'm not really sure how you are making a distinction so let me ask this more succinctly before tossing in the towel because I think you do support development in C for the PRUs.

    1. Does the approach I have used to allocate variables into PRU shared memory work and do you see any issues with it?

    2. Do wi-fi and the IEP interact or conflict?

  • Hi Walter,

    Frankly, it's a bit puzzling because I'd think TI would want to support any project that might result in the sale of more chips.

    Of cause we want to provide the best support on our devices. However it is technically impossible to support any non-TI released software and board designed using our devices. Just give you one example, we don't know what the userspace library versions used in Debian filesystem, we don't know what kernel version runs on Debian, and if any additional kernel patches added by the Beagle community and it is well validated.

    1. Does the approach I have used to allocate variables into PRU shared memory work and do you see any issues with it?

    I am routing your query to our PRU expert for comments.

    2. Do wi-fi and the IEP interact or conflict?

    Our PRU expert might provide a generic comment on this, but probably won't be specific, as we don't have experience on this community's Beaglebone Black Wireless board, as TI Processor SDK Linux package only supports the community's Beaglebone Black board, not its Wireless variant.

  • Understood.   I've just received support before for my projects running Debian so it's never been an issue before.    

    So just so I am clear, what OS should I be running on a Beaglebone Black to expect support from TI too?

  • Hello Walter,

    SOFTWARE SUPPORT FROM TI
    We can answer questions about software that TI developed and tested. For example, we can support Linux questions about the version of Linux distributed in AM335x Linux Processor SDK 6.3 (kernel 4.19) or the next AM335x SDK which releases within a month (kernel 5.4). 

    IEP & WIFI?
    I have not looked into wifi. However, I would be surprised if the PRU's IEP timer was used by something outside of the ICSS. If you were running PRU Ethernet or PRU networking protocols on your PRU cores, then that firmware would use the IEP in a similar manner to how CPSW Ethernet uses the CPTS.

    PRU SUPPORT IN GENERAL
    We support general PRU firmware development questions regardless of the version of Linux you are running. Keep in mind that we would still not be able to answer questions about non-TI software like the PASM compiler or a method of loading the PRU that does not use the RemoteProc driver.

    You can definitely define areas of memory within the PRU and access it in C. For example, this code from our _Getting Started Labs_ defines variables at a specific location in PRU DMEM:

    #include <stdint.h>
    #include "resource_table_empty.h"
    
    /* TODO: define c */
    /* a, b, and c are stored in a defined location in PRU memory */
    #define a  (*((volatile unsigned int *)0x100))
    #define b  (*((volatile unsigned int *)0x104))
    #define c  (*((volatile unsigned int *)0x108))
    
    /*
     * main.c
     */
    void main(void)
    {
    ...
    
    	a = 1;
    	b = 2;
    	
    	/*
    	 * TODO: store the sum of the numbers at memory locations a and
    	 * b in memory location c
    	 */
    	c = a + b;
    ...
    }

    It sounds like you've made some progress since your initial post, so I'll wait for your response before trying to answer any more questions.

    Regards,

    Nick

  • Nick, 

    Thank you for all of this information. It's helpful to a newbie like me.  We are focused on using the Beaglebone Black series right now.  I didn't initially realize that the Beaglebone Black Wireless doesn't use a TI chip but instead uses a chip that has AM355x mask within it.   So, I certainly can't expect TI to support that all in.  

    I don't think the IEP is being used by the wi-fi now.   But the wi-fi is tossing some warnings so I moved off it to a basic Beaglebone Black with a tethered ethernet connection.   The precise timing I'm getting from my program using the IEP is rock solid so I do not think the IEP is impacted by tethered ethernet either.   

    I have successfully allocated some arrays into the shared RAM of the PRUs and that seems to work without a hitch.   Here's my code in case you might see a problem with it.  Basically, I want to place these four arrays in the shared memory starting at the beginning of shared memory and then put them one after the other without any overlap.

    #define SHARE_MEM 0x10000

    #define E_RING_BUFFER_SIZE 60

    volatile unsigned int *DetTSampleSet = (unsigned int *) SHARE_MEM;
    volatile unsigned int *DetBSampleSet = (unsigned int *) (SHARE_MEM + (E_RING_BUFFER_SIZE * sizeof(unsigned int)));
    volatile unsigned int *TimeSampleSet = (unsigned int *) (SHARE_MEM + (E_RING_BUFFER_SIZE * sizeof(unsigned int))*2);
    volatile unsigned int *PressureSampleSet = (unsigned int *) (SHARE_MEM + (E_RING_BUFFER_SIZE * sizeof(unsigned int))*3);

    Later in my code, I use them like this where pnr is an index into the array.

    DetBSampleSet[pnr]=RawAnalog;

    Do you see anything obviously wrong with this approach?  It's working fine so far.

    I am using remoteproc because I definitely want to use something that is supported by TI.   It works for me so far but I have a weird problem that I cannot figure out.   I doubt this is something TI can support but I'll toss it out there just in case.

    My code is all written in C.

    The PRU code comes up and waits for a message from the host code.  The host code sends specific code words through the RPMSG to the PRU and when the PRU gets an interrupt and reads the message, it then executes based on what the message is.  This works fantastic until...

    In one particular action the PRU needs to take, I have two unsigned int variables that I initialize at startup.  If I DO NOT also include them inside a particular case statement and set their values there, the PRU doesn't respond to the message from the host and start the action it's being instructed to do.  It's extremely weird.  I can't imagine this has anything to do with the processor.  I've been trying to declare and initialize these two variables differently but nothing is taking care of the problem.   It's probably a C issue and I'm just not seeing it yet.  So close....

    I appreciate all the help I get from these communities.  I am working alone and it's tough when you hit a problem like this as I am sure that many others can attest to this too.


    Walter

  • Hello Walter,

    I apologize for the delayed response. Let me know if you need us to continue the conversation.

    Regards,

    Nick

  • Nick, 

    I was able to eliminate the problem by renaming the two variables so I guess it could be a problem in the toolchain somewhere.  But I have other variables that use a similar naming structure.

    They were originally Start_of_Pulse and End_of_Pulse.  The code runs but the PRU does not respond to a message sent by the host if the line

    Start_of_Pulse = -1;

    is not present.

    I needed to be able to set Start_of_Pulse to the place in the array where the currently read data is stored.  Like this, 

    Start_of_Pulse = pnr; // pnr is the current position in the ring buffer array.  Both are unsigned int.

    If I changed the line to this, 

    PulseStart = pnr; 

    everything works!

    Very strange.  I've moved on but it would be nice to know what caused this.

    I know people say this all the time but the only thing I would change between 'working' and 'not working' was the line 

    Start_of_Pulse = -1; 

    to 

    Start_of_Pulse = pnr;