LP-CC1352P7: Wi-SUN wfantund never survives more than 24 hours.

I have configured a new wfanctl systemd script according to your staff's suggestion.

Then, wfantund has been running smoothly for 48 hours.

I will keep it running for one more week and see whether it works or not.

Hi Tim,

Let us know of the results of that test.

Regards,

Arthur

Hello Arthur,

I have tested the TI wfantund interface and stack software for about 2 weeks.  These are the result.  Please also refer to my first post and its attachment:

1. For the AM64x Starter Kit with general purpose IC; flashed with the Debian image, the wfantund interface and stack can only survive for a maximum of 4 to 5 days.  Sometimes, it only survives for less than 24 hours!
   
   
2. I did repeat the whole setup on a regular DELL Intel Core i3 Ubuntu 22.04 server.  The wfantund interface and stack only survived for about 2 days!
3. In every case, I am not able to reset or restart the wfantund interface and stack with wfanctl CLI so as to make them up and run again.  A system soft reboot is necessary!
4. Configuring and tuning systemd scripts for wfantund and wfanctl do not help.  I created systemd script for wfanctl by referring to your colleague`s help:
   e2e.ti.com/.../5157072
5. My conclusion is - the present TI wfantund stack is not robust enough for industrial application!

Please help me to repair them!

I have been following TI Wi-SUN development for a few years.  I have also been developing a very beautiful frontend for IoT services.  I could not imagine TI Wi-SUN wfantund turning out to be unstable!  I was planning to promote km IoT service in every possible way by the end of this year and now TI wfantund turns out to be an obstacle and bottleneck!

I am very disappointed and depressed!!!

Hi Tim,

I am very surprised at the results. Do you have logs you could share with us? We want to share this with R&D.

Regards,

Arthur

Tim, we have customers in production and running for a very long time with the TI Wi-SUN stack, but the wfantund component has not gone through the same long-term testing. We will work on it this quarter and include wfantund in long term stability test. I will make sure we update this thread with an update at the right time, so that you can get the performance you expect from this component.

Hi Arthur,

Of course.

Where are the related wfantund log files within my AM64x Starter Kit with general purpose IC; flashed with the Debian image?
www.ti.com/.../SK-AM64B

Yes please!

My target market is km wireless connection with low latency which is still an empty sector.  This is a big potential market for power grid, power tower, water pipe, hospitals, airport facilities, etc.

I am looking forward to your good news!  

Hello,
Here are some logs of related processes started from my customised systemd scripts.  It may or may not help you to debug. 

1. ExecStart=/usr/local/sbin/wfantund --config /etc/wpantund.conf
   
   

   May 19 16:17:41 BeaglePlay wfantund[1085]: [-NCP-]: Stack is up
   May 19 16:17:41 BeaglePlay wfantund[1085]: State change: "offline" -> "associated"
   May 19 16:17:41 BeaglePlay wfantund[1085]: wfantund[1085]: Adding multicast address "ff02::1:3" to NCP
   May 19 16:17:41 BeaglePlay wfantund[1085]: Adding multicast address "ff02::1:3" to NCP
   May 19 16:17:46 BeaglePlay wfantund[1085]: wfantund[1085]: SpinelNCPTask.cpp:86: Requirement Failed (IS_EVENT_FROM_NCP(>May 19 16:17:46 BeaglePlay wfantund[1085]: wfantund[1085]: SpinelNCPTaskSendCommand.cpp:351: Requirement Failed ((mRetV>May 19 16:17:46 BeaglePlay wfantund[1085]: wfantund[1085]: SendCommand task encountered an error: 7 (0x00000007)
   May 19 16:17:46 BeaglePlay wfantund[1085]: SpinelNCPTask.cpp:86: Requirement Failed (IS_EVENT_FROM_NCP(event) && GetIns>May 19 16:17:46 BeaglePlay wfantund[1085]: SpinelNCPTaskSendCommand.cpp:351: Requirement Failed ((mRetVal) == 0)
   May 19 16:17:46 BeaglePlay wfantund[1085]: SendCommand task encountered an error: 7 (0x00000007)

   
   
   
   
2. WorkingDirectory=/usr/local/ti-wisunfantund/ti-wisun-webapp/server/
   ExecStart=/usr/local/bin/npm run wfan -- --host 0.0.0.0
   
   

   May 19 16:18:18 BeaglePlay wfantund[1127]: Configuration file "/etc/wpantund.conf" read.
   May 19 16:18:18 BeaglePlay wfantund[1127]: ../util/tunnel.c:157: Requirement Failed (0 == ioctl(fd, TUNSETIFF, (void*)&>May 19 16:18:18 BeaglePlay wfantund[1127]: ../util/tunnel.c:215: Requirement Failed (0 == ioctl(fd, TUNGETIFF, (void*)&>May 19 16:18:18 BeaglePlay wfantund[1127]: TunnelIPv6Interface: Couldn't get tunnel name! errno=77, File descriptor in >May 19 16:18:18 BeaglePlay wfantund[1127]: Ready. Using DBUS bus ":1.36"
   May 19 16:18:18 BeaglePlay wfantund[1127]: Running as root without dropping privileges!
   May 19 16:18:18 BeaglePlay wfantund[1127]: driver_to_ncp_pump: Socket error on read: File descriptor in bad state
   May 19 16:18:18 BeaglePlay wfantund[1127]: NCPInstance: errno 77 "File descriptor in bad state"
   May 19 16:18:18 BeaglePlay wfantund[1127]: Cleaning up. (gRet = 9)
   May 19 16:18:18 BeaglePlay wfantund[1127]: Stopped.

   
   
   
   

Hello!

Unfortunately I also made some similar experience with wfantund or the border router firmware crashing frequently, and reported multiple times to this forum. While still waiting for TI to fix the root cause, here are some workarounds which make it relatively stable:

• Do not start the webapp! When using only wfantund and wfanctl from the command line, it runs much more stable
• When compiling wfantund, configure with "--disable-debug --enable-static-link-ncp-plugin=spinel"
• Redirect wfantund output to null device: "wfantund 2> /dev/null"

I also wrote a script to start wfantund at boot and automatically reset the CC1352P7 in case it goes out of associated state. It's written for the BeaglePlay GPIOs but should be easily adaptable for AM64. I post it here:

#!/bin/bash

# Border router IPv6 address and netmask
# This address will be assigned to the interface on startup
BORDER_ROUTER_IP=fd12:3456:789::1
BORDER_ROUTER_NETMASK=48

# ti-wisunfantund paths
WFANTUND_DIRECTORY=/usr/local/sbin
WFANTUND_BINARY=wfantund
WFANCTL_DIRECTROY=/usr/local/bin
WFANCTL_BINARY=wfanctl
WFANCTL_CMD=$WFANCTL_DIRECTROY/$WFANCTL_BINARY

############################################################

# Function to reset NCP
reset_ncp() {
    gpioset -m time -u 200000 gpiochip2 14=0    # Pull reset pin of CC1352P7
    gpioget gpiochip2 14 > /dev/null            # Set GPIO back to input
}

# Function to bring up Wi-SUN stack
start_wisun_stack() {
    echo -n "set Interface:Up true" | $WFANCTL_CMD 2> /dev/null
    echo -n "set Stack:Up true" | $WFANCTL_CMD 2> /dev/null
}

reset_ncp
sleep 2

# Start wfantund if it is not already running
pgrep -x $WFANTUND_BINARY > /dev/null || $WFANTUND_DIRECTORY/$WFANTUND_BINARY 2> /dev/null &
sleep 2

# Bring up interface and Wi-SUN stack
start_wisun_stack

# Add ip address to the network interface
ip addr add $BORDER_ROUTER_IP/$BORDER_ROUTER_NETMASK dev wfan0

# Notify systemd that startup is complete
systemd-notify --ready

# Loop to restart NCP in case it goes offline
# Unfortunately, the NCP firmware still goes offline at random intervals,
# in which case it needs to be restarted.
while true
do
    sleep 300
    if [[ $(echo -n "get NCP:State" | $WFANCTL_CMD 2> /dev/null | head -n 1) != 'NCP:State = "associated"' ]]; then
        reset_ncp
        sleep 2
        start_wisun_stack
    fi
done

# Wait for wfantund to quit
wait

And systemd service file to run it:

# Wi-SUN border router service unit file

[Unit]
Description=Wi-SUN border router

[Service]
Type=notify
ExecStart=/usr/local/bin/wisun-border-router

[Install]
WantedBy=network.target

Hope that helps!

Georg

Hello Georg,
You are a big hero in TI Community!  Let me try and see what can happen. 

Hi Tim,

Indeed, when I Erling and I said that we have been running the wfantund for a very long time, I failed to mention that this was without the webapp, which is not meant for production use..

Let me know of your results.

and thank you Georg for the instructions.

Regards,

Arthur

I ran Georg`s bash script from rc.local, it turned out that it ran almost correctly except the channel spacing parameter.  It turned out to be an error.

wfanctl:wfan0> get chspacing
chspacing = "0 kHz"

Anyway, it is a good starting point and Georg`s work is highly appreciated! 

Also, Georg`s bash script already has a respawn mechanism, so its systemd script can be set to

Type=onshot
RemainAfterExit=true

or simply use the rc.local for one time execution.

For debugging, I myself prefer the re.local method.

The channel spacing is a compile time parameter for ns_br firmware, I don't think it's changed by wfantund.

For starting the script, I recommend to run it from systemd instead of rc.local. If you set the type to "notify", systemd waits to be notified for the network device is available. This is useful if other startup scripts depend on it.

BTW., here is a newer version of the script I made last week. It not only resets the CC1352P7 but also restarts wfantund. I'm currently running some long term testing to see which version survives best.

#!/bin/bash

# Border router IPv6 address and netmask
# This address will be assigned to the interface on startup
BORDER_ROUTER_IP=fd12:3456:789::1
BORDER_ROUTER_NETMASK=48

# ti-wisunfantund paths
WFANTUND_DIRECTORY=/usr/local/sbin
WFANTUND_BINARY=wfantund
WFANCTL_DIRECTROY=/usr/local/bin
WFANCTL_BINARY=wfanctl
WFANCTL_CMD=$WFANCTL_DIRECTROY/$WFANCTL_BINARY

############################################################

# Function to bring up border router
start_wisun() {
    gpioset -m time -u 200000 gpiochip2 14=0    # Pull reset pin of CC1352P7
    gpioget gpiochip2 14 > /dev/null            # Set GPIO back to input
    sleep 2

    # Start wfantund if it is not already running
    pgrep -x $WFANTUND_BINARY > /dev/null || $WFANTUND_DIRECTORY/$WFANTUND_BINARY 2> /dev/null &
    sleep 2

    # Bring up Wi-SUN stack
    echo -n "set Interface:Up true" | $WFANCTL_CMD 2> /dev/null
    echo -n "set Stack:Up true" | $WFANCTL_CMD 2> /dev/null

    # Add ip address to the network interface
    ip addr add $BORDER_ROUTER_IP/$BORDER_ROUTER_NETMASK dev wfan0
}

# Bring up Wi-SUN
start_wisun

# Notify systemd that startup is complete
systemd-notify --ready

# Loop to restart NCP in case it goes offline
# Unfortunately, the NCP firmware still goes offline at random intervals,
# in which case it needs to be restarted.
while true
do
    sleep 300
    if [[ $(echo -n "get NCP:State" | $WFANCTL_CMD 2> /dev/null | head -n 1) != 'NCP:State = "associated"' ]]; then
        pkill -x $WFANTUND_BINARY
        start_wisun
    fi
done

# Wait for wfantund to quit
wait

Thank you Georg!

I am testing this newer version.

Good news is - it starts with correct chspacing = "200 kHz".  Every parameter also looks good.

I will let it run without the demo wi-sun web portal and see whether it can survive for more than a week.