This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

TDA4VM: [Inter-core Virtual Ethernet] tapif crashes when receiving Multicast too fast

Part Number: TDA4VM

Dear Ti

    Custom board with SDK 8.4 

     With virtual tap to receive multicast, when the server sending to fast, the tapif app will occur Segmentation fault when call Bufpool_freeBuf.

Logs:
[root@AOS ~]# ip tuntap add mode tap dev tap0
[root@AOS ~]# ip link set address 00:01:02:04:05:06 dev tap0
[root@AOS ~]# ip link set dev tap0 up
[root@AOS ~]# ip addr add 192.168.1.11/24 dev tap0
[root@AOS ~]# ifconfig tap0 up

[root@AOS ~]# tcpdump -i tap0 &
[root@AOS ~]# tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap0, link-type EN10MB (Ethernet), capture size 262144 bytes

[root@AOS ~]# tapif tap0 2 3 4 0xE4000000 0x00800000 1000 4 0xE4800000 0x01800000 2 64 64 &
[root@AOS ~]# Opened TAP Device successfully
Queue Mapping Succeeded
Bufpool Mapping Succeeded
Assigned Queue Handles
Queues have been initialized
Assigned Bufpool Handle
Bufpool Init: Values Initialized
14:02:37.620237 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28
Bufpool Init: Cleared Buffers
Initialized Bufpool
Initialization complete
-------------------------------------------------------------------------
Starting TX and RX Tasks
Started TX Task
Started RX Task
14:02:37.674912 ARP, Request who-has 192.168.0.200 tell 192.168.0.200, length 28
14:02:37.674953 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 44
14:02:37.674975 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 44
14:02:37.675002 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:37.675027 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:37.675051 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:37.675073 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 44
14:02:37.675098 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:37.675115 ARP, Request who-has 172.20.0.1 tell 174.34.224.54, length 46
14:02:37.675141 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675161 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675185 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675205 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675227 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675250 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675271 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675294 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675314 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675335 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675357 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675377 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675398 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675420 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675440 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675462 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675482 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675503 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675524 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675545 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675565 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675620 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675642 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675664 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675685 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675707 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675730 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675750 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675771 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675793 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675813 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675836 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675856 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675877 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675900 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.675919 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 44
14:02:37.675942 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:37.675970 ARP, Request who-has 172.20.0.1 tell 174.34.224.54, length 46
14:02:37.676015 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:37.676032 ARP, Request who-has 172.20.0.1 tell 174.34.224.54, length 46
14:02:37.676057 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676080 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676104 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676127 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676149 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676174 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676194 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676215 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676236 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676256 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676277 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676297 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676317 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676339 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.676359 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677454 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677475 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677495 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677517 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677537 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677557 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677579 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677599 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677621 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677641 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677661 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677682 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677702 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677722 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677744 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677764 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677786 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677806 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677826 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677847 IP 172.20.3.7.30490 > 239.255.3.1.30490: UDP, length 56
14:02:37.677869 IP 172.20.3.9.30490 > 239.255.3.1.30490: UDP, length 72
14:02:38.052015 IP6 :: > ff02::1:ff04:506: ICMP6, neighbor solicitation, who has fe80::201:2ff:fe04:506, length 32
14:02:38.371997 IP6 :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28

[2]+  Segmentation fault         tapif tap0 2 3 4 0xE4000000 0x00800000 1000 4 0xE4800000 0x01800000 2 64 64

Segmentation fault in "rtos\ethfw\apps\tap\tapfirmware.c" ->  "BufpoolTable_Handle[hBuf->poolId].numBufFree++;"

int32_t Bufpool_freeBuf(Bufpool_Buf* hBuf)
{
	int32_t retVal = BUFPOOL_OK;

	assert(hBuf && (hBuf->refCount >=0));

	if(hBuf->refCount > 0)
	{
		--(hBuf->refCount);

		if(0 == hBuf->refCount)
		{
			/* Mark the buffer free */
			hBuf->isUsed = 0;
		}
	}

	/*
	* Here we don't have a direct bufpool handle so
	* refer to the pool using the global table and
	* the poolId available in hBuf
	*/
//	printf("%s, %d, %ld\n", __func__, __LINE__, BufpoolTable_Handle[hBuf->poolId].numBufFree);
	BufpoolTable_Handle[hBuf->poolId].numBufFree++;
//	printf("%s, %d\n\n", __func__, __LINE__);
	return retVal;

}

The pDriverBuf is incorrect at the 256th package。

rtos\ethfw\apps\tap\tapif.c

pDriverBuf =
(Bufpool_Buf *)phy_to_virt((size_t)p_deq_node->DataBuffer,
(uint32_t) bufpool_base_addr, (void *)BufpoolTable_Handle);

static void rx_task(void *arguments)
{
	printf("Started RX Task\n");
	fflush(stdout);
	/* Fetching arguments for tx_task to use */
	struct arg_struct *args = (struct arg_struct *)arguments;
	/* Storing the values */
	IcQ_Handle rxQHandle = args->arg_Q_Handle;
	IcQ_Handle IcQ_globalQTable_Handle = args->arg_IcQ_globalQTable_Handle;
	Bufpool_Handle BufpoolTable_Handle = args->arg_BufpoolTable_Handle;
	int tap_fd = args->arg_tap_fd;
	uint32_t q_polling_interval = args->arg_q_polling_interval;
	uint32_t bufpool_base_addr = args->arg_bufpool_base_addr;
	uint32_t max_rx = args->arg_max_frames;
	/* Variable to store output of read syscall */
	int nbytes = -1;
	/* Character buffer to store frame contents */
	unsigned char tap_buffer[TAP_BUFFER_SIZE];
	/* Variable to store length of received frame */
	uint32_t recv_frame_len;
	/* Variable to store return value of Bufpool_freeBuf function */
	int32_t retVal = -1;
	/* Variable to store number of frames received in current wake */
	/* cycle. */
	uint32_t  rx_frame_count = 0;
	/* Pointer to queue node that will be dequeued from RX Queue */
	IcQ_Node *p_deq_node = NULL;
	/* Pointer to Bufpool_Buf */
	Bufpool_Buf *pDriverBuf;
	static int i = 0;
	while (1) {
		/* Set count of received frames in current wake up cycle */
		/* to zero. */
		rx_frame_count = 0;
		while (rx_frame_count < max_rx) {
			/********** Rx Section **********/
			/* Perform Dequeue to fetch descriptor from Queue */
			p_deq_node = IcQueue_deq(rxQHandle);
			/* If deq_node is not NULL and deq_node points to */
			/* non NULL Buffer Pool */
			if (p_deq_node != NULL) {
				/* p_deq_node is already virtual address */
				/* Convert physical address of buffer to virtual address */
				pDriverBuf =
						(Bufpool_Buf *)phy_to_virt((size_t)p_deq_node->DataBuffer,
						(uint32_t) bufpool_base_addr, (void *)BufpoolTable_Handle);
				recv_frame_len = pDriverBuf->payloadLen;
#if 1
				printf("[%d] phy_addr=0x%x, phy_base_addr=0x%x, refCount=%ld, isUsed=%ld, recv_frame_len=%ld, poolID=%d\n",
					++i,
					p_deq_node->DataBuffer,
					bufpool_base_addr,
					pDriverBuf->refCount,
					pDriverBuf->isUsed,
					recv_frame_len,
					pDriverBuf->poolId
					);
#endif
				/* Copy buffer to user-space */
				for (int index = 0; index < recv_frame_len; index++) {
					tap_buffer[index] = pDriverBuf->payload[index];
				}
				/* Write to Linux Network Stack via TAP device */
				nbytes = write(tap_fd, tap_buffer, recv_frame_len);
				assert(nbytes==recv_frame_len && "Unable to write to network stack");
				/* Increment count of received frames */
				rx_frame_count++;
//				printf("%s, %d, refCount = %d, isused=%d, recv_frame_len=%d\n", __func__, __LINE__, pDriverBuf->refCount, pDriverBuf->isUsed, recv_frame_len);
				/* Release frame memory from shared buffer pool */
				retVal = Bufpool_freeBuf(pDriverBuf);
				/* If retVal!=BUFPOOL_OK : Unable to free memory */
				assert(retVal == BUFPOOL_OK);
			} else {
				break;
			}
			usleep(100000);
		}
		/* Sleep */
		usleep(q_polling_interval);
	}
	/* Clean up */
	free(args);
}

You can see the data in this memory is not right.

[256] phy_addr=0xe4e00080, phy_base_addr=0xe4800000, refCount=0, isUsed=9216, recv_frame_len=96, payloadLen=96, poolID=-2130837248

Log
...
[252] phy_addr=0xe4dfe280, phy_base_addr=0xe4800000, refCount=1, isUsed=1, recv_frame_len=102, payloadLen=102, poolID=3
[253] phy_addr=0xe4dfe880, phy_base_addr=0xe4800000, refCount=1, isUsed=1, recv_frame_len=102, payloadLen=102, poolID=3
[254] phy_addr=0xe4dfee80, phy_base_addr=0xe4800000, refCount=1, isUsed=1, recv_frame_len=102, payloadLen=102, poolID=3
[255] phy_addr=0xe4dff480, phy_base_addr=0xe4800000, refCount=1, isUsed=1, recv_frame_len=102, payloadLen=102, poolID=3
[256] phy_addr=0xe4e00080, phy_base_addr=0xe4800000, refCount=0, isUsed=9216, recv_frame_len=96, payloadLen=96, poolID=-2130837248
Segmentation fault

And if added "usleep(100000);" in the second while loop, it seems better.

Thanks!

  • Hi Ning,

    It seems like TAP interface is receiving UDP traffic, It will be unicast traffic and should flow via virtual network interface (eth1).

    Purpose TAP interface is to forward information related to shared multicast and Broadcast frames, Which are received to Ethfw and then forwarded to A72 via Inter-core communication of TAP interface using shared memory.

    Basically TAP interface reserves Limited space, when we are sending traffic at faster tight it might be taking time to free memory and keep available for coming frames.

    When you add usleep, it is freeing the memory and making availability for in-coming frames.

    But, will take a look at segmentation fault issue.

    Best Regards,
    Sudheer

  • Hi Ning,

    Can you make TAP's number of buffers same as Enet LLD intercore configuration. Make NUM_BUFFERS=1280 in ethfw/apps/tap/tapfirmware.h and verify the same value with BUFPOOL_BUF_MAX in enet/lwipific/inc/bufpool.h
    It should solve the segmentation fault issue.

    Thanks
    Vaibhav Jindal

  • Hi Vaibhav Jindal

     It solve the segmentation fault issue.

    thanks!