network speed

We have seen DM8168 reaching up to 800~900 Mbps. Which psp release/ kernel version you are using?   Are you measuring with TCP or UDP traffic?

hi yogesh,

Which psp release/ kernel version you are using?  

==> rdk3.0

Are you measuring with TCP or UDP traffic?

==> TCP

Can you please try following command before running iperf

ethtool -C eth0 rx-usecs 500

hi Yogesh,

we got following message when board to board connection directly.... 

Set coalesce to 500 usecs.

....

DVR1 Client
[ 3] local 192.168.3.111 port 49165 connected with 192.168.3.112 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 30.8 MBytes 258 Mbits/sec
[ 3] 1.0- 2.0 sec 28.5 MBytes 239 Mbits/sec
[ 3] 2.0- 3.0 sec 31.8 MBytes 266 Mbits/sec
[ 3] 3.0- 4.0 sec 31.4 MBytes 263 Mbits/sec
[ 3] 4.0- 5.0 sec 30.1 MBytes 253 Mbits/sec
[ 3] 5.0- 6.0 sec 30.0 MBytes 252 Mbits/sec
[ 3] 6.0- 7.0 sec 30.2 MBytes 254 Mbits/sec
[ 3] 7.0- 8.0 sec 31.8 MBytes 266 Mbits/sec
[ 3] 8.0- 9.0 sec 33.1 MBytes 278 Mbits/sec
[ 3] 9.0-10.0 sec 56.1 MBytes 471 Mbits/sec
[ 3] 0.0-10.0 sec 334 MBytes 280 Mbits/sec

DVR2 Client
Client connecting to 192.168.2.111, TCP port 5002
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.112 port 39251 connected with 192.168.2.111 port 5002
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 37.5 MBytes 315 Mbits/sec
[ 3] 1.0- 2.0 sec 37.8 MBytes 317 Mbits/sec
[ 3] 2.0- 3.0 sec 36.0 MBytes 302 Mbits/sec
[ 3] 3.0- 4.0 sec 35.9 MBytes 301 Mbits/sec
[ 3] 4.0- 5.0 sec 34.5 MBytes 289 Mbits/sec
[ 3] 5.0- 6.0 sec 50.9 MBytes 427 Mbits/sec
[ 3] 6.0- 7.0 sec 42.8 MBytes 359 Mbits/sec
[ 3] 7.0- 8.0 sec 36.4 MBytes 305 Mbits/sec
[ 3] 8.0- 9.0 sec 36.4 MBytes 305 Mbits/sec
[ 3] 9.0-10.0 sec 37.6 MBytes 316 Mbits/sec
[ 3] 0.0-10.0 sec 386 MBytes 323 Mbits/sec

You are testing full duplex mode. I need to see if anything additionally needs to be enabled for full duplex mode. Can you use window size 256k and run iperf test between DVR1 to DVR2 as below

DVR1: set coalesc to 500

DVR1 : iperf -s -w 256k

DVR 2: iperf -c <DVR1_ip> -w 256k -i <iterations> -t <time_in_sec>

hi Yogesh,

Many thanks for your reply.

Would you mind to show us your result from iperf ??

I dont have 8168 board with me now. I will see if I can get these results for you.

Hi Yogesh,

Many thanks.

I found following description

NVR
Several changes are done to enable customer build an NVR product (mainly in DM816x). This release provides
NVR GUI application (UDWords design) with various optimizations & feature enhancements.
• Updated Ethernet offload driver integrated for DM816x. Ethernet offload driver helps increasing NVR use case
channel density. Refer Ethernet Offload Driver for more details.

==> does it mean the features was included in kernel driver in rdk4.1?  will it influence the performance of iperf under giga LAN ??

The description is about offload driver this does not apply to the default ehternet driver present in kernel. If you are using offload driver It will help you save lot of A8 cycles as well as meet the network bandwidth that you need. (comparable to original ethernet driver) 

In RDK 4.1 / 4.0, please have look at this appnote dvr_rdk\docs\AppNotes\DM81xx_DVR_RDK_Ethernet_Offload_Driver.pdf Let me know if are able to access this document or not.

this has detailed performance numbers measured with iperf for TCP Tx/Rx as well as UDP Tx/Rx. If this doesn't meet your network bandwidth requirements you might have to use default eth driver in kernel. I'm still looking for similar data that of original driver and I will inform you if I could find that.

hi Yogesh,

thanks for your reply. I have one question about following description in DM81xx_DVR_RDK_Ethernet_Offload_Driver.pdf

Limitations
• In DM816x, when eth_offload driver is enabled, c6x DSP cannot be used for any other purpose other than
ethernet offload

==> does it means the SCD function should be closed??

hi Yogesh,

do you know how to know the loading of c6x when offload is running?

hi Yogesh,

By testing rdk4.1, the speed could be 800~900 Mbps if offload was enable.

Does TI have intention to implement this in kernel driver instead of DSP ?? Because it will influence the function in DSP side due to its limitation.

Sorry for late reply I was on leave. 

• In DM816x, when eth_offload driver is enabled, c6x DSP cannot be used for any other purpose other than
ethernet offload

==> does it means the SCD function should be closed??

- No. Actually the offload driver is a low priority task on DSP while SCD (AlgLink) will be a higher priority one so it can pre-empt the offload driver task. Currently (in RDK 4.1) the offload driver is a busy loop looking for packets, semPendTimeout is 0. If you want to run SCD along with offload driver, it is recommended to change pin->semPendTimeout  to a non-zero value (1 / 2) ETH_OFFLOAD_emacOpen().

do you know how to know the loading of c6x when offload is running?

- Load will be 100% on DSP as it is a busy loop.

Does TI have intention to implement this in kernel driver instead of DSP ?? Because it will influence the function in DSP side due to its limitation.

- I'm not sure if I get this question completely. The offload driver is created to be run on slave core (DSP / M3) so that A8 doesn't spend huge cycles on Network Tx/Rx and it is available for other fuctions like hosting GUI and doing HDD operations. Part of the driver is still in kernel which implements required ioctls supported by original ethernet driver. Just the difference is work happens on slave core based on RPC messages passed across. If you dont want DSP to get influenced, you can first increase semPendTimeout  as mentioned above or consider moving offload driver to on of the m3.

hello Yogesh,

    Great thanks for your reply.

I am asked to migrate offload function in RDK4.1 to RDK 3.0.

I'm not sure what's the range that I should modify. Could you help me that?

By my checking, the modification included something as below under m3video in RDK 4.1. 

#grep -r "ETH_OFFLOAD" .

...

./links_m3video/system/system_m3video.c:#if defined(SYSTEM_ETH_OFFLOAD_ENABLE) && !defined (SYSTEM_ETH_OFFLOAD_CORE_DSP)
./links_m3video/system/system_m3video.c:#if defined(SYSTEM_ETH_OFFLOAD_ENABLE) && !defined (SYSTEM_ETH_OFFLOAD_CORE_DSP)
./links_m3video/system/system_m3video.c: ETH_OFFLOAD_init();
./links_m3video/system/system_m3video.c:#if defined(SYSTEM_ETH_OFFLOAD_ENABLE) && !defined (SYSTEM_ETH_OFFLOAD_CORE_DSP)
./links_m3video/system/system_m3video.c: ETH_OFFLOAD_deInit();

But according to following in DM81xx_DVR_RDK_Ethernet_Offload_Driver.pdf, it looks like something conflict.

Is there any thing I miss-understandard?

Features
• In DM816x, offload driver uses the c6x DSP

For DM8168 RDK 4.1 by default offloads ethernet driver to DSP and it works on DSP.

for DM8107 ethernet offload driver is offloaded to M3VIDEO, you need not worry about that in case you are not using DM8107. DM8107 has completely different Ethernet IP compared to DM8168. and the pdf which you are looking at explains only the case of DM8168.

hello Yogesh,

    Great thanks for your reply.

I am asked to migrate offload function in RDK4.1 to RDK 3.0 completely.

I'm not sure what's the range that I should modify. Could you help me that?

By my checking, the modification included something as below in RDK 4.1. 

     #grep -r "ETH_OFFLOAD" .

    ./eth_offload/bios6/inc/cpsw/csl_emac.h:Int32 ETH_OFFLOAD_EMAC_Poll(Void *arg);
    ./eth_offload/bios6/inc/emac/soc.h:#define CSL_EMAC_0_REGS (ETH_OFFLOAD_EMAC_0_REGS)
    ./eth_offload/bios6/inc/emac/soc.h:#define CSL_EMAC_1_REGS (ETH_OFFLOAD_EMAC_1_REGS)
    ./eth_offload/bios6/inc/emac/csl_emac.h:typedef Int32 (* ETH_OFFLOAD_EMAC_POLL) (Void *arg);
    ./eth_offload/bios6/inc/emac/csl_emac.h:Int32 ETH_OFFLOAD_EMAC_Poll(Void *arg);

    ./cfg/ti816x/config_1G_128MLinux.bld: len: ETH_OFFLOAD_SIZE
    ./cfg/ti816x/genaddrinfo.xs: var ETH_OFFLOAD_ADDR_STR = Value2HexString(ETH_OFFLOAD_ADDR);
    ./cfg/ti816x/genaddrinfo.xs: fd.writeLine("#The start address of ETH_OFFLOAD section ");
    ./cfg/ti816x/genaddrinfo.xs:  fd.writeLine(GetShellExportString("ETH_OFFLOAD_ADDR",ETH_OFFLOAD_ADDR_STR));
    ./cfg/ti816x/BIOS_c6xdsp.cfg:Program.sectMap[".eth_offload_shm"] = "ETH_OFFLOAD";
    ...

    ./Makefile:ifeq ($(SYSTEM_ETH_OFFLOAD_ENABLE),yes)
    ./links_c6xdsp/system/system_c6xdsp.c:#if defined(SYSTEM_ETH_OFFLOAD_ENABLE) && defined    (SYSTEM_ETH_OFFLOAD_CORE_DSP)
    ....

Could I just migrate these?

You need not get concerned about internals of offload driver implementation unless you need to modify or enhance the driver. By default the offload driver is enabled in DM8168 it should work as it is.

hi Yogesh,

    thanks for your reply.

I want to migrate the offload function from rdk4.1 to rdk3.0

Did you mean that I could just modify the above mentioned files? or should I modify something in kernel ?

backporting of this driver not advised. I thought you are moving to rdk 4.1, there are changes in kermel to have additional interface eth1 and register offload driver.

hello Yogesh,

     thanks for your reply.

Because it will take a lot of effort for us to migrate to rdk4.1, we "have to" migrate this function from rdk4.1 to rdk3.0.

Please help us to point out the changes in kernel. Really appreciate~~.

Thanks. Super Yogesh

You can yourself find out on kernel tree hosted by dvr rdk. Search for 'offload' on dvrrdk_kernel_rel_04.00.00.03 branch. There shouldn't be many. 

http://arago-project.org/git/projects/?p=linux-dvr-rdk-dm81xx.git;a=summary

hello Yogesh,

thanks for your reply.

backporting of this driver not advised.

> May I know why?

hello Yogesh,

• In DM816x, when eth_offload driver is enabled, c6x DSP cannot be used for any other purpose other than
ethernet offload

==> does it means the SCD function should be closed??

- No. Actually the offload driver is a low priority task on DSP while SCD (AlgLink) will be a higher priority one so it can pre-empt the offload driver task. Currently (in RDK 4.1) the offload driver is a busy loop looking for packets, semPendTimeout is 0. If you want to run SCD along with offload driver, it is recommended to change pin->semPendTimeout  to a non-zero value (1 / 2) ETH_OFFLOAD_emacOpen().

- if we want to use offload function only, how do we disable SCD ?

backporting of this driver not advised.

> May I know why?

We wont be able to support on older releases.

- if we want to use offload function only, how do we disable SCD ?

It should be a parameter in your usecase file (enableSCDAlg in dspAlgParams) or in vsysConfig (enableScd) which can be set by demo.

hi Yogesh,

thanks for your reply.

I have a question.

By checking, these mentioned parameters(enableSCDAlg in dspAlgParams, enableScd in vsysConfig) are in host side, but I didn't found any judgement in DSP side to judge if enable offload by SCD.

Could you help to point out where it is? thanks

hi Yogesh,

I have two more questions

Currently (in RDK 4.1) the offload driver is a busy loop looking for packets, semPendTimeout is 0. If you want to run SCD along with offload driver, it is recommended to change pin->semPendTimeout  to a non-zero value (1 / 2) ETH_OFFLOAD_emacOpen().

1. do you mean set  pin->semPendTimeout  to 1/2 ?  by my checking, 

struct ETH_OFFLOAD_inter {
...
Int32 semPendTimeout;      ==> are you sure  1/2(0.5) is correct? or something I miss-understand??
...

2. do you mean in default state pin->semPendTimeout=0, we just need to make sure SCD was disable? right?

Yogesh Marathe said:

You can yourself find out on kernel tree hosted by dvr rdk. Search for 'offload' on dvrrdk_kernel_rel_04.00.00.03 branch. There shouldn't be many.

http://arago-project.org/git/projects/?p=linux-dvr-rdk-dm81xx.git;a=summary

hi Yogesh,

by your above mention, I got following. And I'm not sure where I should refer to when migrating to RDK3.0.

Could you give me a hint?? Thank you very much.

2013-03-21 Sriramakrishnan A G Redo: ti816x : enable ethernet offload driver for emac1... Redo: ti816x : enable  ethernet offload driver for emac1 interface.ethernet offload driver can claim ownership   of... 
2013-03-21 Sriramakrishnan A G ti816x : enable ethernet offload driver for emac1 interface.ti816x : enable  ethernet offload driver for emac1 interface.ethernet offload driver can claim ownership of... 
2013-03-11 Kedar Chitnis - updated clock816x_data.c to associated "eth_offload... ...clock816x_data.c to  associated "eth_offload" EMAC driver with EMAC clocks - this allows eth_offload driver to make use of EMAC  clocks - if eth_offload driver is not used then this change... 
2010-12-23 Johannes Berg led_class: fix typo in blink API offload is a regular error code that is... 
2010-11-18 John Fastabend net: zero kobject in rx_queue_release features and offloads are toggled.  Presumably this... 

Int32 semPendTimeout;      ==> are you sure  1/2(0.5) is correct? or something I miss-understand??

[YM] by 1 / 2 I meant set semPendTimeout to one or two ms.

2. do you mean in default state pin->semPendTimeout=0, we just need to make sure SCD was disable? right?

[YM] yes in default state you need to disable that, I hope there is nothing else running on dsp

You should be worried about only first three patches (dated 2013)

 Rest all should be there in earlier version of RDK kernel already.

Can you mearsure cpu load of cortex A8 as well whe you hit 600-700 Mbps? Is this TCP or UDP, Tx or Rx ?

hi Yogesh,

Can you mearsure cpu load of cortex A8 as well whe you hit 600-700 Mbps? Is this TCP or UDP, Tx or Rx ?

- 9x%, TCP, Rx 

I have one more question

I got following messages as using rdk4.1  

eth_offload: `' invalid for parameter `shared_mem_phys_addr'
insmod: error inserting '../kermod/eth_offload.ko': -1 Invalid parameters
ifconfig: SIOCGIFFLAGS: No such device
udhcpc: SIOCGIFINDEX: No such device

And modify as below in run_nvr.sh

. ./parseEnv.sh
. ./env_${ENV_DDR_MEM}_${ENV_LINUX_MEM}.sh

# 'eth1' use eth1 interface and do DHCP before moving ahead
# 'eth0' use eth0 interface and do DHCP before moving ahead

eth_if="eth1"

cd scripts/

#if [ "$eth_if" == "eth1" ]; then ./eth_offload_load.sh; fi => I mark all lines behind this and run this scripts manually 
#if [ "$eth_if" == "eth0" ]; then ./eth_org_load.sh; fi

#cd -

#./bin/sys_pri.out --dmm-pri-set A8 0

#$dvrapp_dir/dvrmain --$eth_if -qws hdmi0 hdmi1 &

Did you know why ? Thanks

MyGod said:

I got following messages as using rdk4.1  

eth_offload: `' invalid for parameter `shared_mem_phys_addr'
insmod: error inserting '../kermod/eth_offload.ko': -1 Invalid parameters
ifconfig: SIOCGIFFLAGS: No such device
udhcpc: SIOCGIFINDEX: No such device

And modify as below in run_nvr.sh

. ./parseEnv.sh
. ./env_${ENV_DDR_MEM}_${ENV_LINUX_MEM}.sh

# 'eth1' use eth1 interface and do DHCP before moving ahead
# 'eth0' use eth0 interface and do DHCP before moving ahead

eth_if="eth1"

cd scripts/

#if [ "$eth_if" == "eth1" ]; then ./eth_offload_load.sh; fi => I mark all lines behind this and run this scripts manually 
#if [ "$eth_if" == "eth0" ]; then ./eth_org_load.sh; fi

#cd -

#./bin/sys_pri.out --dmm-pri-set A8 0

#$dvrapp_dir/dvrmain --$eth_if -qws hdmi0 hdmi1 &

Did you know why ? Thanks

After I do following manually before  ./eth_offload_load.sh

     export ETH_OFFLOAD_ADDR="0x40300000"

it will show

[ETH OFFLOAD] Shared memory address = 0x40300000, size =0x0000f400 !!!
[ETH OFFLOAD] GRO is ENABLED !!!
[ETH OFFLOAD] EMAC-1 is ENABLED !!!
[ETH_OFFLOAD] RX CSUM is ENABLED !!!
[c6xdsp ] [ETH_OFFLOAD] OPENED !!!
[c6xdsp ] EMAC_start returned 0
udhcpc (v1.19.4) started
Sending discover...

Thanks

For TCP bandwidth issue enable interrupt coalescing as below and check this again

> ethtool -C eth0 rx-usecs 500

For eth_offload.ko issue, dont run the scripts manually as there can be environment variable issue if any other intermediate scripts are assuming anything about environtment variables, do you see same error when you run run_nvr.sh after choosing eth1?

hi Yogesh,

did you have any idea about following message??

root@dm816x:/opt/dvr_rdk/ti816x# ./ethtool -C eth1 rx-usecs 500
rx-usecs unmodified, ignoring
no coalesce parameters changed, aborting

eth1 interface is offload driver which by default enables coalescing so no need to do this on eth1 as the action will be ignored. I though you were measuring TCP Rx bandwidth with eth0 - default eth driver and hence suggested this.

You are seeing 600-700 Mbps with offload driver or with original eth0 driver?

hi Yogesh,

   Thanks for your reply.

You are seeing 600-700 Mbps with offload driver or with original eth0 driver?

- original eth0 driver

I have one more question.

I see following in run_nvr.sh and not sure if it's related to offload. Would you mind to explain this? Thanks

sysctl -w net.core.rmem_max=33554432
sysctl -w net.core.wmem_max=33554432

This is not related to offload but this is related to CPU cycles optimizations on A8. This is done in order to hold more data in kernel so as to enable user space application on linux to consume data in bulk packets instead of pack by packet. This will enable you to use recvmmsg() api from user sapce applications reducing overall context switches. This is beneficial especially for nvr application.

If you are contrained by memory and do not care about CPU load, it is ok to remove this. 

hi Yogesh,

   Many thanks for your response.

One more question is in FC_RMAN_IRES_c6xdsp.cfg, I see something below, would you mind to explain what's the purpose of following behavior? 

var CacheDisableRegions = [..., "ETH_OFFLOAD"];

DisableMARBits(CacheDisableRegions, Cache);

I'm not excatly sure why this is done but as far as I know  lists of software descriptors which hold TX and RX packets to be processed are placed in ETH_OFFLOAD section. The code that you have pointed out disables cacheability for that particular section from DSP side. This might be done to avoid cache coherency operations on this section as the software descriptors are modified by both A8 & Dsp to maintain state of the packet and any cache coherency issue here may result in either drop of packet or incorrect processing on the either side.

hello super-Yogesh,

    Really thanks for your reply.

Because I don't know how to move these following to rdk3.0, would you mind to give a hint about this ?? 

var CacheDisableRegions = [..., "ETH_OFFLOAD"];

DisableMARBits(CacheDisableRegions, Cache);

ETH_OFFLOAD section has to be uncached on dsp.

hello Super-Yogesh,

 As I see something below, how did I know each meaning of variable? is there any document?

/* CPU access code and data - 0x80000000 cache enable */
Cache.MAR128_159 = 0xFFFFFFFF;
/* TILER memory cache disabled - 0xA0000000*/
Cache.MAR160_191 = 0xFFFFFFFF;
/* memory cache disabled - 0xC0000000*/
Cache.MAR192_223 = 0xFFFFFFFF;
/* memory cache disabled - 0xE0000000*/
Cache.MAR224_255 = 0xFFFFFFFF;

Refer section 1.3  C674x DSP Subsystem in TRM, this section in the end has links to  "TMS320C674x DSP Megamodule Reference Guide" and "TMS320C674x DSP Cache User’s Guide", which indetail explains MAR bits and cache usage. You might have to also look into xdctools documentation for Cache module.

hi Yogesh,

by checking  "TMS320C674x DSP Megamodule Reference Guide", I have a concept for MAR.

I found following in FC_RMAN_IRES_c6xdsp.cfg in rdk4.1

/* CPU access code and data - 0x80000000 cache enable */
Cache.MAR128_159 = 0xFFFFFFFF;

/* TILER memory cache disabled - 0xA0000000*/
Cache.MAR160_191 = 0xFFFFFFFF;             ==> "0" means non-cacheable and why it fill "1" ?

/* memory cache disabled - 0xC0000000*/
Cache.MAR192_223 = 0xFFFFFFFF;            ==> "0" means non-cacheable and why it fill "1" ?

/* memory cache disabled - 0xE0000000*/
Cache.MAR224_255 = 0xFFFFFFFF;           ==> "0" means non-cacheable and why it fill "1" ?

and the related parts shown in rdk3.0 are as below

/* TILER memory cache disabled - 0xA0000000*/
Cache.MAR160_191 = 0x7FFFFFFF;

/* memory cache disabled - 0xC0000000*/
Cache.MAR192_223 = 0x00000000;

/* memory cache disabled - 0xE0000000*/
Cache.MAR224_255 = 0x00000000;

Is there any thing wrong or I mis-understand??

hi Yogesh,

one more question about XDC doc

I look at DVRRDK_04.01.00.02\ti_tools\xdc\xdctools_3_23_03_53\docs\docs\index.html and found the URL links inside almost fail to link.

how do I know the detail usages of each Cache.MARn, for example how do I map following regions to which MARn ??

var CacheDisableRegions = ["SR0","REMOTE_DEBUG_MEM", "ETH_OFFLOAD"];

Don't compare Rdk 4.1 and Rdk 3.0 in this aspect because there might be some features added in 4.1 which needs these sections to be cacheable. As far as ETH_OFFLOAD is uncached it should be ok for offload driver.  Remember I have already mentioned backporting of offload driver is not advisible.

Sorry I'm not cache expert and I would have to find out the answer 

For documentation copy xdctools.chm locally and try to open it will show links.

Hello Yogesh,

thanks for your reply.

I have one more question.

how many occurence of A8 CPU loading when offload was enable?? Could you describe the behavior in detail?

hello Yogesh,

  the offload was enable in eth1 default, how could I change it to eth0?  

how many occurence of A8 CPU loading when offload was enable?? Could you describe the behavior in detail?

[YM] Sorry this question is not clear to me, what do you mean by occurence of A8 CPU loading?

  the offload was enable in eth1 default, how could I change it to eth0?

[YM] You just need to mention eth_if="eth0" in run_nvr.sh