This thread has been locked.

If you have a related question, please click the "Ask a related question" button in the top right corner. The newly created question will be automatically linked to this question.

AM5708: AM5708: Ethernet transmission issue in PRU interfaces

Part Number: AM5708
Other Parts Discussed in Thread: AM5728

Hi:

      I have a custom AM5708 based board and SDK6.02. I have an ethernet board with 2 interfaces (eth1 and eth2) over PRU_2.

There is another ethernet interface over GMAC port (eth0). In all these cases the phyter is DP83822 and the physical media is cable.

The issue only happens in the PRU ethernet interfaces (eth1 and eth2), not in the GMAC interface (eth0) under the same testing conditions.

If there is medium-high transmission rate, the issue comes up when unplugging and plugging back the cable. In this case the interface stops

sending packets.

After some investigation here are my comments:

   1.- This behavior does not happen in GMAC port. It is not affected by high tx/rx traffic.

   2.- PRU ports continue to receive high traffic messages, but they don´t send anything out any more.

   3.- The only way to recover PRU ports are by doing "ifconfig down" and then "ifconfig up".

        In this process the PRU firmwares are loaded again, so they start from an initial conditions.

   4. After debugging the code, the issue seems to be in "prueth.c" in the transmission part.

      Here below you have this piece of code in function "emac_ndo_start_xmit()":

    qid = prueth_get_tx_queue_id(emac->prueth, skb);
    if (emac->port_id == PRUETH_PORT_MII0) {
        /* packet sent on MII0 */
        ret = prueth_tx_enqueue(emac, skb, PRUETH_PORT_QUEUE_MII0,
                    qid);
    } else if (emac->port_id == PRUETH_PORT_MII1) {
        /* packet sent on MII1 */
        ret = prueth_tx_enqueue(emac, skb, PRUETH_PORT_QUEUE_MII1,
                    qid);
    } else {
        goto fail_tx; /* switch mode not supported yet */
    }

    if (ret) {
        if (ret != -ENOBUFS && ret != -EBUSY) {
            if (netif_msg_tx_err(emac) && net_ratelimit())
                netdev_err(ndev,
                       "packet queue failed: %d\n", ret);
            goto fail_tx;
        } else {
            return NETDEV_TX_BUSY;
        }
    }

  Comments about this piece of code:

          1.- If I run this code, when there is high tx traffic and unplugging/plugging the cable, the system goes

               very slowly and the message "RT Throttling actived" appears in the console. After a short

               time, linux crashes.

               To avoid this I have to replace "return NETDEV_TX_BUSY" with "goto fail_tx".

2.- When the problem comes up I see that function "ret = prueth_tx_enqueue()" is returning "-ENOBUFS".

      and no messages are sent out. Reception is fine in this situation; it still continues to receive messages.

     The only way I found to get out of this situation and come back to a normal situation is by issuing "ifconfig eth1 down"

     and then "ifconfig eth1 up". (The same for eth2 interface).


Regards

Billa

  • Hi:

        This issue does not happen if ethernet media is fiber optic.

    In this case it works fine.

    Regards

    Billa

  • Hi:

       I reproduced this issue in the IDK. The setup is as follows:

       1.- AM5728 IDK with Linux RT in both SDK6.02 and SDK6.03 binaries.

       2.- Ports eth2 and eth3. Connected directly to a Ubuntu PC (first tested in eth2 and then in eth3).

       3.- Attached is the script file to setup eth2 and eth3 ("dual_idk.sh").

       4.- In the Ubuntu PC I have the wireshark to sniff messages sent out by IDK..

       5.- Attached is the program (in source and binary mode) to run in the IDK.

            The format of this program to send messages continuously is:

                   ./goose_user_space [interface] [time_between_messages_in_usec] [number_of_messages_to_be_sent]

              i.e. to send 10 messages every 1 second over eth2

                   ./goose_user_space   eth2    1000000    10


              i.e to send messages continuously at the highest rate

                   ./goose_user_space   eth2    0   0

         6.- To make IDK fail, run the above program to send messages continuously at 100 usec or highest rate.

               While the IDK is sending, unplug and plug several times the cable.

              After some tries you will see that the interface green led remains flashing all the time and wireshark in the PC is not receiving any message

    PK�FsQgoose_user_space_in_idk/PK��P P;��h#goose_user_space_in_idk/dual_idk.sh���n�0��<��4ʎR%HY@+��L*44ʲ���Q�[*E�{���#��>��\����9���2
    �����װ
    j�IB�c�rJ@Ӵ��I���a���&���!�4{�_(��Q��ļ�4�ۭfۚy{���G>o��h�ml���y��gs�����P�`�k�5�v^�:��$:`@�!�ח'J�<��d�y%/���+ \`�{E��G��{NR�A�V
    ʪ�� �������@.H����z��'�X�����>x��Yf���K�ɞH�acFU����M��"�������u��F�H��'�N�i7'�D���L�>h��F
    _�&IA#��*ݷ���Ga�sU���X�=g%����y�r�Ă�R��R�%�DzZ���;.�޶���`L�=��o)�i�t�R�F)&L���Z��'��S�[�H�n�ωv��O0;F�k7���s;5�ji/,���?�#C��ҟZ�PK�F�P��[�1(goose_user_space_in_idk/goose_user_space�Zl#E�ٵ_��$`�;���=	p�G5��I�ml'$��{
    EHё(�{�h�$R0�@t���Y�z���M��y������=a���sf^
    kb��nΖA?�9n.�^�4���l	�3K*[R��xA!qI�@�	�B4$��*���p!
    �
    �ƅ����ȋ�mĉX5�"�x��ý�W���I!�xle�p.dg�m�}
    �8s^2��v{����V�P2�J��Djx4�
    �s��Lr����^��.�C�ٴ�Q��x��+WN?�����OF��]��D�%��y9�h?�"/�~��&E���b�������reHmi��#���um�wd�?i��>���p���ɉ[�W�۴��]��k��sۍ������Z�C�ց}�́�w�8��A�t��ʁk�H�80C�ƨ'��\:v��g�|6�c��t*�2z6��R\�"�}�l�e��T��
    �C9�Ȕ�c#���K�c�$�%u=â�t���T<�f�����|d(�H�]��:w����60�NI�QF�W��9gd]��JJs���~�-�zMx�}�V0�Hc�5�.Cߓ��O��HW1�L�s�4�-#�A�F�s;ҋk'���L�z7���큏0?��ȇ��@�55Q���`ʻ���*>{����sgV��p���»���p�-���la��������f���[�ն����eS��.���Uj��.
    ud������@��M*��w��>Y
    y���C<X���_�M��S�v�ݳ�ͅ��ss�{7��B�g�>{at�}��}����l�X�Yc.4]��@�a�wLu�LN_��ISO����t~�хH������O|��]���'�t���Q��n���w�7��+{���|����gI�W�0���{��Y��Ϫ�b,ϧ\�Sf���B{�k��և
    ~#���u
    yax��0‚�]"�(��i�MN��2O�-y^�/)��R���K�q*l�s�i��T�r�a~��1SO��y%<9ݨ�S��u@;�t5�}l��4Yۙ��gp����+�aî��Ҿ3�(�����+��� nqM(_�h�ܬ�|���:a�F{h�|��ވ�п��WkFܞ�G~�h��6�u
    �e^(;��uR� >�����l#��������g=�?��ע�=�A�_���P�c����px�‰�TH�*��"��~���l�u{����Ю���q�I��k��,���4��Xy�r��y�oŘ[�3�uQ���,���_���LK�L��R6;^c�}e՘���sso��9Z~�p&�>��[ی�D����q�2m���'�M\FX���0��"[���}��-\A���U�zԫ�缂���B��y���a�B]�󨟦5̃�e�^��>���Vk�<l�
    �+{�d�|��ðί�5��LNw>Q�'NN��3QP.���~���H��`�}����;�W���ճ�k����θ����^��2��둋�?�
    �	�Ouq6
    Y��[B�#���a��|inm_��&R�Ҹ6��r�=��G���i=����Ҝ�#f�8�9=���rG�?�X�]�����o�	�
    ۷k�v�Zѵ�λ`�RĻб�δ�����
    i�~ʽo,�h���4�:��/�ՎN�54��͏et-<��U$��n�͝�������Z���b�C+ �bM��~��Ϊƹoy�ZR�(�W?ͥ��\�U�U.�a�
    g4�.���>|��O��_��ٯ��|)삮�g�5��in.���AS9WC?���YJ�c������/^�5��(�I�ҙ��9_B��ɼ��
    ;�dt�h_��{��]�^�kϊ�p_.�M���T�1��
    {GW,ۉ���K��x{Gw�{�}wi=�˔˯O�t򎧩�}�������Y��͛n;i��NU���?U��N�hM8�k��t26���֟�jX���@&��O$��y=���Z[p�@k�e�`x�	�Y=@��M7%[0,�v'R�lZ@t+%i�ێ��#&�c���K0����S^��勔r��׺��u|1_K��
    |m��o�i�+���h;f]u¾��[W���+�ߌtK�\�s�$��ϩ&_��m�%�VL���H���φVy�,�ț�9w���D>u�u&��g�a>�����-�6R����r�'��B�e=�C�H{���xn�~�xv�`�*���0[���\>�{9�o��=��)GW�A
    ��=�I���
    I�bЋF�~�n��
    ����Trý�!���KS�JZ�V(of����f#<@�A�<�۳2��e|^���"�;��E�+_�p��.;�_U���;q�e\�L�R���]s�m�To)�6�Ro,;��aNqa��M>��k���O�u�RsE�1/�C��jj�q[�5��4�Q��}e���-��@bk���oU){�\N}E-���j�M^���^�ժu��V������b�`*˘Z��o[�־�5vlP��/a>�>���.Zօ��<�A��J��i�Zt㖄*t�]kpĕm@�^
    1-�4Pt�[��n���b�	Te���`�6i�2��_�	KU�/dn��6�Z�u���e��j|��Q�Z���g��A�r�����_�Y�^:�ނ4k6Ue:^�	����E�CU%㸈H#:ʍ��z&ºɘx���x��VheP_f<�jiQ�
    ���Z[գ��B�7	��.�%R�|0&��5���c�|���f�~G
    �p)$D$���k�n�bD�I�����)���+�ʄ�}i���˅���H`d�-@h�h����e���D2NȊ��k�${ 5L���Ѹ/hO�Иɤx>���0c��Θ��X����zpP���j���>Ÿ[U���{ԣx�ݤ��\M���{Z�M�={�������]�uU���]����URW��Eר�Fk����KY=7N��f�C�ʀ�~rF�U�x�n\��Y���=��t��3]g�g+�s�1.s)-�>d�ۿ��s�Xh��E�b�aF;��}������@;���-0ԟLG���~0��SDfx���N!�O��D4�]���1�S�c��#8�U �y.1�dZ`�V-�!@L��C�m��2�Czv@`�bii�|���8��i~�T��Si�T:b�H�D��)�E$j(ד���C���c����t�"8��i]�'b��Jǂ1g��Y�L�T/n����H4����Y.q��3=5���u�A?7ҏ(`R2%5������tiq��hv &��;IXF�R���}I��i�l8�K���D��_��=];��^w���浾��=��"x�48�X�U3VZ���&�+v�ֺ�Ds����n�b���	��	Z/)c�ʟ0�SL�ke!�s��"�~Q�!�C���g;�6��K�ϕgZCgL,K~a�B�S�.���nVM���".��"�
    �r�x��E\.�E���"��gW
    �f�����*�j��T��o���kp�	$��������똑�ċ��)�5���1�Zf�K\��
    ������������;���#��N��l�A��n�?C��%�W6��W���*��[��z��jO��܈���f��U6���<m��\��}�Y̿���w�߱�\���-���s��:��k�߃0�=��n�c����(�*Mqs�2W�ҟe�����5s���)Oҭ��*[�vAp�w�Vy����g��h�C<��#Ynὕ��)/�X^��[x^��e�g�D� ���S�R� �u��S���1�OVJ��R
    �M�bQ�Z�(����m�6��֢�)����!#X�v�%��e��Ѿo;�_gֲ����w�b��Z��}��3��_�/u�x7>ȁ��~Vy���,,V�Tk>�(k����d{�`<=��=�l�W����:�YU�W2��j)�'���9�&�����K�gh#}����������f��@���Z�j�kN�|L�a�B���s0͖��ޢ�}2�#��؇�s�_>���j0h��}?�İ+��G������<@�����(A\��	z��Ebح�9l����@2�MF����"��QKe�z^�[�[-/�*=K�c�?�#�ᡡ1$�!s�oQ�霎
    ����$�ׇbfGKO0+Vv�Kd��v�D�";�WǞ];-t&��ڽwgGwd�+�ݥ7���ٽK�y$��P��ۿ�������GU%v=�G
    $�+��l�!���h�"�Z��
    Xi�({�|��2��?墄�td0��'u�)�y��_�9
    �r�D�~��`du����F~pfK$�C�a�*��m�;.>s+1�`nl(��g
    =h�ċ������+��H�I4�p"���Y0>��?��Ű�y.�N��l���3�d�2L��Gq��SZ��`��=��h">ʂ��1@/�P�n�E�ѡD���p�vdAL�!L�]4���a�uG�@�hj���T�h����#&������\�cfP���:.�r�w���kr�7��	��},+�ݛ?3-&�8�A�__��3�k��T���x&�6yR���Fyl���Q6���!��Wy>l�l�z��|��I�6��H9R�KU��$Ĩ���qYDi�9����>E�:����%o��w���1Υ�h�3L^��,�}v���M�q������
    �g�o!T���k��"�US�u���
    ^�df~n��t��2��j��9xA�D��W;xR��TO�c�~���U������~{G�O����g�&O�'{��a�v�@�U��������6��g�R�2|e�x6y[.0���%g�v��,"������T������|>/��M���,�o/�@k��vٕ���=
    t��.+��u��]�]u��ݜ�R��նگ�XW�֡E�?��åx隟o���r3�9���PK�F�P�
    VG��*goose_user_space_in_idk/goose_user_space.c�X{o�0��|
    �=�t-[�&�����e�($N�F��������	�Z��5��������g��o�g/�pn�<����KE�g��X�<ל[����(v�`�z���̛�Ѯ����l����.0b��!��S��J��<p�ll�w�}?������ʃ���xtl����^�tT��������bo�w2�|��ţ�2��#EY��w�s��S+d�xj��9�:f���
    M��ԍ
    ӗmG�����F^a�>�xƪ���л�cm��x���,�&�naL���ƺjè$�u#�i����Č4i�Mi詩���Գ9n��@0j[���
    �Ue[�n�e���&���h��Hp��_{��S�MGj�i��$J���$�%��Ίb��	2�2��Iw�;4��B���Hx8|�𔡥�vN{���%e�A���h1�l���(���7��,�z��4t��ta�V����mI�X�<Zxc[�{�|[�C�����JF��(v���~���$vIށx�
    �-b���]�̵[
    zo�Ѝ����i/1�N�Eͷs�N�g$�y��W����2Y
    ��["�r���M�����^��
    J[P�-��ez��$�Npk����v����!�YpdS�-u�����������^�*J[�g��
    r�p]������1�y~��-�W��
    ���܂���cMY)(W��:�ٜ�a�)��v�<7�?`4=gy�¹eK�Ҝq��z�Y�'4�%n�&Ny����"�?8�;�WJ(��3���|sqtea�q�K�ǖ��w�.rC���>w��,fs/�pو�ٞ_�Be%w��@RA#�b���J�<�OIRt�A��]�����2��������ijbkPj���ˮ�R�ȃ�
    ���S	Z���C��� 0!NM9^2]K+6�-Y��?���S���g���!w��Xp�Ü�"Ɓr�<h%V�7��x��s	L|/z%������A�J�Z���Ɗ���p�{�Q{�&��p���1�,X��nuP��Ub�3��ح��p�����7�����[U�4�H�/0�����+�������m�&hY-�2:쩡�UV`s�jY�.kb[KGѹ)����,pq"�U�0��<��Sa�5#�'\5��5-k3>n'�]�5w��n�T�����Y!q	b����G��O�(����
    7��캗�{,�o�+4�?�����7���'�h��YɌަ����JOд�W�'X��A晴�vz���F��X1| �< 9���	�vs��a�^xv�Z�Αj4���+��d!��,I]�S�,��.lj�I���	����Gq�����$z�d1��C�!~9y�~,z�
    :Qz�U��.P/�ތ3U�t�UWd��gU�Q4�5u����Ȁ����i��(��0�;�h��\]k���qʔ���(={�gJ��&!��C���kD]��O��uJ��
    ��	����Ӊ"��d�D-�X����-o���#�Z
    ���`�=7��ôߚ��y��������"�����Ek������fbJ�l_#3lg�x���!�I�O�J5DfK���}��yj���f�gQbG�1f���+T�XRsw������Hc���B��B0وO�ղ<�yvW�v�׀�U6�Ͼ>����
    cv��SF��Ws�v ¿����Y��ܥT��=��v
    �va��03��Ԁ)�J��JV
    ��(';�+����YӔ�ʵ�PKnRrQ)Ǖ�t� goose_user_space_in_idk/MakefileU˱
    � ����A܅NRJ��@.r��1�D���2��g�7��?�7��B��jLE!�2�Ҿ2�F8��$���9_��!@?ػ[c\�J�Ub_?s��@ȄEÉg!'��PK?�FsQ$goose_user_space_in_idk/
     KزCI��Ҁ�CI��1Z�CI��PK?��P P;��h#$ 6goose_user_space_in_idk/dual_idk.sh
     ~���e{�CI��KزCI��PK?�F�P��[�1($ Ogoose_user_space_in_idk/goose_user_space
     �����CI��*	�CI��PK?�F�P�
    VG��*$ �goose_user_space_in_idk/goose_user_space.c
     �5;����CI���|�CI��PK?nRrQ)Ǖ�t� $ goose_user_space_in_idk/Makefile
     �ꋽ�+B�CI��+��CI��PKG�

    Billa

  • Hi:

       The above program was built with VLAN tag in the GOOSE messages. The attached one does not have VLAN tag in the messages

    Billa

    PKLsQgoose_user_space_in_idk/PK�KsQ P;��h#goose_user_space_in_idk/dual_idk.sh���n�0��<��4ʎR%HY@+��L*44ʲ���Q�[*E�{���#��>��\����9���2
    �����װ
    j�IB�c�rJ@Ӵ��I���a���&���!�4{�_(��Q��ļ�4�ۭfۚy{���G>o��h�ml���y��gs�����P�`�k�5�v^�:��$:`@�!�ח'J�<��d�y%/���+ \`�{E��G��{NR�A�V
    ʪ�� �������@.H����z��'�X�����>x��Yf���K�ɞH�acFU����M��"�������u��F�H��'�N�i7'�D���L�>h��F
    _�&IA#��*ݷ���Ga�sU���X�=g%����y�r�Ă�R��R�%�DzZ���;.�޶���`L�=��o)�i�t�R�F)&L���Z��'��S�[�H�n�ωv��O0;F�k7���s;5�ji/,���?�#C��ҟZ�PK�KsQ�}�	[�1(goose_user_space_in_idk/goose_user_space�Z	L+E��mK�(�
    ���{�Jo@EQTD��YK�@���-O0�����x�xD�}_�W4c�D�m�D�����v�+�xą�����{:��'����9g楰&F(��
    ���ps1w��ؖ��d�,�`li���%�i…hH)�U�'�B���W���ۈ��jE�1
    c�x���+��W�(B>&��ʘ�$\���;�l��q�dz?�v{����V�P21J��Djl<͎�s���Lr����^��.�C�ٴ�Q���z藷��麲���M�z��o։v��Kn%�r��q҅,^��J�M�,�2����}G_Y���ʐ��x�fG��ӧ�ֵwTܑ�ᤵ��������+�&o��_ٴ��]��k�or�]r���g���džk�x[�9p����9p܁������܁k�H�80C�ƨ'�d�t�H=�r�l,3�F��T�e�l6���EF��<�d���Gs�C�)��Veӣ,���l,������	��S�|�E"�W$��f��h"�v����)�l�mx4��6��ʯ�qs�Ⱥ�%���8�{�
    [�
    �5<���GZ�<"�A�H�}O��?ixj"]�X3�j�U���F���4��v��0�A�ٙ4&�n��1�;a~�=�]Ӂ�kz�P5}��wգ��>8|��Y��+g���p���»���p�-���la��������f��w?X�U����4�c�Ss.���UjS3.
    ud����ɂ��@��M*“�w��?Yy���C<X���_�M���ӊv��s�·�����{/�q%�s
    �[��cl��0�ciV���5�~���ӝ5S3׮w���;1�}��i::`{�@6���Կ�����.�>ʕ�-���sg_|e�������W`�,
    �
    ���}oS6ǵ�9�]�����+|�,�?U�hu�U#����o��?�v�#/ϷFXp����?5����sP�)�%�K����_
    {=3|�6N����FpOu+��G^����[P�S3��څO�Q��HW���f:L����}�\��6�*�0"�;Î���͐����\
    �ׄ�5����)��
    ���^�������썸�{�R|�f��)y�w��\[����e'��NI��'>��>�6��ē>7�>�w��S��G��|�@�Z��>��K��u,�ۃ~�Q8�
    �Y�cU���O�m�cRXC:ڕښ8�yɑ}�Q��w�%�&�+�C^��"/�s��33?Q�����2]a~��n��dZ�f�w���j��o�����|~�����V8q��ϯe�U"�~�}�8s�6l��S3&.#�Y�K�Yx	��-\Nx��>�� �G�*K=�U�wn��qn�b�܂��s�n��ڹ�O3���o/E]��g��96Ɔ��=f3g>��aX�W���e�fz�,��fv�,(L�x?�x�u��?�>�{�;z�̪3��v�eSs���YW����Ջ�R��c=r��g�E;���.��!�"�~K�w�v�z~$�p�/�m���DjX[�F�\.:���
    =��wZ�Ec��,�鈙 �XN�i�`�ܑ��&i����X��Es�����{�]{�{��yl_�x�:wޙ�7�[�X!-=D�N��3����[Ƿ��ۡ�щ���fҳ���������������8������_�=��@�shd%֔J�7��j��֖篵 u����|���Y�Y��֨pF�2�O��W���t�+�<w�<��@֤��/�]е򌽚�?�ϧ߃�yh*�*�'�?>K�|l^W�����+^�F��
    <��0�K)6z2�?q��D2�3�â���+�t׽���ם�Ų�A=�r��<�����x��a;q``�R�)ށ�]��{�]ZO�2���S?���i�z߻�svggr~gg��N�:��SU�q�Oջ��6Z���@:����Ч
    ��Ve�3�ɦ�I����D^��Dz���<�n�:�\k�s�@VO��&��-�N{�h6� ���������&���zG�&�A�V�N�)/���%J�Z�k]��:�:_S��
    |-��o@.ƽb>��ύ�c�U'�I�u5���b��H�T�5�1׊)�|N5��L4b�/��b�/q��q���*/��y39玡�{b�ȧN�Τ��7�R�����-���id����	g�yYW��#Ҟ(s=�۽�,��,�
    �?!�V�>%�O�^��knO�f��Uv��u_�I���
    I�b�K��~�n��
    ����Trý�!���KSu?-M�
    ›��(��S|P�=������Ej����ꇈ��j��)���WU-#�N�j1��T�-b�Xf"�[��
    ���N�~�S\��v������Sw��\Qn��K���p>����[����4�S��}e�~e��`��5B�J·�����S_QK7��{��0����r�j�eU��)��#�?�JS���m���׳ڎ
    J=��������z�x#�C˺p��gq7��#\Y��5l^�nܒP���g5�����߫a ƒ���
    ]�)|��ۮ��:��2�vl0m7P:�/*����n�?e���Pl�¸�t��f�ü�Y's���}2v��vգ��B��� �AF�MU��m���ol�z��XU�8."҈�r���Ʌ�v2&��5?�Zԗ��ZZ�5�--��V��`[HQ�fC�x���D*���P��ߓ,�OX��L���C�.Ń����Q}
    �MQ��7i��f\��9���by��P|x0M?����PvD_X1� ��x���-�B�c�d������I�pj�~!F��q_ԞH�1�I� |8-�?<`ƒcq�1=9��-������h
    7�4C5}�q��Vs7��G�I�
    ���ZR7󴸛<`{�p{I�]�����	�����%r���&�K�Qˍ���C=��z��7��*�V4���0������st�2Ů*�T�ie�{�p��:K=[9�#�q�Kii�!s��%}���B��,:ˍ1ڝt7o
    �ک���Bh�ѡd:� ��#�l�"2cۧ�t
    �|2�'�I�\�G�A����[����H6��n�Ą��t��(�8�g��X �������M���9��G�I�L>�H� Ǝ��m��YDR�F�q=�=�<�.���1��O��*�|�օ|"F{�t,s�����4N��F�ڑ9�D�������<�S+2��n>�G�F�LJ�ģC["6�՝.-���DVv� 	�
    �����`R�|�2K��)=.+Qj�W�w_�N���*�y�'�~�� 
    �>c��~�q���$w�n���n4�\d�o������~w��v�6@�X��'L��XYȵ��o��_�K �C��ɻD���4�/�>W�i
    �1�,�E�
    }O��nTL��Y5�G�'��L����+��e�%BWq��+��'�E\!�.&���rW	=]��B/�0��(o��.�H\+t]�&t}�1#��W����0�_c�5�&��ށx-^ہx^ׁ�s`́�7F@����g�ƃ�Q;�5�j��?K��l�����U���g5��N�՞�!_���6��mX��y�f'�J��}�Y̿�������ƿr����n�*�M~�(߀�~��X�W�1~�-̚�ǣx�4�-́�\nK�1��o�WS,́�2ǧ<I�.�?�m�����m��3���Y�)��O��H�[xoea�e�K�/�����^��gY�Y<�;H���T��?{��}&k~s������~Ӿ�(���B��f��6|�b�kQ��j�o
    ���,~���ٲ}�h߷����kZ���|��W��w�Ržr�Z�΁�9�V����IQ?�<�e��
    �5j�5���Ӳ�j0��P�[�߫yG����*�+������j��VN����%��_���H��*���o��{�h�Y�;�aA�֧Z������/�ӃGX��)�L���(b�L�:7���\��O���b�71�J#G��##C�?�!P����J׳�p�wG�v�z��x:2�LF��8v��Htl��ң��������Vm��Y�g�|v�
    e��z$>6::�$6d��-�p:��qBar��D�QL��x�	���s�]"��s$�������:���i�3Qd�޽�:{#{ww��@d���w���H$?���5�˙@ɏ�J�z<��4*H�'W6��ٖ-B~�e�1>вE�o�l1����Q�F�~�e2~�E	s��H4O�S��<'�(s$�,���.��H7u����F~pfK$�C�a�*�����;.>s+1�`nb4��g
    =b�ċ������'��K�I4�p"���FX0>��?��Ű�y.�N��l���3�d�2L���q��SZ��`g��A}<gA}Ę	�G�A7�"��h"��pd8@;� ��(���.����޺���
    #y45l�Dj(]4
    f�&������\�cfP����.�r�w���kr�7��	��},+�ݛ?3-&�8�A�__��3���T���x&�6yR���Fyl���Q6���!��Wy>l�l�z��|��I�6��H9R�KU��$Ĩ���qYBi�9����>E�����%ouo���1Υ�i�3L^��,�}v���L�q���^��
    �g�o!T���k��"�U��u�
    ^�df~n��t��2��j��9xA�D��W;xR��TO�c�~���U������~{G�O����o3&O�'���f�v�@�U�����"���6��g�R�6|e��2l�Xdܣ�K����YDV_�9x���W�����16�B^����#��񷍃�i�5ba���K��:,l�~�?m�}i�C��o7g�Ǩu����+3���uh����p^A��[���o�5g���PK�KsQ�����*goose_user_space_in_idk/goose_user_space.c�X{o�0��|
    �=�t-[�&�����e�($N�F��������	�Z��5��������g��o�g/�pn�<����KE�g��X�<ל[����(v�`�z���̛�Ѯ����l����.0b��!��S��J��<p�ll�w�}?������ʃ���xtl����^�tT��������bo�w2�|��ţ�H0�j���,��;����^<5B��[�m�.E_�F��˶��]�Q�s#�0��E<c�mbu��ȱ���w<WIi�n�F�0&���c]�aTqȺ	ĴI�NjbF���4���̇zS���EH �-V}X�Ǫ��Z7�2fu�Y���g4�b$8`˯����#��4hl�E�u��zgE1k����Aߤ;�]A!�`p�?<>Dx��҉m;��E�뒲� ���A��D����}�H��N�F�	w�j:�L�y���`�I��¶$t�Z-��-��[����!\��wl%#�k��Hj�{Q{�$�@�c��KC�.�H�Z����D�F����_'ɢ�۹j'�3e��s�׫���r���K�-�e9NH��&�{Exoo�w����-(�I�?=ۀX�R'�5���T�AK���,8����������������\�w%��سY�9Y�.�p��h���<?f���b�gvEpn���ױ����x�v��l��0�n�;f��0����y�ܲ��xiθWY=�,�X��p��v�<����YGx�uߝ���+%��uez���8����8��%�c�	�
    ���f���U|���b���m�lD�lϯT�����k ��|1tz`%Q���$):��ɮes�HH����TL{�au
    H�Y�5(5�X�e�J)j�A�Yq���m��r	j���/�.������,�E��)���3��a�;�G,��a��G�@�B����_N<�ǹ&��n�����Q�[-�s~c�MJO�ͽ�ߨ�M�Ng8��r���l
    ,�X@�:�]�*��ӊJI�V�I���y��{�U��p�	�*l~���ylvM��]����6`����S����*+�9u�,D�5�������r
    _�K�8��*�M�s~���E�0蚑�����皖����š��Pf7m*`�n�����1������ާ
    Kk�Tކ�y^pv��w�=��?����
    ���뛏����
    K4��dFoSwO�Eho�'hZ������wT�Y;���T#�t�>R������~����E/<;m�k�H5��L���a�Rb����qOo6��$�����~Q��8�o�}S�W��H��Ċ����b?�o�(=���du��Soƙ�B�p��+�\g����(՚�vHoi��Nd���\�4q`��@НE4�|��5TX�8eJ�\a��=}
    ��3%��b��[ߡy��5����'[��:�krK���S�L�|��D�[H�I��n,�QI��7u}�K�ƃr�R0����a�o���<F��އ�^rpx���ࢵ셈�_M�S31�U����3y�}h�ܤߧU�"��|���p��<�B�A3�3�(������BH�*A,��;m}���i��ǏY~�ApH!�l�'�jY�<��@;�k��*��g_����ۆ1���)�ߍ�9�k;�?�_�`GͬDO�R*M��g;�Y�0{V�zqj�t%O~�+��P�����S��iʍ]�Z�PK�KsQ)Ǖ�t� goose_user_space_in_idk/MakefileU˱
    � ����A܅NRJ��@.r��1�D���2��g�7��?�7��B��jLE!�2�Ҿ2�F8��$���9_��!@?ػ[c\�J�Ub_?s��@ȄEÉg!'��PK?LsQ$goose_user_space_in_idk/
     ��y�N����y�N��1Z�CI��PK?�KsQ P;��h#$ 6goose_user_space_in_idk/dual_idk.sh
     ��SN��{ng�N��KزCI��PK?�KsQ�}�	[�1($ Ogoose_user_space_in_idk/goose_user_space
     ��SN����p�N��*	�CI��PK?�KsQ�����*$ �goose_user_space_in_idk/goose_user_space.c
     ��SN���w�N���|�CI��PK?�KsQ)Ǖ�t� $ !goose_user_space_in_idk/Makefile
     ��SN����z�N��+��CI��PKG�

  • Hi Billa,

    Thanks for posting these details.  I'm working with the AM57xx support team to reproduce this issue and understand what's happening.

    We should have more feedback in the coming days.

    Brad

  • Hi Billa,

    I have reproduced your issue on my AM574x IDK using Linux RT SDK 6.03.  I setup both my Linux PC and my IDK with static IP addresses so that I could connect them directly together.  I ran tcpdump from the Linux PC and then started your application on the IDK:

    ./goose_user_space eth2

    After unplugging/plugging the cable several iterations the PC stopped outputting GOOSE packets.  At that point it was hard to do anything on my board, i.e. even Ctrl-C didn't want to quit your program.  Through some combo of Ctrl-C and unplugging/plugging the cable it finally quit.  I could see at that point that ethernet communications weren't functioning at all, e.g. ping doesn't work either.

    Performing "ifconfig eth2 down" and "ifconfig eth2 up" seemed to clear the issue.  The following messages were printed during the "down" and look interesting:

    [ 3418.472865] prueth pruss2_eth eth2: Link is Down
    [ 3418.483265] pruss 4b280000.pruss: unconfigured system_events[63-0] = 00000600,04500000
    [ 3418.491223] pruss 4b280000.pruss: unconfigured host_intr = 0x00000155
    [ 3418.499347] remoteproc remoteproc6: stopped remote processor 4b2b4000.pru
    [ 3418.509484] net eth2: stopped

    I see your logs in this other thread have these exact same bits set:

    https://e2e.ti.com/support/processors/f/791/t/953711

    That seems to be consistent. I will ask the firmware team to comment on those bits.  Also, I plan to keep experimenting with this setup a bit more to better understand the issue.  For example, I want to see if there's a relationship to the goose packets specifically or if I can see similar behavior when sending other packets (e.g. ping, iperf tcp, iperf udp, etc.

    Best regards,
    Brad

  • Quick update...  I think I've found an easier way to reproduce the issue:

    • Linux PC: iperf3 -s
    • IDK: iperf3 -t 30 -c <linux-pc>

    It appeared to get into the bad state the very first time I unplugged/plugged the cable.  I ran a similar test on CPSW ports and it recovered ok.

    Best regards,
    Brad

  • I've also confirmed there's no requirement to be directly connected.  In short:

    1. Start with Linux RT SDK Binaries

    2. Can hook up IDK to a switch using DHCP (i.e. the most typical setup).  Not required to directly connect to PC or use static IP address.

    3. Run iperf3 server on PC; iperf3 client from IDK.

    4. Unplug IDK cable during test, wait a couple seconds, plug it back in.  Seems to completely lockup the interface.

    Best regards,
    Brad

  • Hi,

    Sorry for the delay, I am looking at reproducing and investigating this issue next week.

    Regards

    Vineet

  • Vineet -- one of Billa's co-workers proposed the following firmware patch which has resolved the problem.  He made an adjustment to ti/drv/icss_emac/firmware/icss_dualemac/src/emac_MII_Xmt.asm :

     900 XMT_LB_1OOMbps_MODE:

     901     ; Insert the CRC in the outgoing frame by writing to R30

     902     .if $defined("ICSS_REV1")

     903     M_XMT_INSERT_CRC_ICSS_REV1

     904     .endif      ;ICSS_REV1

     905     .if $defined("ICSS_REV2")

     906     M_XMT_INSERT_CRC_ICSS_REV2

     907     .endif      ;ICSS_REV2

     908

     909     ; Don't allow insertion of next packet in TX Fifo unless it is empty

     910     CLR R22 , R22 , PACKET_TX_ALLOWED

     911     CLR R22 , R22 , Entire_Tx_Data_Not_Pushed

     912

     913     ; Fix issue where PRU MII TX FIFO goes into overflow condition and not able to recover.

     914     ; This issue is caused by clearing of PACKET_TX_ALLOWED in R22 above, coinciding with

     915     ; MII TX FIFO overflow due to sudden loss of link. That causes MII_TX_TASK not be invoked ever.

     916     ; Use existing macro M_XMT_FILL_LEVEL_CALC_ICSS_REV2 that handles MII TX FIFO overflow by

     917     ; M_TX_RESET. Also, SET PACKET_TX_ALLOWED in R22 in error_in_fill_level.

     918     .if $defined("ICSS_REV1")

     919     M_XMT_FILL_LEVEL_CALC_ICSS_REV1     ; calculate the current TX fifo fill level

     920     .endif      ;ICSS_REV1

     921

     922     .if $defined("ICSS_REV2")

     923     M_XMT_FILL_LEVEL_CALC_ICSS_REV2     ; calculate the current TX fifo fill level

     924     .endif      ;ICSS_REV2

     925

     926 xmt_save_context:

    The issue is being tracked inside TI as LCPD-19899 and I have requested that we pull the same patch into the official firmware branch.

  • Thanks Brad and to Customer,

    I have filed a task PINDSW-4691 to track migration of this fix.

    I will now close this E2E thread.

    Regards

    Vineet