FreeRTOS+TCP does not connect to network anymore after a few days

Hello, I have great trouble with an FreeRTOS+TCP application I’m developing. It all worked fine and I communicated with the board over TCP untill suddenly no communication is possible anymore. I cannot connect to the board and not even ping to the board. Then I get: Destination host unreachable. Some background info: Hardware platform: Xilinx Zynq on a Trenz TE0720 module Software: Xilinx SDK 2016.2 Network: Switch with router, pc and board connected to it Settings: Static IP The first time I developed the application making use of the TCP stack for weeks when it suddenly didn’t work anymore, changing nothing on the TCP settings. Then I built a new application from scratch to see what was going wrong, which was working fine again, so I imported my project and continued developing. Now after 4 days developing and testing, I just wanted to conduct a new test, it again cannot be connected to anymore, and the ping returns Destination host unreachable. After IP_Init I set the config to call vApplicationNetworkHook if all is set. Strange thing is, this is still called, only no connection can be made. Can someone please give some direction to look in for? Something on the Zynq itself? Something on the network? Best regards, Giovanni

FreeRTOS+TCP does not connect to network anymore after a few days

Hi Giovanni, Lately there were a few changes to the network-interface software for Zynq. I’m not sure which release you’re using now, but you might want to compare and try the new version that I attached. This is a non-official release, just for testing.

FreeRTOS+TCP does not connect to network anymore after a few days

Hello Hein, Thank you for your answer. It seems the files I have are exactly the same as the files you sent. So unfortunately no progress there. Thanks again, if you have more tips, please share! Best regards, Giovanni

FreeRTOS+TCP does not connect to network anymore after a few days

Sorry, my stupid. The latest /labs release does contain the latest network interface driver for Zynq.
Then I get: Destination host unreachable
In those cases, does it help to do a cold start of the device?
Something on the Zynq itself?
I’m afraid I’m not an expert on that
After IP_Init I set the config to call vApplicationNetworkHook if all is set. Strange thing is, this is still called, only no connection can be made
At first I thought of network buffers being exhausted. But FreeRTOS_IPInit() is being called after a reboot and everything should be OK. Is your device OK? Do you have a second one to try? What about the LAN? Do you use a switch between laptop and device? Can other devices still be reached? Regards, Hein

FreeRTOS+TCP does not connect to network anymore after a few days

Hello Hein,
  • No problem.
  • Every time I conduct a session I have to do a cold restart, or else SDK will not execute the session properly.
  • Indeed, IPInit() is only called after a reboot / restart / cold restart.
  • I have a second device, which does exactly the same.
  • I use a switch between pc and device and the rest of the network stays working as it did before. Also on Wireshark no messages with the devices IP address come through. Restart of the pc / switch or whatsoever make no difference.
For now, I’m able to keep on developing with an older version (from last friday), untill that one will stop working.. I am starting to get clueless.. Thanks again! Best regards, Giovanni

FreeRTOS+TCP does not connect to network anymore after a few days

Hi Giovanni, it is very difficult to get insight into the problem you encounter. Do understand correctly that the symptom is that your device becomes unreachable? It doesn’t respond to any of the network protocols? But can you confirm that the CPU is still running? Are all tasks still running? Maybe you can check that with LED’s, or a serial connection, or maybe in an Eclipse debug session? The TCP/IP stack has been out for a long time and users consider is stable. You’ll have to investigate. It is important to keep on checking the amount of available resources: stack per task, free memory for pvPortMalloc(), the number of free network buffers. And of course, if an exception occurs: let the device make it clear in some way. Usually an application will end in a for(;;) loop, becoming unreachable. Now about the reboots/resets: we should be very clear: what do you do exactly to get your Zynq to life again?

FreeRTOS+TCP does not connect to network anymore after a few days

Hello Hein, I was hoping someone had this problem before and had a direction to search in. CPU is still running, I was testing hardware, which had the highest priority. So I laid down the Ethernet problem and made another thread with a terminal application to test the hardware. Everything works as expected, apart from being reachable from the network. Also the network task is still running, but the listening function has no use, if the device is unreachable. When it still worked, I could ping and connect to the devices after IPInit(). Also DHCP worked and I could track down the bootp messages through Wireshark. After the fault occured not anymore. I’m afraid I have to check resources in memory etc, what you are saying, not my expertise. The application doesn’t end in a endless loop, because it still calls the vApplicationNetworkHook in which I can still ‘do things’.
Now about the reboots/resets: we should be very clear: what do you do exactly to get your Zynq to life again?
1- Shutdown the voltage of the board and power it back up 2- Disconnect and delete the debug session in SDK 3- Start a new debug session in SDK This is the only way to start a proper debug session, else the application will not start properly and will hang somewhere during initialization of the processor. Not that strange, have noticed it with other processors also. Thank you again for your effort!

FreeRTOS+TCP does not connect to network anymore after a few days

If you look with WireShark, you will see that the device sends so-called gratuitous ARP-packets, for example: ~~~ 2 3.508662 Microchi_d4:6a:53 Broadcast ARP 60 Gratuitous ARP for 192.168.2.201 (Request) ~~~ Can you see these ARP-packets? And when the device becomes “unreachable”, do you still see them? About checking resources. In many versions of NetworkInterface.c you will see the following ‘checking’ code: ~~~ /* Define in you FreeRTOSIPConfig.h : */

define ipconfigCHECKIPQUEUE_SPACE 1

/* Add to your prvEMACHandlerTask() : */ static void prvEMACHandlerTask( void *pvParameters ) { UBaseType_t uxLastMinBufferCount = 0; UBaseType_t uxLastMinQueueSpace = 0; UBaseType_t uxLastFreeHeapSpace = 0; UBaseType_t uxCurrentCount;
for( ;; )
{
    uxCurrentCount = uxGetMinimumFreeNetworkBuffers();
    if( uxLastMinBufferCount != uxCurrentCount )
    {
        /* The logging produced below may be helpful
        while tuning +TCP: see how many buffers are in use. */
        uxLastMinBufferCount = uxCurrentCount;
        FreeRTOS_printf( ( "Network buffers: %lu lowest %lun",
            uxGetNumberOfFreeNetworkBuffers(), uxCurrentCount ) );
    }

    #if( ipconfigCHECK_IP_QUEUE_SPACE != 0 )
    {
        uxCurrentCount = uxGetMinimumIPQueueSpace();
        if( uxLastMinQueueSpace != uxCurrentCount )
        {
            /* The logging produced below may be helpful
            while tuning +TCP: see how many buffers are in use. */
            uxLastMinQueueSpace = uxCurrentCount;
            FreeRTOS_printf( ( "Queue space: lowest %lun", uxCurrentCount ) );
        }
    }
    #endif /* ipconfigCHECK_IP_QUEUE_SPACE */
    {
        uxCurrentCount = xPortGetMinimumEverFreeHeapSize();
        if( uxLastFreeHeapSpace != uxCurrentCount )
        {
            /* The logging produced below may be helpful
            while tuning +TCP: see how many buffers are in use. */
            uxLastFreeHeapSpace = uxCurrentCount;
            FreeRTOS_printf( ( "Heap: lowest %lun", uxCurrentCount ) );
        }
    }
    ...
}
} ~~~ It will send out logging like “Network buffers: 24 lowest 20”. Make sure that “lowest” never gets close to zero. The IP-task has a message queue. You will see logging like “Queue space: lowest 19”. This should remain far above zero. Also the lowest amount of free heap will be logged as e.g. : “Heap: lowest 84904” Can you make sure that this logging is sent to a terminal so you never miss it?
1- Shutdown the voltage of the board and power it back up
Is this power-cycle really necessary? Or would steps 2) en 3) be enough? And a final request: could you attach the FreeRTOSIPConfig.h file that you use? Regards.

FreeRTOS+TCP does not connect to network anymore after a few days

Hello Hein, Thanks again! About ARP: When the device is reachable I can see the gratuitous ARP-packets. When the device is not reachable, I cannot see them. So it seems there is nothing coming out of the device (no arp request or something like it). I’m going to check that on hardware level. About logging it gives the following: Network buffers: 64 lowest 64 (and later on it prints a second time and then lowest is 63) Queue Space: lowest 101 Heap lowest: 41820000 (later on, the second time it prints 4179032) If I understand you correctly no problems here. About power cycling: On my development board it was really needed, but now I tried with a full reset en it is not needed anymore. Maybe something on the development board with reset of power. Thanks for the note. Attached: FreeRTOSIPConfig.h Best regards

FreeRTOS+TCP does not connect to network anymore after a few days

Giovanni, Ref your FreeRTOSIPCOnfig.h : I would definitely define: ~~~ /* Use a zero-copy EMAC driver in both directions. */ #define ipconfigZERO_COPY_TX_DRIVER 1 #define ipconfigZERO_COPY_RX_DRIVER 1 ~~~ Although the latter ipconfigZERO_COPY_RX_DRIVER is never mentioned for in the Zynq driver. The Zynq Ethernet driver was a bit difficult to develop because of the cached memories. DMA see the physical memory, while the CPU reads and writes to memory caches. We’ve decided to reserve 1 MB of non-cached memory and dedicate that to the +TCP and +FAT drivers. It will be accessed by both DMA’s. The ipconfigZERO_COPY_TX_DRIVER variant of the Zynq driver was tested better than the memcpy version. Did you have a particular reason to define ipconfigZERO_COPY_TX_DRIVER as 1 ?

FreeRTOS+TCP does not connect to network anymore after a few days

Hein, Unfortunately it did not make a difference. And in your last sentence you meant a reason for being 0 instead of 1. Because it is (was) defined 0. No I did not had a reason and cannot remember changing that definition. But I did some comparing (again and again) and I now got it back working. Not sure yet if it will stay working. In the IP settings nothing changed, also not in software. So I went back to the hardware definition in Xilinx Vivado and there was a clock setting for Ethernet changed. Of which I’m sure the firmware guy nor I have changed it. So I want to investigate if this is a bug in Vivado. This setting influences the Vivado / SDK automatically generated driver causing to still properly ‘set’ the driver and the IP stack run through its initialization functions. But physically there was no connection to the network due to a clock not properly set or synchronized to. As I said, I’m going to investigate it. Though I’ve learned that the ZEROCOPYDRIVER suits better vor Zynq applications, so I will use that. Thank you so much for helping me, I really appriciate it!

FreeRTOS+TCP does not connect to network anymore after a few days

Glad to hear.
As I said, I’m going to investigate it.
When you’ve found it, please report about it so you can help others.
This setting influences the Vivado / SDK automatically generated driver causing to still properly ‘set’ the driver
When there is time, you may want to get rid of the automatic things of Eclipse/SDK and start using a Makefile project. It gives a very precise control of what happens during the make process. Regards, Hein