[ECOS] TCP MBUF loss & crash

["Jon Hartley" <jdh at cea dot com dot au>]

Hi,
we are experiencing MBUF losses using stream sockets with bursty data (about
40 msgs once per sec).
eventually, after a few minutes, all mbufs are used up and we crash.
The problem seems to arise when more than about 209 mbufs are used, the size
of the mbuf pool seems not to matter.
After traffic ceases the number of used mbufs never goes back to zero, even
after 3 minutes.

BOARD: Custom, using mpc860 and QUICC ethernet.
ECOS VERSION: Anonymous, taken   on 14 Feb 2001
     updated: packages/redboot   on 15 June 2001,
		  packages/net       on 18 June 2001,
		  packages/devs/eth  on 18 June 2001,
		  packages/io/eth 18 on 18 June 2001.

The following program demonstrates this problem:

//-------------------------------------------->

#include <cyg/kernel/kapi.h>
#include <network.h>
#include "stdio.h"

#define TCP_NODELAY  0
#define TCP_LINGER
#define NUM_DATA 500

extern void cyg_kmem_print_stats( void );
extern int cyg_kmem_get_mbufs(void);      //added to support.c
extern int cyg_kmem_get_maxmbufs( void ); //added to support.c, returns max
load ever on mbufs

void TestEther()
{
   char data[1000];
   struct sockaddr_in serv_addr;
   int sockfd,count = 0;
   struct sockaddr_in cli_addr;
   int write_sockfd, clilen;

   //cyg_kmem_print_stats();

   sockfd = socket(AF_INET, SOCK_STREAM, 0);

   memset((char *) &serv_addr, 0, sizeof(serv_addr));
   serv_addr.sin_family = AF_INET;
   serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
   serv_addr.sin_port = htons(9001);
   serv_addr.sin_len = sizeof(serv_addr);

   bind(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr));
   listen(sockfd, 1);

   clilen = sizeof(cli_addr);
   write_sockfd = (int) accept((int) sockfd, (struct sockaddr *)&cli_addr,
(int *)&clilen);

   while (count < NUM_DATA)
      {
      int i,len1 = 0,len = 16;
      for (i = 0; i < len; i++)
         data[i] = i;

      if ((len1 = send(write_sockfd,data,len , 0)) != len)
         {
         printf("Send failed: %d of %d\n",len1,len);
         break;

         }

      count++;
      }

   cyg_thread_delay(3000);

   close(write_sockfd);
   close(sockfd);

   printf("sent: %d mbufs: %d max in use:
%d\n",count,cyg_kmem_get_mbufs(),cyg_kmem_get_maxmbufs());
   cyg_kmem_print_stats();

   cyg_thread_delay(200000);
   printf("mbufs: %d\n",cyg_kmem_get_mbufs());
   cyg_kmem_print_stats();



}
//--------------------------------------------------------
============================================================================
===
first we ttry a few tests with a memory pool of 127 mbufs
CYGPKG_NET_MEM_USAGE = 65536.
============================================================================
===

when NUM_DATA is 1000:
------------------------------------------------

sent: 1000 mbufs: 3 max in use: 126
...
mbufs: 0
Network stack mbuf stats:
   mbufs 0, clusters 4, free clusters 4
   Failed to get 0 times
   Waited to get 0 times
   Drained queues to get 0 times
Misc mpool: total   16368, free   13408, max free block 12432
Mbufs pool: total   16256, free   16128, blocksize  128
Clust pool: total   32768, free   22528, blocksize 2048


still fine but when NUM_DATA is upped to 1500:
---------------------------------------------
sent: 1500 mbufs: 11 max in use: 126
Remote communication error: Connection reset by peer.

Apart from the crash which happens only in an mbuf loss situation,
a loss of 8 mbufs is indicated,
allowing 3 for the close which delays about 2 minutes.

============================================================================
===
we now increase the memory pool to 511 mbufs CYGPKG_NET_MEM_USAGE = 262144.
============================================================================
===

				    NUM_DATA is 1050:
--------------------------------------------
sent: 1050 mbufs: 11 max in use: 217
Remote communication error: Connection reset by peer.

Same problem even with more available mbufs!


when NUM_DATA is 1020:
---------------------------------------------
sent: 1020 mbufs: 11 max in use: 212
Network stack mbuf stats:
   mbufs 10, clusters 8, free clusters 3
   Failed to get 0 times
   Waited to get 0 times
   Drained queues to get 0 times
Misc mpool: total   65520, free   61808, max free block 60304
Mbufs pool: total   65408, free   64000, blocksize  128
Clust pool: total  131072, free  112640, blocksize 2048
mbufs: 8
Network stack mbuf stats:
   mbufs 8, clusters 8, free clusters 4
   Failed to get 0 times
   Waited to get 0 times
   Drained queues to get 0 times
Misc mpool: total   65520, free   62560, max free block 61584
Mbufs pool: total   65408, free   64256, blocksize  128
Clust pool: total  131072, free  112640, blocksize 2048


So, no crash this time but still lost 8 mbufs

when NUM_DATA is 1015:
---------------------------------------------

sent: 1015 mbufs: 3 max in use: 209
Network stack mbuf stats:
   mbufs 2, clusters 5, free clusters 4
   Failed to get 0 times
   Waited to get 0 times
   Drained queues to get 0 times
Misc mpool: total   65520, free   61808, max free block 60304
Mbufs pool: total   65408, free   65024, blocksize  128
Clust pool: total  131072, free  118784, blocksize 2048

<CRASH>