[Winpcap-users] Using too many pcap_t handles causes errors?

[Hakan Uluoz]

Hi ,

Gianluca thanks for the info and suggestions, as far as I understand the non-paged kernel pool is the source of my problems. Yes, at first stage I thought a master thread for capturing and workers for processing, but hence this application is realtime ( or let's say near-realtime ) the results ( at least on win systems ) were worse ( though I did not make a complete protocol etc analysis on the packets ) as analysis results were much sooner than the events. As I said I have no trouble with the processor power so running all analysis threads paralely gives me the results as the expected events occur.

 Tried decreasing the mem usage with pcap_setbuff but I did not see any +/- ( honestly I did not test this issue completely , I promıse I'll test it by lowering to 256 KB which I think will be more then enough since RTP packets I am collecting is around 60-70 bytes including protocol overhead). 

Finally, as you suggested I decided to swap to a 64 bit OS, paralelly porting the applications to linux. Natively 64 bit Win OSes do not have the strange 256 MB kernel pool limit, AFAIK the limit is far beyond GBs. One point I need to know is results of using an 32bit dll or application on a 64 bit os( I cannot find the wpcap.dll or libs 64 bit version ). If you are experienced on this topic, will "using 64 bit os and running same applications on it ( or converting the applications to 64 bit but the wpcap.dll  will be still 32 bit )" be enough to avoid the kernel limitation, or still the WoW64 emulation limits you to 256 MB pool? If second is the reality, then compiling a 64 bit version of the pcap lib and/or dlls will be the only solution.  

My best regards and again thanks for the suggestions.

Hakan.


Hi,

  I am building a Win 32 application pair to monitor the SIP flow on a gateway. Basically the machine running the monitor applications is connected to a hub with the gateway. Main application monitors the SIP flow and runs sub-applications according to the SIP informations. Sub-applications monitor the RTP flows on forward and reverse directions. So all sub-applications have 2 pcap_t handles plus the main has 1. All have unique filters on the same adapter. Yes, the sub-applications are better be threads but there are some other restrictions irrelevant to wpcap that forces me to make them as applications.

  The machine runs on Win XP 32 bit with 2 GB of ram, with all unnecessary services removed. Applications run smooth on around 60 channels ( that makes 60*2 + 1 => 121 handles ). But when the channel count exceeds this boundry, findalldevs and open_live starts to fail. The errors are generally :

  Error in pcap_findalldevs: PacketGetAdapterNames: ERROR_INSUFFICIENT_BUFFER(122)

  Unable to open the adapter. <adapter> is not supported by WinPcap - errbuf :(NULL)

  Unable to open the adapter. <adapter> is not supported by WinPcap - errbuf : Cannot determine the network type(0)

  And mostly :
  Unable to open the adapter. <adapter> is not supported by WinPcap - errbuf : driver error: not enough memory to allocate the kernel buffer  
   
  First observations showed that when the total memory consumptions exceed 1GB errors start. Thought to be a paging issue and disabled paging but did not give a cure. As there is around 1 GB free physical ram available, I focussed on the kernel memory usage, and found that the errors start as soon as the non-paged kernel memory usage reaches around 256 MB ( which is the limit for all Win 32 versions AFAIK ). Besides the CPU usage is very acceptable under all circumstances.
   
  Can anyone clarify the reason for these errors I am getting? Is it memory, handle limitations? And I'd be thankful on suggestions on a solution. I already admit that everything has a limit but a way to tripple or double the channel count ( e.g. monitoring 180-120 channels ) would be quite useful.

The problem is non paged pool exaustion: every capture instance by default uses a 1MB kernel buffer that is allocated from the non paged pool, plus some kB for the internal structures (I don't remember exactly how many). 
Several solutions come to my mind:
- after you open the adapter with pcap_open_live, set the kernel buffer size to a smaller one (with pcap_setbuff).
- redesign your application so that you open less pcap_t handles and then dispatch the packets to different threads
- use a 64bit machine.

Have a nice day
GV

   
  As a note I am using the WinPcap 4.0.2. 
   
  My best regards,
   
  Hakan.



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.winpcap.org/pipermail/winpcap-users/attachments/20081212/06a69d2e/attachment.htm