[Winpcap-users] Using too many pcap_t handles causes errors?

Gianluca Varenni gianluca.varenni at cacetech.com
Fri Dec 12 16:29:21 GMT 2008


  ----- Original Message ----- 
  From: Hakan Uluoz 
  To: winpcap-users at winpcap.org 
  Sent: Thursday, December 11, 2008 11:12 PM
  Subject: [Winpcap-users] Using too many pcap_t handles causes errors?


  Hi ,

  Gianluca thanks for the info and suggestions, as far as I understand the non-paged kernel pool is the source of my problems. Yes, at first stage I thought a master thread for capturing and workers for processing, but hence this application is realtime ( or let's say near-realtime ) the results ( at least on win systems ) were worse ( though I did not make a complete protocol etc analysis on the packets ) as analysis results were much sooner than the events. As I said I have no trouble with the processor power so running all analysis threads paralely gives me the results as the expected events occur.

   Tried decreasing the mem usage with pcap_setbuff but I did not see any +/- ( honestly I did not test this issue completely , I promıse I'll test it by lowering to 256 KB which I think will be more then enough since RTP packets I am collecting is around 60-70 bytes including protocol overhead). 

  Finally, as you suggested I decided to swap to a 64 bit OS, paralelly porting the applications to linux. Natively 64 bit Win OSes do not have the strange 256 MB kernel pool limit, AFAIK the limit is far beyond GBs. One point I need to know is results of using an 32bit dll or application on a 64 bit os( I cannot find the wpcap.dll or libs 64 bit version ). If you are experienced on this topic, will "using 64 bit os and running same applications on it ( or converting the applications to 64 bit but the wpcap.dll  will be still 32 bit )" be enough to avoid the kernel limitation, or still the WoW64 emulation limits you to 256 MB pool? If second is the reality, then compiling a 64 bit version of the pcap lib and/or dlls will be the only solution.  

When you work on a 64bit OS (I mean x64/EM64T), all the kernel components must be 64bit. But the user level components can be 32- or 64bit. Even if the application is 32bit, the WinPcap driver is 64bit, so there is no 256MB limitation on the nonpaged pool.

Regarding a 64bit version of the user level components, there's an experimental build of them here

http://www.winpcap.org/install/x64.htm

and I plan to release a new 4.1beta version by the end of the year that includes both 32- and 64- user level components (and associated lib files).

Have a nice day
GV


  My best regards and again thanks for the suggestions.

  Hakan.


  Hi,

    I am building a Win 32 application pair to monitor the SIP flow on a gateway. Basically the machine running the monitor applications is connected to a hub with the gateway. Main application monitors the SIP flow and runs sub-applications according to the SIP informations. Sub-applications monitor the RTP flows on forward and reverse directions. So all sub-applications have 2 pcap_t handles plus the main has 1. All have unique filters on the same adapter. Yes, the sub-applications are better be threads but there are some other restrictions irrelevant to wpcap that forces me to make them as applications.

    The machine runs on Win XP 32 bit with 2 GB of ram, with all unnecessary services removed. Applications run smooth on around 60 channels ( that makes 60*2 + 1 => 121 handles ). But when the channel count exceeds this boundry, findalldevs and open_live starts to fail. The errors are generally :

    Error in pcap_findalldevs: PacketGetAdapterNames: ERROR_INSUFFICIENT_BUFFER(122)

    Unable to open the adapter. <adapter> is not supported by WinPcap - errbuf :(NULL)

    Unable to open the adapter. <adapter> is not supported by WinPcap - errbuf : Cannot determine the network type(0)

    And mostly :
    Unable to open the adapter. <adapter> is not supported by WinPcap - errbuf : driver error: not enough memory to allocate the kernel buffer  
     
    First observations showed that when the total memory consumptions exceed 1GB errors start. Thought to be a paging issue and disabled paging but did not give a cure. As there is around 1 GB free physical ram available, I focussed on the kernel memory usage, and found that the errors start as soon as the non-paged kernel memory usage reaches around 256 MB ( which is the limit for all Win 32 versions AFAIK ). Besides the CPU usage is very acceptable under all circumstances.
     
    Can anyone clarify the reason for these errors I am getting? Is it memory, handle limitations? And I'd be thankful on suggestions on a solution. I already admit that everything has a limit but a way to tripple or double the channel count ( e.g. monitoring 180-120 channels ) would be quite useful.

  The problem is non paged pool exaustion: every capture instance by default uses a 1MB kernel buffer that is allocated from the non paged pool, plus some kB for the internal structures (I don't remember exactly how many). 
  Several solutions come to my mind:
  - after you open the adapter with pcap_open_live, set the kernel buffer size to a smaller one (with pcap_setbuff).
  - redesign your application so that you open less pcap_t handles and then dispatch the packets to different threads
  - use a 64bit machine.

  Have a nice day
  GV

     
    As a note I am using the WinPcap 4.0.2. 
     
    My best regards,
     
    Hakan.





------------------------------------------------------------------------------


  _______________________________________________
  Winpcap-users mailing list
  Winpcap-users at winpcap.org
  https://www.winpcap.org/mailman/listinfo/winpcap-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.winpcap.org/pipermail/winpcap-users/attachments/20081212/e05037f4/attachment.htm


More information about the Winpcap-users mailing list