[Winpcap-users] Better Performance

Charu Agrawal cagrawal at altasens.com
Mon Apr 27 17:14:23 PDT 2009


BroadCom Gig Ethernet.

I tried with kernel buffer set to both 40MB and 16MB.

Thanks

-----Original Message-----
From: winpcap-users-bounces at winpcap.org [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of Gianluca Varenni
Sent: Monday, April 27, 2009 4:17 PM
To: winpcap-users at winpcap.org
Subject: Re: [Winpcap-users] Better Performance


----- Original Message ----- 
From: "Charu Agrawal" <cagrawal at altasens.com>
To: <winpcap-users at winpcap.org>
Sent: Monday, April 27, 2009 3:58 PM
Subject: Re: [Winpcap-users] Better Performance


> Hello David,
>
> I have 2Gig RAM on my machine and speed around 2.67GHz.
> Even if I remove everything in the packet_handler except packetCounter++, 
> I get bad performance. So I doubt it has anything to do with the code
> Inside packet handler.
> I changed the kernel buffer too.
>
> I was wondering if there is anything else that effects performance ?Are we 
> usually able to get 30Mbytes/s frame rate with winpcap ? or is it just my 
> workstation?

What network card are you using?
What is the kernel buffer size that you are using?


Have a nice day
GV

>
> Thanks
> Charu
>
> -----Original Message-----
> From: winpcap-users-bounces at winpcap.org 
> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
> Sent: Friday, April 24, 2009 1:08 PM
> To: winpcap-users at winpcap.org
> Subject: Re: [Winpcap-users] Better Performance
>
> Charu,
>
> Can you give me your hardware specs?  Also, what was your throughput
> with the empty callback function?
>
> Let's do another test...
>
> u_char dummy[2000]; /* I assume your packets (and snaplen) are less than 
> 1500 bytes long */
>
> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr 
> *header, const u_char *pkt_data)
> {
> if (header->len < sizeof(dummy)+sizeof(struct pcap_pkthdr))
> {
> memcpy(dummy, header, sizeof(struct pcap_pkthdr));
> memcpy(dummy+sizeof(struct pcap_pkthdr), pkt_data, header->len);
> }
>
> packetCounter++;
> }
>
> How is the throughput with just the memcpy() overwriting a dummy buffer?
>
> Lastly, I would use MUCH less kernel buffer space (40MB is too much).  I
> would use 8MB - 16MB.  For example:
>
> if (pcap_setbuff(adhandle,16777216) < 0)
>
>
> DC
>
> Gianluca Varenni wrote:
>> ----- Original Message -----
>> From: "Charu Agrawal" <cagrawal at altasens.com>
>> To: <winpcap-users at winpcap.org>
>> Sent: Friday, April 24, 2009 10:01 AM
>> Subject: Re: [Winpcap-users] Better Performance
>>
>>
>>
>>> Hello David,
>>>
>>> If I change the code as you suggested, I do get a performance
>>> improvement. I have now allocated memory outside the loop.
>>> But I still need to use the memcpy function to store all packets in an
>>> array. That still slows it down.
>>>
>>> Anything else I can do to improve performance ? Is there a way to change
>>> the winpcap source code for better performance ?
>>>
>>>
>>
>> WinPcap has been highly optimized over the years for performance. Before
>> looking into optimizing the WinPcap code, I would understand exactly what
>> the bottleneck is.
>> You say that you are copying packets into your allocated memory. Why 
>> can't
>> you process the packets inline, i.e. when you receive them? Also, how 
>> much
>> memory did you allocate for the packets? Preallocating several megabytes
>> ahead of time for the packets can be as dangerous as malloc'ing for every
>> packet. This is because allocating memory just means that you have 
>> reserved
>> a large chunk of address space in your process, not that you have 
>> physically
>> reserved all those megabytes in the physical RAM (i.e. not all that 
>> memory
>> is your working set).
>>
>> Hope it helps
>> GV
>>
>>
>>
>>> Thanks
>>> Charu
>>>
>>> -----Original Message-----
>>> From: winpcap-users-bounces at winpcap.org
>>> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
>>> Sent: Thursday, April 23, 2009 7:03 PM
>>> To: winpcap-users at winpcap.org
>>> Subject: Re: [Winpcap-users] Better Performance
>>>
>>> Charu,
>>>
>>> malloc() is VERY expensive; especially for larger buffers. Are you
>>> saying that if you change the callback function to this...
>>>
>>> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr
>>> *header, const u_char *pkt_data)
>>> {
>>> packetCounter++;
>>> }
>>>
>>> you still can't read 20,000 packets / sec? (I assume each packet is
>>> full, so this would be close to 30MB/sec)
>>>
>>> What type of hardware (CPU, RAM, NIC, motherboard, etc) and O.S. do you
>>> use?
>>>
>>> DC
>>>
>>> Charu Agrawal wrote:
>>>
>>>> Hi,
>>>>
>>>> Yeah. I understand that. But I doubt if that effects the speed much.
>>>>
>>>> I am doing some processing on the array later. But the processing is
>>>> not the bottleneck. It is the packet capture (packet_handler) that is
>>>> the bottleneck. Even if I remove
>>>>
>>>> All the code inside the packet_handler the packet capture still does
>>>> not meet expectations. We have nearly 20,000 packets. And as I
>>>> mentioned before we are looking to achieve a frame rate
>>>>
>>>> Of 30-60 Mbytes/s.
>>>>
>>>> Any ideas ?
>>>>
>>>> Thanks
>>>>
>>>> Charu
>>>>
>>>> *From:* winpcap-users-bounces at winpcap.org
>>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>>
>>> Varenni
>>>
>>>> *Sent:* Tuesday, April 21, 2009 3:33 PM
>>>> *To:* winpcap-users at winpcap.org
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> First of all, allocating some memory at every received packet is a bad
>>>>
>>>> idea. You should preallocate memory at the very beginning.
>>>>
>>>> Then, how many packets do you want to capture? What are you doing with
>>>>
>>>> all those packets in these arrays?
>>>>
>>>> Have a nice day
>>>>
>>>> GV
>>>>
>>>>     ----- Original Message -----
>>>>
>>>>     *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>>
>>>>     *To:* winpcap-users at winpcap.org <mailto:winpcap-users at winpcap.org>
>>>>
>>>>     *Sent:* Tuesday, April 21, 2009 2:20 PM
>>>>
>>>>     *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>>     Hi,
>>>>
>>>>     This is what the code looks like -
>>>>
>>>>     This is all initialization code -
>>>>
>>>>     if ((adhandle= pcap_open_live(d->name, // name of the device
>>>>
>>>>     65536, // portion of the packet to capture.
>>>>
>>>>     // 65536 grants that the whole packet will be captured on all the
>>>>     MACs.
>>>>
>>>>     1, // promiscuous mode (nonzero means promiscuous)
>>>>
>>>>     1, // read timeout
>>>>
>>>>     errbuf // error buffer
>>>>
>>>>     )) == NULL)
>>>>
>>>>     {
>>>>
>>>>     printf("\nUnable to open the adapter. %s is not supported by
>>>>     WinPcap\n", d->name);
>>>>
>>>>     return RETCODE_ERR;
>>>>
>>>>     }
>>>>
>>>>     //Set the filter string to the source ip address
>>>>
>>>>     //Only packets from this source address will be captured.
>>>>
>>>>     //This does slow down the capture a bit.
>>>>
>>>>     NetMask=0xffffff;
>>>>
>>>>     filter = "src 192.168.1.10";
>>>>
>>>>     if(pcap_compile(adhandle, &fcode, filter, 1, NetMask) < 0)
>>>>
>>>>     {
>>>>
>>>>     printf("\nError compiling filter: wrong syntax.\n");
>>>>
>>>>     pcap_close(adhandle);
>>>>
>>>>     return RETCODE_ERR;
>>>>
>>>>     }
>>>>
>>>>     if(pcap_setfilter(adhandle, &fcode)<0)
>>>>
>>>>     {
>>>>
>>>>     printf("\nError setting the filter\n");
>>>>
>>>>     pcap_close(adhandle);
>>>>
>>>>     return RETCODE_ERR;
>>>>
>>>>     }
>>>>
>>>>     //Set the kernel buffer
>>>>
>>>>     if (pcap_setbuff(adhandle,41943040) < 0)
>>>>
>>>>     {
>>>>
>>>>     printf("\n unable to set buffer");
>>>>
>>>>     return RETCODE_ERR;
>>>>
>>>>     }
>>>>
>>>>     Then I start the pcap_loop on a separate thread - This function is
>>>>     called for each packet capture.
>>>>
>>>>     I copy the data for all packets in a global array - image_buf
>>>>
>>>>     void packet_handler(u_char *dumpfileHandler, const struct
>>>>     pcap_pkthdr *header, const u_char *pkt_data)
>>>>
>>>>     {
>>>>
>>>>     //copy the packet data to image_buf and also store the packet
>>>>
>>> length
>>>
>>>>     image_buf[packetCounter] = (unsigned
>>>>     char*)(malloc(header->caplen*sizeof(unsigned char)));
>>>>
>>>>     if(image_buf[packetCounter] == NULL)
>>>>
>>>>     {
>>>>
>>>>     printf("\n unable to allocate buffer");
>>>>
>>>>     return;
>>>>
>>>>     }
>>>>
>>>>     memcpy(image_buf[packetCounter],pkt_data,header->caplen);
>>>>
>>>>     pkt_length[packetCounter]=header->caplen;
>>>>
>>>>     packetCounter += 1;
>>>>
>>>>     }
>>>>
>>>>     Please let me know if you need more details. I can zip and send
>>>>     the code file if needed.
>>>>
>>>>     Thanks
>>>>
>>>>     Charu
>>>>
>>>>     *From:* winpcap-users-bounces at winpcap.org
>>>>     [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>>     Varenni
>>>>     *Sent:* Tuesday, April 21, 2009 1:51 PM
>>>>     *To:* winpcap-users at winpcap.org
>>>>     *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>>     Before trying to modify the WinPcap source code, I would try to
>>>>     understand what the bottlenecks are. What are you doing with the
>>>>     received packets?
>>>>
>>>>     Can you show us the packet processing loop that you have in your
>>>>     code (even in form of pseudocode)?
>>>>
>>>>     Have a nice day
>>>>
>>>>     GV
>>>>
>>>>         ----- Original Message -----
>>>>
>>>>         *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>>
>>>>         *To:* winpcap-users at winpcap.org
>>>>         <mailto:winpcap-users at winpcap.org>
>>>>
>>>>         *Sent:* Monday, April 20, 2009 10:58 AM
>>>>
>>>>         *Subject:* [Winpcap-users] Better Performance
>>>>
>>>>         Hi,
>>>>
>>>>         I am trying to figure out how to capture image frames most
>>>>         efficiently. Each frame is approximately 20Mbytes.
>>>>
>>>>         I am trying to achieve frame rate of 1-2 frames per second.
>>>>         The packets are being received over a Gig eth dedicated link.
>>>>         ( Packets are sent over that link using only one source).
>>>>
>>>>         I am unable to achieve this frame rate using winpcap. Would it
>>>>         be helpful to modify the Winpcap source code for better
>>>>         performance ? Has anyone here have any experience in modifying
>>>>         the winpcap source to achieve higher frame rate capture ?
>>>>
>>>>         Thanks for your help in advance
>>>>
>>>>         Regards
>>>>
>>>>         Charu Agrawal
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>>         _______________________________________________
>>>>         Winpcap-users mailing list
>>>>         Winpcap-users at winpcap.org
>>>>         https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>>     _______________________________________________
>>>>     Winpcap-users mailing list
>>>>     Winpcap-users at winpcap.org
>>>>     https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users 

_______________________________________________
Winpcap-users mailing list
Winpcap-users at winpcap.org
https://www.winpcap.org/mailman/listinfo/winpcap-users


More information about the Winpcap-users mailing list