[Winpcap-users] Better Performance
Gianluca Varenni
gianluca.varenni at cacetech.com
Tue Apr 28 12:34:15 PDT 2009
pcap_next_ex should not give you any performance hit.
Can you post the code that you use with pcap_next_ex?
GV
----- Original Message -----
From: "Charu Agrawal" <cagrawal at altasens.com>
To: <winpcap-users at winpcap.org>
Sent: Tuesday, April 28, 2009 12:02 PM
Subject: Re: [Winpcap-users] Better Performance
> Hello David,
>
> I figured out the reason for the poor performance.
> I was creating a separate thread for the pcap_loop, and the main thread
> calculated when to timeout.
> When I use pcap_loop in the main thread, I get the performance I need.
>
> But the drawback now is, that I cannot get timeout using pcap_loop or
> pcap_dispatch.
>
> This is what I want.
>
> Suppose I say pcap_loop(handle,20000,....)
> I need to capture 20000 packets. But if I capture only 16000 packets in
> say 10s , I want the program to timeout.
>
> I tried to use pcap_next_ex for this, but it slows down the code.
> pcap_loop gives me the needed performance, except the timeout ability.
>
> Is there a workaround ?
>
> Thanks
> Charu
>
> -----Original Message-----
> From: winpcap-users-bounces at winpcap.org
> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
> Sent: Friday, April 24, 2009 1:08 PM
> To: winpcap-users at winpcap.org
> Subject: Re: [Winpcap-users] Better Performance
>
> Charu,
>
> Can you give me your hardware specs? Also, what was your throughput
> with the empty callback function?
>
> Let's do another test...
>
> u_char dummy[2000]; /* I assume your packets (and snaplen) are less than
> 1500 bytes long */
>
> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr
> *header, const u_char *pkt_data)
> {
> if (header->len < sizeof(dummy)+sizeof(struct pcap_pkthdr))
> {
> memcpy(dummy, header, sizeof(struct pcap_pkthdr));
> memcpy(dummy+sizeof(struct pcap_pkthdr), pkt_data, header->len);
> }
>
> packetCounter++;
> }
>
> How is the throughput with just the memcpy() overwriting a dummy buffer?
>
> Lastly, I would use MUCH less kernel buffer space (40MB is too much). I
> would use 8MB - 16MB. For example:
>
> if (pcap_setbuff(adhandle,16777216) < 0)
>
>
> DC
>
> Gianluca Varenni wrote:
>> ----- Original Message -----
>> From: "Charu Agrawal" <cagrawal at altasens.com>
>> To: <winpcap-users at winpcap.org>
>> Sent: Friday, April 24, 2009 10:01 AM
>> Subject: Re: [Winpcap-users] Better Performance
>>
>>
>>
>>> Hello David,
>>>
>>> If I change the code as you suggested, I do get a performance
>>> improvement. I have now allocated memory outside the loop.
>>> But I still need to use the memcpy function to store all packets in an
>>> array. That still slows it down.
>>>
>>> Anything else I can do to improve performance ? Is there a way to change
>>> the winpcap source code for better performance ?
>>>
>>>
>>
>> WinPcap has been highly optimized over the years for performance. Before
>> looking into optimizing the WinPcap code, I would understand exactly what
>> the bottleneck is.
>> You say that you are copying packets into your allocated memory. Why
>> can't
>> you process the packets inline, i.e. when you receive them? Also, how
>> much
>> memory did you allocate for the packets? Preallocating several megabytes
>> ahead of time for the packets can be as dangerous as malloc'ing for every
>> packet. This is because allocating memory just means that you have
>> reserved
>> a large chunk of address space in your process, not that you have
>> physically
>> reserved all those megabytes in the physical RAM (i.e. not all that
>> memory
>> is your working set).
>>
>> Hope it helps
>> GV
>>
>>
>>
>>> Thanks
>>> Charu
>>>
>>> -----Original Message-----
>>> From: winpcap-users-bounces at winpcap.org
>>> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of David Chang
>>> Sent: Thursday, April 23, 2009 7:03 PM
>>> To: winpcap-users at winpcap.org
>>> Subject: Re: [Winpcap-users] Better Performance
>>>
>>> Charu,
>>>
>>> malloc() is VERY expensive; especially for larger buffers. Are you
>>> saying that if you change the callback function to this...
>>>
>>> void packet_handler(u_char *dumpfileHandler, const struct pcap_pkthdr
>>> *header, const u_char *pkt_data)
>>> {
>>> packetCounter++;
>>> }
>>>
>>> you still can't read 20,000 packets / sec? (I assume each packet is
>>> full, so this would be close to 30MB/sec)
>>>
>>> What type of hardware (CPU, RAM, NIC, motherboard, etc) and O.S. do you
>>> use?
>>>
>>> DC
>>>
>>> Charu Agrawal wrote:
>>>
>>>> Hi,
>>>>
>>>> Yeah. I understand that. But I doubt if that effects the speed much.
>>>>
>>>> I am doing some processing on the array later. But the processing is
>>>> not the bottleneck. It is the packet capture (packet_handler) that is
>>>> the bottleneck. Even if I remove
>>>>
>>>> All the code inside the packet_handler the packet capture still does
>>>> not meet expectations. We have nearly 20,000 packets. And as I
>>>> mentioned before we are looking to achieve a frame rate
>>>>
>>>> Of 30-60 Mbytes/s.
>>>>
>>>> Any ideas ?
>>>>
>>>> Thanks
>>>>
>>>> Charu
>>>>
>>>> *From:* winpcap-users-bounces at winpcap.org
>>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>>
>>> Varenni
>>>
>>>> *Sent:* Tuesday, April 21, 2009 3:33 PM
>>>> *To:* winpcap-users at winpcap.org
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> First of all, allocating some memory at every received packet is a bad
>>>>
>>>> idea. You should preallocate memory at the very beginning.
>>>>
>>>> Then, how many packets do you want to capture? What are you doing with
>>>>
>>>> all those packets in these arrays?
>>>>
>>>> Have a nice day
>>>>
>>>> GV
>>>>
>>>> ----- Original Message -----
>>>>
>>>> *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>>
>>>> *To:* winpcap-users at winpcap.org <mailto:winpcap-users at winpcap.org>
>>>>
>>>> *Sent:* Tuesday, April 21, 2009 2:20 PM
>>>>
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> Hi,
>>>>
>>>> This is what the code looks like -
>>>>
>>>> This is all initialization code -
>>>>
>>>> if ((adhandle= pcap_open_live(d->name, // name of the device
>>>>
>>>> 65536, // portion of the packet to capture.
>>>>
>>>> // 65536 grants that the whole packet will be captured on all the
>>>> MACs.
>>>>
>>>> 1, // promiscuous mode (nonzero means promiscuous)
>>>>
>>>> 1, // read timeout
>>>>
>>>> errbuf // error buffer
>>>>
>>>> )) == NULL)
>>>>
>>>> {
>>>>
>>>> printf("\nUnable to open the adapter. %s is not supported by
>>>> WinPcap\n", d->name);
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> //Set the filter string to the source ip address
>>>>
>>>> //Only packets from this source address will be captured.
>>>>
>>>> //This does slow down the capture a bit.
>>>>
>>>> NetMask=0xffffff;
>>>>
>>>> filter = "src 192.168.1.10";
>>>>
>>>> if(pcap_compile(adhandle, &fcode, filter, 1, NetMask) < 0)
>>>>
>>>> {
>>>>
>>>> printf("\nError compiling filter: wrong syntax.\n");
>>>>
>>>> pcap_close(adhandle);
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> if(pcap_setfilter(adhandle, &fcode)<0)
>>>>
>>>> {
>>>>
>>>> printf("\nError setting the filter\n");
>>>>
>>>> pcap_close(adhandle);
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> //Set the kernel buffer
>>>>
>>>> if (pcap_setbuff(adhandle,41943040) < 0)
>>>>
>>>> {
>>>>
>>>> printf("\n unable to set buffer");
>>>>
>>>> return RETCODE_ERR;
>>>>
>>>> }
>>>>
>>>> Then I start the pcap_loop on a separate thread - This function is
>>>> called for each packet capture.
>>>>
>>>> I copy the data for all packets in a global array - image_buf
>>>>
>>>> void packet_handler(u_char *dumpfileHandler, const struct
>>>> pcap_pkthdr *header, const u_char *pkt_data)
>>>>
>>>> {
>>>>
>>>> //copy the packet data to image_buf and also store the packet
>>>>
>>> length
>>>
>>>> image_buf[packetCounter] = (unsigned
>>>> char*)(malloc(header->caplen*sizeof(unsigned char)));
>>>>
>>>> if(image_buf[packetCounter] == NULL)
>>>>
>>>> {
>>>>
>>>> printf("\n unable to allocate buffer");
>>>>
>>>> return;
>>>>
>>>> }
>>>>
>>>> memcpy(image_buf[packetCounter],pkt_data,header->caplen);
>>>>
>>>> pkt_length[packetCounter]=header->caplen;
>>>>
>>>> packetCounter += 1;
>>>>
>>>> }
>>>>
>>>> Please let me know if you need more details. I can zip and send
>>>> the code file if needed.
>>>>
>>>> Thanks
>>>>
>>>> Charu
>>>>
>>>> *From:* winpcap-users-bounces at winpcap.org
>>>> [mailto:winpcap-users-bounces at winpcap.org] *On Behalf Of *Gianluca
>>>> Varenni
>>>> *Sent:* Tuesday, April 21, 2009 1:51 PM
>>>> *To:* winpcap-users at winpcap.org
>>>> *Subject:* Re: [Winpcap-users] Better Performance
>>>>
>>>> Before trying to modify the WinPcap source code, I would try to
>>>> understand what the bottlenecks are. What are you doing with the
>>>> received packets?
>>>>
>>>> Can you show us the packet processing loop that you have in your
>>>> code (even in form of pseudocode)?
>>>>
>>>> Have a nice day
>>>>
>>>> GV
>>>>
>>>> ----- Original Message -----
>>>>
>>>> *From:* Charu Agrawal <mailto:cagrawal at altasens.com>
>>>>
>>>> *To:* winpcap-users at winpcap.org
>>>> <mailto:winpcap-users at winpcap.org>
>>>>
>>>> *Sent:* Monday, April 20, 2009 10:58 AM
>>>>
>>>> *Subject:* [Winpcap-users] Better Performance
>>>>
>>>> Hi,
>>>>
>>>> I am trying to figure out how to capture image frames most
>>>> efficiently. Each frame is approximately 20Mbytes.
>>>>
>>>> I am trying to achieve frame rate of 1-2 frames per second.
>>>> The packets are being received over a Gig eth dedicated link.
>>>> ( Packets are sent over that link using only one source).
>>>>
>>>> I am unable to achieve this frame rate using winpcap. Would it
>>>> be helpful to modify the Winpcap source code for better
>>>> performance ? Has anyone here have any experience in modifying
>>>> the winpcap source to achieve higher frame rate capture ?
>>>>
>>>> Thanks for your help in advance
>>>>
>>>> Regards
>>>>
>>>> Charu Agrawal
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>> _______________________________________________
>>>> Winpcap-users mailing list
>>>> Winpcap-users at winpcap.org
>>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>>
>>>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>>
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
More information about the Winpcap-users
mailing list