[Winpcap-users] Capturing from a'tap'typedeviceusing2networkcards - and how to order the packets

Wed Jun 4 06:37:07 GMT 2008

> And according to my app, I have two different threads, one for capturing 
> packets and another one for processing them. Thus, all database access is 
> separated from the actual capture. Is not this enough?

No. If the database is slow at inserting records, using two threads doesn't 
help. You just move the problem to some other place. Suppose that you 
receive 100 packets per second, that you need to insert 1 record in the DB 
per packet, and that you can make 50 DB insertions per second. Your DB will 
not keep up. Period. If you use two threads, you will probably use a fixed 
size or variable size queue to connect the capture thread with the database 
thread. If your db doesn't keep up, and you are using a fixed size queue, 
the queue will be eventually full, and you will lose packets. If you have a 
variable size queue, the queue will grow grow grow, and eventually you will 
run out of memory.

----->I'm not sure about number of packets per second, but I don't insert 1 record in the DB per packet. I am recording skinny calls. I have one record per call and I access it about 6 times (for the creation and later field updates). And for the queue, I check if it is full and write a log message that up to now has never appeared. Thus, it is never full (max_size is 1073741823)and neither I am run out of memory because the consumer thread seems to be fast enough. It is just CPU which come to full working. 

I will really appreciate any other ideas

María de Fátima Requena Cabot (2488)
+34 91 787 23 00 alhambra-eidos.es

-----Mensaje original-----
De: winpcap-users-bounces at winpcap.org [mailto:winpcap-users-bounces at winpcap.org] En nombre de Gianluca Varenni
Enviado el: miércoles, 04 de junio de 2008 1:44
Para: winpcap-users at winpcap.org
Asunto: Re: [Winpcap-users] Capturing from a'tap'typedeviceusing2networkcards - and how to order the packets

----- Original Message ----- 
From: "Maria de Fatima Requena" <MariaF.Requena at a-e.es>
To: <winpcap-users at winpcap.org>
Sent: Thursday, May 29, 2008 11:33 PM
Subject: RE: [Winpcap-users] Capturing from a 
'tap'typedeviceusing2networkcards - and how to order the packets

> If I remove the 'update in real time' tick, when will it update?. I can't 
> see any menu or button to 'refresh'
>

It won't, in fact. It will dump the packets to disk. When you stop the 
capture, Wireshark will load the capture.

> And according to my app, I have two different threads, one for capturing 
> packets and another one for processing them. Thus, all database access is 
> separated from the actual capture. Is not this enough?

No. If the database is slow at inserting records, using two threads doesn't 
help. You just move the problem to some other place. Suppose that you 
receive 100 packets per second, that you need to insert 1 record in the DB 
per packet, and that you can make 50 DB insertions per second. Your DB will 
not keep up. Period. If you use two threads, you will probably use a fixed 
size or variable size queue to connect the capture thread with the database 
thread. If your db doesn't keep up, and you are using a fixed size queue, 
the queue will be eventually full, and you will lose packets. If you have a 
variable size queue, the queue will grow grow grow, and eventually you will 
run out of memory.

Hope it helps
GV

>
>
>
> María de Fátima Requena Cabot (2488)
> +34 91 787 23 00 alhambra-eidos.es
>
>
> -----Mensaje original-----
> De: winpcap-users-bounces at winpcap.org 
> [mailto:winpcap-users-bounces at winpcap.org] En nombre de Gianluca Varenni
> Enviado el: jueves, 29 de mayo de 2008 19:58
> Para: winpcap-users at winpcap.org
> Asunto: Re: [Winpcap-users] Capturing from a 'tap' 
> typedeviceusing2networkcards - and how to order the packets
>
> 100% CPU usage (and packet drops) means that your application is not fast
> enough at processing packets.
>
> Some things to remember
> - Wireshark usually loses packets at high packet rates because
>  1. it dumps the packets to disk. Disks are SLOW.
>  2. if you have enabled real-time update of the packets in the wireshark
> GUI, that takes time and memory, and can affect performance
> - in one of your previous e-mails, I seem to remember that you were 
> dumping
> some packet contents (or fields) in a database. Unless you are using
>  an embedded DB in memory, databases are SLOW. They are slow because
> normally their backing store is a disk. Hard disks are slow.
>
> Again, I would probably profile your application and try to locate where 
> all
> the CPU load is going.
>
> Have a nice day
> GV
>
> ----- Original Message ----- 
> From: "Maria de Fatima Requena" <MariaF.Requena at a-e.es>
> To: <winpcap-users at winpcap.org>
> Sent: Thursday, May 29, 2008 4:10 AM
> Subject: RE: [Winpcap-users] Capturing from a 'tap' type
> deviceusing2networkcards - and how to order the packets
>
>
>> Sorry, i have forgotten to mention that I'm also having a 100%CPU usage
>> for both my app and Wireshark
>>
>> Thanks
>>
>>
>> María de Fátima Requena Cabot (2488)
>> +34 91 787 23 00 alhambra-eidos.es
>>
>>
>>
>> -----Mensaje original-----
>> De: winpcap-users-bounces at winpcap.org
>> [mailto:winpcap-users-bounces at winpcap.org] En nombre de Maria de Fatima
>> Requena
>> Enviado el: jueves, 29 de mayo de 2008 12:09
>> Para: winpcap-users at winpcap.org
>> Asunto: RE: [Winpcap-users] Capturing from a 'tap' type device
>> using2networkcards - and how to order the packets
>>
>> I have similar problems. I posted some time ago about missing packets
>> (packets that Wireshark captured but my application didn't) but now I am
>> in a different machine with high traffic and both Wireshark and my app
>> miss them.
>>
>> Any ideas?
>>
>>
>> María de Fátima Requena Cabot (2488)
>> +34 91 787 23 00 alhambra-eidos.es
>>
>>
>> -----Mensaje original-----
>> De: winpcap-users-bounces at winpcap.org
>> [mailto:winpcap-users-bounces at winpcap.org] En nombre de Gianluca Varenni
>> Enviado el: miércoles, 28 de mayo de 2008 19:55
>> Para: winpcap-users at winpcap.org
>> Asunto: Re: [Winpcap-users] Capturing from a 'tap' type device using
>> 2networkcards - and how to order the packets
>>
>>
>> ----- Original Message ----- 
>> From: "Michael Shiels" <MaSSoft at massoftware.com>
>> To: <winpcap-users at winpcap.org>
>> Sent: Wednesday, May 28, 2008 7:06 AM
>> Subject: [Winpcap-users] Capturing from a 'tap' type device using 2
>> networkcards - and how to order the packets
>>
>>
>>> This problem has had a couple attempts and each has failed to get the
>>> packets ALWAYS into the proper order.
>>>
>>> It seems some packets hang around for many seconds somewhere in the pcap
>>> world, but I can't say for sure.
>>>
>>> The code is doing everything I could find for high speed/high volume
>>> capturing.
>>>
>>> Specifically
>>>
>>> Timeout of '1' on pcap_open, with flags PCAP_OPENFLAG_PROMISCUOUS
>>>
>>> Pcap_setbuff( 5*1024*1024 )
>>>
>>> Pcap_setmintocopy(0)
>>
>> Putting mintocopy to 0 and setting the timeout to 1ms does *not* help
>> getting better performance. It increases the responsiveness of the
>> application, at the expense of a higher CPU load (because of the larger
>> number of syscalls/second used to bring packets to user level).
>>
>>
>>>
>>> We then have a loop of code that gets the pcap events and does a wait
>>> with
>>> 1
>>> second timeout, then polls both pcap handles till we find no data ON 
>>> BOTH
>>> HANDLES.  That should retrieve all packets is my understanding. We then
>>> delay the packets in some queues for a period of time, that should allow
>>> all
>>> 'hardware/software' buffers to empty I expected.  Then the packets are
>>> merged by timestamp.
>>>
>>> BUT under load it seems that we still some packets stuck somewhere, for
>>> multiple seconds.  WE started by waiting for packets to be older than 5
>>> seconds to merge, and it had problems.  Bumping the process to wait for
>>> 30
>>> seconds before merging packets, seems to work, but it's a supreme
>>> hack/probably not something 100% reliable.
>>>
>>
>> WinPcap does not starve the packets in the driver for all that time (5
>> seconds or so), if you are actively reading data from the devices. How do
>> you "poll both the pcap handler till we find no data"? Do you use two
>> threads to do that?
>> Also, what is the traffic load we are talking about?
>>
>> Have a nice day
>> GV
>>
>>> ANY IDEAS?? HELP!!!!!!
>>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users 

_______________________________________________
Winpcap-users mailing list
Winpcap-users at winpcap.org
https://www.winpcap.org/mailman/listinfo/winpcap-users