[Winpcap-users] Capturing from a'tap'typedeviceusing2networkcards - and how to order the packets

Thu May 29 21:17:50 GMT 2008

Very simple loop that polls each

Yes in theory it could be two threads, but that doesn't really change the
problem I have right now.

That thread just allocates buffers/copies/queues data.  

ANOTHER thread which waits till the data timestamps are at least X seconds
old (5 didn't work, 30 seems to be better) then attempts to merge the data
packet by packet into order by timestamp, as long as one of the queues has
something > X seconds old.  Very simple/basic.  EXCEPT that once in a while
literally there is no data from pcap for X seconds which means we HAVE TO
HAVE POLLED it many times and got nothing, then boom packets that are >X
seconds old.

-----Original Message-----
From: winpcap-users-bounces at winpcap.org
[mailto:winpcap-users-bounces at winpcap.org] On Behalf Of Gianluca Varenni
Sent: May 29, 2008 4:52 PM
To: winpcap-users at winpcap.org
Subject: Re: [Winpcap-users] Capturing from
a'tap'typedeviceusing2networkcards - and how to order the packets

In your original email, you said "then polls both pcap handles till we find 
no data ON BOTH
HANDLES".

How do you "poll both the pcap handles till we find no data"? Do you use two
threads to do that? A round robin algorithm?

Have a nice day
GV

----- Original Message ----- 
From: "Michael Shiels" <MaSSoft at massoftware.com>
To: <winpcap-users at winpcap.org>
Sent: Thursday, May 29, 2008 12:29 PM
Subject: RE: [Winpcap-users] Capturing from a 
'tap'typedeviceusing2networkcards - and how to order the packets

> Hmm.  Good questions.
>
> Right now I am just going by the fact that the packets timestamps on them,
> are 'out of order', not even looking at the packet flow/protocol flow at
> all.
>
> SO the problem you described about packets being stamped wrong is not what

> I
> am running into.  It's more a packets are stuck/queued somewhere, such 
> that
> when I pull pcap_next_ex() there is no data from that card for many 
> seconds,
>>5 in this case, then all of a suddent packets with time stamps over 5
> seconds old are released from within pcap.
>
> -----Original Message-----
> From: winpcap-users-bounces at winpcap.org
> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of Gianluca Varenni
> Sent: Thursday, May 29, 2008 1:51 PM
> To: winpcap-users at winpcap.org
> Subject: Re: [Winpcap-users] Capturing from a 'tap'
> typedeviceusing2networkcards - and how to order the packets
>
>
> ----- Original Message ----- 
> From: "Michael Shiels (MaS Software)" <MaSSoft at massoftware.com>
> To: <winpcap-users at winpcap.org>
> Sent: Thursday, May 29, 2008 4:24 AM
> Subject: RE: [Winpcap-users] Capturing from a 'tap' type
> deviceusing2networkcards - and how to order the packets
>
>
>> Oh I don't think we miss any packets in our situation, they just seem
>> buffered up somewhere and unable to flow evenly through the system.
>>
>> Not sure if there is a list anywhere, but I know way way back in the day
>> when I hand did network analyzers (DOS days) there were certain cards 
>> that
>> were superb for network analysis, since they were extreme high
>> performance,
>> but now sure if such a list exists for WinPCap.
>
> If it's a network card (or NIC driver) problem, what you would see is
> packets with a completely bogus timestamp, i.e. the packet has been 
> received
>
> by the hardware at time X, and WinPcap timestamped it at X+5seconds, 
> because
>
> either the NIC notified its NIC driver after a long delay, or the NIC 
> driver
>
> notified WinPcap of the received packet with such long delay. (Please
> remember that it's WinPcap who timestamps the packets as soon as the 
> packet
> gets delivered by the NIC driver to the WinPcap protocol driver).
>
>>From what you say, it looks like the packets have meaningful timestamps,
> but
> they somewhat get "jammed" between the WinPcap driver and your application
> calling pcap_next_ex (or any other routine to receive packets).
>
> Can you please confirm this?
>
> If this is the case, the choice of a network card vs. another one will not
> make any difference, expecially if you are not working at high packet 
> rate.
>
> Have a nice day
> GV
>
>>
>> -----Original Message-----
>> From: winpcap-users-bounces at winpcap.org
>> [mailto:winpcap-users-bounces at winpcap.org] On Behalf Of Maria de Fatima
>> Requena
>> Sent: May 29, 2008 6:09 AM
>> To: winpcap-users at winpcap.org
>> Subject: RE: [Winpcap-users] Capturing from a 'tap' type device
>> using2networkcards - and how to order the packets
>>
>> I have similar problems. I posted some time ago about missing packets
>> (packets that Wireshark captured but my application didn't) but now I am
>> in
>> a different machine with high traffic and both Wireshark and my app miss
>> them.
>>
>> Any ideas?
>>
>>
>> María de Fátima Requena Cabot (2488)
>> +34 91 787 23 00 alhambra-eidos.es
>>
>>
>> -----Mensaje original-----
>> De: winpcap-users-bounces at winpcap.org
>> [mailto:winpcap-users-bounces at winpcap.org] En nombre de Gianluca Varenni
>> Enviado el: miércoles, 28 de mayo de 2008 19:55
>> Para: winpcap-users at winpcap.org
>> Asunto: Re: [Winpcap-users] Capturing from a 'tap' type device using
>> 2networkcards - and how to order the packets
>>
>>
>> ----- Original Message ----- 
>> From: "Michael Shiels" <MaSSoft at massoftware.com>
>> To: <winpcap-users at winpcap.org>
>> Sent: Wednesday, May 28, 2008 7:06 AM
>> Subject: [Winpcap-users] Capturing from a 'tap' type device using 2
>> networkcards - and how to order the packets
>>
>>
>>> This problem has had a couple attempts and each has failed to get the
>>> packets ALWAYS into the proper order.
>>>
>>> It seems some packets hang around for many seconds somewhere in the pcap
>>> world, but I can't say for sure.
>>>
>>> The code is doing everything I could find for high speed/high volume
>>> capturing.
>>>
>>> Specifically
>>>
>>> Timeout of '1' on pcap_open, with flags PCAP_OPENFLAG_PROMISCUOUS
>>>
>>> Pcap_setbuff( 5*1024*1024 )
>>>
>>> Pcap_setmintocopy(0)
>>
>> Putting mintocopy to 0 and setting the timeout to 1ms does *not* help
>> getting better performance. It increases the responsiveness of the
>> application, at the expense of a higher CPU load (because of the larger
>> number of syscalls/second used to bring packets to user level).
>>
>>
>>>
>>> We then have a loop of code that gets the pcap events and does a wait
>>> with
>>
>>> 1
>>> second timeout, then polls both pcap handles till we find no data ON 
>>> BOTH
>>> HANDLES.  That should retrieve all packets is my understanding. We then
>>> delay the packets in some queues for a period of time, that should allow
>>> all
>>> 'hardware/software' buffers to empty I expected.  Then the packets are
>>> merged by timestamp.
>>>
>>> BUT under load it seems that we still some packets stuck somewhere, for
>>> multiple seconds.  WE started by waiting for packets to be older than 5
>>> seconds to merge, and it had problems.  Bumping the process to wait for
>>> 30
>>> seconds before merging packets, seems to work, but it's a supreme
>>> hack/probably not something 100% reliable.
>>>
>>
>> WinPcap does not starve the packets in the driver for all that time (5
>> seconds or so), if you are actively reading data from the devices. How do
>> you "poll both the pcap handler till we find no data"? Do you use two
>> threads to do that?
>> Also, what is the traffic load we are talking about?
>>
>> Have a nice day
>> GV
>>
>>> ANY IDEAS?? HELP!!!!!!
>>>
>>> _______________________________________________
>>> Winpcap-users mailing list
>>> Winpcap-users at winpcap.org
>>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>>
>>
>> _______________________________________________
>> Winpcap-users mailing list
>> Winpcap-users at winpcap.org
>> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users
>
>
> _______________________________________________
> Winpcap-users mailing list
> Winpcap-users at winpcap.org
> https://www.winpcap.org/mailman/listinfo/winpcap-users 

_______________________________________________
Winpcap-users mailing list
Winpcap-users at winpcap.org
https://www.winpcap.org/mailman/listinfo/winpcap-users