Re: Recent NTP pool traffic increase

[Andreas Ott <andreas () naund org>]

Hi,
On Fri, Dec 16, 2016 at 04:44:04PM +0700, Roland Dobbins wrote:
> > Looking at the source IP distribution, does a significant proportion 
> > of the larger query base seem to originate out-of-region?
>
> And are do they appear to be mostly broadband access networks, or . . . 
> ?

Datapoints are via nfsen (nflow/sflow collection) from a US west coast
network lab that has "three" NTP pool servers, one IPv4 only set to 25
Mbps, the other one IPv4 and IPv6 on the same server both set to 100Mbps
at the NTP pool registration site.

Traffic is about 4 times P95 in the last 3 days from what it was before, and
the increase is IPv4 on the server that has IPv4 and IPv6. IPv6 traffic is
in line with what it used to be, no large increase.

The server with higher bandwidth and IPv4+IPv6 is seeing a large increase
on IPv4, from single hosts that seem to be in broadband networks and a certain
site's crawler that is hosted on AWS. The latter almost looks like someone
hardcoded a config instead of relying on the pool's DNS. 

The top talker abuses something in the protocol, this does not look for real and
I will contact Verizon/FiOS

tcpdump -nvvi hme0 port 123 and host 98.113.213.d|grep "Originator - Transmit Timestamp:"
            Originator - Transmit Timestamp: 2123062516.816546608 (1967/04/12 11:35:16)
            Originator - Transmit Timestamp: 862276608.564645656 (1927/04/30 01:16:48)
            Originator - Transmit Timestamp: 3399899220.431115995 (2007/09/27 16:27:00)
            Originator - Transmit Timestamp: 140873162.935483905 (1904/06/19 11:26:02)
            Originator - Transmit Timestamp: 1878223676.912769495 (1959/07/09 16:47:56)
            Originator - Transmit Timestamp: 2713286246.929585296 (1985/12/24 18:37:26)
            Originator - Transmit Timestamp: 3219464534.831489402 (2002/01/08 07:42:14)
            Originator - Transmit Timestamp: 2210689093.339715993 (1970/01/20 16:18:13)
            Originator - Transmit Timestamp: 3899283084.650125848 (2023/07/25 14:11:24)
[...]


nfdump -M /var/nfsen/profiles-data/live/dmz208_0201:br1  -T  -R 
2016/12/13/nfcapd.201612131630:2016/12/16/nfcapd.201612161630 -n 10 -s record/bytes -A proto,srcip,dstport -6 "dst ip 
j.k.l.235 and proto udp"
Aggregated flows 51346
Top 10 flows ordered by bytes:
Date first seen          Duration  Proto                             Src IP Addr Dst Pt   Packets    Bytes      bps    
Bpp Flows
2016-12-13 16:31:22.608 259394.340  UDP                             98.113.213.d    123    12.3 M    1.1 G    34107     
90  3000
2016-12-13 16:50:31.649 253960.650  UDP                               54.236.1.d    123    126976   11.4 M      359     
90    31
2016-12-13 17:43:29.760 255090.188  UDP                               54.236.1.d    123    114688   10.3 M      323     
90    28
2016-12-13 20:23:39.198 211054.259  UDP                               54.236.1.d    123     90112    8.1 M      307     
90    22
2016-12-13 22:29:12.265 218623.774  UDP                           204.177.184.d    123     61440    5.5 M      202     
90    15
2016-12-14 04:12:44.389 102634.717  UDP                            162.243.191.d    123     61440    5.5 M      431     
90    15
2016-12-13 22:10:33.226 223641.048  UDP                             198.199.99.d    123     53248    4.8 M      171     
90    13
2016-12-13 21:31:18.841 194915.427  UDP                           220.253.150.d    123     53248    4.8 M      196     
90    13
2016-12-13 20:01:40.452 242771.757  UDP                              troublemaker    123     49152    4.4 M      145    
 90    12
2016-12-14 05:21:20.634 208902.664  UDP                               54.236.1.d    123     40960    3.7 M      141     
90    10
Summary: total flows: 60396, total bytes: 21023451720, total packets: 233586118, avg bps: 648125, avg pps: 900, avg 
bpp: 90
Time window: 1970-01-01 00:00:01 - 2016-12-16 16:34:54
Total flows processed: 29676807, Blocks skipped: 0, Bytes read: 1662858132
Sys: 7.730s flows/second: 3839128.8  Wall: 7.722s flows/second: 3842810.0 

Note: "troublemaker" is a host on the internal network that has a known issue
with NTP time keeping, it originates a lot of packets and steps a lot.


Reply to me directly if you want more details.

-andreas
-- 
Andreas Ott   andreas () naund org