Re: NANOG Digest, Vol 46, Issue 15

[Suess13 <suess13 () cfl rr com>]

Juniper core dump issue, patch is on the way.



On Nov 7, 2011, at 11:41 AM, "Steve Dispensa" <dispensa () phonefactor com> wrote:

> Level 3 was down in KC, Chi, and San Jose (at least) for us between
> about 8:10 and 8:40, plus or minus. Brought down SureWest in KC too.
>
> -Steve
>
> > -----Original Message-----
> > From: nanog-request () nanog org [mailto:nanog-request () nanog org]
> > Sent: Monday, November 07, 2011 10:05 AM
> > To: nanog () nanog org
> > Subject: NANOG Digest, Vol 46, Issue 15
> >
> > Send NANOG mailing list submissions to
> >    nanog () nanog org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >    https://mailman.nanog.org/mailman/listinfo/nanog
> > or, via email, send a message with subject or body 'help' to
> >    nanog-request () nanog org
> >
> > You can reach the person managing the list at
> >    nanog-owner () nanog org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of NANOG digest..."
> >
> >
> > Today's Topics:
> >
> >   1. Re: Time Warner Telecom problems (Jon Lewis)
> >   2. Re: Performance Issues - PTR Records (Leigh Porter)
> >   3. Re: Time Warner Telecom problems (Ray Van Dolson)
> >   4. General Internet Instability (Jared Mauch)
> >   5. Re: TATA problems? (Pierre-Yves Maunier)
> >   6. Re: TATA problems? (Leigh Porter)
> >   7. Re: Time Warner Telecom problems (Joe Greco)
> >   8. Re: TATA problems? (Kelly Kane)
> >   9. Re: Time Warner Telecom problems (Blake Hudson)
> >  10. RE: Time Warner Telecom problems (Thomas York)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Mon, 7 Nov 2011 10:12:30 -0500 (EST)
> > From: Jon Lewis <jlewis () lewis org>
> > To: Peter Pauly <ppauly () gmail com>
> > Cc: nanog () nanog org
> > Subject: Re: Time Warner Telecom problems
> > Message-ID: <Pine.LNX.4.61.1111071005540.24418 () soloth lewis org>
> > Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
> >
> > On Mon, 7 Nov 2011, Peter Pauly wrote:
> >
> > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> > suffering
> > > from it too and calls to the NOC have not been answered so far...
> does
> > > anyone have any further information?
> > >
> > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> >
> > I noticed just a little while ago that we're having a lot of DNS fail.
> > Initial findings were that several of the root-servers we were trying
> to
> > reach via our TWTelecom link were unreachable after 2 hops into TWT.
> >
> >  4  64-128-130-233.static.twtelecom.NET (64.128.130.233)  2.399 ms
> 2.298
> > ms  2.338 ms
> >  5  mia2-pr1-xe-1-3-0-0.us.twtelecom.net (66.192.253.18)  11.571 ms
> > 11.552 ms  9.467 ms
> >  6  * * *
> >  7  * * *
> >  8  * * *
> >
> > For instance, a.root-servers.net is pingable from a rackspace server,
> but
> > not from our network (unless I shut off TWT, at which point it is, but
> > it's apparently not the same a.root-servers.net instance rackspace
> sees).
> > I assume this is one of the root-servers being anycast.
> >
> > Shutting off our BGP with TWT didn't appear to help (though the
> > root-servers became reachable)...so I assume there's more going on
> than
> > just TWT routing fail.
> >
> > ----------------------------------------------------------------------
> >  Jon Lewis, MCP :)           |  I route
> >  Senior Network Engineer     |  therefore you are
> >  Atlantic Net                |
> > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
> >
> >
> >
> > ------------------------------
> >
> > Message: 2
> > Date: Mon, 7 Nov 2011 15:29:30 +0000
> > From: Leigh Porter <leigh.porter () ukbroadband com>
> > To: Bj?rn Mork <bjorn () mork no>
> > Cc: "nanog () nanog org" <nanog () nanog org>
> > Subject: Re: Performance Issues - PTR Records
> > Message-ID: <53A4963F-4969-4A60-BF06-E690C7324863 () ukbroadband com>
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> >
> >
> > On 7 Nov 2011, at 14:03, "Bj?rn Mork" <bjorn () mork no> wrote:
> >
> > > Leigh Porter <leigh.porter () ukbroadband com> writes:
> > >
> > > > Indeed, there is no way I would allow that either. But really,
> > > > providing a reverse zone and forward zone to match is a case of
> five
> > > > minutes and a shell script or a DNS that as Steinar said, will
> > > > synthesise results.
> > > >
> > > > It's really not all that difficult..
> > >
> > > No, not at all.  It's just totally pointless.  Any IPv6 address is
> just
> > > as pretty as a synthesized name.  Maybe even prettier. Do you prefer
> > > "2001:db8:1::2" or
> "20010db8000100000000000000000002.rev.example.com"?
> > > If we're going to provide any reverse DNS for end users, then it is
> > > because we can create names which actually improves something.
> > >
> > >
> > > Bj?rn
> >
> > Yup it is pointless.. Mine are all ipadrress.domain which is of
> course,
> > pointless.. I suppose at least somebody would glean that perhaps its a
> > home user rather than a business or server on that address but that's
> all.
> > With IPv6 arguably even more pointless as you say.
> >
> > ______________________________________________________________________
> > This email has been scanned by the MessageLabs Email Security System.
> > For more information please visit http://www.messagelabs.com/email
> > ______________________________________________________________________
> >
> >
> >
> > ------------------------------
> >
> > Message: 3
> > Date: Mon, 7 Nov 2011 07:28:18 -0800
> > From: Ray Van Dolson <rvandolson () esri com>
> > To: nanog () nanog org
> > Subject: Re: Time Warner Telecom problems
> > Message-ID: <20111107152817.GA29715 () esri com>
> > Content-Type: text/plain; charset=us-ascii
> >
> > On Mon, Nov 07, 2011 at 07:04:19AM -0800, Peter Pauly wrote:
> > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> > suffering
> > > from it too and calls to the NOC have not been answered so far...
> does
> > > anyone have any further information?
> > >
> > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> >
> > FWIW, my home TWC connection dropped this morning for about 15 minutes
> > (Southern California around 6:30AM'ish).  Still could ping the default
> > gateway, but packets weren't traversing much beyond that.
> >
> > Didn't investigate further, just headed into work.
> >
> > Ray
> >
> >
> >
> > ------------------------------
> >
> > Message: 4
> > Date: Mon, 7 Nov 2011 10:31:31 -0500
> > From: Jared Mauch <jared () puck nether net>
> > To: Tom Hill <tom () ninjabadger net>
> > Cc: nanog () nanog org
> > Subject: General Internet Instability
> > Message-ID: <B6567BC5-75E9-4E58-AFA9-5ADC0C2A7BDD () puck nether net>
> > Content-Type: text/plain; charset=us-ascii
> >
> > On Nov 7, 2011, at 10:08 AM, Tom Hill wrote:
> >
> > > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > > We seem to be having some problems with our tata links - first seen
> in
> > EU
> > > > about 45 minutes ago, now we're seeing problems in NA.  I'm focused
> on
> > DNS,
> > > > so I'm seeing a lot of timeouts/servfails, but our networking folks
> are
> > > > talking about links dropping.
> > > >
> > > > Anyone else seeing oddness on the NA Internet right now?
> > > >
> > > > http://downrightnow.com/ confirms - something is up.
> > >
> > > There are widespread issues across the Internet; certain versions of
> > > Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
> > > message.
> > >
> > > (That's the running theory at least).
> > >
> > > It's affected multiple service providers, globally, not just those
> > > connected to TATA.
> >
> >
> > Pretty much any major BGP event will impact multiple providers.
> >
> > A threshold you should use to view the general instability (which I
> find
> > valuable, you may as well) is route views data.
> >
> > If you look at the BGP UPDATES archive sizes, you can see when
> something
> > happens, e.g.:
> >
> > http://archive.routeviews.org/bgpdata/2011.11/UPDATES/
> >
> > Take a look at the size of the updates.20111107.1400.bz2 file and the
> 1415
> > file.  They are abnormally large compared to a normal period of time.
> > This shows there were a lot of updates out there being processed and a
> > reference to levels of instability.
> >
> > If you are not feeding route views or similar community projects,
> please
> > consider doing so.  It helps paint the view for those doing analysis.
> >
> > - Jared
> >
> >
> > ------------------------------
> >
> > Message: 5
> > Date: Mon, 7 Nov 2011 16:33:15 +0100
> > From: Pierre-Yves Maunier <nanog () maunier org>
> > To: Tom Hill <tom () ninjabadger net>
> > Cc: nanog () nanog org
> > Subject: Re: TATA problems?
> > Message-ID:
> <CAEGdXrM+2rFXG2=d80kO_1ObtAv=24npNqZsLCTXUxeKS9q9yA () mail gmail com>
> > Content-Type: text/plain; charset=ISO-8859-1
> >
> > 2011/11/7 Tom Hill <tom () ninjabadger net>
> >
> > > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > > We seem to be having some problems with our tata links - first
> seen in
> > EU
> > > > about 45 minutes ago, now we're seeing problems in NA.  I'm
> focused on
> > > DNS,
> > > > so I'm seeing a lot of timeouts/servfails, but our networking
> folks
> > are
> > > > talking about links dropping.
> > > >
> > > > Anyone else seeing oddness on the NA Internet right now?
> > > >
> > > > http://downrightnow.com/ confirms - something is up.
> > >
> > > There are widespread issues across the Internet; certain versions of
> > > Juniper firmware have core dumped after seeing a particular BGP
> 'UPDATE'
> > > message.
> > >
> > > (That's the running theory at least).
> > >
> > > It's affected multiple service providers, globally, not just those
> > > connected to TATA.
> > >
> > > Tom
> > On our side all our 10.3R2.11 core dumped which made all our
> interfaces
> > flapped.
> > I've been told 10.4R1.9 is affected too.
> >
> > --
> > Pierre-Yves Maunier
> >
> >
> > ------------------------------
> >
> > Message: 6
> > Date: Mon, 7 Nov 2011 15:45:18 +0000
> > From: Leigh Porter <leigh.porter () ukbroadband com>
> > To: Pierre-Yves Maunier <nanog () maunier org>
> > Cc: "nanog () nanog org" <nanog () nanog org>
> > Subject: Re: TATA problems?
> > Message-ID: <7994AF08-0622-434F-974F-FC9269469176 () ukbroadband com>
> > Content-Type: text/plain; charset="us-ascii"
> >
> >
> > My 10.4r1.9 boxes died also but I saw interfaces go down whilst bgpd
> > seemed stable.
> >
> > --
> > Leigh
> >
> >
> > On 7 Nov 2011, at 15:34, "Pierre-Yves Maunier" <nanog () maunier org>
> wrote:
> > > 2011/11/7 Tom Hill <tom () ninjabadger net>
> > >
> > > > On Mon, 2011-11-07 at 10:00 -0500, Todd Snyder wrote:
> > > > > We seem to be having some problems with our tata links - first
> seen in
> > EU
> > > > > about 45 minutes ago, now we're seeing problems in NA.  I'm
> focused on
> > > > DNS,
> > > > > so I'm seeing a lot of timeouts/servfails, but our networking
> folks
> > are
> > > > > talking about links dropping.
> > > > >
> > > > > Anyone else seeing oddness on the NA Internet right now?
> > > > >
> > > > > http://downrightnow.com/ confirms - something is up.
> > > >
> > > > There are widespread issues across the Internet; certain versions
> of
> > > > Juniper firmware have core dumped after seeing a particular BGP
> > 'UPDATE'
> > > > message.
> > > >
> > > > (That's the running theory at least).
> > > >
> > > > It's affected multiple service providers, globally, not just those
> > > > connected to TATA.
> > > >
> > > > Tom
> > > On our side all our 10.3R2.11 core dumped which made all our
> interfaces
> > > flapped.
> > > I've been told 10.4R1.9 is affected too.
> > >
> > > --
> > > Pierre-Yves Maunier
> ______________________________________________________________________
> > > This email has been scanned by the MessageLabs Email Security
> System.
> > > For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
> > ______________________________________________________________________
> > This email has been scanned by the MessageLabs Email Security System.
> > For more information please visit http://www.messagelabs.com/email
> > ______________________________________________________________________
> >
> >
> >
> > ------------------------------
> >
> > Message: 7
> > Date: Mon, 7 Nov 2011 09:54:25 -0600 (CST)
> > From: Joe Greco <jgreco () ns sol net>
> > To: ppauly () gmail com (Peter Pauly)
> > Cc: nanog () nanog org
> > Subject: Re: Time Warner Telecom problems
> > Message-ID: <201111071554.pA7FsPHb045359 () aurora sol net>
> > Content-Type: text/plain; charset=us-ascii
> >
> > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> > suffering
> > > from it too and calls to the NOC have not been answered so far...
> does
> > > anyone have any further information?
> > >
> > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> >
> > Actually, it looks to me like they mean "Time Warner", because that's
> > what they said.
> >
> > The company once known as "Time Warner Telecom" has always been a
> > different entity, and hasn't been known as that in some time, now
> > being called "twtelecom."  Much of that company is what was once
> > known as inc.net, a Milwaukee area provider of the '90's.
> >
> > Time Warner Cable appears to have experienced an implosion this
> morning,
> > being out of service for about 11 minutes.  During that time, packets
> > originating here in Milwaukee quickly died in Chicago;
> >
> > 1  76.46.192.1  8.320 ms  9.900 ms  7.974 ms
> > 2  24.160.230.32  7.967 ms  5.975 ms  8.479 ms
> > 3  24.160.229.132  8.471 ms  7.969 ms  10.991 ms
> > 4  24.160.229.193  9.972 ms  9.973 ms
> >    24.160.229.197  9.985 ms
> > 5  * * *
> > 6  * * *
> >
> > while packets destined for RR all seemed to be headed out to SJC, from
> > what I can tell.
> >
> > ... JG
> > --
> > Joe Greco - sol.net Network Services - Milwaukee, WI -
> http://www.sol.net
> > "We call it the 'one bite at the apple' rule. Give me one chance [and]
> > then I
> > won't contact you again." - Direct Marketing Ass'n position on e-mail
> > spam(CNN)
> > With 24 million small businesses in the US alone, that's way too many
> > apples.
> >
> >
> >
> > ------------------------------
> >
> > Message: 8
> > Date: Mon, 7 Nov 2011 07:55:33 -0800
> > From: Kelly Kane <kelly () hawknetworks com>
> > To: Tim Vollebregt <tim () interworx nl>
> > Cc: nanog () nanog org
> > Subject: Re: TATA problems?
> > Message-ID:
> <CAKfXD0zrH9TJkN=7doGe1uAq8zxVszyvtJYfLLTbiC6+UMy4ww () mail gmail com>
> > Content-Type: text/plain; charset=UTF-8
> >
> > On Mon, Nov 7, 2011 at 07:06, Tim Vollebregt <tim () interworx nl> wrote:
> > > On #IX there are rumours about Junos version 10.3R2.11 being core
> dumped
> > and
> > > rebooted, which makes sense.
> >
> > Perhaps related to Juniper PSN-2011-08-327? Did the whole router
> > reboot, or just the service module?
> >
> > We saw one TATA session, and one Abovenet session flap.
> >
> > Kelly
> >
> >
> >
> > ------------------------------
> >
> > Message: 9
> > Date: Mon, 07 Nov 2011 10:02:13 -0600
> > From: Blake Hudson <blake () ispn net>
> > To: nanog () nanog org
> > Subject: Re: Time Warner Telecom problems
> > Message-ID: <4EB80105.8060703 () ispn net>
> > Content-Type: text/plain; charset=UTF-8; format=flowed
> >
> >
> > Joe Greco wrote the following on 11/7/2011 9:54 AM:
> > > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> > suffering
> > > > from it too and calls to the NOC have not been answered so far...
> does
> > > > anyone have any further information?
> > > >
> > > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> > > Actually, it looks to me like they mean "Time Warner", because
> that's
> > > what they said.
> > >
> > > The company once known as "Time Warner Telecom" has always been a
> > > different entity, and hasn't been known as that in some time, now
> > > being called "twtelecom."  Much of that company is what was once
> > > known as inc.net, a Milwaukee area provider of the '90's.
> > >
> > > Time Warner Cable appears to have experienced an implosion this
> morning,
> > > being out of service for about 11 minutes.  During that time,
> packets
> > > originating here in Milwaukee quickly died in Chicago;
> >
> > Using the looking glass from TWtelecom, we saw 30-60min outage
> (roughly
> > 8:30AM to 9:30AM CST) between the Kansas City location and our own
> > server room in Kansas City. Other TWtelecom locations appeared to be
> > unaffected. Perhaps TWtelecom is served by Timewarner or shares
> > equipment in KC. Either way, none of our KC customers who were served
> > via TWtelecom or Timewarner were able to reach us. Packets would hit
> > Level 3 Communications and die in either direction at the border
> between
> > L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
> > http://lglass.twtelecom.net/
> >
> >
> >
> > ------------------------------
> >
> > Message: 10
> > Date: Mon, 7 Nov 2011 11:04:59 -0500
> > From: "Thomas York" <straterra () fuhell com>
> > To: "'Blake Hudson'" <blake () ispn net>,    <nanog () nanog org>
> > Subject: RE: Time Warner Telecom problems
> > Message-ID: <c23401cc9d67$046baea0$0d430be0$@fuhell.com>
> > Content-Type: text/plain;    charset="utf-8"
> >
> > FWIW, We saw issues here in Indianapolis between TWTC and L3 up until
> a
> > few minutes ago.
> >
> > --Thomas York
> >
> > -----Original Message-----
> > From: Blake Hudson [mailto:blake () ispn net]
> > Sent: Monday, November 07, 2011 11:02 AM
> > To: nanog () nanog org
> > Subject: Re: Time Warner Telecom problems
> >
> >
> > Joe Greco wrote the following on 11/7/2011 9:54 AM:
> > > > Gizmodo is reporting problems at Time Warner Telecom .... we're
> > > > suffering from it too and calls to the NOC have not been answered
> so
> > > > far...  does anyone have any further information?
> > > >
> > > > http://gizmodo.com/5857010/massive-time-warner-outage-hits-the-us
> > > Actually, it looks to me like they mean "Time Warner", because
> that's
> > > what they said.
> > >
> > > The company once known as "Time Warner Telecom" has always been a
> > > different entity, and hasn't been known as that in some time, now
> > > being called "twtelecom."  Much of that company is what was once
> known
> > > as inc.net, a Milwaukee area provider of the '90's.
> > >
> > > Time Warner Cable appears to have experienced an implosion this
> > > morning, being out of service for about 11 minutes.  During that
> time,
> > > packets originating here in Milwaukee quickly died in Chicago;
> >
> > Using the looking glass from TWtelecom, we saw 30-60min outage
> (roughly
> > 8:30AM to 9:30AM CST) between the Kansas City location and our own
> server
> > room in Kansas City. Other TWtelecom locations appeared to be
> unaffected.
> > Perhaps TWtelecom is served by Timewarner or shares equipment in KC.
> > Either way, none of our KC customers who were served via TWtelecom or
> > Timewarner were able to reach us. Packets would hit Level 3
> Communications
> > and die in either direction at the border between
> > L3 and TW. FWIW, TW was showing a good BGP route to us and vise versa.
> > http://lglass.twtelecom.net/
> >
> >
> >
> >
> >
> > End of NANOG Digest, Vol 46, Issue 15
> > *************************************