RE: Cisco ASR9902 SNMP polling ... is interesting

[Drew Weaver via NANOG <nanog () lists nanog org>]

One other quick note for the zero people that are interested.

It seems like LPTS is at best half implemented in XR 24 as it seems to have no problem dynamically generating its own 
<<totally hidden>> configuration for configured BGP sessions, etc but it can't look at the fact that there is an ACL on 
the snmp-community and simply white list the IPs attached to that ACL in the LPTS/CoPP policer?

It can't throw a log entry on boot that says "Hi most likely we're just silently discarding traffic due to the default 
LPTS policer".

It can't further go on to specifically say "Hey we know you didn't configure this but we noticed that the SNMP policer 
is dropping like 60% of the requests...."

By default there isn't even a control-plane configuration in IOS XR 24

show run control-plane
Wed Aug  6 13:44:59.244 EDT
% No such configuration item(s)

I bet the smart licensing daemon gets a real nice carveout in the default LPTS policer configuration. Its very 
important that smart licensing never fails to function but basic network telemetry isn’t that important.

Thanks,
-Drew









-----Original Message-----
From: Drew Weaver via NANOG <nanog () lists nanog org> 
Sent: Wednesday, August 6, 2025 1:28 PM
To: 'North American Network Operators Group' <nanog () lists nanog org>
Cc: 'LJ Wobker (lwobker)' <lwobker () cisco com>; Drew Weaver <drew.weaver () thenap com>
Subject: RE: Cisco ASR9902 SNMP polling ... is interesting

Hi there,

It has since been identified that the reason that the traffic is being dropped is the SNMP policer in LPTS seems to 
just be discarding the traffic.

I didn't configure it to do this.
This doesn't show up in the running configuration TAC still hasn't figured that out yet and they still haven't provided 
me with a way to simply whitelist traffic from a single /32 in LPTS 4 weeks later.

So yes, I will admit that I am somewhat ignorant on what you guys call CoPP in this platform but I don't think me being 
ignorant about it is as big as problem as TAC being fully unaware that it exists at all.

Still waiting for TAC to tell me how to whitelist a single /32 in the policer.

In 9 more weeks I'll let you know what the result ends up being.

Thanks though for stopping by.
-Drew



-----Original Message-----
From: LJ Wobker (lwobker) via NANOG <nanog () lists nanog org>
Sent: Tuesday, August 5, 2025 11:46 AM
To: North American Network Operators Group <nanog () lists nanog org>
Cc: LJ Wobker (lwobker) <lwobker () cisco com>
Subject: RE: Cisco ASR9902 SNMP polling ... is interesting

Wow, what a food fight this became.  At risk of wading into the middle school cafeteria and wearing ketchup, I'll 
attempt to possibly return to some semblance of a technical discussion.  For background, I was the first TME here at 
Cisco who worked on the ASR9k program back in the mid-2000s - so my memory might be a bit rusty but at least to some 
degree I can present myself as a knowledgeable source.  I also worked in TAC back in the day so I have some familiarity 
with their processes.  ;-)

In all IOS XR systems, there's an architecture designed to make sure that control plane traffic coming from the very 
high speed interfaces doesn't overwhelm the processing capacity of the system.  The whole thing is relatively complex 
and the exact implementation differs from system to system in the fine details, but the idea is that you want to funnel 
down traffic headed for the RP or linecard CPU so that by the time it gets there you're as confident as you can be that 
the traffic is legitimate and in the right place.

No one uses the same terms for anything, so some terminology...  We (cisco) broadly call the infrastructure "LPTS":  
Local Packet Transport Service.  The act of identifying that a packet needs to go up to the control plane we call 
"punting".  Every modern system from every vendor has SOME form or fashion for this, otherwise it's trivial to melt the 
system with traffic pointed at the control CPU.  But no one uses the same words.

Drew - I'm sorry you don't like the way my router works.  This hurts my feelings, because he's really a pretty good 
little router.  Let's see if we can figure out why.  In this case, there's lots of possible places things can behave in 
ways you don't like.

First question... when you say "we poll SNMP on any interface" -- do you mean you're changing the target IP address for 
where you point the SNMP manager, where sometimes it's the management ethernet address and sometimes a regular 
interface address?  This matters because IN GENERAL (yes, I know...) the system behaves differently here.  Packets 
pointed at the management ethernet are run through a different set of policers than if you're pointed at a data plane 
interface.  IN GENERAL the "best" way to do something like this is with a loopback interface, as the defaults are 
"better" tuned for that config compared to a direct zap at the actual interface IP.  This also has the benefit of 
virtualizing the loopback so you aren't tied to a single point of failure, but that's a separate thing.

I'm not remotely surprised that the behavior is different from the 9901 to the 9902.  At risk of being an apologist for 
my implementation, even within a product family there are always (sometimes stupid) differences in the implementations. 
 

I can ABSOLUTELY ASSURE you that there is nowhere in the code that says "make 62% of the SNMP polls fail because we 
hate Drew".  This is not how our system works... somewhere in the path there's a policer or a meter that is either 
dropping some of the inbound requests, or the SNMP process is choking on something and timing out, or something like 
that.  But there is no such thing on the router side as an SNMP polling timeout - that is a client side thing.  The 
SNMP process on the router gets a request, and it sends a response, that's all.  If something (either external or 
within the labyrinth of internal protections) drops the request on the way in, SNMP never sees it, so it can't respond. 
 Then the client has to figure out what to do, which often is throw a timeout and/or retry -- but this is dependent on 
the implementation of the SNMP client, and there's nothing that the router OS can do about it.

As someone mentioned along the way, the right way to troubleshoot this is to find the commands in XR that will show you 
the counters and potential drops between "the packet arrives at the box" and "SNMP did its thing with the packet".  I 
have to sadly admit that here I'm one of those old-ass Air Force Colonels who USED to be a hot-shit pilot, but now I 
fly a desk.  12 years ago I could have told you chapter and verse what the commands are and where all the drop/meter 
counters live, but father time is undefeated and now I spend time apologizing on NANOG lists instead of having an 
actual lab to work on.  That said, your expectation that someone in TAC can figure out what's happening and explain it 
to you is totally reasonable, and if you're not getting those answers then escalating is correct.  We might not be able 
(or willing) to change the behavior to do things the way you like them, but we absolutely owe you an explanation of 
what's actually happening.  If you can't this from TAC, let me know and I will attempt to shake that tree.

At LEAST the following things would need to be chased down, some of which we'd have to get from the customer side...
* which interface(s) are being polled?  MgmtEth, loopback, physical?  
* at what rate does the SNMP station generate and send request packets?  (Time windows matter here.  A short but very 
fast burst of requests might trip the meter, stuff like that)
* can this rate be changed?
* how much stuff (i.e. MIBs) are you polling? 

Anyway... hopefully that points you at least somewhat in the right direction.

--lj

-----Original Message-----
From: Mel Beckman via NANOG <nanog () lists nanog org>
Sent: Monday, August 4, 2025 10:42 AM
To: Tom Beecher <beecher () beecher cc>
Cc: nanog () lists nanog org; Mel Beckman <mel () beckman org>
Subject: Re: Cisco ASR9902 SNMP polling ... is interesting

Sorry, Tom. I’m not taking the bait.

-mel via cell

On Aug 4, 2025, at 7:02 AM, Tom Beecher <beecher () beecher cc> wrote:


Mel-

You have made multiple technical assertions in this thread that are demonstrably false. Quoting your earlier messages :

  1.  Also, non-management interfaces do packet processing in silicon at the ASIC level and don’t have the capacity to 
do anything more than statistical sampling of packets that require CPU-level processing to retrieve counters and 
generate SNMP responses. 62 % is as good a sampling rate as any other.
  2.  Cisco is likely to say that the control plane is only fully supported on the management port.
  3.  In-band SNMP to data forwarding interfaces violates that separation.

 You have attempted to frame these comments as :

honest and sincere attempts by other members to help identify the possible problem.

While your attempts to help may have been honest and sincere attempts to help the OP, they actually achieved the 
opposite effect. Your incorrect technical assertions , if anything, only hindered the OP's attempt to understand and 
identify their issue. Comment #1 is especially egregious ; you're telling Drew that his observations are *normal*.

Saku made 2 comments that addressed these falsehoods :

It might be easier to contribute, if there is familiarity to the subject matter.

some community member piled on with what can only be described as a bizarre drivel.

The first was a polite way of calling out the technical inaccuracies. The second was a more forceful way of stating 
"what you said was wrong". Most people, when they are corrected on a factual point, tend to reply with "Oh hey, I got 
that wrong, thanks for setting me straight" and move on. You seem to have just ignored it.

There is a massive difference between the following statements :

  1.  You are an idiot. [ Attacking the person ]
  2.  What you said was idiotic. [ Attacking the statements ]

It seems to be that you may be struggling in identifying that difference, and taking *any* criticism as a personal 
attack.

Nobody is bullying you, or anybody else, in this conversation.





On Mon, Aug 4, 2025 at 9:42 AM Mel Beckman via NANOG <nanog () lists nanog org<mailto:nanog () lists nanog org>> wrote:
Thanks. I knew we were not so out to lunch! If you don’t push back on bullies, they take over the community. It crops 
up on nanog periodically. :(

-mel via cell

> On Aug 4, 2025, at 5:54 AM, Joe Loiacono via NANOG <nanog () lists nanog org<mailto:nanog () lists nanog org>> wrote:
>
> Hi Mel, for what it's worth, I could not figure out what they were 
> referring to by Saku's comments. I saw no justification for their 
> complaint. A bit out of character for Saku, also,
>
> Joe
>
> > On 8/2/2025 7:23 PM, Mel Beckman via NANOG wrote:
> > I’ll just let the incivility of you both stand.
> >
> > -mel
> >
> > On Aug 2, 2025, at 3:52 PM, Tom Beecher <beecher () beecher cc<mailto:beecher () beecher cc>> wrote:
> >
> > 
> > Mel-
> >
> > Saku did not call *you* any names. He called your *incorrect statements* in this thread 'bizzard drivel'. Which he 
> > is absolutely correct about. While your intentions may certainly have been to help, your statements here have been 
> > frankly dead wrong and did not accomplish that.
> >
> > Probably just want to take the L here.
> >
> >
> > On Sat, Aug 2, 2025 at 5:34 PM Mel Beckman via NANOG <nanog () lists nanog org<mailto:nanog () lists nanog 
> > org><mailto:nanog () lists nanog org<mailto:nanog () lists nanog org>>> wrote:
> > Saku,
> >
> > What is actually appalling is that a member of NANOG calls “bizarre drivel” the honest and sincere attempts by other 
> > members to help identify the possible problem. There’s no cause to be uncivil, people can disagree without stooping 
> > to name-calling.
> >
> >  -mel
> >
> > > > On Aug 2, 2025, at 11:46 AM, Saku Ytti via NANOG <nanog () lists nanog org<mailto:nanog () lists nanog 
> > > > org><mailto:nanog () lists nanog org<mailto:nanog () lists nanog org>>> wrote:
> > >
> > > On Sat, 2 Aug 2025 at 21:02, Tom Beecher via NANOG 
> > > <nanog () lists nanog org<mailto:nanog () lists nanog org><mailto:nanog () lists nanog org<mailto:nanog () lists 
> > > nanog org>>> wrote:
> > >
> > > > I don't have in depth knowledge of Cisco's SNMP implementations, or 
> > > > even the ASR platform specifically, but if Cisco TAC is telling you 
> > > > this is 'normal', they are completely full of shit, and you should 
> > > > click any and every 'escalate' button you can find.
> > > >
> > > > This almost sounds like a default control plane DDOS policer / LPTS 
> > > > , something like that.
> > > There are various complicated reasons for this, LPTS policer is 
> > > unlikely culprit, but possible. Bug search will show various DDTS 
> > > with poor SNMP performance outcome, most of them are unrelated to LPTS.
> > >
> > > But absolutely correct, the right solution is to escalate. In common 
> > > case this would be SE from your account team, who would fight for 
> > > you internally.
> > >
> > >
> > > It is appalling that OP came to nanog after correctly suspecting TAC 
> > > is gaslighting them, some community member piled on with what can 
> > > only be described as a bizarre drivel.
> > > --
> > >  ++ytti
> > > _______________________________________________
> > > NANOG mailing list
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org
> > > _archives_list_nanog-40lists.nanog.org_message_&d=DwIGaQ&c=euGZstcaT
> > > DllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1
> > > o4LtA3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVb
> > > VNzC8h9aWfc&s=HVfyN6javj5uX9ryxhOPxQSiMh2CkQJi_x885vQNB0M&e=
> > > 7KXUNRGFI5OEVSDEDU2OL5VMY5NBGQCV/
> > _______________________________________________
> > NANOG mailing list
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_
> > archives_list_nanog-40lists.nanog.org_message_C&d=DwIGaQ&c=euGZstcaTD
> > llvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4
> > LtA3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVbVNz
> > C8h9aWfc&s=GtBCB1cT8FNf1-UD3vXAYH3UHRxLVcJgUO3WmSwt7a4&e=
> > F3QHVTISL6LDFTOWG4E3KK54QEDHUIY/
> > _______________________________________________
> > NANOG mailing list
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_
> > archives_list_nanog-40lists.nanog.org_message_O&d=DwIGaQ&c=euGZstcaTD
> > llvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4
> > LtA3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVbVNz
> > C8h9aWfc&s=5ODSQkzz8W9Kr3E9IWdoE9mLIm_bTb8Z0H9sSnuNKSs&e=
> > J7ICXLSPFND32X2XS2U7XIWA6DALSIF/
> _______________________________________________
> NANOG mailing list
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_a
> rchives_list_nanog-40lists.nanog.org_message_E4&d=DwIGaQ&c=euGZstcaTDl
> lvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4Lt
> A3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVbVNzC8h
> 9aWfc&s=8E7xCMB2-Jb4W7oWeB3GOFc7RFZYZYj3W5GlLeJX9BA&e=
> CF2TFV35VSJVFEZZANEWOAJFUUNDL4/
_______________________________________________
NANOG mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_archives_list_nanog-40lists.nanog.org_message_RU6WF77QOECXABP6IDCMVNLAH67X4WNW_&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVbVNzC8h9aWfc&s=7t0FidWZ-eOmjk9WDRw3h78TBRDLNkqVXdQ7GSVnrOc&e=
_______________________________________________
NANOG mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_archives_list_nanog-40lists.nanog.org_message_3NCOGL6SHARKHBT2TJRK4W7ZOP2BO2BW_&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVbVNzC8h9aWfc&s=M4KruocLeATFcohjqA5bbEtk4u9xNX0ZFyQt_OhItjM&e=
_______________________________________________
NANOG mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_archives_list_nanog-40lists.nanog.org_message_LE6LLRVDEOQK3R5JO3G3QSIRYYICRQIZ_&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=ZRQKyw0amYQuJDrOUoUtJCSVZKWvb764kPF4UjLJKuQ_I4NVhCMVbVNzC8h9aWfc&s=HK3eMuL_F8B7YRLvgGYzli-lx8Y-h6JZXJr7pNeDoCg&e=
_______________________________________________
NANOG mailing list
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.nanog.org_archives_list_nanog-40lists.nanog.org_message_PNYIWP2AYG6A5XAUWE2LB7ITRSWE4O5Y_&d=DwIGaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=OPufM5oSy-PFpzfoijO_w76wskMALE1o4LtA3tMGmuw&m=XqO0z1zusbA5mM-cjO6mJQBBYiF6tKyN3QbF_1TuXAbm1aecz0pA7ZBv4LoMY2xZ&s=epw_VYl-NVBHiYGuuePKpKAZPoMzcdYh1MJrXkDLoAI&e=
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/7YBOXDLYINDYBELQJ7UOWJ6INO4QXJV5/