Re: How much do you automate your automation?

["Patrick W. Gilmore via NANOG" <nanog () lists nanog org>]

Certain companies, e.g. hyperscalers, automate changes completely. An engineer / architect decides what do to, e.g. 
upgrade all the XXX routers to OS YYY. Then they hand that off to an operations team which uses a pre-written script 
(really much much much more than a “script” - frequently written by a third team) to tell the “upgrade all device type 
$FOO to OS $BAR”. At which point the system figures which devices get upgraded, separates the fleet into stages, 
decides when each device is touched, pre-drains, upgrades, verifies the upgrade, undrains, verifies traffic moved back, 
proceeds to next device, etc., with possible human ACKs required to move to the next stage or whatever other 
segmentation you like.

Obviously you can make things more specific, such as all device type $FOO in role $BAR, or in geography $BAT, or pretty 
much any other method you can dream up.

It is almost like computers are good at following a complex decision tree with lots of variables. Who knew?

Without this, networks deploying 10s of 1000s of devices could not survive. With it, you can scale the number of 
devices far more quickly than you scale the staff.

Oh, and you can also take down your whole network very very quickly. :-)

-- 
TTFN,
patrick

> On Apr 14, 2026, at 15:36, Jon Lewis via NANOG <nanog () lists nanog org> wrote:
>
> I've been told that at [some of] the largest networks, network engineers "never directly log into network devices".  
> This implies that all configuration changes made to and insights gleaned from the network gear are done via some form 
> of automation.
>
> I assume it's commonplace to have/use Unix CLI tools for executing configuration changes.  I've written such things 
> for the past couple of places I've worked so that we can literally copy&paste from a MOP to a shell session and have 
> a change implemented.  Such tools become extremely handy when you want to make the same change on a few or a few 
> hundred devices.
>
> What I'm wondering is, how common is it to take the next logical step and if you have a planned maintenance window to 
> implement some simple change, do you have an engineer manually make that change, manually execute a script that 
> implements the change, or use old-school automation (at) to schedule a date & time at which the script that 
> implements the change will be run, and optionally have an engineer monitor that the change happened and had the 
> intended results?
>
>
> ----------------------------------------------------------------------
> Jon Lewis, MCP :)              |  I route
> Blue Stream Fiber, Sr. Neteng  |  therefore you are
> _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
> _______________________________________________
> NANOG mailing list https://lists.nanog.org/archives/list/nanog () lists nanog 
> org/message/WKX5XT2LSEL4T3ZGFGRG7Q3KUAR54LVA/

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/HYA3XBFA56S2HYJULUFMBWYHUXSOCHNV/