Re: Accidental ARIN Reallocation

[John Curran via NANOG <nanog () lists nanog org>]

Brian –

I actually just replied to Jon’s email with very much the same sentiment – this was not a “simple error”; 
it was an incident that revealed ARIN has operated with manual processes for management of 4.10 
address space that lacked appropriate controls to catch human error. That’s a very serious problem, 
and we have promptly remedied it with appropriate review and cross-validation.

We will be reviewing this incident and our broader registry change management practices with the ARIN 
Board of Trustees, and I will note your suggestions regarding an independent audit [1] and looking to more
formal, documentation-driven models (i.e. aerospace) for consideration.  

Thanks,
/John

John Curran
President and CEO
American Registry for Internet Numbers


[1] Note that ARIN does periodically engage an independent firm to conduct a full review and audit of the 
     Registration Services Department’s processes and procedures <https://www.arin.net/about/corporate/rsd_audits/>
     but that is focused on ensuring that registry changes are made consistent with the policies described in the 
     Number Resource Policy Manual (NRPM) – as opposed to your suggestion which implies an audit that is 
     done against best practices in change management. 


> On Dec 14, 2025, at 11:12 PM, Brian Russo via NANOG <nanog () lists nanog org> wrote:
>
> Sounds like you’re treating this as a simple error when the root cause is a
> lack of change management practices.
>
> Having a supervisor approve something is meaningless if there is no actual
> cross-validation of the change and associated data sources.
>
> Frankly I agree with others that you are likely not treating this seriously
> enough. I’ll reiterate what others have said - your whole literal purpose
> is to manage these assignments. You failed at your fundamental mission.
>
> I would recommend an independent audit and review of practices, as well as
> seeking inspiration from documentation-driven aerospace maintenance
> practices.
>
> - bri
>
> On Sun, Dec 14, 2025 at 16:08 John Sweeting via NANOG <nanog () lists nanog org>
> wrote:
>
> > See inline (ARIN)
> >
> > John S.
> >
> > From: Jon Lewis via NANOG <nanog () lists nanog org>
> > Date: Sunday, December 14, 2025 at 6:55 PM
> > To: John Curran via NANOG <nanog () lists nanog org>
> > Cc: John Curran <jcurran () arin net>, Jon Lewis <jlewis () lewis org>
> > Subject: Re: Accidental ARIN Reallocation
> >
> > On Fri, 12 Dec 2025, John Curran via NANOG wrote:
> >
> > > Short version – ARIN failed here (as you noted in your post). We’ve
> > > published a public incident report that lays out what happened, the
> > > impact, and what we’re changing:
> > > https://www.arin.net/announcements/20251212/
> >
> > This is a pretty epic failure considering ARIN's purpose is the assignment
> > of unique Internet numbers (and the necessary record keeping to facilitate
> > that function).
> >
> > I assume 23.128.0.0/10 records are maintained partially in the "off line
> > Excel file" because your primary system lacks necessary features to
> > differentiate free space / sparse allocations in 23.128.0.0/10 from
> > "regular" free space?  I assume the sparse allocation strategy here is to
> > allow a member to come back for more 4.10 space and ideally just extend an
> > original /24 to a /23, perhaps getting the next /24 eventually, and
> > finally swap the adjacent /23 and /24 for the covering /22?
> >
> > (ARIN) correct, and the community developed policy requires sparse
> > allocation. From the policy
> > "A /24 will be allocated. ARIN should use sparse allocation when possible
> > within that /10 block. “
> >
> > Things I'm curious about that are not mentioned in the incident report:
> >
> > 1) When the analyst looked in the e-black-book and selected 23.150.164.0
> >    (23.150.164.0/22, I presume) to be used to satisfy the request they
> >    were working on, did they fail to see that this /22 was already in use
> >    for a sparse assignment and 23.150.164.0/24 was already assigned, or
> >    when 23.150.164.0/24 was originally assigned to AS397031, was the
> >    e-black-book not properly updated to reflect the sparse assignment of
> >    23.150.164.0/22?
> >
> > (ARIN) The e-black-book (spreadsheet) was not properly updated which led
> > to the analyst selecting that block.
> >
> > 2) When assigning IP space, is it customary for the analyst to check
> >    current/recent snapshots of the global BGP tables to see if the space
> >    selected for assignment is currently or has recently been advertised?
> >    It's not guaranteed that assigned space will be in the DFZ, but if it
> >    is, that's a pretty good indicator that something is going wrong with
> >    the current assignment of the space.
> >
> > (ARIN) Yes the steps call for the analyst to check for routing, this step
> > was missed.
> >
> > 3) Did the block split automatically delete any/all child subnets of
> >    23.150.164.0/22, or did the analyst manually delete 23.150.164.0/24
> >    from whois (and that automatically deleted the ROA and rDNS
> >    delegation)?
> >
> > (ARIN) The split, in this case, returned 23.150.164.0/24, 23.150.165.0/24
> > and 23.150.166.0/23 so the analyst selected 23.150.164.0/24 for issuance.
> > The next step is to delete it from the ARIN OrgID in order to have it
> > available to allocate to the customer. The fact that it was assigned to
> > OrgID GL-700 versus OrgID ARIN was missed.
> >
> > (ARIN) We have inserted a supervisor mandatory check and approval in the
> > process prior to the delete action being initiated.
> >
> >
> > ----------------------------------------------------------------------
> >  Jon Lewis, MCP :)              |  I route
> >  Blue Stream Fiber, Sr. Neteng  |  therefore you are
> > _________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
> > _______________________________________________
> > NANOG mailing list
> >
> > https://lists.nanog.org/archives/list/nanog () lists nanog org/message/X3WPNTJZUNDJMAOOJ5F3JKI4UAT65C3L/
> > _______________________________________________
> > NANOG mailing list
> >
> > https://lists.nanog.org/archives/list/nanog () lists nanog org/message/6AC5KQD26PVHCN4T5HAFECIZWI67NJX4/
> _______________________________________________
> NANOG mailing list 
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/65OO7CNHXJ2MUBWRGVLRLQF5HXASJ44E/

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/5MEDPCFLWBRA3BNXBHEGLZUMRFUFS5KZ/