Accidental ARIN Reallocation

[Sylvain BAYA via NANOG <nanog () lists nanog org>]

Le 15/12/2025 à 06:48, Hank Nussbacher via NANOG a écrit :
> On 15/12/2025 6:55, John Curran via NANOG wrote:
>
> A masterclass in owning a mistake and handling it properly.

Hi Hank,
Fully agreed! brother, it's where the difference
stands: PmR -- Post-mortem Report!

Incidents are common things; PmR is BCOP
(Best Operational Practice)...it's still rare :'-(

...and this one gives us an opportunity to
ameliorate the BCOP template [*]; imho.
__
[*]: dns-post-mortem <https://archive.psg.com/240614.dns-post-mortem>

...i humbly suggest to John and team, to accept
to update the structure of their PmR, so that,
in its next version, it takes into account all the
feedback/suggestions received on topic.

What could be mine?
That file should be version-controlled & must
follow a staging step prior to be committed...
...imho. It's practically what they decided until
they are able to fully integrate it into normal
pipelines.

...what to do during the staging: cross-check
using some API endpoints such as:

  * Related prefixes 
<https://stat.ripe.net/docs/data-api/api-endpoints/related-prefixes/>
  * Prefix overview 
<https://stat.ripe.net/docs/data-api/api-endpoints/prefix-overview/>
  * RIS first last seen 
<https://stat.ripe.net/docs/data-api/api-endpoints/ris-first-last-seen/>
  * Routing status 
<https://stat.ripe.net/docs/data-api/api-endpoints/routing-status/>
  * RPKI history 
<https://stat.ripe.net/docs/data-api/api-endpoints/rpki-history/>
  * Routing history 
<https://stat.ripe.net/docs/data-api/api-endpoints/routing-history/>
  * Prefix routing consistency 
<https://stat.ripe.net/docs/data-api/api-endpoints/prefix-routing-consistency/>
  * RPKI validation 
<https://stat.ripe.net/docs/data-api/api-endpoints/rpki-validation/>
  * Reverse DNS consistency 
<https://stat.ripe.net/docs/data-api/api-endpoints/reverse-dns-consistency/>
  * 
<https://stat.ripe.net/docs/data-api/api-endpoints/related-prefixes/>Looking 
glass <https://stat.ripe.net/docs/data-api/api-endpoints/looking-glass/>
  * AS routing consistency 
<https://stat.ripe.net/docs/data-api/api-endpoints/as-routing-consistency/>
  *
  * Easy BGP Monitoring with BGPalerter | RIPE Labs 
<https://labs.ripe.net/author/massimo_candela/easy-bgp-monitoring-with-bgpalerter/>
  * An Introduction to IRR Explorer | RIPE Labs 
<https://labs.ripe.net/author/teun/an-introduction-to-irr-explorer/>
  * BGPAlerter | Loud Networking 
<https://loudnetworking.org/2021/07/28/bgpalerter/>
  * .


Shalom,
--sb.


> Regards,
> Hank
>
> > > On Dec 14, 2025, at 6:54 PM, Jon Lewis via NANOG 
> > > <nanog () lists nanog org> wrote:
> > >
> > > On Fri, 12 Dec 2025, John Curran via NANOG wrote:
> > >
> > > > Short version – ARIN failed here (as you noted in your post). We’ve 
> > > > published a public incident report that lays out what happened, the 
> > > > impact, and what we’re changing: 
> > > > https://www.arin.net/announcements/20251212/
> > > This is a pretty epic failure considering ARIN's purpose is the 
> > > assignment of unique Internet numbers (and the necessary record 
> > > keeping to facilitate that function).
> > Jon –
> >
> > I agree completely. This was a failure of ARIN in the performance of 
> > its core mission, and one that resulted in customer impact. The 
> > community is entitled to full transparency in understanding how this 
> > occurred and the steps we have taken to prevent any similar incident 
> > in the future.
> >
> > Analysts have performed this particular allocation process thousands 
> > of times previously without issue, but in this case a resource 
> > analyst made an error that resulted in the reallocation of a 
> > previously assigned NRPM 4.10 address block. Clearly, while that was 
> > the trigger for this incident, the real fault here is that ARIN 
> > should not have any processes that are predicated on perfect human 
> > performance. Prior to this incident, it was my belief that this was 
> > already the case and that assigned resource blocks could not be 
> > impacted by analyst error.
> >
> > That is not the case for the manual processes used in the management 
> > of NRPM 4.10 address blocks. As a result, we have corrected the 
> > process to require a second set of eyes before any change is 
> > committed. Longer term, I prefer to fully automate this process, but 
> > until that can be implemented we will continue with the manual 
> > process, as amended with a mandatory supervisor confirmation step, as 
> > a reasonable and appropriate mitigation.
> >
> > I have experience running several major ISPs and am fully aware that 
> > operators rely on ARIN for flawless performance. Even a single 
> > customer impact is not acceptable, which is why we issued the report 
> > to the community detailing the incident and its resolution. To the 
> > extent that there is any need for additional clarity, please don’t 
> > hesitate to ask – either here on the list (or to me directly as you 
> > prefer.)
> >
> > Thanks,
> > /John
> >
> > John Curran
> > President and CEO
> > American Registry for Internet Numbers
> >
> > _______________________________________________
> > NANOG mailing list
> > https://lists.nanog.org/archives/list/nanog () lists nanog org/message/2F2TMDVELVVFUX6MT6A3566CEOMKLMN4/ 
>
>
> _______________________________________________
> NANOG mailing list 
> https://lists.nanog.org/archives/list/nanog () lists nanog org/message/U33VXD6K4YROJH7T7AQHQYYOCMZWMJVQ/

--

Best Regards !

baya.sylvain [AT cmNOG DOT cm] |
cmNOG's Structure <https://www.cmnog.cm/dokuwiki/Structure> | CAMIX's 
Website 
<https://web.archive.org/web/20240706030230/http://www.camix.cm/> | 
Douala-IX's Looking Glass <https://tools.std.douala-ix.net/lg> |
cmNOG's Surveys <https://survey2.cmnog.cm/> | Subscribe to cmNOG's 
Mailing List <https://lists.cmnog.cm/mailman/listinfo/cmnog> |
__
#LASAINTEBIBLE|#Ephésiens3:17,14-19«17 en sorte que CHRIST habite dans 
vos cœurs par la foi; afin qu'étant enracinés et fondés dans 
L'Amour,»#AMEN,#Maranatha,#MerciJÉSUS!
#‎MaPrière‬ est que tu naisses de nouveau.#Chrétiennement‬

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature

_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/GFL2NVMOK3QS43KTKFUIJ442NF4MBFJT/