nanog mailing list archives
Re: cms.ups.com lame delegation
From: Cuenta Uno via NANOG <nanog () lists nanog org>
Date: Tue, 10 Nov 2026 13:27:53 -0500
What caused the BGP routing changes observed in Ethiopia on November 10, 2026? <https://networkengineering.stackexchange.com/questions/88980/what-caused-the-bgp-routing-changes-observed-in-ethiopia-on-november-10-2026> https://routeviews.cloud/graph2/groups/20261110.0200_WIDE_OUTAGE.ET.4.0221/196.189.112.0_20/20261110.0200/2 I am analyzing BGP routing data from RouteViews and noticed unusual routing changes affecting networks in Ethiopia on November 10, 2026. I would like to understand whether these changes indicate a real network disruption, a routing policy change, or a measurement artifact. What BGP indicators would help distinguish these possibilities? El mar, 10 nov 2026 a la(s) 11:37 a.m., Alex Buie via NANOG ( nanog () lists nanog org) escribió:
Hi all,
Posting in case this saves someone a ticket, and in the hope that
someone from UPS DNS or Cloudflare's resolver team is reading.
TL;DR
- cms.ups.com is delegated to six nameservers. Only two of them
(nsa.ups.com, nsb.ups.com) actually serve the zone. The other four
(auth1/auth2.dns.cogentco.com, cmtu.mt.ns.els-gms.att.net,
cbru.br.ns.els-gms.att.net) are authoritative for ups.com but
answer anything under cms.ups.com with a referral back to the same
six NS.
- Google (8.8.8.8), Quad9 (9.9.9.9) and OpenDNS (208.67.222.222) cope
with it. Cloudflare (1.1.1.1, 1.0.0.1, 1.1.1.2, 1.1.1.3) returns
NOERROR with ANSWER 0, AUTHORITY 0, no SOA and no EDE.
- What users see: UPS WorldShip can't resolve www.ws.cms.ups.com (a
CNAME to wwwwscms.dmz-gslb.ups.com) and fails with DNS error -53670
/ WinHTTP 12007.
- The CNAME has a 7-day TTL, so Cloudflare answers correctly whenever
the cache you hit already has it. That makes it look intermittent.
Uncached names under cms.ups.com failed on Cloudflare every time we
tried.
How we found it
A customer's WorldShip install stopped sending. Their local forwarder
(dnsmasq) had a Cloudflare resolver and Quad9 as upstreams. An empty
NOERROR looks like a legitimate NODATA answer, so dnsmasq passed it
to the client and never tried Quad9. A conditional forward got them
working, but the actual problem is upstream of all of us.
Everything below was run from one vantage point in the eastern US on
2026-11-10 between 16:22 and 16:25 UTC, using dig 9.10.6.
1. The lame delegation
The same query with recursion off, sent to each listed nameserver:
dig @<ns> www.ws.cms.ups.com A +norec
nsa.ups.com 153.2.242.115 aa, CNAME
nsb.ups.com 153.2.244.155 aa, CNAME
auth1.dns.cogentco.com 66.28.0.14 no aa, referral to same 6
auth2.dns.cogentco.com 66.28.0.30 no aa, referral to same 6
cmtu.mt.ns.els-gms.att.net 99.99.99.132 no aa, referral to same 6
cbru.br.ns.els-gms.att.net 68.94.156.132 no aa, referral to same 6
A good server:
$ dig @153.2.242.115 www.ws.cms.ups.com A +norec
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 59834
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
www.ws.cms.ups.com. 604800 IN CNAME wwwwscms.dmz-gslb.ups.com.
A lame server:
$ dig @66.28.0.14 www.ws.cms.ups.com A +norec
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62565
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 1
;; AUTHORITY SECTION:
cms.ups.com. 3600 IN NS cmtu.mt.ns.els-gms.att.net.
cms.ups.com. 3600 IN NS auth2.dns.cogentco.com.
cms.ups.com. 3600 IN NS nsb.ups.com.
cms.ups.com. 3600 IN NS nsa.ups.com.
cms.ups.com. 3600 IN NS cbru.br.ns.els-gms.att.net.
cms.ups.com. 3600 IN NS auth1.dns.cogentco.com.
All four lame servers answer ups.com SOA with aa set, so they are
secondaries for the parent that carry the cms.ups.com delegation but
not the child zone. nsa/nsb serve cms.ups.com SOA serial 2023100422
<(202)%20310-0422>
and answer over both UDP and TCP. The referral is properly signed
(NSEC3 proof of no DS for cms.ups.com), so this is an ordinary
insecure delegation and DNSSEC isn't the problem.
Side note: the Cogent and AT&T secondaries also return different
ups.com SOA serials (2023542354 <(202)%20354-2354> vs 2023541845
<(202)%20354-1845>).
2. Resolver comparison: www.ws.cms.ups.com A, 5 queries each
resolver good empty
1.1.1.1 4 1
1.0.0.1 5 0
1.1.1.2 3 2
1.1.1.3 2 3
8.8.8.8 5 0
9.9.9.9 5 0
208.67.222.222 5 0
When Cloudflare did answer, the CNAME TTL was about 604360, so some
Cloudflare cache had picked up the 604800-second record a few
minutes earlier. That's probably why this "works for me" for some
people.
The same resolver about a minute apart, cached and not:
$ dig @1.1.1.1 www.ws.cms.ups.com A
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11500
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; ANSWER SECTION:
www.ws.cms.ups.com. 604373 IN CNAME wwwwscms.dmz-gslb.ups.com
.
wwwwscms.dmz-gslb.ups.com. 5 IN A 153.2.73.86
$ dig @1.1.1.1 www.ws.cms.ups.com A
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56866
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.ws.cms.ups.com. IN A
;; Query time: 143 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; MSG SIZE rcvd: 47
The empty reply has no SOA in the authority section and no Extended
DNS Error option. The same dig does show Cloudflare's EDE on other
failures: dnssec-failed.org comes back SERVFAIL with OPT=15 (EDE 9).
3. Cache-busting: random nonexistent names under cms.ups.com
The real zone answers these with NXDOMAIN and an SOA. Each query used
a fresh label (nanog-test-<8 random hex>.cms.ups.com), so no resolver
could have it cached.
resolver result (5 queries each)
1.1.1.1 NOERROR, ANSWER 0, no SOA x5
1.0.0.1 NOERROR, ANSWER 0, no SOA x5
1.1.1.2 NOERROR, ANSWER 0, no SOA x5
8.8.8.8 NXDOMAIN + SOA x5
9.9.9.9 NXDOMAIN + SOA x5
208.67.222.222 NXDOMAIN + SOA x5
Five more against 1.1.1.1 were also empty NOERROR (20 of 20). With
+cd, 4 of 5 were still empty and 1 returned NXDOMAIN, so turning off
validation doesn't fix it.
If Cloudflare chose among the six servers at random, we'd expect
about a third of uncached lookups to reach nsa/nsb and succeed. We
saw none in 20. From outside we can't tell why.
4. Reproduce it yourself
# lame vs good authoritative
dig @66.28.0.14 www.ws.cms.ups.com A +norec
dig @153.2.242.115 www.ws.cms.ups.com A +norec
# uncached name, Cloudflare vs Quad9
n=test-$RANDOM$RANDOM.cms.ups.com
dig @1.1.1.1 $n A +noall +comments | grep status
dig @9.9.9.9 $n A +noall +comments | grep status
Asks
- UPS DNS folks: please either load cms.ups.com on the Cogent and
AT&T secondaries or remove them from the cms.ups.com delegation.
- Cloudflare resolver folks: when every referral for a zone cut
points back at itself, could 1.1.1.1 keep trying the remaining NS,
or at least return SERVFAIL with an EDE (22, "No Reachable
Authority", seems to fit) instead of an empty NOERROR? Forwarders
like dnsmasq treat NOERROR/NODATA as final and won't fail over to
another upstream.
- Everyone else: if you get WorldShip tickets from sites that use
Cloudflare DNS, this is probably the cause. Until UPS fixes the
delegation, forwarding cms.ups.com to any other resolver works,
e.g. in dnsmasq:
server=/cms.ups.com/9.9.9.9
Happy to share more output off-list.
*Alex Buie*Principal Network Engineer
7300 State Highway 121, Suite 800
McKinney, Texas 75070
D: 469-884-0225 <(469)%20884-0225> | www.cytracom.com
_______________________________________________
NANOG mailing list
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/Z5FSEH25UCV5C7GAEU2GDLXZFH6C5S7N/
_______________________________________________ NANOG mailing list https://lists.nanog.org/archives/list/nanog () lists nanog org/message/3KBDLX2CQDOFYE3LXE5JJNICEWRS4V2H/
Current thread:
- cms.ups.com lame delegation Alex Buie via NANOG (Nov 10)
- Re: cms.ups.com lame delegation Cuenta Uno via NANOG (Nov 10)