Risks Digest 34.27

[RISKS List Owner <risko () csl sri com>]

RISKS-LIST: Risks-Forum Digest  Tuesday 28 May 2024  Volume 34 : Issue 27

ACM FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS (comp.risks)
Peter G. Neumann, founder and still moderator

***** See last item for further information, disclaimers, caveats, etc. *****
This issue is archived at <http://www.risks.org> as
  <http://catless.ncl.ac.uk/Risks/34.27>
The current issue can also be found at
  <http://www.csl.sri.com/users/risko/risks.txt>

  Contents:
Unions Raise Safety Concerns Over Remote-Controlled Trains
 (The New York Times)
'I was misidentified as shoplifter by facial recognition tech' (BBC)
Facebook account takeovers are targeting people you know, turning friendship
 into fraud (CBC)
What Does an AI Do When It Sees an Optical Illusion?
 (Scientific American)
AI-powered hate content is on the rise, experts say (Matthew Kruk)
The order in which data is fed to LLMs can make a big difference (PGN)
Windows Total "Recall" -- aka *keylogger* -- is security nightmare
Crowds Flocked to the New York-Dublin Livestream. Then Things Got Racy.
 (WSJ)
The Harsh Truth Behind Samsung's Phone Repair Program (Florence Ion)
Congress Just Made It Basically Impossible to Track Taylor Swift’s
 Private Jet (Gizmodo)
Elon Musk wants our help with a [minor|huge] problem. (Rob Slade)
Re: A woman was dragged by a self-driving Cruise taxi in San Francisco
 (Geoff Kuenning, Wol)
Re: Half of calls to gambling helpline were for help (Amos Shapir)
Re: I stumbled upon LLM Kryptonite and no one wants to fix it (Steve Bacher)
Re: MITRE ATLAS on obscurity (Jared Richo et al.)
Abridged info on RISKS (comp.risks)

----------------------------------------------------------------------

Date: Mon, 27 May 2024 14:53:13 -0400
From: Gabe Goldberg <gabe () gabegold com>
Subject: Unions Raise Safety Concerns Over Remote-Controlled Trains
 (The New York Times)

Railroad unions are raising safety concerns about the growing use of
remote-controlled trains after a rash of fatal accidents.

Remote-control locomotives are not autonomous like a self-driving car, but
they do lack the highly trained engineer who sits high in the cab at the
front of the locomotive on traditional trains, scanning the track ahead.

Instead, the train is most often controlled by a single remote-control
operator who may or may not be aboard, running the engine, brakes and other
mechanisms from a body-worn remote-control device that is connected to the
locomotive by a computer. In some cases, a second operator may also help
guide the train.

Unlike Teslas or other automated cars, which have various onboard cameras
and navigation sensors, remote trains have no such equipment -— they depend
on what the operator can see from wherever they are standing.

“With remote-control operations, it’s just that: There’s no requirement to
have the person in the cabin of the locomotive,” said John Esterly, a union
leader in Ohio. “They may be on another end of the locomotive.  They may be
1,000 feet away controlling it from the other end. That’s the fundamental
difference with remote trains: that lack of a set of eyes in the cab.”

In some cases, the remote operator is not on the train at all. In operations
within a rail yard or very near one, there are special protocols in place,
and the remote operator may be standing as far as several thousand feet from
the train. In these designated “remote control zones,” there is no
requirement that the person piloting the train have a view of the tracks
ahead.

These zones can stretch for several miles, documents show. It was in one of
these areas that the train in Buffalo was operating when it hit the boy —
and no one was onboard.

“As the Rail Safety Advisory Committee again reviews the use of RCL
technology, railroads are confident that the data will show what it always
has,” said Jessica Kahanek, a spokeswoman for the Association of American
Railroads, a group that represents the freight train industry.  “Remote
locomotives are just as safe as conventional ones.”

https://www.nytimes.com/2024/05/27/us/train-safety-crashes-union.html?smid=nytcore-ios-share&referringSource=articleShare&sgrp=c-cb

------------------------------

Date: Sun, 26 May 2024 18:12:58 -0600
From: Matthew Kruk <mkrukg () gmail com>
Subject: 'I was misidentified as shoplifter by facial recognition tech'
 (BBC)

https://www.bbc.com/news/technology-69055945

Sara needed some chocolate -- she had had one of those days - so wandered
into a Home Bargains store.
"Within less than a minute, I'm approached by a store worker who comes up
to me and says, 'You're a thief, you need to leave the store'."

Sara - who wants to remain anonymous -- was wrongly accused after being
flagged by a facial-recognition system called Facewatch.

She says after her bag was searched she was led out of the shop, and told
she was banned from all stores using the technology.

"I was just crying and crying the entire journey home.  I thought, 'Oh,
will my life be the same? I'm going to be looked at as a shoplifter when
I've never stolen'."

Facewatch later wrote to Sara and acknowledged it had made an error.

Facewatch is used in numerous stores in the UK -- including Budgens, Sports
Direct and Costcutter - to identify shoplifters.

------------------------------

Date: Mon, 27 May 2024 06:57:58 -0600
From: Matthew Kruk <mkrukg () gmail com>
Subject: Facebook account takeovers are targeting people you know,
 turning friendship into fraud (CBC)

https://www.cbc.ca/news/canada/new-brunswick/facebook-account-taken-over-friends-scam-1.7205356

For three days, Lesa Lowery says she could do nothing but watch as a
fraudster impersonated her on Facebook, swindling her friends out of
thousands of dollars for goods that didn't exist.

The entire time Meta -- the company behind the social media site that has
billions of users worldwide -- ignored the crime.

"I just felt helpless," said Lowery, who told Go Public her account was
taken over by the fraudster in early March. "I literally sat there and
cried," she said.

"I felt really bad for everybody whose money was taken." She'd connected
with hundreds of people on Facebook, many of whom she'd lost touch with in
person.

------------------------------

Date: Tue, 28 May 2024 01:08:21 +0000
From: Richard Marlon Stein <rmstein () protonmail com>
Subject: What Does an AI Do When It Sees an Optical Illusion?
 (Scientific American)

https://www.scientificamerican.com/article/optical-illusions-can-fool-ai-chatbots-too/

"To deploy AI systems responsibly, we need to understand their
vulnerabilities and blind spots as well as where human tendencies will and
won’t be replicated, says Joyce Chai, a computer science professor and AI
researcher at University of Michigan and senior author of the preprint
presented at the December 2023 conference. “It could be good or bad for a
model to align with humans,” she says. In some cases, it’s desirable for a
model to mitigate human biases. AI medical diagnostic tools that analyze
radiology images, for instance, would ideally not be susceptible to visual
error."

Attribute and identify the AI as image interpreter, and disclose the AI's
risk management attribute scores per some standard, such as NIST's AI Risk
Management Framework (https://doi.org/10.6028/NIST.AI.100-1).

Disclosing the author's origin should pique public interest. The
accompanying AI RMF attribute scores will defy public interpretation for all
but the cognoscenti.

------------------------------

Date: Sun, 26 May 2024 17:38:34 -0600
From: Matthew Kruk <mkrukg () gmail com>
Subject: AI-powered hate content is on the rise, experts say (WiReD)

The clip is of a real historical event -- a speech given by Nazi dictator
Adolf Hitler in 1939 at the beginning of the Second World War.

But there is one major difference. This viral video was altered by
artificial intelligence, and in it, Hitler delivers antisemitic remarks in
English.

A far-right conspiracy influencer shared the content on X, formerly known as
Twitter, earlier this year, and it quickly racked up more than 15 million
views, Wired magazine reported in March.

It's just one example of what researchers and organizations that monitor
hateful content are calling a worrying trend.

They say AI-generated hate is on the rise.

------------------------------

Date: Mon, 27 May 2024 14:17:16 PDT
From: Peter Neumann <neumann () csl sri com>
Subject: The order in which data is fed to LLMs can make a big difference

Manipulating SGD with Data Ordering Attacks,
Ilia Shumailov et al. (including the late Ross Anderson)
https://arxiv.org/pdf/2104.09667

------------------------------

Date: Mon, 27 May 2024 01:04:25 +0000
From: Henry Baker <hbaker1 () pipeline com>
Subject: Windows Total "Recall" -- aka *keylogger* -- is security nightmare

Whose bright idea was this?  Has Clippy gone to the dark side?

Every sysadmin should instantaneously block this "feature".

We can only hope that the clueless person at Microsoft who
greenlighted this "Recall" "feature" will be recalled him/her/itself
and immediately fired.

But Microsoft will probably blame this idiocy on some AI rather
than an actual human being...

https://www.theregister.com/2024/05/22/windows_recall/

Giving Windows total recall of everything a user does is a privacy minefield

It's only a preview, and maybe it should stay there ... forever

Richard Speed Wed 22 May 2024 // 13:11 UTC

Build Microsoft's Windows Recall feature is attracting controversy
before even venturing out of preview.

Like so many of Microsoft's AI-infused products, Windows Recall will
remain in preview while Microsoft refines it based on user feedback &ndash;
or simply gives up and pretends it never happened.

The principle is simple. As noted earlier, Windows takes a snapshot of
a user's active screen every few seconds and dumps it to disk. The
user can then scroll through the archive of snapshots to find what
were doing some time back, or query an AI system to recall past
screenshots by text.

The Windows 11 feature is supposed to eventually expand to allow users
to pull up anything that happened recently on their Copilot+ PC and
interact with or use it again, as the system logs all app activity,
communications, and so on, as well as by-the-second screenshots, to
local storage for search and retrieval.

Microsoft, which was just scolded by the US government for lax
security, said: "Recall will also enable you to open the snapshot in
the original application in which it was created, and, as Recall is
refined over time, it will open the actual source document, website,
or email in a screenshot. This functionality will be improved during
Recall's preview phase."

Improvements will certainly be needed, particularly in how the
function deals with privacy.

Taking aside the fact that BitLocker will only come into play on
Windows 11 Pro or Enterprise devices &ndash; everyone else must make do with
"data encryption" &ndash; Windows Recall has the potential to be a privacy
nightmare.

According to Microsoft, all the processing takes place on a customer's
device, and the snapshots stay there. The IT giant also says that for
the relatively small number of users running its Edge browser &ndash; with a
market share of just under 13 percent, according to Statcounter &ndash;
InPrivate sessions won't be snapped, nor will DRM content.

It will not hide information such as passwords or financial
account numbers. That data may be in snapshots stored on your
device

Microsoft said in its FAQs that its snapshotting feature will vacuum
up sensitive information: "Recall does not perform content
moderation. It will not hide information such as passwords or
financial account numbers. That data may be in snapshots stored on
your device, especially when sites do not follow standard Internet
protocols like cloaking password entry."

But that's OK &ndash; a user can opt to filter out sites, right? Only if
you're using Edge. In the deeper documentation for the service,
Microsoft said: "To filter out a website from a snapshot, you must be
using Microsoft Edge."

"Recall won't save any content from your private browsing activity
when you're using Microsoft Edge or a Chromium-based browser."
So, at least it's more than Edge when it comes to respecting private
tabs. Tarquin Wilton-Jones, a developer and privacy expert at Vivaldi,
a Chromium-based browser vendor, earlier expressed hope that the
automatic respecting of the InPrivate mode &ndash; or Incognito mode for
Chrome &ndash; would apply outside of Edge.

"It almost certainly will not respect any browser's attempts to clear
browsing data, where the browser could historically have been in any
screenshots," he added.

Recall stores not just browser history, but also data that users
type into the browser with only very coarse control over what gets
stored

"It also cannot respect GDPR requests to delete personal data exposed
in an application when the source data is deleted by a data
controller, and for this reason, it is clearly a massive privacy risk
for any organization that handles private data. Who knows what other
private data, or sensitive information, it might store in a freely
accessible format?"

Mozilla's Chief Product Officer Steve Teixeira told The Register:
"Mozilla is concerned about Windows Recall. From a browser
perspective, some data should be saved, and some shouldn't. Recall
stores not just browser history, but also data that users type into
the browser with only very coarse control over what gets stored. While
the data is stored in encrypted format, this stored data represents a
new vector of attack for cybercriminals and a new privacy worry for
shared computers.

"Microsoft is also once again playing gatekeeper and picking which
browsers get to win and lose on Windows &ndash; favoring, of course,
Microsoft Edge. Microsoft's Edge allows users to block specific
websites and private browsing activity from being seen by
Recall. Other Chromium-based browsers can filter out private browsing
activity but lose the ability to block sensitive websites (such as
financial sites) from Recall.

"Right now, there's no documentation on how a non-Chromium based,
third-party browser, such as Firefox, can protect user privacy from
Recall. Microsoft did not engage our cooperation on Recall, but we
would have loved for that to be the case, which would have enabled us
to partner on giving users true agency over their privacy, regardless
of the browser they choose."

Jake Moore, Global Cybersecurity Advisor at ESET, noted that while the
feature is not on by default, its use "opens up another avenue for
criminals to attack."

In essence, a keylogger is being baked into Windows as a feature
Moore warned that "users should be mindful of allowing any content to
be analysed by AI algorithms for a better experience."

Cybersecurity expert Kevin Beaumont was scathing in his assessment of
the technology, writing: "In essence, a keylogger is being baked into
Windows as a feature."

AI expert Gary Marcus was blunter: "F^ck that. I don't want my
computer to spy on everything I ever do."

Probe incoming

To add to Microsoft's woes, a spokesperson for the UK's Information
Commissioner's Office said today: "We expect organisations to be
transparent with users about how their data is being used and only
process personal data to the extent that it is necessary to achieve a
specific purpose. Industry must consider data protection from the
outset and rigorously assess and mitigate risks to people's rights and
freedoms before bringing products to market.

"We are making enquiries with Microsoft to understand the safeguards
in place to protect user privacy."

At present, Windows Recall feels like it was put together with
insufficient thought.

Microsoft has said that "Recall is a key part of what makes Copilot+
PCs special."

However, as Microsoft has pointed out, it remains in preview.
Enterprises are unlikely to go anywhere near it until the privacy and
security questions it raises have been answered. The GDPR aspect alone
makes it a non-starter for all but the most determined of
organizations.

Microsoft's customers and Windows enthusiasts alike have been
clamoring for something in the operating system to make all the AI
hype worthwhile. But, in its current form, Windows Recall is not it.

  [Also trashed roundly by Lauren Weinstein.  PGN]

------------------------------

Date: Mon, 27 May 2024 10:37:00 -0400
From: Monty Solomon <monty () roscom com>
Subject: Crowds Flocked to the New York-Dublin Livestream. Then Things
 Got Racy. (WSJ)

A public-art installation fostered connections across the Atlantic. But when
behavior took an inappropriate turn, both cities turned the screens off.

https://www.wsj.com/us-news/new-york-dublin-portal-shut-down-return-5e0bfd84

------------------------------

Date: Sun, 26 May 2024 21:08:43 -0400
From: Monty Solomon <monty () roscom com>
Subject: The Harsh Truth Behind Samsung's Phone Repair Program
 (Florence Ion)

Samsung makes fixing its phone with genuine parts more expensive and
requires repair shops to snitch on customers.

https://gizmodo.com/harsh-truth-samsung-phone-repair-program-ifixit-1851500413

------------------------------

Date: Sun, 26 May 2024 21:05:20 -0400
From: Monty Solomon <monty () roscom com>
Subject: Congress Just Made It Basically Impossible to Track Taylor Swift’s
 Private Jet (Gizmodo)

Legislation just signed into law has made it exceedingly to difficult to
track private jet activity.

https://gizmodo.com/congress-just-made-it-way-harder-to-track-taylor-swift-1851492383

------------------------------

Date: Tue, 28 May 2024 07:51:55 -0700
From: Rob Slade <rslade () gmail com>
Subject: Elon Musk wants our help with a [minor|huge] problem.

Neuralink, his attempt at a brain implant, which may a) help seriously
disabled people move and communicate with much greater facility, b) help
gamers spend much more time in immersive battles and seriously
[realistic|unrealistic|fantasy] artificial pornography, c) allow our
hallucinating AI Singularity Overlords to control us much more easily, has
run into a problem with limitations on the speed of data transmission.  He
needs someone to come up with some kind of data compression that allows for
greater than two hundred times reduction in bandwidth.
https://newsletters.cbc.ca/c/119rjIcMdG5aHEEj8KIvsulzvelyOA

OK, first off, I recall someone who had a *great* idea for fabric dying.
Black is notoriously hard to do.  So, someone came up with the idea of using
carbon dying for fabric, and went to a chemist to find a solvent for carbon.
Since the only known solvent for carbon is liquid iron, it was a bit of an
ask.  I suspect Musk is making a similar level of ask.

But I am well aware that we, as human beings, are extremely ingenious.  I
suspect someone *might* come up with a compression method on that order.

And that's where the trouble might start.

Compression is either lossy or lossless.  If someone comes up with a
lossless compression method for this particular application, it will be
because they have developed a new and tremendously useful understanding of
the brain, and how it works.  If so, I'm all in.  That'll be a tremendous
boost in a great many areas.

But it's much, much more likely that somebody will come up with a lossy
compression algorithm, since that'll be a shortcut, and convenient.  Now,
looking just at the "helping the disabled" part of this idea, what we are
trying to do is help those who have mobility and communication challenges
"live and move and have [their] being" (to seriously misquote, completely
out of context of the original) with the assistance of Neuralink.  And if we
don't understand what we are losing, in this process, how do we know what we
are losing on behalf of those who are using the system?  How are those who
may have serious communications problems anyways, to let us know that we
have imprisoned them in a system which does not allow them to cry for help
about certain things?

------------------------------

Date: Sun, 26 May 2024 23:37:19 -0700
From: Geoff Kuenning <geoff () cs hmc edu>
Subject: Re: A woman was dragged by a self-driving Cruise taxi in
 San Francisco (RISKS-34.26)

If a person is crossing in a crosswalk, there is no legal right to enter the
intersection even if the light is green.  That makes sense: some people have
disabilities that make it impossible to cross the entire street during the
time alloted by the "Walk" sign.

There was an incident in LA a few years ago where a cop ticketed an elderly
woman stepped off the curb the moment it was legal to do so, but couldn't
get across in time.  He was roundly pilloried, and I believe the law was
changed to prohibit such unfair tickets.

(There is also no right to obey a green light if an emergency vehicle is
present.  Good sense trumps simple rules.  Unfortunately autonomous vehicles
don't yet have good sense.)

  [The latest version of the California Driver's Handbook stresses this
  point as a change in the law.  PGN]

------------------------------

Date: Tue, 28 May 2024 07:23:44 +0100
From: Wols Lists <antlists () youngman org uk>
Subject: Re: A woman was dragged by a self-driving Cruise taxi
 in San Francisco (RISKS-34.26)

> As always, it pays to go beyond the headline.

And even this is misleading.

There is a set of traffic lights outside Kings College Hospital in Denmark
Hill, which everybody walking to the hospital from the station will use.

We always wait for the green man to come on before we start crossing.  (If
we're not already waiting at the red, we don't attempt to cross).  This
gives us just enough time to get to the middle of the crossing (where there
is no island) before the lights for the traffic go green again.

Fortunately, we've never had any problem with cars not waiting, but it
sounds like autonomous cars might be a real danger...

------------------------------

Date: Mon, 27 May 2024 12:12:30 +0300
From: Amos Shapir <amos083 () gmail com>
Subject: Re: Half of calls to gambling helpline were for help
 placing mobile bets (RISKS-34.25)

This reminds me of a story of the 1970's: A university advertised their
addiction prevention hotline with the slogan "Have a craving for a joint?
Call our addiction help hotline!".  They canceled the ad after it turned
out that 90% of callers asked how much they charge for an ounce...

(How times have changed...  When I tried to search for original reports of
this, all I got were ads for hotlines which actually do sell cannabis.)

------------------------------

Date: Mon, 27 May 2024 07:01:33 -0700
From: Steve Bacher <sebmb1 () verizon net>
Subject: Re: I stumbled upon LLM Kryptonite and no one wants to fix it
 (The Register, RISKS 34.26)

The link in the RISKS-34.26 entry is broken.  This one works:

  https://www.theregister.com/2024/05/23/ai_untested_unstable/

------------------------------

Date: Sun, 26 May 2024 21:10:09 -0600
From: "Jared E. Richo" <jericho () attrition org>
Subject: Re: MITRE ATLAS on obscurity

  [I think this interchange in another group was very worthy of RISKS. PGN]

Dan, Perhaps I am misreading you, but I don't read this as 'reversing' the
maxim of "obscurity is not security". While obscurity, on its own, is not
security, it absolutely has a place as part of security. Citing that quote
or any variation of it, needs more qualification here.

There's a not-so-fine line where information disclosure goes from a
non-issue to a concern, but why volunteer any single bit of information that
may assist attackers? Some say a remote path disclosure vulnerability is too
trivial to really assist an attacker, others thing it absolutely can be an
issue depending on the system, if it can be used as part of an exploit
chain, etc.

On 5/26/2024 8:25 PM, dan () geer org wrote:
> I am tempted to suggest that the proliferation of ML reverses the
> old saw "obscurity is not security" and, in fact, several of the
> ATLAS points read that way to me, e.g.,
>
> https://atlas.mitre.org/mitigations/AML.M0000
>
> "Limit the public release of technical information about the machine
> learning stack used in an organization's products or services.
> Technical knowledge of how machine learning is used can be leveraged
> by adversaries to perform targeting and tailor attacks to the target
> system. Additionally, consider limiting the release of organizational
> information - including physical locations, researcher names, and
> department structures - from which technical details such as machine
> learning techniques, model architectures, or datasets may be inferred."

------------------------------

Date: Sat, 28 Oct 2023 11:11:11 -0800
From: RISKS-request () csl sri com
Subject: Abridged info on RISKS (comp.risks)

 The ACM RISKS Forum is a MODERATED digest.  Its Usenet manifestation is
 comp.risks, the feed for which is donated by panix.com as of June 2011.
=> SUBSCRIPTIONS: The mailman Web interface can be used directly to
 subscribe and unsubscribe:
   http://mls.csl.sri.com/mailman/listinfo/risks

=> SUBMISSIONS: to risks () CSL sri com with meaningful SUBJECT: line that
   includes the string `notsp'.  Otherwise your message may not be read.
 *** This attention-string has never changed, but might if spammers use it.
=> SPAM challenge-responses will not be honored.  Instead, use an alternative
 address from which you never send mail where the address becomes public!
=> The complete INFO file (submissions, default disclaimers, archive sites,
 copyright policy, etc.) has moved to the ftp.sri.com site:
   <risksinfo.html>.
 *** Contributors are assumed to have read the full info file for guidelines!

=> OFFICIAL ARCHIVES:  http://www.risks.org takes you to Lindsay Marshall's
    delightfully searchable html archive at newcastle:
  http://catless.ncl.ac.uk/Risks/VL.IS --> VoLume, ISsue.
  Also, ftp://ftp.sri.com/risks for the current volume/previous directories
     or ftp://ftp.sri.com/VL/risks-VL.IS for previous VoLume
  If none of those work for you, the most recent issue is always at
     http://www.csl.sri.com/users/risko/risks.txt, and index at /risks-34.00
  ALTERNATIVE ARCHIVES: http://seclists.org/risks/ (only since mid-2001)
 *** NOTE: If a cited URL fails, we do not try to update them.  Try
  browsing on the keywords in the subject line or cited article leads.
  Apologies for what Office365 and SafeLinks may have done to URLs.
==> Special Offer to Join ACM for readers of the ACM RISKS Forum:
    <http://www.acm.org/joinacm1>

------------------------------

End of RISKS-FORUM Digest 34.27
************************