#engineer-ai

[mostlysignssomeportents]

How I got scammed

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2024/02/05/cyber-dunning-kruger/#swiss-cheese-security

I wuz robbed.

More specifically, I was tricked by a phone-phisher pretending to be from my bank, and he convinced me to hand over my credit-card number, then did $8,000+ worth of fraud with it before I figured out what happened. And then he tried to do it again, a week later!

Here's what happened. Over the Christmas holiday, I traveled to New Orleans. The day we landed, I hit a Chase ATM in the French Quarter for some cash, but the machine declined the transaction. Later in the day, we passed a little credit-union's ATM and I used that one instead (I bank with a one-branch credit union and generally there's no fee to use another CU's ATM).

A couple days later, I got a call from my credit union. It was a weekend, during the holiday, and the guy who called was obviously working for my little CU's after-hours fraud contractor. I'd dealt with these folks before – they service a ton of little credit unions, and generally the call quality isn't great and the staff will often make mistakes like mispronouncing my credit union's name.

That's what happened here – the guy was on a terrible VOIP line and I had to ask him to readjust his mic before I could even understand him. He mispronounced my bank's name and then asked if I'd attempted to spend $1,000 at an Apple Store in NYC that day. No, I said, and groaned inwardly. What a pain in the ass. Obviously, I'd had my ATM card skimmed – either at the Chase ATM (maybe that was why the transaction failed), or at the other credit union's ATM (it had been a very cheap looking system).

I told the guy to block my card and we started going through the tedious business of running through recent transactions, verifying my identity, and so on. It dragged on and on. These were my last hours in New Orleans, and I'd left my family at home and gone out to see some of the pre-Mardi Gras krewe celebrations and get a muffalata, and I could tell that I was going to run out of time before I finished talking to this guy.

"Look," I said, "you've got all my details, you've frozen the card. I gotta go home and meet my family and head to the airport. I'll call you back on the after-hours number once I'm through security, all right?"

He was frustrated, but that was his problem. I hung up, got my sandwich, went to the airport, and we checked in. It was total chaos: an Alaska Air 737 Max had just lost its door-plug in mid-air and every Max in every airline's fleet had been grounded, so the check in was crammed with people trying to rebook. We got through to the gate and I sat down to call the CU's after-hours line. The person on the other end told me that she could only handle lost and stolen cards, not fraud, and given that I'd already frozen the card, I should just drop by the branch on Monday to get a new card.

We flew home, and later the next day, I logged into my account and made a list of all the fraudulent transactions and printed them out, and on Monday morning, I drove to the bank to deal with all the paperwork. The folks at the CU were even more pissed than I was. The fraud that run up to more than $8,000, and if Visa refused to take it out of the merchants where the card had been used, my little credit union would have to eat the loss.

I agreed and commiserated. I also pointed out that their outsource, after-hours fraud center bore some blame here: I'd canceled the card on Saturday but most of the fraud had taken place on Sunday. Something had gone wrong.

One cool thing about banking at a tiny credit-union is that you end up talking to people who have actual authority, responsibility and agency. It turned out the the woman who was processing my fraud paperwork was a VP, and she decided to look into it. A few minutes later she came back and told me that the fraud center had no record of having called me on Saturday.

"That was the fraudster," she said.

Oh, shit. I frantically rewound my conversation, trying to figure out if this could possibly be true. I hadn't given him anything apart from some very anodyne info, like what city I live in (which is in my Wikipedia entry), my date of birth (ditto), and the last four digits of my card.

Wait a sec.

He hadn't asked for the last four digits. He'd asked for the last seven digits. At the time, I'd found that very frustrating, but now – "The first nine digits are the same for every card you issue, right?" I asked the VP.

I'd given him my entire card number.

Goddammit.

The thing is, I know a lot about fraud. I'm writing an entire series of novels about this kind of scam:

https://us.macmillan.com/books/9781250865878/thebezzle

And most summers, I go to Defcon, and I always go to the "social engineering" competitions where an audience listens as a hacker in a soundproof booth cold-calls merchants (with the owner's permission) and tries to con whoever answers the phone into giving up important information.

But I'd been conned.

Now look, I knew I could be conned. I'd been conned before, 13 years ago, by a Twitter worm that successfully phished out of my password via DM:

https://locusmag.com/2010/05/cory-doctorow-persistence-pays-parasites/

That scam had required a miracle of timing. It started the day before, when I'd reset my phone to factory defaults and reinstalled all my apps. That same day, I'd published two big online features that a lot of people were talking about. The next morning, we were late getting out of the house, so by the time my wife and I dropped the kid at daycare and went to the coffee shop, it had a long line. Rather than wait in line with me, my wife sat down to read a newspaper, and so I pulled out my phone and found a Twitter DM from a friend asking "is this you?" with a URL.

Assuming this was something to do with those articles I'd published the day before, I clicked the link and got prompted for my Twitter login again. This had been happening all day because I'd done that mobile reinstall the day before and all my stored passwords had been wiped. I entered it but the page timed out. By that time, the coffees were ready. We sat and chatted for a bit, then went our own ways.

I was on my way to the office when I checked my phone again. I had a whole string of DMs from other friends. Each one read "is this you?" and had a URL.

Oh, shit, I'd been phished.

If I hadn't reinstalled my mobile OS the day before. If I hadn't published a pair of big articles the day before. If we hadn't been late getting out the door. If we had been a little more late getting out the door (so that I'd have seen the multiple DMs, which would have tipped me off).

There's a name for this in security circles: "Swiss-cheese security." Imagine multiple slices of Swiss cheese all stacked up, the holes in one slice blocked by the slice below it. All the slices move around and every now and again, a hole opens up that goes all the way through the stack. Zap!

The fraudster who tricked me out of my credit card number had Swiss cheese security on his side. Yes, he spoofed my bank's caller ID, but that wouldn't have been enough to fool me if I hadn't been on vacation, having just used a pair of dodgy ATMs, in a hurry and distracted. If the 737 Max disaster hadn't happened that day and I'd had more time at the gate, I'd have called my bank back. If my bank didn't use a slightly crappy outsource/out-of-hours fraud center that I'd already had sub-par experiences with. If, if, if.

The next Friday night, at 5:30PM, the fraudster called me back, pretending to be the bank's after-hours center. He told me my card had been compromised again. But: I hadn't removed my card from my wallet since I'd had it replaced. Also, it was half an hour after the bank closed for the long weekend, a very fraud-friendly time. And when I told him I'd call him back and asked for the after-hours fraud number, he got very threatening and warned me that because I'd now been notified about the fraud that any losses the bank suffered after I hung up the phone without completing the fraud protocol would be billed to me. I hung up on him. He called me back immediately. I hung up on him again and put my phone into do-not-disturb.

The following Tuesday, I called my bank and spoke to their head of risk-management. I went through everything I'd figured out about the fraudsters, and she told me that credit unions across America were being hit by this scam, by fraudsters who somehow knew CU customers' phone numbers and names, and which CU they banked at. This was key: my phone number is a reasonably well-kept secret. You can get it by spending money with Equifax or another nonconsensual doxing giant, but you can't just google it or get it at any of the free services. The fact that the fraudsters knew where I banked, knew my name, and had my phone number had really caused me to let down my guard.

The risk management person and I talked about how the credit union could mitigate this attack: for example, by better-training the after-hours card-loss staff to be on the alert for calls from people who had been contacted about supposed card fraud. We also went through the confusing phone-menu that had funneled me to the wrong department when I called in, and worked through alternate wording for the menu system that would be clearer (this is the best part about banking with a small CU – you can talk directly to the responsible person and have a productive discussion!). I even convinced her to buy a ticket to next summer's Defcon to attend the social engineering competitions.

There's a leak somewhere in the CU systems' supply chain. Maybe it's Zelle, or the small number of corresponding banks that CUs rely on for SWIFT transaction forwarding. Maybe it's even those after-hours fraud/card-loss centers. But all across the USA, CU customers are getting calls with spoofed caller IDs from fraudsters who know their registered phone numbers and where they bank.

I've been mulling this over for most of a month now, and one thing has really been eating at me: the way that AI is going to make this kind of problem much worse.

Not because AI is going to commit fraud, though.

One of the truest things I know about AI is: "we're nowhere near a place where bots can steal your job, we're certainly at the point where your boss can be suckered into firing you and replacing you with a bot that fails at doing your job":

https://pluralistic.net/2024/01/15/passive-income-brainworms/#four-hour-work-week

I trusted this fraudster specifically because I knew that the outsource, out-of-hours contractors my bank uses have crummy headsets, don't know how to pronounce my bank's name, and have long-ass, tedious, and pointless standardized questionnaires they run through when taking fraud reports. All of this created cover for the fraudster, whose plausibility was enhanced by the rough edges in his pitch - they didn't raise red flags.

As this kind of fraud reporting and fraud contacting is increasingly outsourced to AI, bank customers will be conditioned to dealing with semi-automated systems that make stupid mistakes, force you to repeat yourself, ask you questions they should already know the answers to, and so on. In other words, AI will groom bank customers to be phishing victims.

This is a mistake the finance sector keeps making. 15 years ago, Ben Laurie excoriated the UK banks for their "Verified By Visa" system, which validated credit card transactions by taking users to a third party site and requiring them to re-enter parts of their password there:

https://web.archive.org/web/20090331094020/http://www.links.org/?p=591

This is exactly how a phishing attack works. As Laurie pointed out, this was the banks training their customers to be phished.

I came close to getting phished again today, as it happens. I got back from Berlin on Friday and my suitcase was damaged in transit. I've been dealing with the airline, which means I've really been dealing with their third-party, outsource luggage-damage service. They have a terrible website, their emails are incoherent, and they officiously demand the same information over and over again.

This morning, I got a scam email asking me for more information to complete my damaged luggage claim. It was a terrible email, from a noreply@ email address, and it was vague, officious, and dishearteningly bureaucratic. For just a moment, my finger hovered over the phishing link, and then I looked a little closer.

On any other day, it wouldn't have had a chance. Today – right after I had my luggage wrecked, while I'm still jetlagged, and after days of dealing with my airline's terrible outsource partner – it almost worked.

So much fraud is a Swiss-cheese attack, and while companies can't close all the holes, they can stop creating new ones.

Meanwhile, I'll continue to post about it whenever I get scammed. I find the inner workings of scams to be fascinating, and it's also important to remind people that everyone is vulnerable sometimes, and scammers are willing to try endless variations until an attack lands at just the right place, at just the right time, in just the right way. If you think you can't get scammed, that makes you especially vulnerable:

https://pluralistic.net/2023/02/24/passive-income/#swiss-cheese-security

Image: Cryteria (modified) https://commons.wikimedia.org/wiki/File:HAL9000.svg

CC BY 3.0 https://creativecommons.org/licenses/by/3.0/deed.en

[probablyasocialecologist]

Google is now the only search engine that can surface results from Reddit, making one of the web’s most valuable repositories of user generated content exclusive to the internet’s already dominant search engine. If you use Bing, DuckDuckGo, Mojeek, Qwant or any other alternative search engine that doesn’t rely on Google’s indexing and search Reddit by using “site:reddit.com,” you will not see any results from the last week. DuckDuckGo is currently turning up seven links when searching Reddit, but provides no data on where the links go or why, instead only saying that “We would like to show you a description here but the site won't allow us.” Older results will still show up, but these search engines are no longer able to “crawl” Reddit, meaning that Google is the only search engine that will turn up results from Reddit going forward. Searching for Reddit still works on Kagi, an independent, paid search engine that buys part of its search index from Google. The news shows how Google’s near monopoly on search is now actively hindering other companies’ ability to compete at a time when Google is facing increasing criticism over the quality of its search results. And while neither Reddit or Google responded to a request for comment, it appears that the exclusion of other search engines is the result of a multi-million dollar deal that gives Google the right to scrape Reddit for data to train its AI products.

July 24 2024

[slothnobility]

💋💋💋

lemme know if you bingo'd on this lolol i must know for science

[dianapocalypse]

generative AI is the future

[javert]

ok look. what are you actually imagining that these engineering students are using chatgpt to do? because my experience of engineering school in every class that actually mattered was that the entire grade in the class came from

going physically into an exam room

taking a test with a pencil and paper and a graphing calculator

sometimes it was open note/open textbook but it was never open computer

there is no room here to use chatgpt

if somehow you are using chatgpt to explain the material to you in a new way beforehand and then memorizing it so that you can go into the test and answer the math questions correctly, that is simply called "learning the material" lol

[nixcraft]

I asked Google "who ruined Google" and they replied honestly using their AI, which is now forced on all of us. It's too funny not to share!

[elbiotipo]

It's so funny when people say "not all AI sucks! only generative AI!" because generative AI is genuinely an amazing technology.

You know why those early AI images like crayon and such were so strange and dreamlike? It's because generative algorithms actually do generate those images. They don't copypaste like a collage, images are created pixel-by-pixel. Generative AIs are actually systems that assimilate concepts, associate them to images, are able to translate instructions in plain human text instead of code and create new things from it (this was seen as pure science fiction less than 5 years ago). This is why AI images now have better quality, because new models are able to understand more concepts and implement them. Because the idea with generative AI isn't and shouldn't for it to be able to just copy-paste images or text, it's the ability to generate new images or text from learned concepts.

This post gets, in a very easy, understandable way, into the details on how this works. And I hope you do give it a read no matter your stand on this:

This, as I always say, was considered pure science fiction, a thing that would not exist until at least the 2100s if at all, and it is now here. And not only by corporations, but open-source models are being researched by the minute.

No, I do not care for AI corporations and I don't care for what they're mostly trying to use AI for (advertising and customer service). I care about what can become of this technology. Advertising and mass produced shit will be shit, no matter if it's done by human or AI. Do I expect an advertisement to be shit because it uses AI? No, I expect it to be shit because it is an advertisement.

What will be interesting, and I think we will see more in the future when the utterly poisoned current discourse about AI calms down, will be when artists with interesting concepts and a good handle of these tools start to create new things, much like synthesizers or photographers didn't ruin music or art, because there was always an artist behind the tool in the first place. Someone is doing those prompts to create something. Your question should be who and why.

[shermyy]

Link Click x LIMBO

[nostalgebraist]

Trying to picture how myself from early 2015 would react if I told him that 10 years later, he'd be freaking out and doomscrolling the news because

"Elon Musk and President Donald Trump, acting through the new federal organization 'DOGE,' are trying to implement Mencius Moldbug's plan to 'reboot the government' by mass-retiring government employees"

[carsthatnevermadeitetc]

Citroën 2CV Safari, 2024. Renders for a proposal by EB Design to create a restomod 2CV desert racer taking inspiration from the twin-engine 2CV Sahara.

[godbirdart]

i've been so tired of google's new legal liability bot sitting on top of the existing highlighted result, taking up page space and either parroting exactly what the highlighted result said or offering completely unrelated or incorrect results, that i actually cheered when this came up:

"oh but AI is experimental -"

the bot told people to eat glue on pizza. you can talk yourself blue in the face about the bot's learning curve and how "it'll be improved with time", but maybe a bot being touted as the latest and greatest in scouring the internet for accurate information should not come with a permanent glaring disclaimer of "it's still learning / results may not be accurate!" as the generative AI ouroboros continues to keep on chewing.

[two-pears-in-a-trenchcoat]

"i am going to caboose you" says the evil caboose man

[teknewsfeed]

It's the real life Wall-e..

[stainedlilac]

Fuck you *takes your Angel Engine design and makes it my own because it's AI slop*

[viridianriver]

'Artificial Intelligence' Tech - Not Intelligent as in Smart - Intelligence as in 'Intelligence Agency'

I work in tech, hell my last email ended in '.ai' and I used to HATE the term Artificial Intelligence. It's computer vision, it's machine learning, I'd always argue.

Lately, I've changed my mind. Artificial Intelligence is a perfectly descriptive word for what has been created. As long as you take the word 'Intelligence' to refer to data that an intelligence agency or other interested party may collect.

But I'm getting ahead of myself. Back when I was in 'AI' - the vibe was just odd. Investors were throwing money at it as fast as they could take out loans to do so. All the while, engineers were sounding the alarm that 'AI' is really just a fancy statistical tool and won't ever become truly smart let alone conscious. The investors, baffingly, did the equivalent of putting their fingers in their ears while screaming 'LALALA I CAN'T HEAR YOU"

Meanwhile, CEOs were making all sorts of wild promises about what AI will end up doing, promises that mainly served to stress out the engineers. Who still couldn't figure out why the hell we were making this silly overhyped shit anyway.

SYSTEMS THINKING

As Stafford Beer said, 'The Purpose of A System is What It Does" - basically meaning that if a system is created, and maintained, and continues to serve a purpose? You can read the intended purpose from the function of a system. (This kind of thinking can be applied everywhere - for example the penal system. Perhaps, the purpose of that system is to do what it does - provide an institutional structure for enslavement / convict-leasing?)

So, let's ask ourselves, what does AI do? Since there are so many things out there calling themselves AI, I'm going to start with one example. Microsoft Copilot.

Microsoft is selling PCs with integrated AI which, among other things, frequently screenshots and saves images of your activity. It doesn't protect against copying passwords or sensitive data, and it comes enabled by default. Now, my old-ass-self has a word for that. Spyware. It's a word that's fallen out of fashion, but I think it ought to make a comeback.

To take a high-level view of the function of the system as implemented, I would say it surveils, and surveils without consent. And to apply our systems thinking? Perhaps its purpose is just that.

SOCIOLOGY

There's another principle I want to introduce - that an institution holds insitutional knowledge. But it also holds institutional ignorance. The shit that for the sake of its continued existence, it cannot know.

For a concrete example, my health insurance company didn't know that my birth control pills are classified as a contraceptive. After reading the insurance adjuster the Wikipedia articles on birth control, contraceptives, and on my particular medication, he still did not know whether my birth control was a contraceptive. (Clearly, he did know - as an individual - but in his role as a representative of an institution - he was incapable of knowing - no matter how clearly I explained)

So - I bring this up just to say we shouldn't take the stated purpose of AI at face value. Because sometimes, an institutional lack of knowledge is deliberate.

HISTORY OF INTELLIGENCE AGENCIES

The first formalized intelligence agency was the British Secret Service, founded in 1909. Spying and intelligence gathering had always been a part of warfare, but the structures became much more formalized into intelligence agencies as we know them today during WW1 and WW2.

Now, they're a staple of statecraft. America has one, Russia has one, China has one, this post would become very long if I continued like this...

I first came across the term 'Cyber War' in a dusty old aircraft hanger, looking at a cold-war spy plane. There was an old plaque hung up, making reference to the 'Upcoming Cyber War' that appeared to have been printed in the 80s or 90s. I thought it was silly at the time, it sounded like some shit out of sci-fi.

My mind has changed on that too - in time. Intelligence has become central to warfare; and you can see that in the technologies military powers invest in. Mapping and global positioning systems, signals-intelligence, of both analogue and digital communication.

Artificial intelligence, as implemented would be hugely useful to intelligence agencies. A large-scale statistical analysis tool that excels as image recognition, text-parsing and analysis, and classification of all sorts? In the hands of agencies which already reportedly have access to all of our digital data?

TIKTOK, CHINA, AND AMERICA

I was confused for some time about the reason Tiktok was getting threatened with a forced sale to an American company. They said it was surveiling us, but when I poked through DNS logs, I found that it was behaving near-identically to Facebook/Meta, Twitter, Google, and other companies that weren't getting the same heat.

And I think the reason is intelligence. It's not that the American government doesn't want me to be spied on, classified, and quantified by corporations. It's that they don't want China stepping on their cyber-turf.

The cyber-war is here y'all. Data, in my opinion, has become as geopolitically important as oil, as land, as air or sea dominance. Perhaps even more so.

A CASE STUDY : ELON MUSK

As much smack as I talk about this man - credit where it's due. He understands the role of artificial intelligence, the true role. Not as intelligence in its own right, but intelligence about us.

In buying Twitter, he gained access to a vast trove of intelligence. Intelligence which he used to segment the population of America - and manpulate us.

He used data analytics and targeted advertising to profile American voters ahead of this most recent election, and propogandize us with micro-targeted disinformation. Telling Israel's supporters that Harris was for Palestine, telling Palestine's supporters she was for Israel, and explicitly contradicting his own messaging in the process. And that's just one example out of a much vaster disinformation campaign.

He bought Trump the white house, not by illegally buying votes, but by exploiting the failure of our legal system to keep pace with new technology. He bought our source of communication, and turned it into a personal source of intelligence - for his own ends. (Or... Putin's?)

This, in my mind, is what AI was for all along.

CONCLUSION

AI is a tool that doesn't seem to be made for us. It seems more fit-for-purpose as a tool of intelligence agencies, oligarchs, and police forces. (my nightmare buddy-cop comedy cast) It is a tool to collect, quantify, and loop-back on intelligence about us.

A friend told me recently that he wondered sometimes if the movie 'The Matrix' was real and we were all in it. I laughed him off just like I did with the idea of a cyber war.

Well, I re watched that old movie, and I was again proven wrong. We're in the matrix, the cyber-war is here. And know it or not, you're a cog in the cyber-war machine.

(edit -- part 2 - with the 'how' - is here!)