git-commit-die - Tumblr blog

[git-commit-die]

Making a post after n years of inactivity because I'm afraid this blog will get deleted in an anti-spam effort if I don't.

[git-commit-die]

ChatGPT, LLMs, Plagiarism, & You

This is the first in a series of posts about ChatGPT, LLMs, and plagiarism that I will be making. This is a side blog, so please ask questions in reblogs and my ask box.

Why do I know what I'm talking about?

I am a machine engineer who specializes natural language processing (NLP). I write code that uses LLMs every day at work and am intimately familiar with OpenAI. I have read dozens of scientific papers on the subject and understand how they work in extreme detail. I have 6 years of experience in the industry, plus a graduate degree in the subject. I got into NLP because I knew it was going to pop off, and now here we are.

Yeah, but why should I trust you?

I've been a Tumblr user for 8 years. I've posted my own art and fanart on the site. I've published writing, both original and fanfiction, on Tumblr and AO3. I've been a Reddit user for over a decade. I'm a citizen of the internet as much as I am an engineer.

What is an LLM?

LLM stands for Large Language Model. The most famous example of an LLM is ChatGPT, which was created by OpenAI.

What is a model?

A model is an algorithm or piece of math that lets you predict or make mimic how something behaves. For example:

The National Weather Service runs weather models that predict how much it's going to rain based on data they collect about the atmosphere

Netflix has recommendations models that predicts whether you'd like a movie or not based on your demographics, what you've watched in the past, and what other people have liked

The Federal Reserve has economic models that predict how inflation will change if they increase or lower interest rates

Instagram has spam models that look at DMs and automatically decide whether they're spam or not

Models are useful because they can often make decisions or describe situations better than a human could. The weather and economic models are good examples of this. The science of rain is so complicated that it's practically impossible for a human to make sense of all the numbers involved, but models are able to do so.

Models are also useful because they can make thousands or millions of decisions much faster than a human could. The recommendations and spam models are good examples of this. Imagine how expensive it would be to run Instagram if a human had to review every single DM and decide whether it was spam.

What is a language model?

A language model is a model that can look at a piece of text and tell you how likely it is. For example, a language model can tell you that the phrase "the sky is blue" is more likely to have been written than "the sky is peanuts."

Why is this useful? You can use language models to generate text by picking letters and words that it gives a high score. Say you have the phrase "I ate a" and you're picking what comes next. You can run through every option, see how likely the language model thinks it is, and pick the best one. For example:

I ate a sandwich: score = .7

I ate a $(iwnJ98: score = .1

I ate a me: score = .2

So we pick "sandwich" and now have the phrase "I ate a sandwich." We can keep doing this process over and over to get more and more text. "I ate a sandwich for lunch today. It was delicious."

What makes a large language model large?

Large language models are large in a few different ways:

Under the hood, they are made of a bunch of numbers called "weights" that describe a monstrously complicated mathematical equation. Large language models have a ton of the weights--as many as tens of billions of them.

Large language models are trained on large amounts of text. This text comes mostly from the internet but also includes books that are out of copyright. This is the source of controversy about them and plagiarism, and I will cover it in greater detail in a future post.

Large language models are a large undertaking: they're expensive and difficult to create and run. This is why you basically only see them coming out of large or well-funded companies like OpenAI, Google, and Facebook. They require an incredible amount of technical expertise and computational resources (computers) to create.

Why are LLMs powerful?

"Generating likely text" is neat and all, but why do we care? Consider this:

An LLM can tell you that:

the text "Hello" is more likely to have been written than "$(iwnJ98"

the text "I ran to the store" is more likely to have been written than "I runned to the store"

the text "the sky is blue" is more likely to have been written than "the sky is green"

Each of them gets us something:

LLMs understand spelling

LLMs understand grammar

LLMs know things about the world

So we now have an infinitely patient robot that we can interact with using natural language and get it to do stuff for us.

Detecting spam: "Is this spam, yes or no? Check out rxpharmcy.ca now for cheap drugs now."

Personal language tutoring: "What is wrong with this sentence? Me gusto gatos."

Copy editing: "I'm not a native English speaker. Can you help me rewrite this email to make sure it sounds professional? 'Hi Akash, I hope...'"

Help learning new subjects: "Why is the sky blue? I'm only in middle school, so please don't make the explanation too complicated."

And countless other things.

[git-commit-die]

currently suffering death by a thousand leetcodes as i look for a new job

[git-commit-die]

Tumblr headcanon: that manscaping brand paid for X thousand clicks and the reason we've been plagued with the ads for so long is because no one is clicking them.

[git-commit-die]

As the staff post about ad-free tumblr continues to get thousands of notes telling staff to fuck off in the tags, I wanna remind you that this website's days are numbered. Tumblr is still unprofitable and by some modern-day miracle none of its acquiring companies pulled the plug on this money pit. But it will happen if it continues its trajectory.

Whether you like it or not, Tumblr needs to make money off you somehow in order to stay up. It either serves ads or asks for money to use it. This has been a paradigm on the web longer than many of you have been alive. It's Tumblr's job to make money right now because it's well past its grace period of being a black hole for cash. This has actually always been Tumblr's job, since it is a corporation, but that's capitalism for ya.

If you want Tumblr to be here for free and you want to continue to use it, you do yourself a disservice by opposing any changes Tumblr makes in order to pay for its costs. When this site finally goes belly-up then you're gonna be Tumblr-less until whatever startup takes its place and the cycle repeats itself.

If you think it should just ask for donations Wikipedia-style, remember that if and when that happens, there will be users repeating the same tired bullshit about giving Tumblr any money.

[git-commit-die]

@dummie-vaughn (responding as a reblog bc this is a sideblog)

Actually, Samsungs are Androids. Android is an operating system. Think of it like how Microsoft Windows can run on computers made by HP, Dell, Acer, etc. Android can run on phones made by Samsung, Google, OnePlus, etc. (On iPhones, the operating system is called iOS, but you can only run that on Apple's phones.)

As for which phone I'd recommend, that depends on your budget. My phone is a Google Pixel 6, which starts at $600. I'm a big fan of the Google Pixel phones. For $350, you can get the also great Google Pixel 4a (a friend who's an Android dev has that one). And for even lower budgets, Wirecutter recommends the OnePlus Nord N200 5G for $240 or the Motorola Moto G Play (2021) for $170.

Like I said in my original post, Androids and iPhones have advantages and disadvantages, and that's a decision you'll have to make for yourself. Here are a few:

• iPhones have better privacy and security

• Androids tend to be cheaper

• iPhones receive software updates for longer, while Androids tend to lose support after a few years

• Apple is anti-right-to-repair, making iPhones more expensive to get fixed, but because of their popularity it is easier to find someone willing to fix it

• Android has a more open ecosystem of apps (though this comes with the risk of installing malware if you aren't careful)

• Usually, iPhones "just work," making them a great choice for the less technically savvy (eg my parents both have one)

• Androids tend to allow much more customization, while iPhones do things in one way and there's no way to change it

• Apple devices work very well with the Apple ecosystem, so they're an obvious choice if you already have a MacBook, iPad, AppleTV, etc

• Because the Apple userbase tends to be wealthier, you can expect a more premium app experience. For example, the drawing apps on iOS are so much better than the ones on Android.

• Android tends to get innovative features much earlier than iOS. For example, Google introduced realtime voice translation years ago, and Apple only just added it this year.

• Apple is arguably a more ethical company than others due to its strong commitments to charity and the environment

(These are all generalizations since, as I said, many companies make Android phones and they are very diverse.)

Personally, despite having worked at Apple, I have an Android phone because I place a lot of value on being able to customize my experience. But I'm a software engineer, so I don't mind when things don't "just work." You may have other priorities.

Everyone saying "Tumblr should just take their app off the Apple App Store" drives me nuts. Apple has built an extremely locked-down ecosystem that makes it nearly impossible to install apps without the App Store. There are only two ways to side-load apps on iOS:

1. Jailbreak your phone, which is not viable for the average user.

2. Install the app via TestFlight, which still requires that Apple have approved it.

Tumblr does not have a choice in this. Either they follow Apple's restrictions, or there is no Tumblr app on iOS.

Note that Google does not have similar policies in the Google Play Store, and that even if they did, side-loading apps on Android is trivially easy. If this really bothers you, then don't make your next phone an iPhone. (Android phones have their own problems, of course--Apple legitimately does have a much better track record on privacy, but that's a tradeoff you'll have to make for yourself.)

Source: I'm a former Apple software engineer

[git-commit-die]

Everyone saying "Tumblr should just take their app off the Apple App Store" drives me nuts. Apple has built an extremely locked-down ecosystem that makes it nearly impossible to install apps without the App Store. There are only two ways to side-load apps on iOS:

1. Jailbreak your phone, which is not viable for the average user.

2. Install the app via TestFlight, which still requires that Apple have approved it.

Tumblr does not have a choice in this. Either they follow Apple's restrictions, or there is no Tumblr app on iOS.

Note that Google does not have similar policies in the Google Play Store, and that even if they did, side-loading apps on Android is trivially easy. If this really bothers you, then don't make your next phone an iPhone. (Android phones have their own problems, of course--Apple legitimately does have a much better track record on privacy, but that's a tradeoff you'll have to make for yourself.)

Source: I'm a former Apple software engineer

[git-commit-die]

You know what? Fuck you.

*downgrades your firmware*

[git-commit-die]

next meeting I have with an engineer who can't communicate their way out of a wet paper bag is going to end with me suffocating them with a wet paper bag

[git-commit-die]

dumbasses in the replies don't know end-to-end encryption means the company can't read your data

Private blogging, it’s a vibe. But some innermost thoughts deserve to be just that: inner. Very, very inner. Now, imagine the perfect complementary app to this one (hellsite). With absolute encryption (for superior secrecy levels), an allowance of thirty photos per entry (for all the moments), even email and text integration (text yourself whenever you feel like it). Yeah, that’s what this is:

Day One (not a hellsite) has joined our family at Automattic. That means Day One (not a hellsite) has joined your family, Tumblr. And that means the universe is telling you to try journaling.

There’s no such thing as cringe in a private journal; it’s all just ~intimacy~. And checking in with yourself in a solo space that looks nice and feels nice. With zero chance of your secrets, the Story of You, or your WIPs reaching other readers unless, or until, you want them to. (No, really. It’s all utterly encrypted.)

Try it out, and never accidentally post something wildly personal on main again. (You’re free to keep doing it on purpose here, though.)

[git-commit-die]

no you