#encoding

[david-goldrock]

Okay so this is a background post about Text encoding, ASCII and Unicode

Text encoding is the process of turning characters to numbers. text encoding allows one to save text as computer data, and to move this data around.

It was understood very early on, that if every user will define their own encoding, no interface could use the data of another because one interface's "a" would be another interface's "p", and so the text would be read as gibberish.

and so, a long time ago (in the 1960s), in a continent far, far away, a standard for text encoding was invented: the American Standard for Character Interface Interchange: ASCII.

ASCII used the fact that in english, almost no characters exist, and so only needed to use 128 characters: each character took 7 bits (1s and 0s), and was sent over a wire. (notice, not everything is a character, there are also character like "delete" and "go down a line" here. this is not for displaying, this is for every interfaces)

Something to remember for later: the number 0 is encoded as NULL, basically "nothing". This is useful because sometimes you want to enter text with an unspecified length, and so you stick a NULL in the end, and the interface reading it reads until it sees a NULL, and all is well. this will be important later

Standard explained, technical info for nerds, go to the next red section to pass

ASCII is a wonderful standard. remember: everything in electronics is easier with powers of 2 (1,2,4,8,16,32 etc.) because of the way we save data (if you want I can explain this further); the first 32 characters are the control characters. want to check if something isn't a control character? check if it's 128 or bigger than 32, and you're done (both powers of 2). the lowercase characters are 32 + their uppercase counterpart. all the numbers have a byte in common. truly, a marvel of engineering.

Standard explanation end

All was well until computers hit the scene not too long after, and used bytes. a byte is basically a whole number whose value can be only from 0-255. they are the standard building block of computer memory, and they have 8 bits.

some countries, like France, used encodings compatible with ASCII, and used the final bit to encode their language's characters. different countries used different versions of encodings, some countries (like Japan) had multiple encodings for the same characters. each encoding used a different number of bits, and different letters for each bit.

But that is fine since, well, how often do you need a computer in London to use an interface in Tokyo? all is well.

Then the World Wide Web happens, and suddenly computers speaking different languages read and write complete garbage everywhere.

So an organization called the Unicode Consortium tries to solve the problem, and to create a unified symbol for all languages. They called the standard utf-8

This standard supports 1,114,112 different characters. at present, only around 10% of this data capability is actually used. this includes dead languages, and emojis (which is a wonderful story)

Standard explained, technical info for nerds, go to the next red section to pass

Issues to tackle in a universal text encoding standard:

The protocol must be backwards compatible with ASCII: if you are writing text in English, which is the language most users used, because ASCII is the standard for this language, your new standard must be readable as ASCII as well

The protocol must never send 8 zeros in a row, except for the NULL character, otherwise old computers will stop reading in the middle

You must be able to minimize space wasted: to create a universal standard one can just make every character 32 bytes long and call it a day, but you would waste a bunch of space that way, and space is expensive

You must be able to pass from letter to letter easily. no saving the index of each character in some sort of list.

english characters are just ASCII. no thinking there. the first bit is set to 0 and so it is very easy to spot

if not, here's what you do:

the first byte has its first bit set to 1, so it's not ASCII. from that point onwards, you count the number of remaining ones until a zero appears. in this case, 1. this is how many more bytes will come. from there on, the rest is data. the first 2 bits of every next byte would start with 10 until the character ends

let's say your character is 2 bytes long, here is how you would represent it:

110somec , 10haract

and when removing the headers, you'll have

somecharact

which will be some character.

let's say your character is 3 bytes long, here is how you would represent it:

1110some , 10charac , 10ter___

and when removing the headers, you'll have

somecharacter___

which will be another character.

if you wanna go back 1 character? just go back bytes until you find one that starts with something other than 10

no excess Nulls will appear because the only way to get 8 zeros

Standard explanation end

[rtcd-io]

Rtcd.io is a simple yet powerful online toolbox for encoding, encryption, data conversion, image processing and more. Totally free and no registration needed. Check it out at https://rtcd.io/

[amicus-19]

working on fixing the streaming version of where's lazlo so that it doesn't have all the bad issues with the artifacting. will upload to the big mega folder with all the other stuff once it's finished

[cerulity]

The reason that QR codes on volatile pixel canvases (like r/place or ourworldofpixels) usually work perfectly is the exact same reason that logos can be placed inside of the codes. QR codes have error correction that allows them to fully recover a certain percentage of data (dependent on the amount of resilience set when creating the QR code). In the case of public canvases, sneakily changing a few pixels won't actually do anything, because the incorrect pixels will be corrected. In the case of logos, a set amount of data is destroyed to fit the logo, and whatever that logo scans as is corrected.

[netscapenavigator-official]

So, like I said I was, I'm re-encoding all the videos on my Jellyfin server as AV1 to save space. I've been doing that for about 4-5 days, straight, at this point. So, I'm gonna give a progress update at just how far I've gotten in those 4-5 days.

So far, I've managed to re-encode a singular folder that was about 13.72 GB.

I still have many other folders to go, that all total up to about 109.12 GB.

I'm going to be here forever.

On the bright side, re-encoding the videos as AV1 fixed some of the corruption that caused a lot of them to become unplayably stuttery.

Also, what I have finished re-encoding was once 10.15 GB. It is now 7.70 GB. So, while it is taking forever, I have managed to save over two whole gigabytes by simply running my videos through a program that essentially makes my computer get hot.

So with this level of success without even scratching the surface, I'm excited to see how much space I'll save when all is finally completed.

[stream-academy]

"My stream looks bad."

This is a pretty common issue that creators face and it all stems from one common misconception: 1080p doesn't mean quality.

The above video is in fact 1080p, but as you can see there's definitely something wrong about it. Here's that exact same freezeframe but at 480p:

So what's happening here? Bitrates.

A Crash Course Through Video Compression

Computer video has historically been a really hard problem to solve. Computers used to be slow (Like, really slow). Computer scientists (read: nerds) figured out that the human eye is bad and have developed techniques that abuse this fact.

Dithering tricks your eye into "seeing" more colors than are actually present. Here's a monotone (two colors!) that appears to also have shades of grey even though there aren't any:

Pretty cool, eh? By using clever placement of the black dots an entire range of grays emerge even though there aren't any.

You can even throw out little bits of detail to save space and the eye will be none the wiser. These are the core principles behind video compression.

Why do we even need to compress video anyways?

The short answer: internet speeds still aren't fast enough. The long answer: distribution costs.

If you are a video site, such as Twitch or YouTube, it is in your best interests to save as much money as possible for every live stream. So, every platform places limits on bitrate.

Okay, get to the point already. What is bitrate and how do I get my stream to look good?

Video compression usually has two modes: constant bitrate (CBR), and variable bitrate (VBR).

For regular video recordings it's usually recommended you use variable bitrate (note that CRF/CQ is a form of variable bitrate). This allows you to save space where you can but increase usage where it's necessary to help maintain visual quality.

But for live-streaming you use constant bitrate (the reason is due to how networks actually operate and how celluar/mobile networks handle streaming. It's complicated). With CBR, you tell the encoder that you want the result to always be at X bits per second regardless of the resulting quality (good or bad). So that means that during moments of low movement (in game, in an IRL setting, or even on your desktop) the encoder doesn't have to work nearly as hard to fit the current frame into the allotted rate. During high movement moments the encoder has to work extra hard to find those visual shortcuts — such as dithering, detail removal, or macroblocks (shudders) — to keep everything within bounds.

Obviously, removing detail will hurt the quality of an image. And doing that repeatedly will result in that reduced image quality you saw in the first 1080p screenshot. For that screenshot, I encoded the video at 500kbps, which is obviously way too low (I did that for for clickbait). For the 480p screenshot I encoded it using a VBR technique called Constant Quality. The size difference between those two files paints a better picture: the 480p version is 12x larger than the 1080p version despite 1080p having 8x as many pixels as 480p.

Alright, so what do you recommend I use?

Well factors like resolution, framerate, your internet, and your computer's hardware all play a role in determining what's best for you.

Here's a table bitrates for resolution and framerate.

Just note that your internet upload speed needs to be large enough for that bitrate AND whatever else you may use the internet for at the same time.

And that's it! Your stream should look loads better! And that's it for this installment of Stream Academy. Feel free to ask us questions about streaming or for help with any problems you've been having along the way. If you're looking for widgets and alerts for your live-stream journey, then you should look into using Casterlabs (that's us! 😁)

Image Sources

Throneful (2021, January 8th). "Apex Legends (2021) - Gameplay (PC UHD) [4K60FPS]" [Video]. YouTube. https://www.youtube.com/watch?v=UPtoIy-oPWQ

Renderedartwork (2012, December 21st). "Michelangelo's David dithered using the Void-and-Cluster technique." [Image]. Wikimedia Foundation. https://commons.wikimedia.org/wiki/File:Michelangelo%27s_David_-_Void-and-Cluster.png

[fourwind-films]

Avid Media Composer Transcoding Instructions

These are instructions to transcode in Avid Media Composer for a professional network workflow. It’s been used for television programs on HBO, PBS, and other networks.

For cheaper cameras like the SONY AS7, CANON 5D series or often most cameras that shoot in the H.264 codec, you need to relabel the linked clips before you import them. This is just to ensure that each clip has a unique and individual name to help later on with the Onlining process. Add the Disk Label with an underscore to the very beginning of each clip. An example is as follows: Original Name: C001 changed to the Modified Name: AxS01_20180918_NEG01_1of1_C001.

In a new bin, right click. In the drop down menu choose Input > Source Browser.

In the pop-up window, navigate to the desired camera card. Click on the root of the camera card. Double check that the target bin at the bottom right of the window is the bin you want to populate with the clips in and press the link button.

Check to take out the data tracks and any blank audio tracks that come in with the clip. This is common extra data that is imported into Avid that are annoying to edit with from SONY cameras like the FS7 as well as other brands of cameras. Do your research and know your camera and what was recorded by talking to the camera operators or reading the camera reports.

Once the clips have populated in your bin, highlight all of the clips and right click. In the drop down menu select Modify > Unlink Media. (The clips need to be Offline/The AMA link broken in order to modify the clip)

Select all the clips again, right click and select Modify > Modify Clip. 

At the top of the new pop up window, click the drop down menu and Set Tracks. Uncheck the D (Data) Track at the bottom and deselect any blank audio tracks (TBD which are blank). 

Once you’ve disabled the Data track and blank audio tracks, click OK. 

Right click inside the bin and select Import > Source Browser again from the drop down menu. Navigate back to the card folder and click the link. The clips should appear online in the bin now, but without the Data track and any audio tracks you also disabled.

Once all of the linked clips are brought in with proper names and data tracks have been disabled, you can assign a Disk Label to each clip to assist in final online relinking. To do this copy the designated Disk Label and paste it clip by clip in the Disk Label column.

[gaurja24510045]

The Strange Magic of Memory: How Songs and Smells Take Us Back

Have you ever been walking through a store, and suddenly a song plays that just stops you in your tracks? In an instant, you’re traveling back to a moment in time you didn’t even realize you had tucked away — maybe it’s a high school trek, or the initial few months of your relationship when you were all about your partner, or maybe even when you were depressively struggling to get through the first year if college with no friends around to support.

Or do you ever just randomly get a whiff of something cooking, or smell a flower that immediately transports you back in time to summer vacations in middle school at your grandma’s place? This phenomenon, which is almost universally experienced, is not just coincidence or random nostalgia. There’s actually some science behind why this happens, it’s called the transfer-appropriate processing effect.

What this effect says is that memory works better when the conditions at the time of recalling them are the same as the conditions when you first experienced it. Episodic memory, what the memory of events in your life is called, is not isolated. It’s deeply embedded in the spatiotemporal context. It’s not just about what you consciously remember — it’s about the tiny, background details too: the sounds, the smells, the environment. When your brain recognizes those familiar cues, it becomes much easier to pull up those stored memories. 

Music has always been one of the strongest memory triggers for me. I have playlists from different phases of my life, and sometimes I’ll put them on shuffle just to see where they take me. One song instantly throws me back to my senior year of college — late nights cramming at the library, drinking bottles on bottles of terrible sting , just blindly hoping I get through the night. It’s not just the memory of studying that comes back; it’s the full feeling of being there. I can almost feel the AC on my skin, the musty smell of the library sofas. Smells do it too. The scent of this particular brand of sunscreen immediately reminds me of swim lessons at the nearby pool, the disgusting changing rooms, the nicest driver uncle who used to take me there, and the strict coach who I used to be terrified of. And of course, visual cues are quite the time machine. An ad from back then, or spotting my old favourite sweater will take me back to when all of the kids in the neighbourhood would get together to trouble everyone with ding dong ditch!

But, it’s not just about what’s happening in the surroundings, though. Transfer-appropriate processing effect also includes the physical context. Physical feelings and emotions also come rushing back in presence of such cues. You may feel your stomach turn when you hear your school’s anthem, which was sung in assemblies you were punished for reaching late to. When the external conditions match up with those old memories, it creates a bridge that makes it easier to retrieve them. 

Lately, I’ve been wondering if this effect, which so certainly associates the context of the memories with their content, can be used strategically. If we know certain cues help lock in memories, maybe it makes sense to intentionally try and save memories by organising them in specific contexts. Maybe, now that I am about to be at a new campus for an internship, I will wear a specific perfume everyday, so each time I wear it later, I am brought back to memories of the stay. Maybe each time I go on a trio with my friends, there will be a special playlist dedicated to it!

[russellmoreton]

DSC_0277a Speculative Encoding : Transactive Memory by Russell Moreton

[nebulastarss]

Giggling and kicking my feet as I unnecessarily encode the entire alphabet for a 5th time in the name of fanfiction

[alpha1convert]

Follow Alpha1Convert on all our socials!

https://x.com/Alpha1Convert/

https://www.instagram.com/alpha1convert/

https://www.pinterest.com/alpha1convert/

https://www.threads.net/@alpha1convert

https://www.youtube.com/channel/UCbFkgRJpUmFzr1Z7NaOVimQ

https://rumble.com/c/Alpha1Convert

https://alpha1convert.start.page

[tenth-sentence]

In Indigenous culture, symbols, patterns and images are used to encode knowledge in supra-rational ways.

"Right Story, Wrong Story: Adventures in Indigenous Thinking" - Tyson Yunkaporta

[saying-french-twenty-times]

Something noteworthy about h265 and many other encodings is that while they do reduce filesize, they never get so complex that I've had something like, buffering due to my hardware needing a sec to decode stuff. Which makes me wonder, if that is something that Computer Scientists have to account for when designing these encodings, and what could be achieved if we allowed for some I guess "local buffering" just to see how small we can get a video. Of course I understand it isn't sensible for most cases, but I am just curious, and I am not knowledgeable enough in the department to answer my own question

[netscapenavigator-official]

Finally pirated acquired Game Changer Season 6. All of the episodes are encoded in H.264. By converting them to AV1 I just turned Episode 1 from 1.37 GB to 374 MB. Literally a whole gigabyte gone. Something I don't have to deal with or waste storage space on. All because my computer got a little toasty. I love modern codecs so much.

[rabatfirstsight]

URL percent encoding

If you want to use password with special character, like an exclamation mark, you need to use percent encoding which is often called URL encoding.