#Data Technology

[elucidata]

Scaling Data Pipelines for High-throughput Bioinformatics

Data pipelines are an essential tool in bioinformatics. They enable the standardization of data-cleaning and complex data-processing which in turn results in better reproducibility. Most data pipelines use some kind of framework for executing its tasks. Modern pipeline orchestration frameworks do a good job of abstracting and separating pipeline logic from infrastructure complexity. But that complexity remains to be managed at some level.

Source

[pebblegalaxy]

Celebrating Excellence: Highlights from the 5th Annual Data Breakthrough Awards Program

Celebrating Excellence: Highlights from the 5th Annual Data Breakthrough Awards Program #DataBreakthrough #InnovationAwards #DataTechnology #Excellence #TechLeaders

Recognizing Exceptional Innovators: Highlights from the 5th Annual Data Breakthrough Awards Program The fifth annual Data Breakthrough Awards program has unveiled its distinguished winners, spotlighting the exceptional achievements in data technology across the globe. Organized by Data Breakthrough, an esteemed market intelligence entity, the awards recognize the forefront leaders, products, and…

[ahb-writes]

(from The Mitchells vs. the Machines, 2021)

[nasa]

Hot New Planetary System Just Dropped.

We hope you like your planetary systems extra spicy. 🔥

A new system of seven sizzling planets has been discovered using data from our retired Kepler space telescope.

Named Kepler-385, it’s part of a new catalog of planet candidates and multi-planet systems discovered using Kepler.

The discovery helps illustrate that multi-planetary systems have more circular orbits around the host star than systems with only one or two planets.

Our Kepler mission is responsible for the discovery of the most known exoplanets to date. The space telescope’s observations ended in 2018, but its data continues to paint a more detailed picture of our galaxy today.

Here are a few more things to know about Kepler-385:

All seven planets are between the size of Earth and Neptune.

Its star is 10% larger and 5% hotter than our Sun.

This system is one of over 700 that Kepler’s data has revealed.

The planets’ orbits have been represented in sound.

Now that you’ve heard a little about this planetary system, get acquainted with more exoplanets and why we want to explore them.

Make sure to follow us on Tumblr for your regular dose of space!

[jastervhett]

The Return…Of Boba Fett's Armor! - The Mandalorian: Chapter 14 (Season 2 Episode 6)

(❤️ this scene & dialogue)

Din Djarin: (Approaching Boba & Fennec) This is all that survived…(Shows them his Beskar spear) Boba Fett: (Instantly recognizing what the spear is made of) Beskar. Din Djarin: (Nods yes) Boba Fett: I want you to take a look at something….

Boba Fett: (Raises his left gauntlet & activates its holographic data read-out display. Looking at the holographic data, which is written in Mando'a….) Boba Fett: My chain code has been encoded in this armor for 25 years. (Indicating hologram) See, this is me…Boba Fett. Boba Fett: (Points at one holographic symbol) This is my father, Jango Fett. (Looks up from the display at Din Djarin)

Din Djarin: (Considering Boba's words, observing the display Boba has just shown him & hearing the name of Mandalorian Bounty Hunter Jango Fett….) Din Djarin: (Din immediately understands why this man has claimed the Mandalorian armor he is now wearing) Your father was a foundling.

Boba Fett: Yes. (Deactivates the display) He even fought in the Mandalorian Civil Wars. Din Djarin: Then that armor belongs to you. Boba Fett: (Nods yes) I appreciate it's return….

THIS scene confirmed SO much: *IS Jango a Mandalorian? YES! *DOES Boba read/understand Mando'a? Uh-huh! *DOES Boba Know his Mandalorian Heritage? Yup! *AFTER All of these years is Boba FINALLY shown on screen being a bad-a$$ with & without his Mandalorian Armor? HELL YES!!!

As a long-time fan of the Mando'ade/ Boba Fett/ Jango Fett/ Jaster Mereel/ True Mandalorians & a big fan of The Mandalorian, it was so satisfying to watch this episode & witness the above-referenced exchange between Din Djarin & Boba Fett.

Boba Fett IS canonically alive! It's official!

(AND this scene FINALLY shuts down all of the "Jango Fett really wasn't a Mandalorian" osik).

[dinosaurspen]

CDC 6600 Supercomputer

[enneadau]

EMERGENCY AUTHOR UPDATE

I feel like this needs to be warned about. Everything on Ao3 that isn't set to private, HAS been data scraped and fed to 3 data sites that provide data for AI training, including writing and artwork.

Yes, this includes my entire Ennead series and everything else I've ever written and posted. As well as anything you all have written but not made private.

Ao3's legal team is fighting it and one site has made the data unavailable, but the other two aren't based in the USA so the fight is harder.

This is frustrating and upsetting news, especially for those of us who now need to pick between our Guest readers who have supported us for a long time and protecting the hard work that we've put our hearts and souls into and I just ask that we support each other and our choices during this time.

The link here has more details but from now on, until I can be sure there's a way to protect my work, which I've spent decades writing and planning, my stories will be posted for members of the site only.

[unbfacts]

[thegikitiki]

The World at Your Fingertips...

Control Data Corporation, 1968

[elucidata]

[science70]

Controller about to insert an Octopack of cartridges into a Control Data Corporation CDC Mass Storage System, 1975.

[bobcat-pie]

[mostlysignssomeportents]

Cloudburst

Enshittification isn’t inevitable: under different conditions and constraints, the old, good internet could have given way to a new, good internet. Enshittification is the result of specific policy choices: encouraging monopolies; enabling high-speed, digital shell games; and blocking interoperability.

First we allowed companies to buy up their competitors. Google is the shining example here: having made one good product (search), they then fielded an essentially unbroken string of in-house flops, but it didn’t matter, because they were able to buy their way to glory: video, mobile, ad-tech, server management, docs, navigation…They’re not Willy Wonka’s idea factory, they’re Rich Uncle Pennybags, making up for their lack of invention by buying out everyone else:

https://locusmag.com/2022/03/cory-doctorow-vertically-challenged/

But this acquisition-fueled growth isn’t unique to tech. Every administration since Reagan (but not Biden! more on this later) has chipped away at antitrust enforcement, so that every sector has undergone an orgy of mergers, from athletic shoes to sea freight, eyeglasses to pro wrestling:

https://www.whitehouse.gov/cea/written-materials/2021/07/09/the-importance-of-competition-for-the-american-economy/

But tech is different, because digital is flexible in a way that analog can never be. Tech companies can “twiddle” the back-ends of their clouds to change the rules of the business from moment to moment, in a high-speed shell-game that can make it impossible to know what kind of deal you’re getting:

https://pluralistic.net/2023/02/27/knob-jockeys/#bros-be-twiddlin

To make things worse, users are banned from twiddling. The thicket of rules we call IP ensure that twiddling is only done against users, never for them. Reverse-engineering, scraping, bots — these can all be blocked with legal threats and suits and even criminal sanctions, even if they’re being done for legitimate purposes:

https://locusmag.com/2020/09/cory-doctorow-ip/

Enhittification isn’t inevitable but if we let companies buy all their competitors, if we let them twiddle us with every hour that God sends, if we make it illegal to twiddle back in self-defense, we will get twiddled to death. When a company can operate without the discipline of competition, nor of privacy law, nor of labor law, nor of fair trading law, with the US government standing by to punish any rival who alters the logic of their service, then enshittification is the utterly foreseeable outcome.

To understand how our technology gets distorted by these policy choices, consider “The Cloud.” Once, “the cloud” was just a white-board glyph, a way to show that some part of a software’s logic would touch some commodified, fungible, interchangeable appendage of the internet. Today, “The Cloud” is a flashing warning sign, the harbinger of enshittification.

When your image-editing tools live on your computer, your files are yours. But once Adobe moves your software to The Cloud, your critical, labor-intensive, unrecreatable images are purely contingent. At at time, without notice, Adobe can twiddle the back end and literally steal the colors out of your own files:

https://pluralistic.net/2022/10/28/fade-to-black/#trust-the-process

The finance sector loves The Cloud. Add “The Cloud” to a product and profits (money you get for selling something) can turn into rents (money you get for owning something). Profits can be eroded by competition, but rents are evergreen:

https://pluralistic.net/2023/07/24/rent-to-pwn/#kitt-is-a-demon

No wonder The Cloud has seeped into every corner of our lives. Remember your first iPod? Adding music to it was trivial: double click any music file to import it into iTunes, then plug in your iPod and presto, synched! Today, even sophisticated technology users struggle to “side load” files onto their mobile devices. Instead, the mobile duopoly — Apple and Google, who bought their way to mobile glory and have converged on the same rent-seeking business practices, down to the percentages they charge — want you to get your files from The Cloud, via their apps. This isn’t for technological reasons, it’s a business imperative: 30% of every transaction that involves an app gets creamed off by either Apple or Google in pure rents:

https://www.kickstarter.com/projects/doctorow/red-team-blues-another-audiobook-that-amazon-wont-sell/posts/3788112

And yet, The Cloud is undeniably useful. Having your files synch across multiple devices, including your collaborators’ devices, with built-in tools for resolving conflicting changes, is amazing. Indeed, this feat is the holy grail of networked tools, because it’s how programmers write all the software we use, including software in The Cloud.

If you want to know how good a tool can be, just look at the tools that toolsmiths use. With “source control” — the software programmers use to collaboratively write software — we get a very different vision of how The Cloud could operate. Indeed, modern source control doesn’t use The Cloud at all. Programmers’ workflow doesn’t break if they can’t access the internet, and if the company that provides their source control servers goes away, it’s simplicity itself to move onto another server provider.

This isn’t The Cloud, it’s just “the cloud” — that whiteboard glyph from the days of the old, good internet — freely interchangeable, eminently fungible, disposable and replaceable. For a tool like git, Github is just one possible synchronization point among many, all of which have a workflow whereby programmers’ computers automatically make local copies of all relevant data and periodically lob it back up to one or more servers, resolving conflicting edits through a process that is also largely automated.

There’s a name for this model: it’s called “Local First” computing, which is computing that starts from the presumption that the user and their device is the most important element of the system. Networked servers are dumb pipes and dumb storage, a nice-to-have that fails gracefully when it’s not available.

The data structures of source-code are among the most complicated formats we have; if we can do this for code, we can do it for spreadsheets, word-processing files, slide-decks, even edit-decision-lists for video and audio projects. If local-first computing can work for programmers writing code, it can work for the programs those programmers write.

Local-first computing is experiencing a renaissance. Writing for Wired, Gregory Barber traces the history of the movement, starting with the French computer scientist Marc Shapiro, who helped develop the theory of “Conflict-Free Replicated Data” — a way to synchronize data after multiple people edit it — two decades ago:

https://www.wired.com/story/the-cloud-is-a-prison-can-the-local-first-software-movement-set-us-free/

Shapiro and his co-author Nuno Preguiça envisioned CFRD as the building block of a new generation of P2P collaboration tools that weren’t exactly serverless, but which also didn’t rely on servers as the lynchpin of their operation. They published a technical paper that, while exiting, was largely drowned out by the release of GoogleDocs (based on technology built by a company that Google bought, not something Google made in-house).

Shapiro and Preguiça’s work got fresh interest with the 2019 publication of “Local-First Software: You Own Your Data, in spite of the Cloud,” a viral whitepaper-cum-manifesto from a quartet of computer scientists associated with Cambridge University and Ink and Switch, a self-described “industrial research lab”:

https://www.inkandswitch.com/local-first/static/local-first.pdf

The paper describes how its authors — Martin Kleppmann, Adam Wiggins, Peter van Hardenberg and Mark McGranaghan — prototyped and tested a bunch of simple local-first collaboration tools built on CFRD algorithms, with the goal of “network optional…seamless collaboration.” The results are impressive, if nascent. Conflicting edits were simpler to resolve than the authors anticipated, and users found URLs to be a good, intuitive way of sharing documents. The biggest hurdles are relatively minor, like managing large amounts of change-data associated with shared files.

Just as importantly, the paper makes the case for why you’d want to switch to local-first computing. The Cloud is not reliable. Companies like Evernote don’t last forever — they can disappear in an eyeblink, and take your data with them:

https://www.theverge.com/2023/7/9/23789012/evernote-layoff-us-staff-bending-spoons-note-taking-app

Google isn’t likely to disappear any time soon, but Google is a graduate of the Darth Vader MBA program (“I have altered the deal, pray I don’t alter it any further”) and notorious for shuttering its products, even beloved ones like Google Reader:

https://www.theverge.com/23778253/google-reader-death-2013-rss-social

And while the authors don’t mention it, Google is also prone to simply kicking people off all its services, costing them their phone numbers, email addresses, photos, document archives and more:

https://pluralistic.net/2022/08/22/allopathic-risk/#snitches-get-stitches

There is enormous enthusiasm among developers for local-first application design, which is only natural. After all, companies that use The Cloud go to great lengths to make it just “the cloud,” using containerization to simplify hopping from one cloud provider to another in a bid to stave off lock-in from their cloud providers and the enshittification that inevitably follows.

The nimbleness of containerization acts as a disciplining force on cloud providers when they deal with their business customers: disciplined by the threat of losing money, cloud companies are incentivized to treat those customers better. The companies we deal with as end-users know exactly how bad it gets when a tech company can impose high switching costs on you and then turn the screws until things are almost-but-not-quite so bad that you bolt for the doors. They devote fantastic effort to making sure that never happens to them — and that they can always do that to you.

Interoperability — the ability to leave one service for another — is technology’s secret weapon, the thing that ensures that users can turn The Cloud into “the cloud,” a humble whiteboard glyph that you can erase and redraw whenever it suits you. It’s the greatest hedge we have against enshittification, so small wonder that Big Tech has spent decades using interop to clobber their competitors, and lobbying to make it illegal to use interop against them:

https://locusmag.com/2019/01/cory-doctorow-disruption-for-thee-but-not-for-me/

Getting interop back is a hard slog, but it’s also our best shot at creating a new, good internet that lives up the promise of the old, good internet. In my next book, The Internet Con: How to Seize the Means of Computation (Verso Books, Sept 5), I set out a program fro disenshittifying the internet:

https://www.versobooks.com/products/3035-the-internet-con

The book is up for pre-order on Kickstarter now, along with an independent, DRM-free audiobooks (DRM-free media is the content-layer equivalent of containerized services — you can move them into or out of any app you want):

http://seizethemeansofcomputation.org

Meanwhile, Lina Khan, the FTC and the DoJ Antitrust Division are taking steps to halt the economic side of enshittification, publishing new merger guidelines that will ban the kind of anticompetitive merger that let Big Tech buy its way to glory:

https://www.theatlantic.com/ideas/archive/2023/07/biden-administration-corporate-merger-antitrust-guidelines/674779/

The internet doesn’t have to be enshittified, and it’s not too late to disenshittify it. Indeed — the same forces that enshittified the internet — monopoly mergers, a privacy and labor free-for-all, prohibitions on user-side twiddling — have enshittified everything from cars to powered wheelchairs. Not only should we fight enshittification — we must.

Back my anti-enshittification Kickstarter here!

If you’d like an essay-formatted version of this post to read or share, here’s a link to it on pluralistic.net, my surveillance-free, ad- free, tracker-free blog:

https://pluralistic.net/2023/08/03/there-is-no-cloud/#only-other-peoples-computers

Image: Drahtlos (modified) https://commons.wikimedia.org/wiki/File:Motherboard_Intel_386.jpg

CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0/deed.en

—

cdsessums (modified) https://commons.wikimedia.org/wiki/File:Monsoon_Season_Flagstaff_AZ_clouds_storm.jpg

CC BY-SA 2.0 https://creativecommons.org/licenses/by-sa/2.0/deed.en

[i-j0s]

[jastervhett]

Star Wars Republic Commando Intro Remake by Oleksandr Maziura

I've just come across this recently. This looks SO cool & incredibly detailed. If ANY OG Lucasarts Star Wars game deserves a remake, it's this gem of a video game, Star Wars Republic Commando (2005). Thank you so much Oleksandr Maziura for this Republic Commando Intro Remake.

In addition: Taun We's hand gesture was a really nice touch (showing her care & empathy towards Baby Boss). Also, after encountering that A-DSD Advanced Dwarf Spider Droid in the Training Simulator, I remember when playing the game I'd IMMEDIATELY toss a good ol' Detonator towards 'em, target 'em & then order ALL Squad Members to concentrate fire on THAT Target.

Back in the day I used to ❤️ using the Engage Target Command and see my Squad just unload their array of Blaster, Sniper & Anti-Armor rounds (in addition to their Thermal & EC (Electro-Static Charge) Detonators) while my Target's health just slowly drained away. Nothing beats Concentrated Firepower!