#Data Platform

[jcmarchi]

Unlock the other 99% of your data - now ready for AI

New Post has been published on https://thedigitalinsider.com/unlock-the-other-99-of-your-data-now-ready-for-ai/

Unlock the other 99% of your data - now ready for AI

For decades, companies of all sizes have recognized that the data available to them holds significant value, for improving user and customer experiences and for developing strategic plans based on empirical evidence.

As AI becomes increasingly accessible and practical for real-world business applications, the potential value of available data has grown exponentially. Successfully adopting AI requires significant effort in data collection, curation, and preprocessing. Moreover, important aspects such as data governance, privacy, anonymization, regulatory compliance, and security must be addressed carefully from the outset.

In a conversation with Henrique Lemes, Americas Data Platform Leader at IBM, we explored the challenges enterprises face in implementing practical AI in a range of use cases. We began by examining the nature of data itself, its various types, and its role in enabling effective AI-powered applications.

Henrique highlighted that referring to all enterprise information simply as ‘data’ understates its complexity. The modern enterprise navigates a fragmented landscape of diverse data types and inconsistent quality, particularly between structured and unstructured sources.

In simple terms, structured data refers to information that is organized in a standardized and easily searchable format, one that enables efficient processing and analysis by software systems.

Unstructured data is information that does not follow a predefined format nor organizational model, making it more complex to process and analyze. Unlike structured data, it includes diverse formats like emails, social media posts, videos, images, documents, and audio files. While it lacks the clear organization of structured data, unstructured data holds valuable insights that, when effectively managed through advanced analytics and AI, can drive innovation and inform strategic business decisions.

Henrique stated, “Currently, less than 1% of enterprise data is utilized by generative AI, and over 90% of that data is unstructured, which directly affects trust and quality”.

The element of trust in terms of data is an important one. Decision-makers in an organization need firm belief (trust) that the information at their fingertips is complete, reliable, and properly obtained. But there is evidence that states less than half of data available to businesses is used for AI, with unstructured data often going ignored or sidelined due to the complexity of processing it and examining it for compliance – especially at scale.

To open the way to better decisions that are based on a fuller set of empirical data, the trickle of easily consumed information needs to be turned into a firehose. Automated ingestion is the answer in this respect, Henrique said, but the governance rules and data policies still must be applied – to unstructured and structured data alike.

Henrique set out the three processes that let enterprises leverage the inherent value of their data. “Firstly, ingestion at scale. It’s important to automate this process. Second, curation and data governance. And the third [is when] you make this available for generative AI. We achieve over 40% of ROI over any conventional RAG use-case.”

IBM provides a unified strategy, rooted in a deep understanding of the enterprise’s AI journey, combined with advanced software solutions and domain expertise. This enables organizations to efficiently and securely transform both structured and unstructured data into AI-ready assets, all within the boundaries of existing governance and compliance frameworks.

“We bring together the people, processes, and tools. It’s not inherently simple, but we simplify it by aligning all the essential resources,” he said.

As businesses scale and transform, the diversity and volume of their data increase. To keep up, AI data ingestion process must be both scalable and flexible.

“[Companies] encounter difficulties when scaling because their AI solutions were initially built for specific tasks. When they attempt to broaden their scope, they often aren’t ready, the data pipelines grow more complex, and managing unstructured data becomes essential. This drives an increased demand for effective data governance,” he said.

IBM’s approach is to thoroughly understand each client’s AI journey, creating a clear roadmap to achieve ROI through effective AI implementation. “We prioritize data accuracy, whether structured or unstructured, along with data ingestion, lineage, governance, compliance with industry-specific regulations, and the necessary observability. These capabilities enable our clients to scale across multiple use cases and fully capitalize on the value of their data,” Henrique said.

Like anything worthwhile in technology implementation, it takes time to put the right processes in place, gravitate to the right tools, and have the necessary vision of how any data solution might need to evolve.

IBM offers enterprises a range of options and tooling to enable AI workloads in even the most regulated industries, at any scale. With international banks, finance houses, and global multinationals among its client roster, there are few substitutes for Big Blue in this context.

To find out more about enabling data pipelines for AI that drive business and offer fast, significant ROI, head over to this page.

[amalgjose]

Key points to be considered before choosing a modern Industrial IoT Platform

Digitalization is the key upgrade that every industry needs at this era. The industrial revolution has evolved to the current state through various phases. These are the key phases of Industrial revolution and we call it with a version in the order it is listed below (Industry 4.0, Industry 5.0 etc.) Mechanization Electrification Automation Digitalization Personalization Most of the…

[123albert]

https://selfgrowth.com/articles/top-10-factors-to-consider-when-building-a-data-platform-in-2024

Crafting a data platform that delivers results requires attention to detail. Learn about the key factors that will influence the success of your platform and how to address them.

[marklovesdata]

SnowDQ – Unveiling the Data Quality Tool for High Quality and Accurate Data

The infographic offers a glimpse into the importance of data quality, exploring how it shapes the destiny of businesses by enabling precise decision-making and fostering a culture of innovation.

Read more: Data Quality Tool for High-Quality Data

[mostlysignssomeportents]

Reverse engineers bust sleazy gig work platform

If you'd like an essay-formatted version of this post to read or share, here's a link to it on pluralistic.net, my surveillance-free, ad-free, tracker-free blog:

https://pluralistic.net/2024/11/23/hack-the-class-war/#robo-boss

A COMPUTER CAN NEVER BE HELD ACCOUNTABLE

THEREFORE A COMPUTER MUST NEVER MAKE A MANAGEMENT DECISION

Supposedly, these lines were included in a 1979 internal presentation at IBM; screenshots of them routinely go viral:

https://twitter.com/SwiftOnSecurity/status/1385565737167724545?lang=en

The reason for their newfound popularity is obvious: the rise and rise of algorithmic management tools, in which your boss is an app. That IBM slide is right: turning an app into your boss allows your actual boss to create an "accountability sink" in which there is no obvious way to blame a human or even a company for your maltreatment:

https://profilebooks.com/work/the-unaccountability-machine/

App-based management-by-bossware treats the bug identified by the unknown author of that IBM slide into a feature. When an app is your boss, it can force you to scab:

https://pluralistic.net/2023/07/30/computer-says-scab/#instawork

Or it can steal your wages:

https://pluralistic.net/2023/04/12/algorithmic-wage-discrimination/#fishers-of-men

But tech giveth and tech taketh away. Digital technology is infinitely flexible: the program that spies on you can be defeated by another program that defeats spying. Every time your algorithmic boss hacks you, you can hack your boss back:

https://pluralistic.net/2022/12/02/not-what-it-does/#who-it-does-it-to

Technologists and labor organizers need one another. Even the most precarious and abused workers can team up with hackers to disenshittify their robo-bosses:

https://pluralistic.net/2021/07/08/tuyul-apps/#gojek

For every abuse technology brings to the workplace, there is a liberating use of technology that workers unleash by seizing the means of computation:

https://pluralistic.net/2024/01/13/solidarity-forever/#tech-unions

One tech-savvy group on the cutting edge of dismantling the Torment Nexus is Algorithms Exposed, a tiny, scrappy group of EU hacker/academics who recruit volunteers to reverse engineer and modify the algorithms that rule our lives as workers and as customers:

https://pluralistic.net/2022/12/10/e2e/#the-censors-pen

Algorithms Exposed have an admirable supply of seemingly boundless energy. Every time I check in with them, I learn that they've spun out yet another special-purpose subgroup. Today, I learned about Reversing Works, a hacking team that reverse engineers gig work apps, revealing corporate wrongdoing that leads to multimillion euro fines for especially sleazy companies.

One such company is Foodinho, an Italian subsidiary of the Spanish food delivery company Glovo. Foodinho/Glovo has been in the crosshairs of Italian labor enforcers since before the pandemic, racking up millions in fines – first for failing to file the proper privacy paperwork disclosing the nature of the data processing in the app that Foodinho riders use to book jobs. Then, after the Italian data commission investigated Foodinho, the company attracted new, much larger fines for its out-of-control surveillance conduct.

As all of this was underway, Reversing Works was conducting its own research into Glovo/Foodinho's app, running it on a simulated Android handset inside a PC so they could peer into app's data collection and processing. They discovered a nightmarish world of pervasive, illegal worker surveillance, and published their findings a year ago in November, 2023:

https://www.etui.org/sites/default/files/2023-10/Exercising%20workers%20rights%20in%20algorithmic%20management%20systems_Lessons%20learned%20from%20the%20Glovo-Foodinho%20digital%20labour%20platform%20case_2023.pdf

That report reveals all kinds of extremely illegal behavior. Glovo/Foodinho makes its riders' data accessible across national borders, so Glovo managers outside of Italy can access fine-grained surveillance information and sensitive personal information – a major data protection no-no.

Worse, Glovo's app embeds trackers from a huge number of other tech platforms (for chat, analytics, and more), making it impossible for the company to account for all the ways that its riders' data is collected – again, a requirement under Italian and EU data protection law.

All this data collection continues even when riders have clocked out for the day – its as though your boss followed you home after quitting time and spied on you.

The research also revealed evidence of a secretive worker scoring system that ranked workers based on undisclosed criteria and reserved the best jobs for workers with high scores. This kind of thing is pervasive in algorithmic management, from gig work to Youtube and Tiktok, where performers' videos are routinely suppressed because they crossed some undisclosed line. When an app is your boss, your every paycheck is docked because you violated a policy you're not allowed to know about, because if you knew why your boss was giving you shitty jobs, or refusing to show the video you spent thousands of dollars making to the subscribers who asked to see it, then maybe you could figure out how to keep your boss from detecting your rulebreaking next time.

All this data-collection and processing is bad enough, but what makes it all a thousand times worse is Glovo's data retention policy – they're storing this data on their workers for four years after the worker leaves their employ. That means that mountains of sensitive, potentially ruinous data on gig workers is just lying around, waiting to be stolen by the next hacker that breaks into the company's servers.

Reversing Works's report made quite a splash. A year after its publication, the Italian data protection agency fined Glovo another 5 million euros and ordered them to cut this shit out:

https://reversing.works/posts/2024/11/press-release-reversing.works-investigation-exposes-glovos-data-privacy-violations-marking-a-milestone-for-worker-rights-and-technology-accountability/

As the report points out, Italy is extremely well set up to defend workers' rights from this kind of bossware abuse. Not only do Italian enforcers have all the privacy tools created by the GDPR, the EU's flagship privacy regulation – they also have the benefit of Italy's 1970 Workers' Statute. The Workers Statute is a visionary piece of legislation that protects workers from automated management practices. Combined with later privacy regulation, it gave Italy's data regulators sweeping powers to defend Italian workers, like Glovo's riders.

Italy is also a leader in recognizing gig workers as de facto employees, despite the tissue-thin pretense that adding an app to your employment means that you aren't entitled to any labor protections. In the case of Glovo, the fine-grained surveillance and reputation scoring were deemed proof that Glovo was employer to its riders.

Reversing Works' report is a fascinating read, especially the sections detailing how the researchers recruited a Glovo rider who allowed them to log in to Glovo's platform on their account.

As Reversing Works points out, this bottom-up approach – where apps are subjected to technical analysis – has real potential for labor organizations seeking to protect workers. Their report established multiple grounds on which a union could seek to hold an abusive employer to account.

But this bottom-up approach also holds out the potential for developing direct-action tools that let workers flex their power, by modifying apps, or coordinating their actions to wring concessions out of their bosses.

After all, the whole reason for the gig economy is to slash wage-bills, by transforming workers into contractors, and by eliminating managers in favor of algorithms. This leaves companies extremely vulnerable, because when workers come together to exercise power, their employer can't rely on middle managers to pressure workers, deal with irate customers, or step in to fill the gap themselves:

https://projects.itforchange.net/state-of-big-tech/changing-dynamics-of-labor-and-capital/

Only by seizing the means of computation, workers and organized labor can turn the tables on bossware – both by directly altering the conditions of their employment, and by producing the evidence and tools that regulators can use to force employers to make those alterations permanent.

Image: EFF (modified) https://www.eff.org/files/issues/eu-flag-11_1.png

CC BY 3.0 http://creativecommons.org/licenses/by/3.0/us/

[rainbowstargazerlilies]

Let's Try This Again!

One note on this post over this post over the next day or so (until Tuesday night, Eastern Time), is equal to 1 minute I'll need to write over the next few weeks. I'll set a goal on TrackBear (a site I've been using to track my word counts recently) and post how things have been going that way. Have fun!

[thatgirlonstage]

The thing about tumblr is that there’s a panic about how the site is dying and falling apart literally every other week so eventually if you’ve been here long enough you just get zen about it. Like if anything specific is your breaking point do whatever you want but personally they’re gonna have to pry me out of the vents like a feral raccoon before I leave. Anyway if you’re new here and you see people talk about how something is the end of tumblr and you’re afraid they’re correct I just want you to know I’ve been here through probably like 300 ends of tumblr. I’m not saying it will never happen for real but statistically I remain skeptical.

[chaos0pikachu]

"why is 4mins getting a second season? who asked for this"

y'all hating b/c 4mins is that bitch, Megan Thee Stallion Yuki Chiba in Mumashi, fries dipped in a chocolate milkshake, Cher in Clueless

and like, also one of VIU's top performing shows on the platform but saying 4mins was having a hot girl summer is funnier so

[chasingfictions]

guys I don’t know what the fuck happened to the meta ad algorithm but every single ad it is showing me between stories is like . severe idf propaganda. anti vaxism. pro life conspiracy theories . so that’s fun that’s cool. like i have been trying to use less Instagram and they said okay we’ve got just the thing . here’s israeli soldiers holding up huge ass guns next to a newborn baby . here’s your friend posting a story about Israeli war crimes in Gaza but don’t worry we’re gonna follow it up with an ad about how Israel is a land of happy smiling people come visit. now here’s someone talking about how exciting it is to find an unvaccinated surrogate for crunchy family planning. now here’s a lady talking about how abortion is bad for women’s mental health ….. and it’s like ok u guys win I am clicking out of stories now :)

[probablyasocialecologist]

Take the planetary computer to its logical end under platform capitalism: every inch of the earth is mapped and monitored. Carbon flows are predicted. A red flag fire warning for a forest in Australia triggers an automatic sell-off of carbon futures; someone’s bank account is crushed while they sleep. Now imagine the same platform is tracking species. Now do people. A fluctuation in the weather forecasts migrants: send more boats to Lampedusa.

All this is simultaneously hyperbolic and a logical extension of current trends. Maybe take it into what Shoshana Zuboff calls surveillance capitalism—nature’s behavioral surplus fabricated into prediction products that anticipate what it will do, which are traded in behavioral futures markets.

Automated machine processes not only know our behavior but also shape our behavior at scale. With this reorientation from knowledge to power, it is no longer enough to automate information flows about us; the goal now is to automate us.

This births a new species of power Zuboff calls “instrumentarianism”—shaping human behavior toward others’ ends. Now instead of human beings, do birds. Now do fish. Now do trees. If all this data is blackboxed, unknowable, and used to make a profit for a mega platform, that’s a horrific future—though if it was going to come to pass, you’d think it would have more hype than it does today.

Holly Jean Buck, Ending Fossil Fuels: Why Net Zero is Not Enough

[kbluebirdart]

I think at this point, No matter which platform you go to, you can't escape. Do you think they hire hundreds of programmers and set up hundreds of servers just to provide free services?

Nope! Twitter/X is not the product; the users themselves are the product

We are the product...

You become one as soon as you register and create an account on any social media platform. : (

[renjunniez]

helping edit research papers for the college journal and my hatred for Generative AI has never been greater. it's like the fucking plague in here i want AI to DIE

[tenicola]

ᕕ( ᐛ )ᕗ Sup ya’ll

It’s been a lonnnnnnng while since I updated this art blog, I know. I fully intended to upload things by the end of 2024, but the holidays left me busy and exhausted.

There’s a lot to say about the last year and my goals for 2025, but for now, I'll use the next week to play some catch-up with sharing my artwork each evening. This will just include the major art and characters I got to in the year, but for other updates and tidbits, feel free to visit and follow me on Bluesky!

[tvmblrsillyman]

@ nebulare.bsky.social <- finally set one up

[aaami]

:)

[anti-rop]

how can you champion free speech and then celebrate when millions of voices on tiktok are censored. hypocrite.

I didn’t want to talk about politics on this blog but, oh well, here we go. Response under the cut.

Let me preface this: I’ve never been a fan of TikTok and when talk of a ban first started to come onto the scene 6 years ago, I thought it was a good thing, for a multitude of reasons but I won't go into all of it. I'll focus on what the proposed ban and SCOTUS corresponded to. This is a topic of US national security and the type of precedents it sets for foreign companies operating in the US. I thought it would be good to act now [2019] rather than later [2025] because looking at the growth curve, it was a service that would easily become so popular that lawmakers would find themselves in an impossible position and a ban would never happen. 

Unfortunately, that’s exactly what’s happened. Again, in my opinion, now a horrible precedent exists. To any foreign government out there, the message is that you are allowed to enter US markets under any pretense, with zero reciprocity for US companies, and as long as you are popular and influential enough the US government and population will go out of its way to facilitate your access

If we are going to go to such extraordinary lengths for a foreign company and government the US must make a demand of absolute reciprocity, in my opinion. Meta, X, Google, Snapchat, and other US-based technology companies must be allowed total market access in China immediately with zero control by the Chinese Government (because that is what they have done through ByteDance owning Tiktok). When the Chinese government inevitably laughs at this demand, ask yourself why. They correctly see Meta, X, Facebook, and Google as instruments of US soft power and as cultural contamination of their civic ideal which undermines their hold on power.

However, we seem to naively believe we're immune from the same influence and have waited so long to act now that we face terrible choices. The one we've made inevitably means we will have a natural experiment now of what it means to allow a government that actively seeks to undermine our civic institutions with the most powerful known technological tool to do so. And the fact that the CCP and ByteDance decided to “shut it down” rather than divest it tells us everything we need to know. No free enterprise would willingly shut off access to 170 million users. 

Also, we should be concerned that millions of Americans acted like drug addicts going through withdrawal when they couldn't access a social media app for roughly 12 hours. That is also cause for great concern. But that's a conversation for another day.