#unstructured data

[hhmglobal]

AI bridges the mental health data gap using NLP & unstructured data—featured in healthcare industry publications & hospital & healthcare journals.

[jcmarchi]

The Sequence Opinion #537: The Rise and Fall of Vector Databases in the AI Era

New Post has been published on https://thedigitalinsider.com/the-sequence-opinion-537-the-rise-and-fall-of-vector-databases-in-the-ai-era/

The Sequence Opinion #537: The Rise and Fall of Vector Databases in the AI Era

Once regarded as a super hot category, now its becoming increasingly commoditized.

Created Using GPT-4o

Hello readers, today we are going to discuss a really controversial thesis: how vector DBs become one of the most hyped trends in AI just to fall out of favor in a few months.

In this new gen AI era, few technologies have experienced a surge in interest and scrutiny quite like vector databases. Designed to store and retrieve high-dimensional vector embeddings—numerical representations of text, images, and other unstructured data—vector databases promised to underpin the next generation of intelligent applications. Their relevance soared following the release of ChatGPT in late 2022, when developers scrambled to build AI-native systems powered by retrieval-augmented generation (RAG) and semantic search.

This essay examines the meteoric rise and subsequent repositioning of vector databases. We delve into the emergence of open-source and commercial offerings, their technical strengths and limitations, and the influence of traditional database vendors entering the space. Finally, we contrast the trajectory of vector databases with the lasting success of the NoSQL movement to better understand why vector databases, despite their value, struggled to sustain their standalone identity.

The Emergence of Vector Databases

[advisedskills]

Is unstructured data causing more problems for your business than you realize?

Companies that dismiss this hidden resource face unresolved challenges and lost opportunities.

Explore how unstructured data might be stalling growth-and the steps you can take to get ahead.

Read the article.

#UnstructuredData #DataAnalytics #BusinessInsights #Innovation #AdvisedSkills

[vectordatabasecode]

Advanced Retrieval Techniques

Retrieval-Augmented Generation with Citations - Explore how augmentation with citations can significantly improve the depth and reliability of generated content.

Similarity Metrics for Vector Search - Understand different metrics that drive the effectiveness of vector searches, crucial for refining retrieval systems.

Local Agentic RAG with Langraph and Llama3 - Discover the integration of local datasets with advanced retrieval frameworks for enhanced performance.

Multimodal RAG with CLIP, Llama3, and Milvus - A deep dive into a multimodal approach, combining textual and visual data for rich content generation.

Practical Guides for Developers

A Beginner's Guide to Using Llama 3 with Ollama, Milvus, LangChain - Perfect for developers new to our frameworks, offering step-by-step guidance.

Getting Started with a Milvus Connection and Getting Started: Pgvector Guide for Developers Exploring Vector Databases - These guides are essential for setting up and beginning work with vector databases.

Educational Articles on Embedding Techniques and Applications

Sparse and Dense Embeddings - A look at different embedding types, offering insights into their use-cases and benefits.

Mastering BM25: A Deep Dive into the Algorithm and Application in Milvus - An in-depth exploration of BM25, a core algorithm for understanding document relevance.

Comparing SPLADE Sparse Vectors with BM25 - Comparative analysis that helps in selecting the right tool for specific retrieval tasks.

Training Your Own Text Embedding Model - Empower your projects by creating custom models tailored to your specific data needs.

Implementing and Optimizing RAG

Guide to Chunking Strategies for RAG and Experimenting with Different Chunking Strategies via LangChain - Both resources provide strategic insights into segmenting text for better retrieval outcomes.

Optimize RAG with Rerankers: The Role and Tradeoffs - Detailed discussion on the optimization of retrieval systems for balance between accuracy and performance.

[shrkdd]

80% of data is unstructured. Manage it like a pro!

Unstructured data is a diverse array of information that is not stored in a traditional database. This includes emails, images, videos, and more. While unstructured data can be a valuable asset for businesses, it can also pose a significant challenge if it is not managed effectively.

In this webinar, industry experts will share their insights on how to manage unstructured data like a pro. They will discuss:

The different types of unstructured data and the associated risks

Strategies for identifying, organizing, and extracting value from unstructured data

Best practices for data security and compliance

This webinar is for business leaders, data professionals, and anyone who wants to learn more about how to manage unstructured data like a pro.

Register for the webinar today!

#unstructureddata #manageunstructureddata #datasecurity #dataprivacy #datagovernance #datacompliance #dataanalytics #datavisualization #datascience #bigdata

[rajaniesh]

Maximize Efficiency with Volumes in Databricks Unity Catalog

With Databricks Unity Catalog's volumes feature, managing data has become a breeze. Regardless of the format or location, the organization can now effortlessly access and organize its data. This newfound simplicity and organization streamline data managem

[simple-logic]

#PollTime What database type handles unstructured data?

A) NoSQL 📂 B) SQL 🗄️ C) Docker 🐳 D) API 🔗

Comments your answer below👇

💻 Explore insights on the latest in #technology on our Blog Page 👉 https://simplelogic-it.com/blogs/

🚀 Ready for your next career move? Check out our #careers page for exciting opportunities 👉 https://simplelogic-it.com/careers/

[innovativeroutinesinternational]

Structured, Semi- & Unstructured Data Masking

DarkShield classifies, finds, and deletes PII in RDBs and flat files, too, plus: free text, JSON, XML, HL7/X12, Parquet and log files; MS Office (Word, Excel and Powerpoint) and PDF documents, NoSQL DBs, as well as DICOM and other image formats. Visit Us: https://www.iri.com/products/iri-data-protector

[kawaiiwizardtale]

How to reduce product returns with Digital shelf analytics

Discover how Digital shelf analytics can help minimize product returns to transform your retail success. Dive in for actionable strategies. Read more https://xtract.io/blog/how-to-reduce-product-returns-with-digital-shelf-analytics/

[verticalcarousel]

Ohalo | Data X-Ray Unstructured Data Discovery Tool

Accelerate accurate data discovery and gain in-depth visibility.Discover, examine and take action on your unstructured data files with a powerful unstructured data discovery tool.

Most organizations have substantial amounts of unstructured data which is unknown. Eliminate blind spots for security, privacy and governance teams by providing visibility to how sensitive data is stored, shared, and used - even in the cloud. Where do you start?

Answering the most essential questions:

Where is my most critical, sensitive data?

What policies should be enforced on this data?

Who has access to the data and is it secure?

Explore Data X-Ray today! - https://www.ohalo.co/discovery

[mourning-again-in-america]

I've never been to Costco, but from the perspective of someone working grocery, most of the time a customer is in line for significantly more than average, it's because there's some sort of communications fuckup between systems, especially if people go through proper channels.

It used to be that if someone walked up with a bag of fried chicken from the hot deli without a pricetag, I'd have to call the deli, wait for them to pick up, ask them for the price, manually key in the price, call my manager to approve the manual price entry, and they'd have to physically walk over to my machine to approve it. Similarly for if a customer has a coupon in their app that they forgot to clip and they don't mention that they never clipped it, I have to call grocery for a price check, grocery has to pick up, find the item, give me the price and my manager might have to OK the alteration (naturally, these steps can and are skipped thanks to my phone having an SKU scanner that works on 99% of items). But similarly, if something just isn't in the system and I can't do a phone lookup for it (happens about once a week), I have to go through the above script, which costs a large amount per customer relative to the average cost per customer.

Also, there's a large amount of social items that people can buy. My grocery store sells transit passes, western union mail orders, lottery tickets, fresh flowers, and balloons inflated on-site, each of which require higher than average time per customer.

To conclude, I think most of their savings probably comes from having a better and simpler backend than most grocery companies, but having a backend that works 99.99 percent of the time saves a lot of unnecessary work for frontend people (and exasperation of customers), coupled with shaving off items that systemically have a high mean worker time per customer makes a much more efficient system.

[ohnoitstbskyen]

I know this is something you'll probably address when/if you do a video about it, but any thoughts on Riot updating the Masque of the Black Rose skins on PBE? (Just announced, model/SFX updates coming to live next patch)

I know they've patched skins before, but I can't recall them ever doing it to an entire release crop. I'm glad they're trying to give people their money's worth, I guess, but I'm really not looking forward to the possibility of a "buy it now and hope it gets improved later" standard, like with the Sahn-Uzal gacha.

I've got a video coming up on the 3BSkyen channel later reacting to the changes. Although word of caution, it is over an hour long and full of unstructured Yapping™ - I'll probably do a short about it also a bit later.

But yeah it's... not usually a great sign when a company has to repeatedly patch its cosmetics due to bad feedback. It's a sign that the artists don't have the time and resources to do as many rounds of feedback and revisions as they'd usually do, I think, and that the company is pivoting to new priorities. Especially the battle pass skins have been shovelware slop, things which are there to fill a spot on a track rather than to have or express any interesting visual ideas.

Hopefully, the revisions are also a sign that someone internally at Riot has managed to put together an argument, or a set of data, that has convinced whatever C-suite jackass decides these things to allow them more time and resources to do better work.

[kuntya]

Notes from the No Kings Protest

Boomers and Gen X SHOWED UP. I didn't see them at the previous protests

It felt massive. Not the biggest I've ever been to, but impressively huge

Unlike previous protests, this one was very unstructured. Instead of group chanting ("I can't hear you!! Yell louder!!!!"), this had a lot of sitting with your friends in the grass

Local organizations had little booths where you could pick up fliers. This was cool. I hadn't heard of all of them.

Live music -- not inescapably loud. Excellent.

Merch was sold. Maybe that's fine? Who vets who gets to sell?

I saw more explicitly anti-capitalist language than I expected. That's interesting.

People, I need you to take privacy more seriously. Most faces were uncovered. Phones everywhere. All of the orgs had info gated behind QR codes. Abysmal. Trump is literally creating a database of Americans' private data. Start becoming untrackable NOW.

Overall, excellent vibes. Maybe the call for Trump to be impeached will become more mainstream.

[mariacallous]

Palantir, the software company cofounded by Peter Thiel, is part of an effort by Elon Musk’s so-called Department of Government Efficiency (DOGE) to build a new “mega API” for accessing Internal Revenue Service records, IRS sources tell WIRED.

For the past three days, DOGE and a handful of Palantir representatives, along with dozens of career IRS engineers, have been collaborating to build a single API layer above all IRS databases at an event previously characterized to WIRED as a “hackathon,” sources tell WIRED. Palantir representatives have been onsite at the event this week, a source with direct knowledge tells WIRED.

APIs are application programming interfaces, which enable different applications to exchange data and could be used to move IRS data to the cloud and access it there. DOGE has expressed an interest in the API project possibly touching all IRS data, which includes taxpayer names, addresses, social security numbers, tax returns, and employment data. The IRS API layer could also allow someone to compare IRS data against interoperable datasets from other agencies.

Should this project move forward to completion, DOGE wants Palantir’s Foundry software to become the “read center of all IRS systems,” a source with direct knowledge tells WIRED, meaning anyone with access could view and have the ability to possibly alter all IRS data in one place. It’s not currently clear who would have access to this system.

Foundry is a Palantir platform that can organize, build apps, or run AI models on the underlying data. Once the data is organized and structured, Foundry’s “ontology” layer can generate APIs for faster connections and machine learning models. This would allow users to quickly query the software using artificial intelligence to sort through agency data, which would require the AI system to have access to this sensitive information.

Engineers tasked with finishing the API project are confident they can complete it in 30 days, a source with direct knowledge tells WIRED.

Palantir has made billions in government contracts. The company develops and maintains a variety of software tools for enterprise businesses and government, including Foundry and Gotham, a data-analytics tool primarily used in defense and intelligence. Palantir CEO Alex Karp recently referenced the “disruption” of DOGE’s cost-cutting initiatives and said, “Whatever is good for America will be good for Americans and very good for Palantir.” Former Palantir workers have also taken over key government IT and DOGE roles in recent months.

WIRED was the first to report that the IRS’s DOGE team was staging a “hackathon” in Washington, DC, this week to kick off the API project. The event started Tuesday morning and ended Thursday afternoon. A source in the room this week explained that the event was “very unstructured.” On Tuesday, engineers wandered around the room discussing how to accomplish DOGE’s goal.

A Treasury Department spokesperson, when asked about Palantir's involvement in the project, said “there is no contract signed yet and many vendors are being considered, Palantir being one of them.”

“The Treasury Department is pleased to have gathered a team of long-time IRS engineers who have been identified as the most talented technical personnel. Through this coalition, they will streamline IRS systems to create the most efficient service for the American taxpayer," a Treasury spokesperson tells WIRED. "This week, the team participated in the IRS Roadmapping Kickoff, a seminar of various strategy sessions, as they work diligently to create efficient systems. This new leadership and direction will maximize their capabilities and serve as the tech-enabled force multiplier that the IRS has needed for decades.”

The project is being led by Sam Corcos, a health-tech CEO and a former SpaceX engineer, with the goal of making IRS systems more “efficient,” IRS sources say. In meetings with IRS employees over the past few weeks, Corcos has discussed pausing all engineering work and canceling current contracts to modernize the agency’s computer systems, sources with direct knowledge tell WIRED. Corcos has also spoken about some aspects of these cuts publicly: “We've so far stopped work and cut about $1.5 billion from the modernization budget. Mostly projects that were going to continue to put us down the death spiral of complexity in our code base,” Corcos told Laura Ingraham on Fox News in March. Corcos is also a special adviser to Treasury Secretary Scott Bessent.

Palantir and Corcos did not immediately respond to requests for comment

The consolidation effort aligns with a recent executive order from President Donald Trump directing government agencies to eliminate “information silos.” Purportedly, the order’s goal is to fight fraud and waste, but it could also put sensitive personal data at risk by centralizing it in one place. The Government Accountability Office is currently probing DOGE’s handling of sensitive data at the Treasury, as well as the Departments of Labor, Education, Homeland Security, and Health and Human Services, WIRED reported Wednesday.

[all-chords-in-sync]

I still find myself tempted by the concept of a week consisting of seven consecutive rest days.

Yet, in perusing the Express's data bank, I've encountered a number of novel concepts.

It might be better to say that the concepts are not novel in and of themselves -- rather, that I have read works in which they are applied in novel ways (at least as far as my personal experience goes).

Namely, the terms "enrichment" and "comfort zone" are ones I've heard before, albeit only on occasion. I can't say I have extensive interest in behavioral science, hence my lack of familiarity with these terms. That said, it's all too easy to meander across a staggering range of research topics that are, at best, only tangentially connected... but I digress.

In any case, there are anecdotes of people who follow schedules consisting of those seven consecutive rest days I've previously proposed. It seems that the resulting lack of structure in one's life can, depending on the circumstances, lead to extraordinary, mind-numbing tedium. As it turns out, stress, like many things, is beneficial in moderation -- in this sense, I mean regular engagement with novel concepts and stimuli.

...Much like my exploration of the data bank, I suppose.

Perhaps it's not unreasonable to think we should also set aside time for non-rest activities -- ones that give us a sense of engagement and fulfillment. This would be a logical approach to staving off the boredom that often sets in during a lengthy interval of unstructured time.

...Having mused for this long, I can understand how behavioral science is an entire discipline worthy of study. Perhaps the concept of seven consecutive rest days warrants further research.