#NaturalLanguage

[aeolife]

Unlocking the power of Natural Language in AEO to boost visibility, engagement, and rankings across all platforms

[govindhtech]

NL2SQL With Gemini And BigQuery: A Step-by-Step Guide

Conversion of natural language to SQL

Beginning to use Gemini and BigQuery for NL2SQL (natural language to SQL)

The intriguing new technology known as Natural Language to SQL, or NL2SQL, was created by combining the classic Structured Query Language (SQL) with Natural Language Processing (NLP). It converts inquiries written in common human language into structured SQL queries.

The technology has enormous potential to change how we engage with data, which is not surprising.

With the help of NL2SQL, non-technical users like marketers, business analysts, and other subject matter experts can engage with databases, examine data, and obtain insights independently without requiring specific SQL expertise. Even SQL experts can save time by using NL2SQL to create sophisticated queries, which allows them to devote more time to strategic analysis and decision-making.

On the ground, how does that appear? Imagine having instant access to a chat interface where you can ask inquiries and receive real-time replies, or

“How many units were sold overall this month?”

“What are the main factors influencing the shift in APAC sales when comparing Q1 and Q2 sales?”

In the past, this would have required an expert to extract information from databases and turn it into business insights. By lowering obstacles to data access, it can democratize analytics by utilizing NL2SQL.

However, a number of obstacles prevent NL2SQL from being extensively used. We’ll look at NL2SQL solutions on Google Cloud and implementation best practices in this blog.

Data quality issues in practical applications

Let us first examine some of the factors that contribute to the difficulty of implementing NL2SQL.

Real-world production data poses a number of difficulties, even if NL2SQL performs best in controlled settings and straightforward queries. These difficulties include:

Data formatting variations: The same information can be expressed in a variety of ways, such as “Male,” “male,” or “M” for gender, or “1000,” “1k,” or “1000.0” for monetary amounts. Additionally, many organizations use poorly defined acronyms of their own.

Semantic ambiguity: Large Language Models (LLMs) frequently lack domain-specific schema comprehension, which results in semantic ambiguity. This can cause user queries to be misinterpreted, for example, when the same column name has many meanings.

Syntactic rigidity: If semantically correct queries don’t follow SQL’s stringent syntax, they may fail.

Unique business metrics: NL2SQL must manage intricate business computations and comprehend table relationships via foreign keys. To translate the question effectively, one must have a sophisticated understanding of the tables that need to be connected and modeled together. Additionally, there is no one standard approach to determine the business KPIs that each corporation should use in the final narrative report.

Client difficulties

Users�� questions are frequently unclear or complicated, so it’s not only the data that can be unclear or poorly formatted. These three frequent issues with user inquiries may make NL2SQL implementation challenging.

Ambiguous questions: Even questions that appear to be clear-cut can be unclear. For example, a query looking for the “total number of sold units month to date” may need to specify which date field to use and whether to use average_total_unit or running_total_unit, etc. The perfect NL2SQL solution will actively ask the user to select the correct column and use their input when creating the SQL query.

Underspecified questions: Another issue is queries that are not detailed enough. For example, a user’s question concerning “the return rate of all products under my team in Q4” does not provide enough details, such as which team should fully grasp the question. An optimal NL2SQL solution should identify areas of ambiguity in the initial input and ask follow-up questions to obtain a comprehensive representation of the query.

Complex queries that require a multi-step analysis: Numerous questions require several stages of analysis. Consider figuring out the main causes of variations in sales from quarter to quarter, for instance: A good NL2SQL solution should be able to deconstruct the study into digestible parts, produce interim summaries, and then create a thorough final report that answers the user’s question.

Dealing with the difficulties

In order to address these issues, Google designed Gemini Flash 1.5 as a routing agent that can categorize queries according to their level of complexity. It can enhance its results by applying methods like contribution analysis models, ambiguity checks, vector embeddings, and semantic searches after the question has been classified.

It reacts to instructions in a JSON format using Gemini. Gemini can act as a routing agent, for instance, by responding to the few-shot prompt.

Direct inquiries

The right column names in scope can be clarified for direct inquiries by utilizing in-context learning, draft SQL ambiguity checks, and user feedback loops. Additionally, simple questions can be guaranteed to generate SQL that is clear.

For straightforward inquiries, its method does the following:

Gathers quality question/SQL pairings.

Keeps samples in BigQuery rows.

Enables the question to have vector embeddings

Leverages BigQuery vector search to extract related examples based on the user’s query.

Adds the table structure, question, and example as the LLM context.

Produces a draft SQL

Executes a loop that includes a SQL ambiguity check, user feedback, refinement, and syntax validation.

Performs the SQL

Uses natural language to summarize the data.

Gemini appears to perform well on tasks that check for SQL ambiguity, according to its heuristic testing. Google started by creating a draft SQL model that had all of the table structure and context-related questions. This allowed Gemini to ask the user follow-up questions to get clarification.

Key driver analysis

Key driver analysis is another name for multi-step reasoning-based data analysis in which analysts must separate and organize data according to every possible combination of attributes (e.g., product categories, distribution channels, and geographies). Google suggests combining Gemini and BigQuery contribution analysis for this use case.

Key driver analysis adds the following steps to the ones done with direct questions:

The routing agent refers users to a key driver analysis special handling page when they ask a query about it.

From ground truth stored in a BigQuery vector database, the agent retrieves similar question/SQL embedding pairings using BigQuery ML vector search.

After that, it creates and verifies the CREATE MODEL statement in order to construct a report on contribution analysis.

Lastly, the SQL that follows is executed in order to obtain the contribution analysis report:

The final report appears as follows:

With Gemini, you can further condense the report in natural language:

Implementing NL2SQL on Google Cloud

Even though this can sound difficult, Google Cloud provides a comprehensive set of tools to assist you in putting an effective NL2SQL solution into place. Let’s examine it.

BigQuery vector search is used for embedding and retrieval

By using BigQuery for embedding storage and retrieval, it is possible to quickly find instances and context that are semantically meaningful for better SQL production. Vertex AI’s text embedding API or BigQuery’s ML.GENERATE_EMBEDDING function can be used to create embeddings. It is simple to match user queries and SQL pairs when BigQuery is used as a vector database because of its inherent vector search.

Contribution analysis using BigQuery

Contribution analysis modeling can find statistically significant differences throughout a dataset, including test and control data, to identify areas of the data that are producing unanticipated changes. A section of the data based on a combination of dimension values is called a region.

To help answer “why?” questions, the recently unveiled contribution analysis preview from BigQuery ML enables automated explanations and insight development of multi-dimensional data at scale. Regarding your data, “What happened?” and “What’s changed?”

The contribution analysis models in BigQuery, in summary, facilitate the generation of many queries using NL2SQL, hence increasing overall efficiency.

Ambiguity checks with Gemini 

The process of translating natural language inquiries into structured SQL queries is known as NL2SQL, and it is often unidirectional. Gemini can assist in lowering ambiguity and enhancing the output statements in order to boost performance.

When a question, table, or column schema is unclear, you may utilize Gemini 1.5 Flash to get user input by asking clarifying questions. This will help you improve and refine the SQL query that is produced. Additionally, Gemini and in-context learning can be used to expedite the creation of SQL queries and results summaries in natural language.

Top NL2SQL techniques

For an advantage in your own NL2SQL endeavor, take a look at the following advice.

Start by determining which questions require attention: Depending on the final report’s goal, answering a question may seem straightforward, but getting the intended response and storyline frequently requires several steps of reasoning. Before your experiment, gather the expected natural language ground truth, SQL, and your query.

Data purification and preparation are essential, and using LLMs does not replace them. As needed, establish new table views and make sure that useful descriptions or metadata are used in place of business domain acronyms. Before going on to more complicated join-required questions, start with straightforward ones that just need one table.

Practice iteration and SQL refinement with user feedback: Google’s heuristic experiment demonstrates that iteration with feedback is more effective following the creation of an initial draft of your SQL.

For queries with multiple steps, use a custom flow: Multi-dimensional data explanations and automated insight development can be made possible by BigQuery contribution analysis models.

Next up?

A big step toward making data more accessible and useful for everyone is the combination of NL2SQL, LLMs, and data analytic methods. Enabling users to communicate with databases through natural language can democratize data access and analysis, opening up improved decision-making to a larger group of people in every company.

Data, size, and value can now be rationalized more easily than ever thanks to exciting new innovations like BigQuery contribution analysis and Gemini.

Read more on govindhtech.com

[worldtodai]

NVIDIA capitalizes on artificial intelligence becoming mainstream, but has it meteoric rise exceeded fundamentals?

NVIDIA Corporation, a once-familiar name in the gaming industry, has skyrocketed to unprecedented heights in recent years, largely thanks to the burgeoning field of artificial intelligence (AI). As AI applications become increasingly prevalent, NVIDIA’s specialized hardware, particularly its graphics processing units (GPUs), have emerged as indispensable tools for training and deploying these…

[mysocial8onetech]

Exploring GOLD: where geometry meets language! This open-source model processes symbols and geometric primitives separately, delivering precise solutions. From math education to research and development, GOLD transforms problem-solving. Its efficient utilization of large language models and separate processing techniques make it a game-changer. Dive into the world of GOLD and explore its potential.

[copperchips]

Computers have come a long way since the days of punch cards. Scientists are always trying to make it easier for us to interact with machines. AI is changing the way we interact with computers and making it possible for us to do things we never thought possible.

[sifytech]

The Role of AI in Driving Digital Transformation

From process automation to personalized customer experiences, AI offers a myriad of benefits that businesses can leverage to stay competitive. Read More. https://www.sify.com/digital-transformation/the-role-of-ai-in-driving-digital-transformation/

[philosophystudentorg]

Sense-Reference Theory Simplified: A Frege Philosophy Guide | PhilosophyStudent.org #shorts

Dive into Gottlob Frege’s groundbreaking distinction between sense and reference in natural language. Please Visit our Website to get more information: https://ift.tt/yMBgntE #sensereference #gottlobfrege #naturallanguage #linguistictheory #philosophyoflanguage #shorts from Philosophy Student https://www.youtube.com/watch?v=VDtRlps-K-0

[tutorsindia152]

Semantic Search Using Natural Language Processing – Tutors India

Semantic search may provide further business benefits by merging NLP with an intuitive user interface and making it straightforward for anyone to interact with and get the results they’re looking for. 

For More Info (Source):

[fahmeenaodetta]

Change

More practice today.

Today I am thinking of change. Prices change, people change, places change, and so on. Price increases, such as the increase in gas prices, are usually negative changes for the consumer; people don't like them very much. From economic theory, I know that consumers sometimes (maybe want to) find substitutes for goods when the price increases.

Consider diagramming. At one point, UML was the predominant language/notation used for creating diagrams. Then there was BPMN. Now, systems will assist with the creation of diagrams. I was an evaluator of a prototype to create class diagrams from requirements. I thought the diagrams created by the system were a good starting point for Analysts who need to create these diagrams. I created and submitted models/diagrams to show how the diagrams may differ when a person - the Analyst - creates it. The author Tiantian Tang actually thanked me in the acknowledgements. That was cool.

The paper:

From Natural Language to UML Class Models: An Automated Solution Using NLP to Assist Requirements Analysis By Tiantian Tang https://theses.liacs.nl/pdf/2020-2021-TangTiantian.pdf

I realize that I have knowledge of the "old" diagrams (DFDs and so on) as well as newer UML and BPMN diagrams. I probably should pay more attention to diagramming. The following blog has examples of diagrams I prepared for a job search process:

http://moorefo1.blogspot.com/

[govindhtech]

Start Using Gemini In BigQuery Newly Released Features

Gemini In BigQuery overview

The Gemini for Google Cloud product suite’s Gemini in BigQuery delivers AI-powered data management assistance. BigQuery ML supports text synthesis and machine translation using Vertex AI models and Cloud AI APIs in addition to Gemini help.

Gemini In BigQuery AI help

Gemini in BigQuery helps you do these with AI:

Explore and comprehend your data with insights. Generally accessible (GA) Data insights uses intelligent queries from your table information to automatically and intuitively find patterns and do statistical analysis. This functionality helps with early data exploration cold-start issues. Use BigQuery to generate data insights.

Data canvas lets BigQuery users find, transform, query, and visualize data. (GA) Use natural language to search, join, and query table assets, visualize results, and communicate effortlessly. Learn more at Analyze with data canvas.

SQL and Python data analysis help. Gemini in BigQuery can generate or recommend SQL or Python code and explain SQL queries. Data analysis might begin with natural language inquiries.

Consider partitioning, clustering, and materialized views to optimize your data infrastructure. BigQuery can track SQL workloads to optimize performance and cut expenses.

Tune and fix serverless Apache Spark workloads. (Preview) Based on best practices and past workload runs, autotuning optimizes Spark operations by applying configuration settings to recurrent Spark workloads. Advanced troubleshooting with Gemini in BigQuery can identify job issues and suggest fixes for sluggish or unsuccessful jobs. Autotuning Spark workloads and Advanced troubleshooting have more information.

Use rules to customize SQL translations. (Preview) The interactive SQL translator lets you tailor SQL translations with Gemini-enhanced translation rules. Use natural language prompts to define SQL translation output modifications or provide SQL patterns to search and replace. See Create a translation rule for details.

Gemini in BigQuery leverages Google-developed LLMs. Billion lines of open source code, security statistics, and Google Cloud documentation and example code fine-tune the LLMs.

Learn when and how Gemini for Google Cloud utilizes your data. As an early-stage technology, Gemini for Google Cloud products may produce convincing but false output. Gemini output for Google Cloud products should be validated before usage. Visit Gemini for Google Cloud and ethical AI for details.

Pricing

All customers can currently use GA features for free. Google will disclose late in 2024 how BigQuery will restrict access to Gemini to these options:

BigQuery Enterprise Plus version: This edition includes all GA Gemini in BigQuery functionalities. Further announcements may allow customers using various BigQuery editions or on-demand computation to employ Gemini in BigQuery features.

SQL code assist, Python code assist, data canvas, data insights, and data preparation will be included in this per-user per-month service. No tips or troubleshooting in this bundle.

84% of enterprises think generative AI would speed up their access to insights, and interestingly, 52% of non-technical users are already using generative AI to extract insightful data, according to Google’s Data and AI Trends Report 2024.

Google Cloud goal with Google’s Data Cloud is to transform data management and analytics by leveraging their decades of research and investments in AI. This will allow businesses to create data agents that are based on their own data and reinvent experiences. Google Cloud unveiled the BigQuery preview of Gemini during Google Cloud Next 2024. Gemini offers AI-powered experiences including data exploration and discovery, data preparation and engineering, analysis and insight generation throughout the data journey, and smart recommendations to maximize user productivity and minimize expenses.

Google Cloud is pleased to announce that a number of Gemini in BigQuery capabilities, including as data canvas, data insights and partitioning, SQL code generation and explanation, Python code generation, and clustering recommendations, are now generally available.

Let’s examine in more detail some of the features that Gemini in BigQuery offers you right now.

What distinguishes Gemini in BigQuery?

Gemini in BigQuery combines cutting-edge models that are tailored to your company’s requirements with the best of Google’s capabilities for AI infrastructure and data management.

Context aware: Interprets your intentions, comprehends your objectives, and actively communicates with you to streamline your processes.

Based on your data: Constantly picks up fresh information and adjusts to your business data to see possibilities and foresee problems

Experience that is integrated: Easily obtainable from within the BigQuery interface, offering a smooth operation across the analytics workflows

How to begin using data insights

Finding the insights you can gain from your data assets and conducting a data discovery process are the initial steps in the data analysis process. Envision possessing an extensive collection of perceptive inquiries, customized to your data – queries you were unaware you ought to ask! Data Insights removes uncertainty by providing instantaneous insights with pre-validated, ready-to-run queries. For example, Data Insights may suggest that you look into the reasons behind churn among particular customer groups if you’re working with a database that contains customer churn data. This is an avenue you may not have considered.

With just one click, BigQuery Studio’s actionable queries may improve your analysis by giving you the insights you need in the appropriate place.

Boost output with help with Python and SQL codes

Gemini for BigQuery uses simple natural language suggestions to help you write and edit SQL or Python code while referencing pertinent schemas and metadata. This makes it easier for users to write sophisticated, precise queries even with little coding knowledge, and it also helps you avoid errors and inconsistencies in your code.

With BigQuery, Gemini understands the relationships and structure of your data, allowing you to get customized code recommendations from a simple natural language query. As an illustration, you may ask it to:

“Generate a SQL query to calculate the total sales for each product in the table.”

“Use pandas to write Python code that correlates the number of customer reviews with product sales.”

Determine the typical journey duration for each type of subscriber.

BigQuery’s Gemini feature may also help you comprehend intricate Python and SQL searches by offering explanations and insights. This makes it simpler for users of all skill levels to comprehend the reasoning behind the code. Those who are unfamiliar with Python and SQL, or who are working with unknown datasets, can particularly benefit from this.

Analytics workflows redesigned using natural language

Data canvas, an inventive natural language-based interface for data curation, wrangling, analysis, and visualization, is part of BigQuery’s Gemini package. With the help of data canvas, you can organize and explore your data trips using a graphical approach, making data exploration and analysis simple and straightforward.

For instance, you could use straightforward natural language prompts to collect information from multiple sources, like a point-of-sale (POS) system; integrate it with inventory, customer relationship management (CRM) systems, or external data; find correlations between variables, like revenue, product categories, and store location; or create reports and visualizations for stakeholders, all from within a single user interface, in order to analyze revenue across retail stores.

Optimize analytics for swiftness and efficiency

Data administrators and other analytics experts encounter difficulties in efficiently managing capacity and enhancing query performance as data volumes increase. BigQuery’s Gemini feature provides AI-powered suggestions for partitioning and grouping your tables in order to solve these issues. Without changing your queries, these suggestions try to optimize your tables for quicker returns and less expensive query execution.

Beginning

Phased rollouts of the general availability of Gemini in BigQuery features will begin over the following few months, starting today with suggestions for partitioning and clustering, data canvas, SQL code generation and explanation, and Python code generation.

Currently, all clients can access generally accessible (GA) features at no additional cost. For further details, please refer to the pricing details.

Read more on govindhtech.com

[mysocial8onetech]

If you are looking for a tool that can help you with your coding tasks using natural language prompts and queries, you should try FalCoder-7B. It is an open coding assistant that leverages the power of Falcon-7B, a large-scale language model that can generate high-quality text in various domains and languages. In this article, you will discover how FalCoder-7B can help you code faster and smarter using natural language. FalCoder-7B allows you to interact with Falcon-7B, a large-scale language model that was trained on billions of tokens of code and natural language data.

[pantechlearning]

Natural language processing (NLP) is an Artificial Intelligence branch that helps computer systems in analysis and synthesis of natural human language. In recent years, NLP techniques have witnessed rapid growth in quality as well as usability. According to surveys, Natural Language Processing is one of the fastest growing skills in the global job market. Tech companies are also applying this method to gain detail understanding of the consumers. Therefore, NLP Projects can help research fellows and students in their research and studies.

One of the best ways to improve your knowledge is doing projects in NLP. Also, there is a huge increase in the demand for people who can program computers to understand human speech. If you wish to improve your NLP skills, you need to get your hands on NLP projects.

How NLP Works?

NLP Works on the basis of Syntactic analysis and also Semantic analysis.

Syntax can be defined as the positioning of words in a sentence. They give grammatical meaning to the sentence. In NLP, syntactic analysis is using to find out how the natural language aligns with the grammatical rules.

Semantics is the study of meaning in languages. It includes applying computer algorithms to find out the meaning and interpretation of words. It also includes the study of how sentences are structured.

What is the importance of NLP Projects?

Natural language processing (NLP) helps computer systems to communicate with humans in their own language. For example, NLP makes computers possible to read text content, hear speech data, interpret them, and find out which parts are important. NLP is important because it helps to sort out ambiguity in language. It also adds important numeric structure to the data for applications used for recognition of speech or analyzing text.

Analysing Social Media is a great example of NLP use. Brands can track conversations online and also they can understand what customers are saying. Voice assistance systems in electronic devices use NLP to respond to vocals of the users. You can also filter and classify your emails with NLP. This is by checking text in emails and stopping spam even before they enter your inbox. NLP can recognize and predict diseases based on electronic health records and patient’s own speech.

Where can you find the best Projects?

Pantech eLearning is an Online Learning Service provider at T.Nagar in Chennai. We are providing some of the best NLP Projects.

Given below is the list of Top Projects we are providing:

Hotel review rating classification

Election Results prediction based on Twitter data

Arabic Natural Language Processing

Hashtag Clustering

Tkinter Chat-Bot Application

Student Placement Prediction

Text Summarization using NLP

Rating Prediction using Machine Learning

Visit our Website and Book your Projects Now.

[freeonlinecoursesudemy]

Natural Language Processing Real-World Projects in Python

Requirements - Basic knowledge of programming is recommended. However, You can follow my Basics of Python Course which is free of cost therefore, the course has no prerequisites, and is open to anyone with basic programming knowledge. Students who enroll in this course will master data science and directly apply these skills to solve real world challenging business problems. Description Are you looking to land a top-paying job in Data Science , AI & Natural Language Processing? Or are you a seasoned AI practitioner who want to take your career to the next level? Or are you an aspiring data scientist who wants to get Hands-on  Data Science and Artificial Intelligence? If the answer is yes to any of these questions, then this course is for you! Data Science is one of the hottest tech fields to be in right now! The field is exploding with opportunities and career prospects. Data Science is widely adopted in many sectors nowadays such as banking, healthcare, Airlines, Logistic and technology. In business, Data Science is applied to optimize business processes, maximize revenue and reduce cost. The purpose of this course is to provide you with knowledge of key aspects of data science applications in business in a practical, easy and fun way. The course provides students with practical hands-on experience using real-world datasets. 1.Task #1 @Predict Customer Sentiments  : Develop an AI model to predict Customer Sentiments of Amazon.. 3.Task # Read the full article