econolytics-blog

[econolytics-blog]

How To Simplify Understanding Of Algorithms Like Gradient Descent

When I first started out learning about machine learning algorithms, it turned out to be quite a task to gain an intuition of what the algorithms are doing. Not just because it was difficult to understand all the mathematical theory and notations, but it was also plain boring. When I turned to online tutorials for answers, I could again only see equations or high-level explanations without going through the detail in a majority of the cases.

It was then that one of my data science colleagues introduced me to the concept of working out an algorithm in an excel sheet. And that worked wonders for me. Any new algorithm, I try to learn it in an excel at a small scale and believe me, it does wonders to enhance your understanding and helps you fully appreciate the beauty of the algorithm.

Let me explain the above using an example.

Most of the data science algorithms are optimization problems and one of the most used algorithms to do the same is the Gradient Descent Algorithm.

Read Full Article at https://www.econolytics.in/blog/how-to-simplify-understanding-of-algorithms-like-gradient-descent/

[econolytics-blog]

Neural Networks In Retail Industry

There are wide applications of neural networks in industry. This post is an attempt to intuitively explain one of the applications of word2vec in retail industry.

Natural language processing is an exciting field. Quite a few new algorithms are being developed resulting in innovative ways of solving traditional problems.

One of the problems that researchers were working on is the challenge of identifying similar words to a given word. This way we would be in a position to say, whether two sentences are mentioning about similar context & perform a variety of tasks.

Traditional ways of text mining:

Traditionally, we are used to one hot encode each word to represent it in multidimensional space. For example, if the sentence that we have reads:

“I enjoy working on data” – we have 5 words: “I”, ”enjoy”, ”working”, ”on”, ”data”

One hot encoding provides an index to each word & converts the sentence (of 5 words) into a vector – i.e.,

“I”                – (1,0,0,0,0)

“enjoy”       – (0,1,0,0,0)

“working”  – (0,0,1,0,0)

“on”            – (0,0,0,1,0)

“data”         – (0,0,0,0,1)

The major drawback of this way of one hot encoding is that a word that has a very similar meaning to any of the above words would be given a different index.

For example, if we have a word (“like”) which is very similar to “enjoy” will have a different index.

Moreover, new words cannot be taken into account as they were not available in the original one hot encoding.

The intuition of word2vec:

Word2vec solves the problem of similar words with different indices using a small trick on the surrounding words.

Read Full Article at https://www.econolytics.in/blog/neural-networks-in-retail-industry/

[econolytics-blog]

Introduction To Neural Networks, Advantages And Applications

Let's begin by first understanding how our brain processes information:

In our brain, there are billions of cells called neurons, which processes information in the form of electric signals. External information/stimuli is received by the dendrites of the neuron, processed in the neuron cell body, converted to an output and passed through the Axon to the next neuron. The next neuron can choose to either accept it or reject it depending on the strength of the signal.

Read Full Article at https://www.econolytics.in/blog/introduction-to-neural-networks-advantages-applications/

[econolytics-blog]

Data scientist career path and opportunities

It is the lucrative career choice you should think to pursue. With its $123,000 median salary, Glassdoor named it as the number one job position in the market with the highest satisfaction score of 4.2 hits out of 5. Harvard called it the sexiest job of the 21st century. But let us consider all the other reasons why one should consider data scientist career path as the most in demand?

 Big data is becoming more valuable as online digital connectivity continues to soar. Digital systems are leaving behind traces of big data from various domains that connect consumers and businesses or organizations.

 Most big organizations store multiple petabytes of data from various business circles. The process to refine this data into meaningful information would involve a “mashup” of several analytical efforts that require skilled personnel. If handled strategically, big data can produce valuable information that informs and fuel the business/organization productivity.

 A data analyst uses systems developed by data engineers/architects to mine (through analytics) big data to generate insights that propel and improve the business decision-making and profit gains.

Data scientist career path: the lucrative job of the 21st century.

 There will always be big data opportunity and the trends of data scientists and analysts are soaring high day after day. Harvard business review named data scientist as the sexiest job of the 21st century.

 There are lot of enthusiasm for big data especially those focusing on technologies that tame it as easy as possible. Think of Hadoop and related open source tools (frameworks for distributed file system processing), cloud computing, and data visualization among others.

 While these tools are breakthroughs in analyzing big data, there is a shortage of data scientists. There is a high demand for professionals with the skill set (and the mindset) to put these big data opportunities to good use. The demand has raced ahead of supply in some sectors.

 Data scientist career path. What does a data scientist do?

A hiring manager once said, “I need someone who understands data”. It’s as simple as that and yet companies struggle to find the right match.

The demand for data science skills is disrupting the job market. As per IBM projects that by 2020, the need for all data professionals will increase to 2.72M jobs in the United States.  

 To attract the right data scientists, organizations need to be able to clearly define what their business needs are. Data never stop flowing in this digital realm. Data scientists are able to bring structure and analysis to large quantities of formless big data. They need to identify rich data sources, join them with other potentially incomplete data sources, and clean the resulting set. The end results should be a stream of flawless, seamless orders of meaningful information that provides business with the right direction.

While data scientists discover news paths for big data analysis, they are faced with technical limitations but most often never bog down their search for innovative solutions. They always find the productive path to fashion their own tools with prospects of conducting their analysis more effectively while minimizing the cost.

 Data scientist make discoveries, communicate what they’ve learned from their systems (including big data), and suggest its implications for new business directions.

 Their skills involve being creative in displaying information visually and making the patterns they find clear and compelling for the end user. At its highest realms, they advise executives and product managers on the implications of the data for their products, processes, and decisions.

At the bottom line, a data scientist job implies some statistics & modeling knowledge (from math and data visualization), combined with programming skills (including database languages) that ultimately result in actionable insights for businesses decision-making.

Finding the right data scientist career path

Data science career path involves diving deep into the data science pipeline (objectives, process etc.), roles, and job opportunities.

The main objective of a data scientist is to go through a sequence of steps in a systematic manner to achieve the desired results. Each step is a contributing factor that involves the creation of models, their validation, evaluation, and potential refinement of big data into the final results. The output must be insightful information presented in charts, graphics, and other information representation forms that’s structured, compelling, and easy to interpret by the managers and executives.

 Business objective for the data scientist involves identifying business issue and/or attractive market opportunity, clearly understand what’s to be accomplished in order to help the business gain a competitive edge.

From that scenario, the role of a data scientist is not always technical. They don’t just program and perform statistics at its core stages of big data process. They need to possess contextual skills (as data analysts) for planning and reporting from all data analytics stages.

Data scientist career path: Roles

A data scientist makes value out of big data. They proactively fetch information from various sources (as inputs) and analyzes it for better understanding (information) about how the business performs, and builds AI tools (software) that automate certain processes.

 Data scientists are multi-talented professionals, their role is a crossover between many different disciplines.  They can be programmers, statisticians, analysts, as well as being good data communicators. If you’re passionate about career path of a data scientists, there are many opportunities you can get there.

 Since the data science field is broad and often involves a lot of confusions. The definition of the job and its roles is convoluted. Data scientist roles is a mix of various occupations like big data engineer, data software engineer, hackers, data analyst, business intelligence (BI) analyst, marketing analyst etc. Their expertise in their roles depends on the scope of the job requirements.

 Here are some of the roles/ job description of the data scientist.

Help the company discover the     information hidden in vast amounts of big data

Help the company make smarter decisions     to deliver even better products and/or services. They interpret and manage     data and solve complex problems using expertise in a variety of data     niches

Focus on applying data mining     techniques, doing statistical analysis, and building high-quality     prediction systems integrated with the company products the services.

Build systems that help the company     achieve their goals by using machine learning techniques and so on.

Improve and extend the features of big     data systems used the company for better data analysis and improved     decision-making.

Develop internal A/B testing procedures     and a lot more.

Data scientist career path: How to become a data scientist

Now that you’ve had some go forward ideas of what a data scientist can do, the big question becomes – how to become a data scientist.

Profession research and job description/roles

First, before you begin any career, researching the profession thoroughly allows you to get a clearer picture of how to get involved.

Remember to clearly understand the job description – data scientists are required to use algorithms and statistical techniques to turn big data into insightful information.  Have a knowledge of the industry the hiring company is operating in. keep in mind that as a data scientist, you must possess effective communication skills. You must be able to communicate information in effectively as it should be.

 Data scientist career path: Key Responsibilities

Responsibilities of a data scientist depend on the organization you’re working in. according to Toptal, among other responsibilities, key data scientists responsibilities may include:

Selecting features, building and     optimizing classifiers using machine learning techniques

Data mining using state-of-the-art     methods

Develop machine learning models and     analytical methods.

Extending the company’s data with     third-party sources of information when needed

Enhancing data collection procedures to     include information that is relevant for building analytic systems

Processing, cleansing, and verifying the     integrity of data used for analysis

Doing ad-hoc analysis and presenting     results in a clear manner

Creating automated anomaly detection     systems and constant tracking of its performance

Data scientist career path: skills

Data scientist’s career path entails wrangling with big data. They apply all their analytic skills to uncover hidden data solutions to business challenges. It’s a heavy task that needs a huge amount of structures and unstructured data paints. They must clean, massage and organize data with their formidable skills in statistics, programming, and math.

 According to data-flair training, a data scientist utilize their knowledge of statistics and modeling to convert data into actionable insights about everything from product development to customer retention to new business opportunities.

 They must possess both technical and non-technical skills to perform their job in an effective manner. They must apply the tools needed to capture data, for data pre-processing, data analytics & pattern recognition, as well as data presentation and visualization.

 Examples of data scientist skills;

Must learn about SQL engines like Apache     Hive, Impala, Spark-SQL, Flink-SQL etc.

Knowledge of big data technologies. Get to know the first generation of tools like Apache Hadoop and its ecosystem like Flume, pig, hive, and so on.

Unix knowledge

Python – an interpreted, object-oriented programming language with dynamic semantics.

Knowledge of statistical data analytics     language R (highly recommended)

Skills on machine learning algorithms for advanced data analytics, productive analytics, advanced pattern matching and so on. Machine learning tools are available in the market like weka, nltk, etc.

Advanced skills in data visualization tools like Tableau, JMP. R also has support for data visualization (such     as ggplot2, lattice, rCharts, google charts, shiny for web apps for     presentations, etc.)

Non-technical skills include excellent communication skills, Business acumen and analytical problem solving to get optimum output.

Data scientist career path: Certifications

There are excellent data scientist certification programs that are widely recognized and reputed if you’re looking to land at the big companies that hire professional data scientists.

 Here are few Data Scientist certifications that focus on useful skills:

Cloudera Certified Professional: Data     Scientist (CCP: DS). CCP: DS is aimed at data scientists to demonstrate advanced skills     in working with big data. Candidates are drilled in 3 exams – Descriptive and Inferential Statistics, Unsupervised Machine Learning, and Supervised     Machine Learning. All candidates must prove their skill set by developing     a production-ready data science solution under real-world conditions. Here     is everything you need to know about Cloudera Data Scientist     Certification.

Certified Analytics Professional (CAP). Created in 2013 by     the Institute for Operations Research and the Management Sciences     (INFORMS) for data scientists. Their certification includes the framing of     business and analytics problems, data, and methodology, model building,     deployment and lifecycle management.

EMC: Data Science Associate (EMCDSA). Demonstrate the ability to apply common techniques and tools required for big data analytics. Candidates are judged on their business acumen and technical expertise in tools such as “R”, Hadoop, and Postgres, etc.

Develop Your Career

Data science is always “in demand” as companies begin to realize the importance of their data analytics to make informed decisions. There is always the look-out for talented data scientist’s professionals from all work industries.

 Data scientist career path involves building a strong professional network. Join online data scientist forums and competition platforms such as those hosted by Kaggle, Topcoder and the Defence Science Technology Laboratory (DSTL). Keep watch of top data science site including Data Science Central, SmartData Collective, What's The Big Data, insideBIGDATA and so on.

 Be on the lookout on job listing sites like data scientist jobs, KD Nuggets, Kaggle, and much more.

 If you are looking for a freelance data analyst projects or a data analyst jobs, you can register to Econolytics i.e. a data analyst marketplace where you can find the data scientist freelance projects.

[econolytics-blog]

Top Data Analyst/Scientist Interview Questions and Answers

In this blog, we will cover the major data analyst/scientist interview questions and answers. The demand for data analysts is growing in today’s advancement in technology. When looking to hire a data analyst, here are the top data analyst/science interview and answers to ask during an interview.

1. What are the top responsibilities of a Data Analyst?

Each profession has its own unique way of handling responsibilities for the smooth running of tasks/processes of businesses or organizations.

Responsibilities of a data analyst may include;

Understanding the data structure and sources relevant to the business;

Being able to extract the data from these sources in a timely & efficient manner;

Identify, evaluate and implement services and tools from external sources to support the validation of data and cleansing;.

Develop and support various reporting processes of the business;

Perform an audit of data and resolve any business associated issues for clients;

Ensure database security by developing access system user levels;

Analyze, identify & interpret process trends or patterns primarily in complex data sets and trigger alerts for the business teams;

Evaluating historical data and making forecasts for growing the business;

Developing and validating predictive models to improve business processes and identify key growth strategies.

2. What are the key skills required for a data scientist?

Mathematics/Statistics Knowledge; A Data scientist should be able to work on statistical concepts seamlessly. Without a good hold on Statistics, a data scientist will not be able to understand basics such as cleaning and manipulating data.

Programming skills:  Should be familiar with computer software and tools including; scripting language (Matlab, Python), Spreadsheet (Excel) and Statistical Language (SAS, R, SPSS), Querying Language (SQL, Hive, Pig). Other computer skills include; big data tools (Spark, Hive HQL), programming (JavaScript, XML), and so on.

Logical Deduction: This is a skill that comes with experience. The data scientist should be able to immediately identify anomalies and be able to draw out strategies from trends. Without this skill, a data scientist is not able to add value to the business.

Besides these skills, domain knowledge is increasingly becoming a requirement for a data scientist. Example: Credit Risk, supply chain management, etc.

Attention to details, decision making and problem-solving, communication skills, are some of the soft skills that a data scientist must develop.

3. Summarize the various steps in an analytics project

Defining the objective function;

Identifying key sources of data required for the analysis;

Data preparation & cleaning;

Data modelling

Model Validation

Implementation and tracking (deployment and monitoring the results)

4. Define Data Cleansing (data cleaning)

Refers to the process of detecting and correcting (or removing) corrupt or inaccurate records from a record set, table, or database. Data cleansing also refers to identifying incomplete, incorrect, inconsistent or irrelevant parts of the record set, table, or database data and then replacing, modifying, or deleting the dirty data. In model development, data cleaning also means identifying anomalies in the data that cannot be represented consistently by one model. Example: For income estimation models, very high values of income that are not consistent with the data should be either removed or capped to a maximum limit.  The aim is to enhance the quality of data.

5. What are the best practices for data cleaning?

Best practices for data cleaning includes;

Understanding the range (Min./Max.), mean, median and plotting a normal curve;

Identifying outliers in the data and treating them;

Missing value treatment;

6. Explain what is logistic regression?

Logistic regression is a statistical method for examining a dataset consisting of one or more independent variables that define an outcome.

7. Give some of the best tools useful for data analysis

Solver

NodeXL

KNIME

R Programming

SAS

Weka

Apache Spark

Orange

Io

Talend

RapidMiner

OpenRefine

Tableau

Google Search Operators

Google Fusion Tables

Wolfram Alpha’s

Pentaho

8. What is the difference between data mining and data profiling?

Data profiling is the process of analyzing the data available from an existing information source like a database and collecting statistics or informative summaries about that data. It may be information on various attributes like discrete value, value range etc.

Data mining is the process of discovering patterns in large data sets involving methods at the intersection of machine learning, statistics, and database systems. It can be focusing on cluster analysis, dependencies, sequence discovery, detection of unused records and others.

9. What are some common problems faced by data analyst?

Problems include;

Data storage and quality

Identifying overlapping data

Common misspelling

Duplicate data entries

Varying value representations

Missing values

Illegal values

Security and privacy of data

10. What are Hadoop and MapReduce?

It’s the name of the programming framework developed by Apache for processing large data set, for an application in a distributed computing environment.

You can read more about Hadoop and Map Reduce here

Read Full Article Here: https://www.econolytics.in/blog/data-analyst-scientist-interview-questions-answers/