tusharsarde-blog

[tusharsarde-blog]

Installation Tensorflow and NLTK (Natural Language Processing) on Ubuntu and Windows

Very simple steps to Tensorflow and NLTK libraries on Ubuntu or Windows

  Step 1 – Make sure you have python installed

For Python 2

python --version

For Python 3

python3 --version

  Step 2 – Make sure your pip or pip3 and python-dev packages are installed

For python 2

$ sudo apt-get install python-pip python-dev

For python 3

$ sudo apt-get install python3-pip python3-dev

Check the pip version

pip…

[tusharsarde-blog]

Installing Jupyter Notebook on Ubuntu in 3 Steps

Very simple steps to install Jupyter Notebook on Ubuntu

Step 1 – Make sure you have python installed

For Python 2

python --version

For Python 3

python3 --version

Step 2 – Make sure your pip or pip3 and python-dev packages are installed

For Python 2

pip --version

If pip not found then

For python 2

sudo apt-get -y install python python-pip python-dev

Step 3 – Install Jupyter Notebook

For python 2

p…

[tusharsarde-blog]

Windows or MAC or Android Can not Connect To Network After Changing WiFI Password

Windows or MAC or Android Can not Connect To Network After Changing WiFI Password

By default laptop or mobile devies saves wifi password or PIN. To remove the old password or PIN follow below steps

Windows 10:

Start button.

Settings.

Network & Internet.

Select your Wi-Fi from the left panel

Go down and click Manage Wi-Fi settings

New windows will open, go down – Under Manage known networks, select your WiFi network.

Finally click on Forget button. It will reset the password,…

[tusharsarde-blog]

How to run Hive HQL files in Spark - Hive on Spark

How to run Hive HQL files in Spark – Hive on Spark

How to run Hive HQL files in Spark

Hive on Spark

  In this tutorial I will explain how to execute .hql or .sql file. I am using spark-2.2.0-bin-hadoop2.7 standalone version.

You can download spark from here – Spark Download

  Write sample .hql file, in my case i have following example file.

tushar@home:~/spark-2.2.0-bin-hadoop2.7$ cat exhiveql.hql

show databases; use default; show tables; create…

[tusharsarde-blog]

Splunk Fundamental 1 - Splunk User Certification Exam Guide Tips

Splunk Fundamental 1 – Splunk User Certification Exam Guide Tips

Splunk Fundamental 1 – Splunk User Certification Exam Guide Tips

I have been preparing for Splunk 6.x user certification exam since last 2 weeks, In this post i will share my experience.

I will not share any questions that had been asked during the exam. But I will share some tips and experience so that you can clear this exam in your first attempt.

Unfortunately, I have failed today in my first…

[tusharsarde-blog]

Hive Services

Hive Services

There are total six Hive services currently run in Hadoop ecosystem.

Hive CLI (Command Line Interface) Service:

This service is used to manage tables, run hive queries. It is the default service in Hive. It is the most common way of interacting with the hive.

Example:

Go to command prompt and type: bin/hive

Hive>

Some CLI options

$bin/hive –help –service cli

It will show you the…

[tusharsarde-blog]

Hive Tutorials

Hive Tutorials

I highly recommend everyone to buy and read the book to get clear theoretical knowledge about the Hive. You can find book below.

This short tutorial will explain you how to create table in Hive, how to import data into hive table. For more advanced hive create table commands please refer my further hive tutorials. I hope your hive is already up and running.

Creating table in Hive

[tusharsarde-blog]

Hive namespaces (Variables and Properties)

Hive namespaces (Variables and Properties)

Hive namespaces (Variables and Properties)

There are total four namespaces available in Hive, we will go through each one.

hivevar

This is useful in defining our own custom variables for example

Hive> set hivevar:toodey=hello;

Hive> set toodey;

Now we can use this in any Hive query statement as we have seen in my previous tutorial about create table.

CREATE TABLE toodey(t STRING, $(toodey)…

[tusharsarde-blog]

Datatypes in apache hive

Datatypes in apache hive

Datatypes in apache hive

What are the most common datatypes we used in apache hive, in this tutorial you will find the list of datatypes that are most common in use!

Hive supports Primitive Data Types

TINYINT

It’s a 1 byte integer.

SMALLINT

It’s a 2 byte integer

INT

It’s a 4 byte integer

BIGINT

It’s a 8 byte integer

BOOLEAN

This is either TRUE or FALSE

FLOAT

It’s a floating point number example –…

[tusharsarde-blog]

Hive JDBC metastore

Hive JDBC metastore

In hive the default database for storing metadata is derby database but limitation to this is it only serves only one client request at a time, it will not handle multiple client requests.

To solve this limitation hive also provides connectivity with JDBC driver that is hive can store all its metadata related information like table schema etc.

Below is a sample MySQL…

[tusharsarde-blog]

Why Apache hive is a schema on read not schema on write

Why Apache hive is a schema on read not schema on write

Why apache hive called schema on read not schema on write?

In relation database systems, let’s say we perform any insert or update operations in this case database has full control over the storage and database can enforce the schema as data is written, this statement in relation databases called schema on write.

However, hive has no control over the storage (Ex. HDFS), there are many possible…

[tusharsarde-blog]

Create table in Hive using octal code

Create table in Hive using octal code

Create table in Hive using octal code

Reference Example is take from Programming in Hive book

Let’s create a table in hive

CREATE TABLE employee ( name  STRING, salary FLOAT, 

subordinates ARRAY<STRING>

deductions MAP<STRING, FLOAT>

address STRUCT<street:STRING, city:STRING, state:STRING,zip:INT>)

ROW FORMAT DELIMITED

FIELDS TERMINATED BY ‘\001’

COLLECTION ITEMS TERMINATED BY ‘\002’

MAP KEYS…

[tusharsarde-blog]

How to add multiple partition in hive

How to add multiple partition in hive

How to add multiple partition in hive

The simple example to add partition would be

ALTER TABLE employee ADD IF NOT EXISTS

PARTITION (countryName= ‘US’) LOCATION ‘/employee/US’

PARTITION (countryName= ‘EU’) LOCATION ‘/employee/EU’

How to drop a partition in hive

ALTER TABLE employee DROP IF EXISTS PARTITION(countryName= ‘US’)

How to rename a column in hive

ALTER TABLE employee CHANGE COLUMN name…

[tusharsarde-blog]

Partition internal or managed tables in hive

Partition internal or managed tables in hive

How to partition managed or internal hive table?

Let’s say we need to partition the data country wise.

Hive> CREATE TABLE countryDetails ( cid INT, countryName STRING ) PARTITIONED BY (countryName STRING)

So in this case the country wise directory partitioned will be get created, so for US and EU there will be two separate directories will get created.

How to retrieve or read data from hive…

[tusharsarde-blog]

What is Manage tables and External tables in hive?

What is Manage tables and External tables in hive?

Manage tables in hive:

Manage tables in hive also called internal tables, this table uses hive default warehouse directory to store the table data. (hive.metastore.warehouse.dir)

One disadvantage is, hive doesn’t allow another tools to share the data that is hive has the ownership of data and that’s where the external tables are came into picture.

External tables in hive:

External tables in hive…

[tusharsarde-blog]

Apache hive tips and tricks

Apache hive tips and tricks

Display hive databases which starts with the given name using LIKE

Hive> SHOW DATABASES LIKE ‘e.*’;

employee

emp

…

How to create hive database in specific or custom location

Hive> CREATE DATABASE employee LOCATION ‘/my/emp/datafiledir’

In this case location is our HDFS location.

We can also add comments while creating hive database

Hive> CREATE DATABASE employee LOCATION ‘/my/emp/datafiledir’…

[tusharsarde-blog]

How to load data into hive partition table

How to load data into hive partition table

How to load data into hive partition table

LOAD DATA INPATH ‘/employee/emp.txt’ OVERWRITE  INTO TABLE employee PARTITION (countryName= ‘US’, state = ‘CH’)

How to insert data into hive tables using select query

INSERT OVERWRITE TABLE employee PARTITION (countryName = ‘US’, state = ‘CH’) SELECT * FROM old_emp oemp WHERE oemp.countryName = ‘US’ AND oemp.state = ‘CH’;

If we don’t want to overwrite…