1) Installing MongoDB¶

goto https://www.mongodb.com/try/download/community

and download based on your OS.

2) double click on the downloaded file to install.

2) Starting MongoDB

1. Go the services section and then start the MongoDB service if not already started

2. Go to bin directory of the MongoDB installation and run the ‘mongo’ command

C:\>cd Program Files\MongoDB\Server\4.4\bin

If command runs successfully, it means the server is up and running.

3) MongoDB Basics:

Difference Between MongoDB and Relational DB

MongoDB	RDBMS
Database	Database
Collection	Table
Document	Row
Field	column
Embedded Documents	Table Join
Primary Key (Default key "_id" )	Primary Key

4) Basic Commands

show dbs: to show all the databases

use <DBName>: creates a new database with the name if not already present or starts using the database given as database name.

db.createCollection(‘<collectionName>’) : to create a new collection. Collections are analogous to tables.

show collections to show all the collections in a database.

db.<collectionName>.insert({“key”:”value”}) : to insert new record into the table.

db.<collectionName>.find(): To show all the records from a table.

db.<collectionName>.update({“key”:”value”}, ({“key”:”new_value”}) : to update an existing record in the table.

db.<collectionName>.remove({“key”:”value”}) : to remove record(s) from the table matching the criteria.

5) Using python for MongoDB

Install Pymongo library

pip install pymongo

import pymongo
dbConn = pymongo.MongoClient("mongodb://localhost:27017/")
# connecting to the locally running MongoDB Instance

dbname='hk_DB1'
db = dbConn[dbname] # connecting to the database named hkDB1 present in the mongoDB
                    # if the database is not present, it'll autoamtically create it.

# show all the databases
print(dbConn.list_database_names())
o/p:
['admin', 'config', 'hk_DB1', 'local']

# check if the databse exists in mongoDB
dblist=dbConn.list_database_names() # obtaining all the database names
db_name='hk_DB1'

if dblist.index(db_name)==-1:
    print ("This Databse doesn't exist")
else:
    print ("This Databse exists")

o/p:
This Databse exists



# connecting to a collection(collections are tables in MongoDB)
collection_name='hk_collection1'
collection=db[collection_name]  # connectig to the collection

# cheking if a collection exists
if collection_name in db.list_collection_names():
    print("The collection exists.")
else:
    print("The collection doesn't exist.")

O/p:
The collection exists.


# inserting a row into collection
hk_row1 = {'id': '1234',
 'name': 'hk',
 'profession': 'SE'
 } # creating key value pairs for inserting into the DB

x = collection.insert_one(hk_row1)  # inserting record into the collection
x.inserted_id  #gives unique of insert

O/P:
ObjectId('6059d389bb9fd7f87181318d')



# inserting multiple rows once
hk_rows = [
    {'id': '1234',
     'name': 'hk',
     'profession': 'SE'},
    {
        'id': '5678',
        'name': 'ab',
        'profession': 'associate'},

    {'id': '9123',
     'name': 'bc',
     'profession': 'lead'}
]
# inserting multiple records into the collection
x = collection.insert_many(hk_rows)



# retieving specific record from collection
result= collection.find({})
result[1] #printing the third record

O/p:
{'_id': ObjectId('6059d4c2bb9fd7f871813192'),
 'id': '5678',
 'name': 'ab',
 'profession': 'associate'}






# find brings out all the records, to overcoem this use limit
result_total= collection.find({}).limit(2)
for res in result_total:
    print(res)
O/p:
{'_id': ObjectId('6059d4c2bb9fd7f871813191'), 'id': '1234', 'name': 'hk', 'profession': 'SE'}
{'_id': ObjectId('6059d4c2bb9fd7f871813192'), 'id': '5678', 'name': 'ab', 'profession': 'associate'}


# retrieveing specific columns
result_some= collection.find({}, {'id','name'}).limit(2) # retrieveing two columns
# The second parameter in find() specifies which columns to choose
for res in result_some:
    print(res)
O/p:
{'_id': ObjectId('6059d4c2bb9fd7f871813191'), 'id': '1234', 'name': 'hk'}
{'_id': ObjectId('6059d4c2bb9fd7f871813192'), 'id': '5678', 'name': 'ab'}


# finding the rows based on given criteria
my_db_query={'name':'hk'}

result1= collection.find(my_db_query) # printing rows where name = hk
for res in result1:
    print(res)

O/p:
{'_id': ObjectId('6059d4c2bb9fd7f871813191'), 'id': '1234', 'name': 'hk', 'profession': 'SE'}




# Deleting records from mongo DB:
# finding the rows based on given conditon
my_db_query={'name':'hk'}
x=collection.delete_one(my_db_query) # the deletion step
print(x.deleted_count)

Data-warehouse Concepts:

1. Data model:

Data model is an abstract model that organizes elements of data and standardizes how they relate to one another.

Data models are made up of entities, which are the objects or concepts we want to track data about, and they become the tables in a database.

2. Data Modelling:

Relational data modelling:

3NF:

--Faster writes.

--Highly normalised data.

--Every table has Primary key, Date timestamp

--cascading effect on child tables.

Dimensional data modelling:

Dimensional data modelling (Ralph Kimball) can result in a design called a star schema, which has denormalized tables, and it is used for building reporting and analytical systems.

--Fast reads.

--Good for reporting.

--Granularity is lowest level up to which data will be stored in a Fact Table in context with specific Dimension table.

Data Vault:

Hub: (emp_id)

Satellite: (emp_name, sal, DOB)

Link: link to hubs

--agile Datawarehouse.

3. Datawarehouse:

Data warehouse (DW or DWH), also known as an enterprise data warehouse (EDW).

retail Datawarehouse

ELT, ETL based Datawarehouse.

4. Datamart:

data mart is a subset of the data warehouse (like sales data mart in

retail Datawarehouse)

5. Star schema:

consists of one or more fact tables referencing any number of dimension tables.

fact tables in a star schema are in third normal form(3NF) whereas dimensional tables are de-normalized.

3NF is used to reduce the data duplication. It is also used to achieve the data integrity.

6. Snowflake schema:

centralized fact tables which are connected to multiple dimensions.

"Snowflaking" is a method of normalizing the dimension tables in a star schema.

7. Fact Table:

Keys columns, Measures columns.

8. Metadata:

Metadata in data warehouse defines the warehouse Entities/objects.

9. Delete duplicate records from table?

Method 1:

begin;

create table table_new as select distinct * from table;

alter table table rename to table_old;

alter table table_new rename to table;

drop table table_old;

commit;

Method2

begin;

-- First identify all the rows that are duplicate

CREATE TEMP TABLE duplicate_saleids AS

SELECT saleid

FROM sales

WHERE saledateid BETWEEN 2224 AND 2231

GROUP BY saleid

HAVING COUNT(*) > 1;

-- Extract one copy of all the duplicate rows

CREATE TEMP TABLE new_sales(LIKE sales);

INSERT INTO new_sales

SELECT DISTINCT *

FROM sales

WHERE saledateid BETWEEN 2224 AND 2231

AND saleid IN(

SELECT saleid

FROM duplicate_saleids

);

-- Remove all rows that were duplicated (all copies).

DELETE FROM sales

WHERE saledateid BETWEEN 2224 AND 2231

AND saleid IN(

SELECT saleid

FROM duplicate_saleids

);

-- Insert back in the single copies

INSERT INTO sales

SELECT *

FROM new_sales;

-- Cleanup

DROP TABLE duplicate_saleids;

DROP TABLE new_sales;

COMMIT;

10. Find duplicate records from a table.

SELECT id, COUNT(*)

FROM users

GROUP BY id

HAVING COUNT(*) > 1;

HAVING is important here because unlike WHERE, HAVING filters on aggregate functions.

8. What is SCD in data warehouse?

A Slowly Changing Dimension (SCD) is a dimension that stores and manages both current and historical data over time in a data warehouse. It is considered and implemented as one of the most critical ETL tasks in tracking the history of dimension records.

--current table

CREATE TABLE basket(id int PRIMARY KEY,

fruit VARCHAR(50) NOT NULL);

INSERT INTO basket values(1,'apple');

INSERT INTO basket values(2,'orange');

INSERT INTO basket values(3,'banana');

--create stage table

create table stage_basket(like basket);

--load data to stage table

INSERT INTO stage_basket values(4,'pear');

INSERT INTO stage_basket values(3,'banana1');

--Start the transaction before deleting / inserting data

begin transaction;

delete from basket

using stage_basket

where basket.id = stage_basket.id;

insert into basket select * from stage_basket;

end transaction;

Slowly Changing Dimension (SCD):

SCD type1 - overwrite data in dimensions.

SCD type2 –

SCD type3 –

SCD type4 –

SCD type5 –

SCD type6 –

Delete Query:

DELETE FROM employees WHERE employee_ID = 3;

Alias in where, Having clause?

We can not use alias in Where, Having Clause.

What is View in a Database?

Views are a logical virtual table created by “select query” but the result is not stored

anywhere in the disk and every time we need to fire the query when we need data, so always we get updated or the latest data from original tables.

What is Materialized View in a Database?

Materialized views are also the logical view of our data-driven by the select query

but the result of the query will get stored in the table or disk, also the definition

of the query will also store in the database.

List duplicates in a fact table?

Select col1, col2, col3, count(*) as duplicate_col from fact_table

GROUP BY col1, col2, col3 HAVING count(*) > 1;

Queries:

Create Tables and insert data?

create TABLE test1 (id integer, Fname varchar(15), LName varchar(20));

insert INTO test1 values (1, 'subash', 'Bose');

insert INTO test1 values (2, 'vir', 'savarkar');

insert INTO test1 values (3, 'Bhagat', 'Singh');

insert INTO test1 values (4, 'Baji', 'Bhosale');

insert INTO test1 values (5, 'hk', 'Bose');

create TABLE test2 (id integer, dept varchar(15), Sal integer);

insert INTO test2 values (1, 'HR', 10000);

insert INTO test2 values (2, 'HR', 10000);

insert INTO test2 values (3, 'IT', 20000);

insert INTO test2 values (4, 'IT', 20000);

insert INTO test2 values (5, 'IT', 20000);

select all rows from both tables?

select * from test1 left join test2 on test1.id = test2.id;

select all rows from test1 table where dept=IT?

select t.id, t.Fname, t.LName from test1 as t left join test2 on t.id = test2.id where test2.dept = 'IT';

Select emp all details if salary greater than avg salary of all employees?

select * from test1 as t left join test2 as t2 on t.id = t2.id

where t2.sal > (select avg(sal) from test2);

create TABLE emp1 (emp_id INTEGER, emp_name varchar(15), job_name varchar(15),

manager_id integer, hire_date date, salary integer, dep_id integer)

insert into emp1 values(68319, 'emp1', 'PRESIDENT' ,NULL , 1991-11-18, 6000.00, 1001);

insert into emp1 values(66928, 'emp2', 'MANAGER' , 68319, 1991-05-01, 2750.00, 3001);

insert into emp1 values(67832, 'emp3', 'MANAGER' , 68319, 1991-06-09, 2550.00, 1001);

insert into emp1 values(65646, 'emp4', 'MANAGER' , 68319, 1991-04-02, 2957.00, 2001);

insert into emp1 values(67858, 'emp5', 'ANALYST_ ' , 65646, 1997-04-19, 3100.00, 2001);

insert into emp1 values(69062, 'emp6', 'ANALYST_ ' , 65646, 1991-12-03, 3100.00, 2001);

insert into emp1 values(63679, 'emp7', 'CLERK___ ' , 69062, 1990-12-18, 900.00 , 2001);

select employee details if he is aa manger

SELECT * FROM emp1 WHERE emp_id IN (SELECT manager_id FROM emp1);

Query to get counts for joins?

create TABLE t1 (emp_id integer, dept_id);

insert into t1 values(1, 1);

insert into t1 values(2, 2);

insert into t1 values(3, 3);

insert into t1 values(4, 1);

create TABLE t2 (emp_id integer, dept_id);

insert into t2 values(1, 1);

insert into t2 values(2, 2);

insert into t2 values(3, 3);

insert into t2 values(4, 1);

insert into t2 values(4, 4);

Unix

Basics:

$ pwd ---displays current working directory path

$ uname ----displays operating system name

Linux

$ uname –r ----displays kernel version name

3.13.0-24-generic

$ hostname -----dispalys server name

hj-glbtal8591

$ hostname -i ----displays servers IP address

10.51.226.42

$ hostname --gives host name

cdtssseabcd.abcorp.com

$> uptime -- #To see the server uptime, boot time, users and load

06:11:56 up 12 days, 10:06, 110 users, load average: 71.49, 73.23, 72.12

Who is Logged In?

$ users

root

$ whoami

root

$ logname ---displays current username

$ ---user working prompt

# ----admin working prompt

$ exit ----logout from current user account

$ su - username ---to switch from one user to another user account

$man <command_name> ---gives manual pages of acommand

$clear ----clears screen

$wc <file_name> ---to count words, lines, bytes

$cd - ---to go to last working directory.

Viewing list of files:

$ls ---lists all files and directories in current directory

$ls -l ---long list format

$ls -a ---list invisible files

$ls -r ----lists files in reverse order

$ls -t -------lists all files based on time

$ls <directory_name> ---to list files in a directory

$ls -la < directory_name > ---to list files in a directory

$ ls *{jpg,JPG} --lists files ending with jpg and JPG

Listing commands in current shell:

$ ls /bin

To use single quotes in a command use back slash (\)

---------------------------------- --------------------------------------

Tab key –use this for auto completion of commands and directory names and file names

------------------------------------------------------- -----------------------------------------------------------

Commands:

Command details:

1) $ cat --help

2) $ info cat

3) $ man cat

Cat:

For creating a file.

$ cat > hari -----creates hari file

$ cat < hari ----- for opening hari file

$ cat >> hari -----for appending the hari file

$ cat -n <file_name> ---‘-n’ gives line number’s

How to open multiple files:

$ cat <file1> <file2> <file3>

$cat -b persistent ----gives line numbers

1 persistent

2 its a level 3 company

3 established in 1990

$ cat > .<file1> ----creates hidden file(‘.’ Is used for creating hidden files)

$ cat /dev/null > hk_test1 --to empty a file

Stat:

$ stat hk_test1 --gives stats for a file

Touch:

It is used for creating empty files, more over it is used for changing file time stamp.

$ touch <file1> <file2> <file3> ---creates multiple files

Creating multiple files:

$ touch memo{1,2,3,4,5}

$ ls m*

memo1 memo2 memo3 memo4 memo5

Deleting files and creating Directories:

$ rm <file1> -----deletes file1

$ rm –i <file1> -----deletes file1 with confirmation

$ rm * -----removes all files

$ mkdir <dir_name> -----creates dir_name directory

$ rmdir <dirname> ----deletes directory

$ rm -r <dirname> -----deletes entire directory structure even if it has files.

Copy:

$ cp <source_file> <targetfile> ----source file must me existing and target may be new or existing

Taking backup to desired location:

cp /etc/postgresql/9.3/main/abcd.conf /root/abcd.conf_back_up

Relative path: from current directory to target directory

Absolute path: from root directory to target directory

MV:

To move a file or directory (or) to rename a file or directory.

$ mv hk1 /root/hari ---hk1 is renamed to hari and moved to root

$ mv hk1 hk2 ----hk1 is renamed to hk2

$ mv .hk1 hk1 -----hk1 hidden file is unhidden

SED:

$sed -n 2p <file_name> ---gives 2^nd line

Replacing strings using sed:

$ sed -i -e 's/PERSISTENT/psl/g' hari1 ---replaces in hari1 file

$ sed -i -e 's/PERSISTENT/psl/g' hari1

Tr:

$ tr " " "," < hari1 ---replaces space with coma and opens file

PERSISTENT

ITS,A,LEVEL,3,COMPANY

ESTABLISHED,IN,1990

Vi Editor:

Vi has 2 searches 1) string 2) character

/ and ? Are used for string search

Eg: ?hari (then enter) ---searches hari string in the file

$vi –R <file_name> ---opens an existing file in read only mode

$view <file_name> ----opens an existing file in read only mode

:q! ---quit vi w/o saving

:w --save

:wq! ---save and quit from vi

cc ---removes contents of the line, leaving you in insert mode

U ---restore current line

u ---undo thelast change to the file

:f <filename> ---copies current_file to filename

:e <filename> ---opens new file filename

:cd dirname ----changes to directory

Running commands in Vi:

Command mode à :! ls

For replacing a string in vi:

:s/pattern/replace/

:s/search/replace/g

Deleting all lines in a file in vi editor:

:1,$d

:%d

Deleting specific lines in a file in vi editor:

:4,$-2d ---leaves first 3 and last 2 lines and deleting the rest.

:4,$d ---leaving first3, rest of the lines are deleted.

:4,$-1d ---leaving first 3 and last 1line, remaining are deleted

How to go to 16 line in Vi:

Esc à 16G

ESCàG ---goes to last line

How to un highlight Searched words in Vi:

goto vi file àESC à :noh àenter

part2

Changing file Permissions:

$ chmod +x hari ---giving execution permission to hari

$chmod -111 hari ---removing execute permission from user, group, others

Change file permissions:

Default file permissions are 644, now we will change to 744

$ ls –l persistent

-rw-r--r-- 1 root root 53 Jan 19 19:50 persistent ----without execution permission

$ chmod 744 persistent

$ ls -l persistent

-rwxr--r-- 1 root root 53 Jan 19 19:50 persistent ----execute permissions received

$? ----gives exit status of the last command executed

$! ---process number of the last background command

Meta characters:

* and ? are two meta characters.

We use * to match zero or more characters.

The command could not be located because '/bin' is not included in the PATH environment variable

$export PATH=/usr/bin:/bin ---works fine with this

.bashrc

.bash_history:

Has all the previously executed commands.

cat .bash_history

SFTP:

Getting file from another UNIX server through sftp:

root@hj-glbtal8591:~# sftp root@11.11.255.55 (user_id@ip_address) ---logs into SFTP

root@10.51.225.55's password:

Connected to 10.51.225.55.

sftp> cd /etc/postgresql/9.3/main ---go to the folder where file is located

sftp> ls –l ---check whether file is there or not

-rw-r----- 1 postgres postgres 4649 Oct 21 15:56 abc.conf ---file exists

-rw-r----- 1 postgres postgres 1636 Oct 20 20:01 bcd.conf

sftp> get abc.conf ---getting the file

Fetching /etc/postgresql/9.3/main/abc.conf to abc.conf

/etc/postgresql/9.3/main/pg_hba.conf 100% 4649 4.5KB/s 00:00

sftp> quit ------quit from sftp

root@hj-glbtal8591:~# ls –l ---check whether file has come

total 2 ---total no:of files in that particular folder

-rw-r----- 1 root root 4649 Jan 16 18:05 pg_hba.conf

drwxr-xr-x 3 root root 4096 Jan 12 21:18 unnati

root@hj-glbtal8591:~#

How do I put (upload) file into another UNIX server using SFTP?

root@hj-glbtal8591:~# sftp root@11.55.225.55

root@11.55.225.55's password:

Connected to 11.55.225.55.

sftp> put pg_hba.conf

sftp> quit

Some SFTP commands:

sftp> help (or) sftp> ? ---shows available commands in SFTP

sftp > !ls ---to list files in local directory (your machine)

sftp > ls ---to list files in remote directory (remote machine)

sftp> mget *.xls ---Get multiple files on a local system

sftp> !pwd --- Local working directory

sftp> pwd ----Remote working directory(print the current directory on the ftp server)

sftp> put pg_hba.conf ---put pg_hba.conf file in remote directory

sftp> mput *.xls ----to put multiple files on remote directory

sftp > cd path ----Change remote directory to 'path'

sftp> bye --- Quit sftp

sftp> exit ---- Quit sftp

sftp> quit ---- Quit sftp

sftp> mkdir persistent ---Create remote directory

sftp> !mkdir persistent --- Create local directory(in your machine)

sftp> rmdir persistent ----remove remote directory

sftp> version -----shows FTP version

sftp> ! ---go back to local directory/ quit sftp

sftp> !cat > persistent ---Execute 'command' in local shell/local machine

sftp> rm path ---Delete remote file

Uninstalling software:

$ apt-get purge <postgresql-server-dev-9.3>

Installing software:

$sudo apt-get install <python3-minimal>

wget command examples:

The quick and effective method to download software, music, video from internet is using wget command.

$ wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.1.tar.gz

$sudo http_proxy='http://name:pwd@brproxy.company.co.in:8080' apt-get install git

run .bashrc on your current terminal session to source ~/.export file.

echo "source ~/.exports" >> ~/.bashrc ???????

Connecting and Working on remote host:

ssh (SSH client) is a program for logging into a remote machine and for executing commands on a remote machine.

root@hj-glbtal8591:~# ssh -l root 10.51.226.43 ---connecting to remote host

root@hj-glbtal8592:~# exit ---goes back to original host

Awk command examples:

$ awk '{print $2,$5;}' persistent ---prints 2 and 5 fields from persistent file

Part3

Vim command examples:

$vim +10 pg_hba.conf ---goes to the 10 line of pg_hba.conf

$Cd - ---takes back to previous working directory

$echo $SHELL ---Gives shell name

Converting all charters in a file to upper case (dd command):

$dd if=persistent of=persistent_new conv=ucase --if= input file , of=output file, conv=convert, ucase=uppercase

$dd if=persistent_new of=persistent_new1 conv=lcase ---creates new file persistent_new1 with lowercase chareacters

$file * ---gives all files names and their details in the current directory

echo "hi krishna, how are you?" ----prints same

echo "hi krishna,\n how are you?" ----- \n is used for next line

echo "hi krishna,\r how are you?" ----- \r is used for removing, preceding it

echo "hi krishna,\t how are you?" ------ \t is used for giving four spaces

echo "hi krishna,\b how are you?" -------\b is for back space

echo "\033[1mhi krishna,how are you?\033[0m" ----\033[1m is used for printing in bold

\033[0m at the end is used for discontinuing boldness

echo "\033[7mhi krishna,how are you? \033[0m" ----\033[7m is used for printing background in black

Zip unzipping files:

$ gzip hari2 ---to zip a file

$ gunzip hari2 ----to unzip a file

$ TeradataStudioExpress64__linux_x86_64.16.00.01.00-1.tar.gz ----file to be unzipped and untarred

$ gunzip TeradataStudioExpress64__linux_x86_64.16.00.01.00-1.tar.gz ---to unzip a file

$ tar -xf TeradataStudioExpress64__linux_x86_64.16.00.01.00-1.tar ----to untar a tar file

$ tar -xzf TeradataStudioExpress64__linux_x86_64.16.00.01.00-1.tar ----to unzip and untar a tar file

ZIP a folder:

$ tar zcvf nameofcompressedfile.tar.gz /path/to/folder/compressthisfolder

--we can’t zip a directory in UNIX, hence archive using tar as above.

Scripting:

Using commands in script:

$vi using_command.sh ---step1 create a file

#using commands in a script ----step2 writing script in vi

pwd

ls –la

$sh using_command.sh ---step3 executing script

/root/Krishna -----output

total 136

drwxr-xr-x 2 root root 4096 Jan 20 17:54 .

drwx------ 7 root root 4096 Jan 20 17:54 ..

Script to read a name:

$vi read.sh

echo "enter your name"

read your_name ---‘read ‘ is used for taking input

echo "hi $your_name how are you"

$sh read.sh

enter your name

krishna

hi krishna how are you

Script to rename a file and open:

$vi rename_file.sh

echo "enter file_name"

read file_name ---Giving the required name (creates new file file_name)

mv persistent $file_name ---persistent file renamed to file_name

cat $file_name

$sh rename_file.sh

Script for reading two file names and renaming file1 to file2:

$vi rename_file1.sh

echo "enter file name1"

read file_1 -----reads file1 which already exists in server

echo "enter file name2"

read file_2 -----reads file2, which is a new file

mv $file_1 $file_2 ----renames file1 to file2

cat $file_2 ----opens file2

Positional parameters:

$ set Krishna from Kamalapuram

$ echo $1 ---gives 1^st string

Krishna

$ echo $2 ---gives 2^nd string

from

$ echo $3 ---gives 3^rd string

Kamalapuram

Script for set Positional parameter:

$ vi posional_parameters.sh

set Krishna from village1

echo $1

echo $2

echo $3

$ sh posional_parameters.sh

Krishna

from

Kamalapuram

Set can be used in a script as well as command prompt.

When set is given different sentence then, previous parameters are over written.

$ set hari from village2

$ echo $* ---we get all the strings

hari from village2

Note: Executable files are in Green color.

Assigning positional parameters from a file:

$ cat > test

hi Krishna, how are you?

$ set `cat test`

$ echo $*

hi Krishna, how are you?

While loop Script:

$vi while_loop.sh

#script for while loop

a=0

while [ $a -le 5 ]

echo $a

a=`expr $a + 1`

done

$sh while_loop.sh

For loop Script:

$ vi for_loop.sh

#for loop to read text and print it

for i in 1 2 3 4

do echo $i

done

$ sh for_loop.sh

Script to create a file and enter data into it:

$ vi file_creation.sh

#create a file and populate it

cat > hk

echo "from psl" >> hk

$ sh file_creation.sh

Ctrl+d

Script using Break:

$ vi break.sh

#using break to stop a process

a=0

while [ $a -lt 10 ]

echo $a

if [ $a -eq 5 ]

then break

a=`expr $a + 1`

done

$sh break.sh

Script for arithmetic operators:

$ vi arithmetic_operators.sh

#arithmatic operators

a=32 b=15

echo `expr $a + $b`

echo `expr $a - $b`

echo `expr $a \* $b` ----we have to give \ before *

echo `expr $a / $b`

echo `expr $a % $b` ------gives remainder

$ sh arithmetic_operators.sh

480

Note: expr works only with integer values.

Order of operation: Bracket, division, multiplication, modular division, addition and subtraction (BODMAS)

Working with floating numbers:

$vi floating.sh

a=10.5

b=3.5

c=`echo $a + $b | bc` ---bc is used for working with floating numbers

d=`echo $a - $b | bc`

echo $c $d

Script if user is ‘root’ then script is executed

$ vi unix1.sh

#if user is root then script is executed

a=`whoami`

if [ $a = 'root' ]

then

echo "authorised"

else

echo "not authorised"

exit

echo "hello"

date

$ sh unix1.sh

authorised

hello

Tue Jan 31 18:19:49 IST 2017

If Script in korn shell:

$ vi test7.ksh --step1: open file in vi

#!/bin/ksh --step2: code in vi

a=10

b=10

c=10

if [ $a == $b ] && [ $b == $c ]

then

echo "a is equal to b and b =c "

else

echo "None of the condition met"

$ ./test7.ksh --step3: execution of korn script

#!/bin/ksh

a=Pass

b=Pass

c=2

if [ $a == Pass ] && [ $b == Pass ] && [ $c=2 ]

then

echo "Pass "

else

echo "Fail"

o/p:

cdts10hdbe01d:abwdusr:/home/abwdusr> ./test7.ksh

Pass

Script to check date is valid or not:

x=22/02/22222

if [[ $x == [0-3][0-9]/[0-1][0-9]/[0-9][0-9][0-9][0-9] ]]

then echo "valid"

else echo "invalid"

cdts10hdbe01d:abwdusr:/home/abwdusr> ./test11.ksh

invalid

Script to check Column is numeric or not:

#!/bin/ksh

export a=`awk -F',' '{print $3}' prod_test.csv | awk ' $1 ~ "^[0-9][0-9]*$"' |wc -l`

export b=`awk -F',' '{print $2}' prod_test.csv | awk ' $1 ~ "^[0-9][0-9]*$"' |wc -l`

c=`expr $a + $b`

print $c

Child Vs Parent Process:

Any process can be a parent and child process at the same time. The only exception is the init process, which is always marked with PID (process ID) 1. Therefore, init is a parent of all processes running on your Linux system.

$ ps -p 1
PID TTY TIME CMD
1 ? 00:00:02 init

Any process created will normally have a parent process from which it was created and will be considered as a child of this parent process. For example:

$ echo $$
27861
$ bash
$ echo $$
28034
$ ps --ppid 27861
PID TTY TIME CMD
28034 pts/3 00:00:00 bash

Line 1: print a PID for a current shell - 27861
Line 2: create a new child process from the process ID 27861
Line 3: print a PID for a current shell - 28034
Line 4: with use of the ps command print the child process of PID 27861

When creating a new child process an export command simply ensures that any exported variables in the parent process are available in the child process.

To find out default login shell:

$ which bash

/bin/bash

(or)

$ who am i

chris pts/0 2011-11-26 07:19 (:0.0)

$ grep chris /etc/passwd

cnegus:x:13597:13597:ravi Negus:/home/cnegus:/bin/bash

File last modified time:

$ date -r <file_name>

Find where particular command came from:

You can use the type command.

e.g.: $ type cat

cat is a tracked alias for /bin/cat

To view your history list:

Use the history command

e.g.: $ history 0 -- --after logging in it shows from 1^st used command to last used command

Sequential commands/using multiple commands:

$ date ; uname ; whoami

Wed Jul 26 10:56:51 EDT 2017

Linux

Sdtdusr

Part4

Mailing:

$ mail -s "hi how r u" hk@in.company.com

Ctrl+d ---to add mail id in CC

Ctrl+c

(or)

$ echo "Mail body" | mail -s "Subject" hk@in.company.com --enter

echo "For testing" | mailx -s "Testing" hk@in.company.com --mail and mailx works as same

Script to send mail:

vi test.ksh

#!/bin/ksh

MESSAGE="hi How are u?"

SUBJECT="Greeting"

TOADDR="hk@in.company.com"

FROM="DONOTREPLY"

echo $MESSAGE | mail -s "$SUBJECT" $TOADDR -- -f $FROM

./test.ksh

Script to send CSV file:

#!/bin/ksh

MESSAGE="hi How are u?"

SUBJECT="Greeting"

TOADDR="hk@in.company.com"

FROM="DONOTREPLY"

echo "$MESSGE" | mail -s "$SUBJECT" -a "/development/abc/data/ABC_PRDCT.csv" $TOADDR -- -f $FROM

--any file can be send like this

To find a file whose location is not known:

$ vi $(find /home | grep xyzzy) ----doesn’t workin Kornshell

Export Usage:

$ m=hk1/hk2/hk3 ----assign m with the directory path hk1/hk2/hk3

$ export m ----exporting m

$ cd $m -----going to path hk1/hk2/hk3

Echo:

$ echo $$ ---returns process ID of the current shell

$ echo $(date) ---------prints date

Thu Jul 27 06:45:54 EDT 2017

Note: Variables can contain the output of a command or command sequence. You can accomplish this by preceding the command with a dollar sign and open parenthesis, and following it with a closing parenthesis. For example, MYDATE=$(date)

$ echo date -----here date is treated as string

date

Calculation:

$ echo "I am $[2012 - 1957] years old." ---works in Bash Shell

I am 55 years old.

Echo:

$ hari=krishna

$ echo ${hari:-"no data"}

krishna

$ hari=

$ echo ${hari:-"no data"}

no data

Find a file using echo:

$ echo *txt

--------------------------------------------------------------- ----------------------

$ echo "This is \" a quote and this is \\ a backslash"

This is " a quote and this is \ a backslash

So backslash itself must be escaped to show that it is to be taken literally.

$ alias ls

alias ls='ls --color=auto' ---The --color=auto option causes different types of files and directories to be displayed in different colors.

Counting no: of files in directory and assigning it to a variable:

$ num_files=$(ls -la | wc -l )

$ echo $num_files

When assigning variables, use only the variable name (for example, BALANCE). When you reference variable,

meaning you want the value of the variable, precede it with a dollar sign (as in $CurBalance). The result of the

latter is that you get the value of the variable, not the variable name itself.

$ echo "count :: ${num_files}"

count :: 5

$ hk1=${num_files-""}

$ echo $hk1

$ hk2=${num_files-"hi"}

$ echo $hk2

$ THIS="Example"

$ THIS=${THIS:-"Not Set"}

$ THAT=${THAT:-"Not Set"}

$ echo $THIS

Example

$ echo $THAT

Not Set

Shell Program1:

$ vi file4

read -p "Your name? " name

if [[ $name = $USER ]]; then

echo "Hello, me."

else

echo "Hello, $name."

$ sh file4

Your name? hk

Hello, hk.

--------------------------------------------------------------- -------------------------------------------------------------

Passing a Parameter in shell Script:

1) Create a ksh file in the pathà /development/abc/exe/

vi hari_practice1.ksh

. $1

echo "Hi Hari !" > ${DATA_FOLDER}/hari_practice1.txt

exit 0

2) Create a parm file in the pathà /development/abc/parms/

Vi hari_practice1.parms

export DATA_FOLDER=/development/abc/data

Note: To run ksh and parameter file.

/development/abc/exe/test_esp.ksh /development/abc/parms/esp_test.parms

3) Now check the file in the path /development/abc/data

--------------------- ------------------------------------------------- ----------------------------------------- ------------------

Usage of NOHUP:

-- nohup prevents the command from being aborted if you log out or exit the shell.

$nohup run_script.sh > hk1.out & --& used for running the command in background and note the process id

----------------------- ------------------------------------ --------------------------

back-ground Process:

$ ps l <PID> --to check back-ground Process

$ps -f ---gives background running jobs

$kill -9 <process_id> ---to kill background running process.

$cat hari& ---add at the end of a command to run the process in the back ground

------------------------------- -------------------------------------------------- ---------------------------------------------------

Command to change the shell:

$ chsh -s /bin/ksh

dos2unix:

$ dos2unix /home/tptuusr/.profile --converts a file from dos to UNIX compatible

How to check whether dos2unix is performed on a file? :

$ cat -ve <file_name> ---you will see $ at the end

Script to create a file by reading input from user:

$ vi script.sh

#!/bin/sh
echo "What is your name?"
read USER_NAME
echo "Hello $USER_NAME"
echo "I will create you a file called ${USER_NAME}_file"
touch "${USER_NAME}_file"

For loop:

#!/bin/sh

for i in hello 1 * 2 goodbye

echo "Looping ... i is set to $i"

done

Output:

Looping ... i is set to hello

Looping ... i is set to 1

Looping ... i is set to (name of first file in current directory)

... etc ...

Looping ... i is set to (name of last file in current directory)

Looping ... i is set to 2

Looping ... i is set to goodbye

While Loop:

#!/bin/sh

while [ "$INPUT_STRING" != "bye" ]

echo "Please type something in (bye to quit)"

read INPUT_STRING

echo "You typed: $INPUT_STRING"

done

What happens here, is that the echo and read statements will run indefinitely until you type "bye" when prompted

Script to list all files in a directory:

#!/bin/sh

for i in *

echo "$i"

done

While loop:

#!/bin/sh

echo "Please talk to me ..."

while : ###colon always evaluates to true

read INPUT_STRING

case $INPUT_STRING in

hello)

echo "Hello yourself!"

;;

bye)

vi echo "See you again!"

break

;;

echo "Sorry, I don't understand"

;;

esac

done

echo

echo "That's all folks!"

Usage of Typeset:

$ typeset -xu h=ram

$ echo $h

RAM

Curl:

--curl is used for downloading a file from web

---------------------------------------------- -------------------

--------------------------------------- -----------------------------------------------

$ rm -rf --rm removes a file in directory, r –removes all files in a directory recursively -- forcefully

Ping:

$ ping 8.8.8.8 ---ping lets you know other machine on the internet is live

---8.8.8.8 is Google’s public DNS SERVER

Bc:

$ bc ----it is a calculator program, we can do calculations here , press CTRL+D after using ‘bc’

--------------------- --------------------------------------------------------------------

Searching Large files:

$ less <file _name> --- we can search page at a time

--- press ESC q to exit or :q

--- /production to search production and press n for next occurrence

Even in vi /<word_name> works

----------------------------------- -------------------------------------------------------------------------------------------------------

Note:

File names can contain any character except / . If file contains spaces or any special characters then put file name in ‘quotes’ or precede each special character with \

------------------------------------------------------------- --------------------------------------------------

View:

$ view <file_name>

ESCàG ---goes to last line in view

Getting total columns from a file:

export Field_count=`head -1 ${FILE} | sed 's/[^,]//g' | wc -c`

Validate a date in unix:

To get Echo screen set below ones:

#!/bin/ksh
#set -vx
#shopt -s -o nounset
#umask 113

numeric check on columns

awk -F',' '{print $2}' test1.csv | awk ' $1 ~ "^[0-9][0-9]*$"' |wc -l ---gives total columns with only numeric data

**************** ***********

value:

$ a=-10

$ b=${a#-}

$ echo $b

Give row numbers to a file in unix:

$ awk '{printf "%d~%s\n", NR, $0}' test_dtl

Script to add row numbers in other folder than current:

#!/bin/ksh

pwd echo "gives current path in script"

cd /home/abwdusr/hk1/

echo "adding row number starts for all files"

awk '{printf "%d~%s\n", NR, $0}' dtl.csv > dtl_rowno.csv

awk '{printf "%d~%s\n", NR, $0}' prod.csv > prod_rowno.csv

echo "adding row number ends"

script to remove files in some folder:

#!/bin/ksh

cd /home/abwdusr/hk1 #Have to parameterize

rm dtl_rowno.csv prod_rowno.csv

unzipping thru script:

#!/bin/ksh

cd /home/abcdusr/hk1

gunzip dtl.csv

gunzip prod.csv

echo "unzipping ends"

unzips if any file is .gz format

Script to create a directory and create a file in it:

#!/bin/ksh

echo "Setting up Env"

cd /home/abcdusr

echo "Current Directory `pwd`"

if [ ! -d "/home/abcdusr/hk2" ]

then

mkdir hk2

echo "removing all the old files"

rm -f /home/abwdusr/hk2/*.*

cd /home/abwdusr/hk2

echo "hi" > test1

echo "

Script Execution Completed @ `date`"

Search This Blog

Mongo DB

Mongo DB

1) Installing MongoDB¶

2) Starting MongoDB

3) MongoDB Basics:

4) Basic Commands

5) Using python for MongoDB

SQL & Data-warehouse

Data-warehouse Concepts:

Unix

Unix

Basics:

Who is Logged In?

Viewing list of files:

Commands:

Command details:

Cat:

Stat:

Touch:

Deleting files and creating Directories:

Copy:

Taking backup to desired location:

MV:

SED:

Tr:

Vi Editor:

part2

Changing file Permissions:

SFTP:

Uninstalling software:

Installing software:

Connecting and Working on remote host:

Awk command examples:

Part3

Vim command examples:

Zip unzipping files:

Scripting:

If Script in korn shell:

Script to check date is valid or not:

Script to check Column is numeric or not:

Child Vs Parent Process:

To find out default login shell:

File last modified time:

Find where particular command came from:

To view your history list:

Part4

Mailing:

Script to send mail:

Script to send CSV file:

To find a file whose location is not known:

Echo:

Passing a Parameter in shell Script:

Usage of NOHUP:

back-ground Process:

Command to change the shell:

dos2unix:

Usage of Typeset:

Curl:

Ping:

Bc:

View:

Getting total columns from a file:

Validate a date in unix:

To get Echo screen set below ones:

numeric check on columns

Give row numbers to a file in unix:

Script to add row numbers in other folder than current:

script to remove files in some folder:

unzipping thru script:

Script to create a directory and create a file in it:

Total Pageviews

Followers