1 Python Advanced

Python methods:

# Instance method
# function vs method
class Class1:
    def func():
        print('hi')
print(type (Class1.func))  # access funtion directly from class (function)

object = Class1()
print(type (object.func)) ##function becomes a method when called via object/instance 
# of a class.

# Classmethod
class Person:
    def __init__(self, first_name, last_name, age):
        self.first_name = first_name
        self.last_name = last_name
        self.age = age

    def full_name(self):
        return f"{self.first_name} {self.last_name}"

    @classmethod    
    def other(cls):
        return Person('John', 'Cena', 25)

obj1 = Person.other()
print(obj1.full_name())

#
# Static method
class multiplier:
    
    # static method are used for grouping similar functions 
    # or uitility methods under a class
    @staticmethod
    def mutli_by_2(c):
        return 2*c + 100
            
    @staticmethod
    def mutli_by_5(f):
        return 5*f + 100

m = multiplier.mutli_by_2(25)
print(m)

m1 = multiplier.mutli_by_5(50)
print(m1)

Higher order functions:

# Higher order functions
def sum1(x):
    return x + 1

def sum2(func, x):
    result = func(x)
    return result

print(sum2(sum1,5))

O/p:

Closure:

If a variable goes out of scope or if function is removed from the current namespace. Even then if variable is remembered then this concept is called Clossure.

Closusre requirements:

1- Nested function

2- Nested function refer value from Outer function(enclosing function)

3- Outer function should return nested function

# Outer enclosing function
def sum1(a):
    
    # Nested function
    def sum2():
        print(a)
    return sum2  # Returning Nested function
int1 = sum1(5)
int1()

O/p:

Decorators:

A function which takes another function as an argument and extends its behavior without changing the original function is called a Decorator. (add functionality to an existing code).

from functools import wraps

class decorator:
    def __init__(self, n):
        self.n = n

    def __call__(self, fn):
        @wraps(fn)
        def wrapper(*args, **kwargs):
            print(self.n*'*')
            result = fn(*args, **kwargs)
            print(result)
            print(self.n*'*')
            return result
        return wrapper

@decorator(10)
def sum(a, b, c):
    return a + b + c

sum(5, 6, 4)

O/p:

**********

Python Generators:

# Generator function
n = 0
def my_gen():
    global n
    n += 1
    print('Printed first')
    yield n

    n += 1
    print('Printed 2nd')
    yield n

for item in my_gen():
    print(item)

O/p:
Printed first
1
Printed 2nd
2

2 SQL Server DB Connection
1) start the services in management studio

SQL server configuration manger start services

2) start the sql server

3) create DB, query
CREATE TABLE EMPLOYEE (
NAME CHAR(20) NOT NULL,
AGE INT,
INCOME FLOAT );

insert into EMPLOYEE values ('emp1', 31, 90000.00);

select * from EMPLOYEE;

Step 1: Install pyodbc
(base) C:\Users\user1>pip install pyodbc

Step2: Retrieve the server name
Server name: User\User1SQL

Step 3: Obtain the database name
Database name: sql_db

Step 4: Get the table name
Table name: dbo.EMPLOYEE

Connect Python IDE to SQL Server:

import pyodbc

conn = pyodbc.connect('Driver={SQL Server};'
'Server=User1\User1SQL;'
'Database=sql_db;'
'Trusted_Connection=yes;')

cursor = conn.cursor()
cursor.execute('SELECT * FROM sql_db.dbo.EMPLOYEE')

for row in cursor:
print(row)

##Insert values to SQL server Table from Python:

import pyodbc
conn = pyodbc.connect('Driver={SQL Server};'
'Server=User1\User1SQL;'
'Database=sql_db;'
'Trusted_Connection=yes;')

cursor = conn.cursor()
cursor.execute('SELECT * FROM sql_db.dbo.EMPLOYEE1')

cursor.execute('''
INSERT INTO sql_db.dbo.EMPLOYEE1 (NAME, AGE, INCOME)
VALUES
('Bob',55,85000),
('Jenny',66,86000)
''')
conn.commit()

Loading data from SQL Server to Python pandas dataframe
import pandas as pd
import pyodbc

conn = pyodbc.connect('Driver={SQL Server};'
'Server=User1\User1SQL;'
'Database=sql_db;'
'Trusted_Connection=yes;')
#query = "SELECT [BusinessEntityID],[FirstName],[LastName],
# [PostalCode],[City] FROM [Sales].[vSalesPerson]"

query = "SELECT * FROM sql_db.dbo.EMPLOYEE1"
df = pd.read_sql(query, conn)

df.head(3)

3 Random password Generation

import random

# Function to shuffle characters of a string
def shuffle(string):
  tempList = list(string)  # string to list
  random.shuffle(tempList) # Random shuffling list
  return ''.join(tempList) # join the random shffled templist to String

char1_uppercase=chr(random.randint(65,90))
char2_lowercase=chr(random.randint(97,122))
char3_special  =chr(random.randint(35,38))
char4_lowercase=chr(random.randint(97,122))
char5_lowercase=chr(random.randint(97,122))
char6_lowercase=chr(random.randint(97,122))
char7_lowercase=chr(random.randint(97,122))
char8_lowercase=chr(random.randint(97,122))

password = char1_uppercase + char2_lowercase + char3_special +\

           char4_lowercase + char5_lowercase + char6_lowercase  + \

           char7_lowercase + char8_lowercase

password = shuffle(password)
print("password is :", password)

password is : fkmTh#bm

GIT Basics

1 Intro

Git is a version control system (VCS) for tracking changes in computer files and coordinating work on those files among multiple people. It is primarily used for software development, but it can be used to keep track of changes in any files.

Top Version control systems:

1) Source code control system (SCCS)-1972

2) Revision Control System (RCS)-1982

3) Concurrent Control system (CVS)-1986-1990

4) Apache Subversion (SVN)-200

5) BitKeeper SCM-2000-closed source

6) GIT-April2005- distributed version control-open source

Single master repository in non-distributed version control system

Distributed Version Control system: Each user maintains own repository. No need to communicate with central repository.

1.1 Git Basics

Git installation: http://git-scm.com/downloads

$which git --states where git is located

/mingw64/bin/git

$git --version ---gives git version

git version 2.11.0.windows.1

Git Architecture

Normal VCS has two-tree architecture  repository

 Working

GIT has three-tree architectureRepository

Staging Index

Working directory

2 Git configuration:

System level configuration:

$git config --system

User level configuration:

$git config --global user.name “******”

$git config --global user.email “*****@domain.com”

To see configured list?

$git config --list to see the configured file

$git config user.name to get configured username

To configure editor:

$git config -- global core.editor "notepad.exe"

$ git help

$ git help log

3 Initializing Git project

After installing GIT and configuring, next step is to initialize GIT to a project (initializing a repository) means track the project.

1.Create a Working Directory and Initialize:

$mkdir first-git-project) Create a working directory

$cd first_git_project go to working Directory folder

$git init

Initialized empty Git repository in C:/Users/first_git_project/.git/

Above is the directory where git looks into.

ls shows directory as empty

$ls –la shows some files.

$ls

$ls -la .git Shows list of files git is tracking and it is git’s working space only exception is config file.

2. Adding, committing changes to the project:

$ git add . adds all files from working directory to staging index

$ git commit -m "initial commit" adds files from staging to repository

changes added to the project.

$ git commit –a adds and commits at a time.

What we do in repository?

1) Make changes

2) Add changes

3) Commit changes.

3. Commit log:

$ git log

commit 2cdc32affb3c0bc8d43a92a655ae3973f8e4afef --this is commit id, each commit has unique id

git log –is used for seeing the changes that are committed.

$git help log to get the manual pages

$ git log -n 1 -- gives 1 commit change

$ git log --since=2016-12-01 --gives commits from the date given

$ git log --until=2016-12-01 --gives up to the date given

$ git log ¬¬--author=”Krishna” --gives particular user log

Note:

$ git log --since="2016-12-01" --gives the commits since the date

$ git log --until="2016-12-30" --gives the commits until the date

We can also grep the log.

$ git log --grep="hari"

Patch:

$ git log --p -- Patch it gives diff of files.

$ git log --stat --summary –gives summary details of file addition and deletion

$ git log --format=online ¬¬ gives complete SHA

$ git log --format=short short SHA

$ git log --format=medium Medium SHA

$ git log --format=full full SHA

$ git log --format=fuller more info than full

$ git log --format=email more info than fuller

$ git log --format=raw gives full log details

$ git log –graph --show graph of each one of our commits

4.Committing Specific file and checking status:

$ git add file.txt adds specific file to staging

$ git commit file.txt adds file from staging to repository

$ git status gives status of files

$ git commit -m "add git file to project"

5. Editing files in git and compare:

After editing any file just do add command in git, then commit.

If you want to be reminded about the changes before committing?

‘diff’ command is used in unix to compare to files.

$ git diff for comparing file in repository with the file in working directory

If n files are modified, still git diff shows all modifications.

$ git diff --staged shows difference of staging and repository file

6. Deleting files from GIT:

Method 1:

Step 1: Directly deleting from the repository directory in system. (Select and delete)

Step2:

Step3: $ git commit -m "delete file"

Method 2:

Step1: $ git rm delete.txt

Step2: $ git commit -m "delete file"

7. Renaming and moving files:

Method1:

Step1: rename file in folder

Step2: $ git add primary_file.txt

Step3: $ git rm first_file.txt

Method2:

$ git mv second_file.txt secondary_file.txt

Moving a file into directory:

$ git mv third_file.txt first_directory/third_file.txt

$ git commit –a adds and commits at a time.

8. Undo changes in git:

Undo changes to a file in working directory:

$ git checkout -- index.html

-- 2 hyphen’s indicate stick to same directory.

Undo changes to a file in staging index:

To un-stage things from staging index to working directory.

$ git reset HEAD file_name.txt ---HEAD specifies look at the head pointer of the file.

Undo changes to a file in repository:

We can change only the last commit as SHA changes for every commit.

Amending last commit

Step1: $ git add git.txt -- adding to staging

Step2: $ git commit --amend –m “added name”

Note: Opening a file: cat file¬¬_name.txt

Creating a file: cat > hari.txt

Git revert: data added in file will be deleted and deleted data will be added to the file from repository.

$ git revert part_of_SHA

Eg: $ git revert 0fe9184f28c0c16cd9c

9. Using reset to undo many commits:

Powerful and very dangerous.

Soft reset: does not change staging index or working directory, they contain the files in their later revised state. But repositories can be set back to earlier version.

E.g.: git reset --soft 0fe9184f28c0c16cd9 (we can revert back to the original head pointer if we have the back up of recent head pointer)

Note: git diff --staged

Mixed reset (default): moves the head pointer to specified commit.

And it also changes staging index to match repository.

It doesn’t change working directory (means it has all the changes that were made).

E.g.: git reset --mixed 0fe9184f28c0c16cd9

Hard reset: It changes staging index and working directory to match repository.

E.g.: git reset --hard 0fe9184f28c0c16cd9

(But, we can get back to the prior of reset --hard if we have the SHA last commit before change)

Note: Cat .git/refs/heads/master  gives head(SHA number) of most recent commit.

Note: $ git commit --am “modifie”

--a  says commit all changes that have been made.

10. How to remove untracked files from working directory?

$ git clean

$ git clean -n (test run, which shows files which could be deleted)

$git clean -f ()

11. Ignoring Files:

Ignoring files inside working directory:

Create a special file in the root of working directory .gitignore

!index.php  don’t ignore the index.php file.

Step1: adding file to directory.

$ cat > .gitignore

Step2: Add the files to .gitignore that to be ignored.

Eg:

$ cat >> .gitignore

temp.txt

step 3: temp.txt file will be ignored.

Note:

temp.txt --ignores temp.txt file

*.zip --ignores files ending in .zip

*.gz --ignores files ending in .gz

log/*.log --ignores files ending with .log in log directory

log/*.log.[0-9] ----ignores files ending with .log with number in log directory

assets/videos/ ---ignores videos in assets directory

!assets/videos/tour_*.mp4 --doesn’t ignore the videos in videos directory ending with tour_*.mp4

Finally add .gitignore to repository by add and commit.

Note:

https://help.github.com/articles/ignoring-files

https://github.com/github/gitignore

Git will not ignore the files which were already tracked. Then, how the file can be ignored?

$git rm --cached tempfile2.txt -- the file will be removed from staging index and it will be present in working directory and repository

$git add .ignore

$git commit -m “removed tempfile2.txt from staging index”

12. Ignoring files globally:

Like ignoring files in all repositories.

User specific instead of repository specific

Step1:

krishna_narra@BLD23312 MINGW64 /c/Users --in Users folder we create the .gitignore_global

$ cat > .gitignore_global

.trashes --adding .trashes files to be ignored globally across all repositories

Step2: config the global file

$git config --global core.excludesfile ~/.gitignore_global (give ~ or full path like /users/.gitignore_global)

Tracking empty directories:

Git is designed to be a file-tracking system

- Tracks files

- Tracks the directories it takes to get to files.

- Ignores directories with no files.

If a file is added to the empty directory then the directory will be tracked.

$ touch hari/.gitkeep --creating empty file in hari directory/folder

$git add hari/

$ git commit -m "add 'empty hari directory' with .gitkeep file"

Now the directory will be tracked.

13. Navigating the commit tree:

Tree-ish:

How to reference a commit?

We can do it by,

1) full SHA-1 hash

2) short SHA-1 hash

-at least 4 characters

-unambiguous (8-10 characters) (for small projects usually we have 4 to 5 commits, so it is easy to search)

3) HEAD pointer

4) Branch reference, tag reference

5) Ancestry

Parent commit:

-HEAD^, acf87504^, master^

-HEAD~1, HEAD~

Grandparent commit:

-HEAD^^, acf87504^^, master^^

-HEAD~2

great-grandparent commit:

-HEAD^^^, acf87504^^^, master^^^

-HEAD~3

To list out a tree:

$ git ls-tree HEAD

100644 blob cd4ec433b469fa490d6a677470eeec3ff938bf8c .gitignore

100644 blob 9c5293d5fd24120e2a41934e39758bb02e9aae53 git.txt

100644 blob 7507afe34a0d88a4e3dbc0ca429d79bacc122e12 hari.txt

040000 tree d564d0bc3dd917926892c55e3706cc116d5b165e hari

100644 blob 11cbcce05dcf757d273b50d2eb8d14f2814bb19a krishna.sh

Note: blob is any kind of file and tree is a directory.

$ git ls-tree HEAD hari/

100644 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 hari/.gitkeep

$ git ls-tree HEAD^ hari/ --gives status of hari folder in previous state ie one commit back

To look specific commits:

$git show 3bf2fb29 --gives complete details of particular SHA

$ git help show -- to look into manual pages

Show works with blobs, trees, tags, and commits

$git diff ---it returns changes between working directory and staging index.

$git diff –staged or $git diff --cached ----it returns difference between staging index and repository

$git diff SHA

$git diff –b SHA --- -b is for space changes and –w is for all space changes.

14. Branching:

Branches are cheap: means they are easy to create, easy to delete and easy to work with.

- Try new ideas by creating sub branches

- Isolate features or sections of work

https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging

Creating a Branch/switching:

$ git branch shows list of branches in our local machine

$ cat .git/HEAD shows where head points to

$ git branch new_feature creates new branch

$ git checkout 'hot_fix' switches to new branch ‘new_feature’

$git checkout -b new-branch Creating a new branch and ‘checking out’ same time

b indicates both create and switch to new branch

Work on the fixes in this branch

Note: working directory should be mostly clean in order to switch between directories.

15. All Steps to push to repo:

1. Working in Master branch:

Clone the git repo to local.

$cd git_project # move to the project directory

# Work on code.

$git add . #add files to staging

$git commit -m "commit1" #add files to repo

$git push #Push to repo

2. Working in a branch:

$cd git_project  move to the project directory

$git remote set-url origin git@github.com:USERNAME/REPOSITORY.git set remote repo

$git init  initializing the current working directory

Branch:

$git branch shows list of branches in our local machine

$git branch branch1 create new branch

$git checkout 'branch1' switches to new branch ‘branch1’

# Work on code or pull from master.

$git add . add files to staging

$git commit -m "commit1" add files to repo

Fetch (any other changes by others) from master:

$git fetch --all

Rebase (link b/n branch and master):

$git rebase master

Move to master and merge the branch:

$git checkout master

$git merge branch1

Push the branch:

$git push --set-upstream origin master

Delete the hotfix branch:

$git branch -d branch1

3.1. Resolve Merge conflict on same line and same file:

In branch_conflcit1:

Edit file main2.py 2^nd line

$git add .

$git commit -m ‘commit from your branch’

In master:

Edit file main2.py 2^nd line

$git add .

$git commit -m ‘commit from master’

$git merge branch_conflict1

====Merge conflict error====

In your branch or master branch your working, do as below:

Remove lines:

<<<<<<< HEAD (current change)

=======

>>>>>>> branch_conflict1 (incoming change)

$git add main2.py

$git commit -m ‘commit after resolving merge conflict’

$git push

3.2. Removed file, merge conflicts:

$git checkout branch1 à2^nd user branch

$git rm main2.py àfile deleted

$git add .

$git commit -m ‘commit for merge conflict’

$git checkout master àyour branch

#Edit file & add code.

$git add .

$git commit -m ‘commit from master’

$git merge branch1

CONFLICT (modify/delete): main2.py deleted in branch1 and modified in HEAD. Version HEAD of main2.py left in tree.

Automatic merge failed; fix conflicts and then commit the result.

Add the file or remove the file.

$ git add main2.py

$git merge branch1

$ git push --set-upstream origin master

4. Git Pull

4.1 Pull remote branch to your branch

$git branch branch1

$git checkout branch1

Pull all files:

$git pull origin master

Pull specific file:

$git fetch –all

$git checkout origin/master -- README.md àpulling README.md file

5. Reset Last commit

$git reset HEAD~ èto undo files added, commited

Remote mail from bit bucket doesn’t match with local mail id.

$git commit --amend --author=”git_name <git_mail>”

Implementation of a Project using GIT in Pycharm

Step 1: Click "VCS" panel which is present on the top of pycharm.

Then select the "Enable Version Control Integration".

Step2: Popup comes-- Select Git from drop down --ok --

This initializes the Project for Git

Step3: Select the Green tik(right upper corner) for committing the code

Step4: Popup comes select the un versioned files (input the commit

message)-- click commit

Step5: Click VCS-- click Import into Version Control and then

click on Share project on GitHub.

Code Update to Git:

Step1: Select commit from left side of Git window.

Step2: Tick the modified file, enter commit message, then commit

Step3: select log master, Right click on master then,

window Popups then select file and push

Step4: Goto VCS -- update project--

Tick merge incoming changes--ok

Python, PySpark, Cloud, Unix, SQL

Search This Blog

999_Errors

Errors

Python Advanced

Connect Python IDE to SQL Server:

import pyodbc

conn = pyodbc.connect('Driver={SQL Server};'
'Server=User1\User1SQL;'
'Database=sql_db;'
'Trusted_Connection=yes;')

cursor = conn.cursor()
cursor.execute('SELECT * FROM sql_db.dbo.EMPLOYEE')

for row in cursor:
print(row)

##Insert values to SQL server Table from Python:

3 Random password Generation

Image Scrapping

Image Scrapping

Git Project

GIT Basics

1 Intro

2 Git configuration:

3 Initializing Git project

5. Editing files in git and compare:

6. Deleting files from GIT:

7. Renaming and moving files:

8. Undo changes in git:

9. Using reset to undo many commits:

10. How to remove untracked files from working directory?

11. Ignoring Files:

12. Ignoring files globally:

13. Navigating the commit tree:

4. Git Pull

5. Reset Last commit

Implementation of a Project using GIT in Pycharm

Total Pageviews

Followers

Search This Blog

999_Errors

Errors

Python Advanced

Connect Python IDE to SQL Server:

import pyodbc conn = pyodbc.connect('Driver={SQL Server};' 'Server=User1\User1SQL;' 'Database=sql_db;' 'Trusted_Connection=yes;')cursor = conn.cursor()cursor.execute('SELECT * FROM sql_db.dbo.EMPLOYEE')for row in cursor: print(row)

##Insert values to SQL server Table from Python:

3 Random password Generation

Image Scrapping

Image Scrapping

Git Project

GIT Basics

1 Intro

2 Git configuration:

3 Initializing Git project

5. Editing files in git and compare:

6. Deleting files from GIT:

7. Renaming and moving files:

8. Undo changes in git:

9. Using reset to undo many commits:

10. How to remove untracked files from working directory?

11. Ignoring Files:

12. Ignoring files globally:

13. Navigating the commit tree:

4. Git Pull

5. Reset Last commit

Implementation of a Project using GIT in Pycharm

Total Pageviews

Followers

import pyodbc

conn = pyodbc.connect('Driver={SQL Server};'
'Server=User1\User1SQL;'
'Database=sql_db;'
'Trusted_Connection=yes;')

cursor = conn.cursor()
cursor.execute('SELECT * FROM sql_db.dbo.EMPLOYEE')

for row in cursor:
print(row)