Python, PySpark, Cloud, Unix, SQL: 2022

170 Python Other topics

13 Other Topics:

Python Shallow Copy, Deep Copy, Copy using =Operator

#Copy using = operator
list1 = [[1,2],[3,4],[5,6]]

list2 = list1   # Creates variable list2 referring Object list1

print('list1 before modification:', list1)
print('list2 before modification:', list2)
print('ID of list1:', id(list1))

list1[1][1]=10

print('list1 after modification:', list1)
print('list2 after modification:', list2)
print('ID of list2:', id(list2))

O/p:

list1 before modification: [[1, 2], [3, 4], [5, 6]]

list2 before modification: [[1, 2], [3, 4], [5, 6]]

ID of list1: 82228360

list1 after modification: [[1, 2], [3, 10], [5, 6]]

list2 after modification: [[1, 2], [3, 10], [5, 6]]

ID of list2: 82228360

list1 and list2 has same ID. If we modify values any, changes will appear in the other also.

To overcome this we have,

Shallow copy

Deep copy.

Copy module is used for both shallow and deep copy.

Shallow Copy:

# shallow copy
import copy
list1 = [[1,2],[3,4],[5,6]]
list2 = copy.copy(list1)  #list2 is shallow copy of list1. ( it is created as new object ) 
          # which has reference of the original elements of list1.                          
print('list1 before modification:', list1)
print('list2 before modification:', list2)
print('ID of list1:', id(list1))

list1[1][1]=10

print('list1 after modification:', list1)
print('list2 after modification:', list2)    
print('ID of list2:', id(list2))

list1.append([7,8])

print('list1 after sublist addition:', list1)
print('list2 after sublist addition:', list2)  # appended sublist wont reflect here

Output:

list1 before modification: [[1, 2], [3, 4], [5, 6]]

list2 before modification: [[1, 2], [3, 4], [5, 6]]

ID of list1: 93070824

list1 after modification: [[1, 2], [3, 10], [5, 6]]

list2 after modification: [[1, 2], [3, 10], [5, 6]]

ID of list2: 93070696

list1 after sublist addition: [[1, 2], [3, 10], [5, 6], [7, 8]]

list2 after sublist addition: [[1, 2], [3, 10], [5, 6]]

Deep Copy:

# Deep copy
import copy
list1 = [[1,2],[3,4],[5,6]]
list2 = copy.deepcopy(list1)  # list2 is deep copy of list1. ( it is created as new object)
# and recursively adds the copies of nested objects present in the original elements.
# means list2 elements wont get effected when list1 elements modified
print('list1 before modification:', list1)
print('list2 before modification:', list2)
print('ID of list1:', id(list1))
print('ID of list2:', id(list2))

list1[1][1]=10
print('list1 after modification:', list1)
print('list2 after modification:', list2)


list1.append([7,8])
print('list1 after sublist addition:', list1)
print('list2 after sublist addition:', list2)

Output:

list1 before modification: [[1, 2], [3, 4], [5, 6]]

list2 before modification: [[1, 2], [3, 4], [5, 6]]

ID of list1: 86296296

ID of list2: 86295816

list1 after modification: [[1, 2], [3, 10], [5, 6]]

list2 after modification: [[1, 2], [3, 4], [5, 6]]

list1 after sublist addition: [[1, 2], [3, 10], [5, 6], [7, 8]]

list2 after sublist addition: [[1, 2], [3, 4], [5, 6]]

162 Python Class

12) Python Class

#!/usr/bin/python
# creating class
class Employee:
   'base class for all employees'
   
   def __init__(self, f_name, l_name, age):
      self.f_name = f_name
      self.l_name = l_name
      self.age = age
         
   def displayEmployee(self):
      print ("First Name : ", self.f_name, "Last Name :", self.l_name, "Age: ", self.age)

#creating instance of objects
"This would create first object of Employee class"
emp1 = Employee("John", "Cena", 45)
"This would create second object of Employee class"
emp2 = Employee("Peter", "Parker", 37)

# calling/accessing attributes
emp1.displayEmployee()
emp2.displayEmployee()

O/p:

First Name : John Last Name : Cena Age: 45

First Name : Peter Last Name : Parker Age: 37

Class Inheritance:

161 Python Errors & Exceptions

Python Errors & Exceptions

Syntax error:

i=[1,2,3]
for x in i print (x)

O/p:

for x in i print (x)

SyntaxError: invalid syntax

160 Python Dates

10). Python Dates

from time import time
time()  # gives ticks
o/p:
1617804792.9688406

import time
print(datetime.datetime.now())
time.sleep(5)           #delay by 5sec
print(datetime.datetime.now())
O/p:
2021-04-07 20:01:56.065809
2021-04-07 20:02:01.067447

date = datetime.datetime.now().strftime("%d-%m-%y")
print(date, '\n')
O/p:
29-03-21

date1 = datetime.datetime.now().strftime("%d%m%y")
print(date1, '\n')
O/p:
290321

date2 = datetime.datetime.now().strftime("%Y")
print(date2, '\n')
O/p:
2021

date2 = datetime.datetime.now().strftime("%y")
print(date2)
O/p:
21

#adding or subtraing days to todays date
import datetime
yesterday = datetime.datetime.now() - datetime.timedelta(days=1)
print(yesterday)
O/p:
2021-04-06 20:18:56.601772

158 Python Regular Expressions

8). Regular Expressions

import re
line = "Python is a great language"

match = re.search('^P', line)  # '^' Matches char start of the string
print(match)
O/p:
< re.Match object;span = (0, 1), match = 'P' >


match = re.search('e$', line)  # '$' matches char end of the string
print(match)
O/p:
< re.Matchobject; span = (25, 26), match = 'e' >


match = re.search('[a-z]', line)  # matches a single character
print(match)
< re.Match object; span = (1, 2), match = 'y' >

#example
line = "Python is a great language"
match = re.search('[^a-z]', line)  # matches a single character (here we get 1st space)
print(match)
O/p:
< re.Match object; span = (6, 7), match = ' ' >


line = 'python is a Great Language'
match = re.search('[a-z]+[^a-z]+[a-z]+[a-z]', line)  # [a-z] gives 1st char + [^a-z] takes space :o/p is 'python '
print(match)  # + [a-z] matches 1 char + [a-z] matches 1 char :final o/p is:'python is'
< re.Match object; span = (0, 9), match = 'python is' >

match = re.search('py*', line)  # Matches preceding character zero or more times
print(match)

line = 'aa aa aaa aaaa'
match = re.search('a{3,4}', line)  # Matches the preceding character minimum m times, and maximum n times
print(match)
O/p:
< re.Match
object;
span = (6, 9), match = 'aaa' >


#exmaple
line = 'aa aa aaa aaaa'
match = re.search('a{4}', line)  # Matches the preceding character exactly m times
print(match)
O/p:
< re.Match object; span = (10, 14), match = 'aaaa' >


#exmaple
line = 'ac abc abcd'
match = re.search('ab?', line)  # Matches the preceding character zero or one times
print(match)
o/p:
< re.Match
object;
span = (0, 1), match = 'a' >

#exmaple
line1 = 'python is a Great Language'
match = re.search(r'[\w]+[\w]', line1)  # \w Matches a word character (a-zA-Z0-9) :
match.group()
O/p:
'python'

#exmaple
match = re.search(r'[\w]+@[\w]+\.[\w]+', 'Please mail it to ayushi_wasthere@gmail.com')
match.group()
'ayushi_wasthere@gmail.com'


#exmaple
line1 = 'python is a Great Language'
match = re.search('\w*', line1)  # Matches a word character (a-zA-Z0-9) :
print(match)
O/p:
< re.Matchobject; span = (0, 6), match = 'python' >



#example
match = re.search('\w+', line1)  # Matches a word character (a-zA-Z0-9) :
print(match)
o/p:
< re.Match object; span = (0, 6), match = 'python' >


#example
line2 = '@python@ is a eay to learn language'
match = re.search('\@\w+\@', line2)
print(match)
O/p:
< re.Match object; span = (0, 8), match = '@python@' >

#example
match = re.search('\w+\@', line2)
print(match)
o/p:
< re.Match object; span = (1, 8), match = 'python@' >

#### logfile exmaple
line3 = '07-04-2021-22:00:00 dev login in error'
match = re.search('\w+', line3)
print(match)
O/P:
< re.Match object; span = (0, 2), match = '07' >


#exmaple
match = re.search('[a-z]+\w', line3)
print(match)
O/p: 
< re.Match object; span = (20, 23), match = 'dev' >


#example
match = re.search('\d+', line3)
print(match)
O/p:
< re.Match object; span = (0, 2), match = '07' >


#exmaple
match = re.search('\d+-\d+-\d+', line3)
print(match)
O/p:
< re.Match object; span = (0, 10), match = '07-04-2021' >


#example
file = open(r'D:\0_Data\regex\regex1.txt', 'r+')
error = []
for i in file:
    match = re.search('\d+-\d+-\d+', i)
    error.append(match)
    print(match)
O/p:
< re.Match object; span = (0, 10), match = '07-04-2021' >
< re.Match object; span = (0, 10), match = '08-04-2021' >
< re.Match object; span = (0, 10), match = '09-04-2021' >
< re.Match object; span = (0, 10), match = '10-04-2021' >
< re.Match object; span = (0, 10), match = '11-04-2021' >
< re.Match object; span = (0, 10), match = '12-04-2021' >


#example
file = open(r'D:\0_Data\regex\regex1.txt', 'r+')
error_date = []
for i in file:
    match = re.search('\d+-\d+-\d+', i)
    error_date.append(match.group())
error_date

O/p:
['07-04-2021',
 '08-04-2021',
 '09-04-2021',
 '10-04-2021',
 '11-04-2021',
 '12-04-2021']

157 Python Functions

7) Functions

1-Function is defined using the def keyword.

2-Arguments/parameters:

Arguments/parameters are specified in parenthesis after function name.

n number of arguments can be added.

3- function_docstring is optional.

4-return exists a fucntion

# Python fucntion syntax
def functionname( parameters ):
   "function_docstring"
   function_suite
   return [expression]

# simple python function
def hi():
    print ('hello')

hi() #to call function

Pass by reference vs value:

Parameters/arguments in Python are passed by reference.

Means if you change what a parameter refers to within a function, change also reflects back in the calling function.

# Pass by reference vs value
def paas_by_ref( mylist,mylist1 ):
   "This changes a passed list into this function"
   mylist.append([1,2,3]);
   mylist1.extend([1,2,3]);
   print ("Values inside function: ", mylist)
   print ("Values inside function mylist1: ", mylist1)
   return

# calling paas_by_ref function
mylist = [10,20,30];
mylist1 = [10,20,30];
paas_by_ref( mylist, mylist1  );
# paas_by_ref(  );
print ("Values outside function: ", mylist)
print ("Values outside function mylist1: ", mylist1)

Return statement:

def return_func( arg1, arg2 ):
   # Add both the parameters and return them."
   total = arg1 + arg2
   print ("Inside function : ", total)
   return total;

# # Now you can call sum function
return_func( 10, 20 );

Function Arguments:

You can call a function by using following types of arguments:

• Required arguments

• Keyword arguments

• Default arguments

• Variable-length arguments

# Required arguments
def args_func( str ):
   "This prints a passed string into this function"
   print (str)
   return;

# Now you can call printme function
args_func()  # pass args here otherwise gets error

# keyword arguments
def args_func (name1, name2):
   "This prints a passed info into this function"
   print ("Name1: ", name1)
   print ("NAme2: ", name2)
   return;

# Call  function
args_func (name1='a', name2="b")

# Default arguments
def keyword_args (name1, name2='b'):
"This prints a passed info into this function"
print ("Name1: ", name1)
print ("NAme2: ", name2)
return;

# Call function
keyword_args (name1='a')

# Variable length arguments
def args_func (name1, *name2):
   "This prints a passed info into this function"
   print ("Name1: ", name1)
   print ("Name2: ", name2)
   return;

# Call  function
# for *name2, we can pass 0 or n args
args_func ('a')

155 Python Loops

5) Loops

5.1 for loop

for i in [1,2,3]:

print (i)

for loop with else,

ints = [1, 2, 3, 4]

for d in [0,1,2,3]:

if d == 3:

break

else:

print("No num left.")

print("1 num left")

5.2) while loop

m=5

i=0

while i < m:

i=i+1

print(i)

5.3 Nested loops

Nested loops is “one loop inside another loop”.

Python Decision making

4) Decision making

Evaluates expression with outcome as True or False.

4.1 Block Structure and Whitespace

code executed when a specific condition met is defined in a "block."

block structure in python is done by indentation.

use 4 spaces(1TAB) for each level of block indentation.

If Statement:

#!/usr/bin/python
var1 = 10
if var1:
   print ("Got desired value")
   print (var1)

IF Else Statement:

var = 0
if var:
   print ("Got true expression value")
   print (var)
else:
   print ("Got false expression value")
   print (var)

O/p:

Got false expression value

elif statement:

if expression1:
   statement(s)
elif expression2:
   statement(s)
elif expression3:
   statement(s)
else:
   statement(s)

if elseif:

if expression1:
   statement(s)
   if expression2:
      statement(s)
   elif expression3:
      statement(s)
   elif:
      statement(s)
   elif expression4:
      statement(s)
else:
   statement(s)

Nested loops:

n = int(input('enter number :'))

if n % 2 != 0:
    print('Odd number')
elif n % 2 == 0:
    if (n >= 2 and n <= 99):
        print('Two digit even number')
    elif (n >= 100 and n <= 999):
        print ('3 digit even number')
    else:
        print('>= 4 digit  even number ')
else:
    print('done')

3. Variables, Dynamic Typing, Dynamic binding

Variables:

A variable in Python is defined through assignment. Assign values to variables using ' = '.

name = 'person1'
age = 30

print ('name :', name)
print ('age :', age)

O/p:

name : person1

age : 30

Dynamic Typing:

No need to declare variable type, value that a variable points to has a type, the variable has no type.

Dynamic Binding or Late Binding

When compiler is not able to resolve the call/binding at compile time, such binding is known as Dynamic or late Binding. The type of object is determined at the run time so this is known as dynamic binding.

Strong Typing:

Example:

'Python ' + str(3.7)

152 Python Data Types

2.0 Python Data Types

Python has five standard data types –

Numeric types: Sequences: Sets: Mappings: Boolean:

Int str set dict

Long byte frozen set

Float list

Complex tuple

1) Numeric types:

Integers:

Integers are any numbers without decimals

#Int assignment exmaple

a=1

print(a)

o/p:

Long:

no long in python 3

Float

# float

a = 1.0

type(a)

Complex:

Complex literals can be created by using the notation x + yj

x =real component

y = imaginary component.

2) Sequences

2.1 Strings

strings are sequence.

Strings are immutable.

[] are used for Indexing. Indexing starts at 0.

String literals can be defined with any of single quotes ('), double quotes (") or triple quotes (''' or """)

2.2 List

4 built-in data types in Python are List, Tuple Set and Dictionary.

List is a sequence, it can hold different objects. (no fixed type constraint )

Lists are mutable.

Lists are constructed with Square brackets0 - "[]"

We can use " : " to index string.

list1= [1, 'str', 1.2]

indexing:

2.3 Tuple:

Tuples are constructed with Parentheses - '()'

Immutable

Example usage of tuples, in days of week, dates in calendar.

For Data integrity, tuples are used.

Tuples are faster compared to list.

tup1 = (1,2,3)  # tuple creation
print (tup1[:2]) # indexing tuple

O/p:

(1, 2)tuple

byte (binary type)

3) Sets:

set

frozen set

Sets are unordered collection of unique elements.

Built in data type which stores collections of data.

Items in set are unordered, unchangeable, unique.

Set items can be of any data type.

s = set()   # set creation
s.add(1)   # adding elemt to set
print(s)

Output:

{1}

set1= {'apple', 1, 2.3}           # set creaiton
print(set1)

Output:

{2.3, 'apple', 1}

set creation from list:

list1 = ['1', '2', '1','2','1','2']
set2 = set(list1)
print(set2)

Output:

{'2', '1'}

Access set items:

We, can't access items in a set by referring index or key. But, we can loop over.

set1= {'apple', 1, 2.3}     # set creaiton

set1[0]

O/p:

TypeError: 'set' object is not subscriptable

set1= {'apple', 1, 2.3}     # set creaiton
for i in set1:
    print(i)

O/p:

2.3

apple

Add new item to set:

set1= {1, 2.3, 'Bng'}     # set creaiton

set1.add('new-item')    #adding item to set
print(set1)

O/p:

{1, 2.3, 'new-item', 'Bng'}

Joining Sets:

set1 = {'a', 'b', 'c'}
set2 = {1, 2, 3}

unioned_set = set1.union(set2)
print (unioned_set)

O/p:

{1, 2, 'a', 3, 'c', 'b'}

4)Mappings:

Dictionary

2.5 Dictionaries:

Dictionary is a mapping.

Mappings are a collection of objects that are stored by a "key".

Mapping store objects by their relative position.

Dictionary consists of a key with an associated value. Value can be any python object

dict1 = {1:'key1', 'key2': 'value2', 'key3': 'value3'}
dict1

O/p:

{1: 'key1', 'key2': 'value2', 'key3': 'value3'}

Dictionary comprehension:

dict_comprehension = {x:x**2 for x in range(5)}
dict_comprehension

O/p:

{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

5) Boolean:

Boolean can be defined by typing True/False without quotes

# Boolean Example
bool1 = True
bool2 = False

print (type(bool1))

O/p:

Note: Mutability:

int, float, bool, str, tuple, unicode are immutable.

list, set, dict are mutable.

599_AWS Interview Questions

0. AWS

Q. What is AWS?

-Amazon Web Services is a Cloud Provider

-AWS provide servers and services that can be used on demand and scale easily.

Q. SSO?

Single Sign ON – it is like using 1 account for many logins.

Q. AWS Cloud Use Cases?

AWS used in

- Enterprise IT,

- Data analytics

- Website hosting

- Gaming

- Storage

- Mobile

- Social Apps

Q. AWS Well-Architected tool?

An AWS Well-Architected tool provided consistent approach for reviewing your architecture against AWS best practices.

AWS Well-Architected Tool àWorkloads à Define workload

Q. Pricing in AWS?

It is based on 3 factors.

1) Compute (CPU, resources, memory)

2) Storage

3) Outbound Data transfer (like downloading froms3)

Q. How to Choose region?

1) Compliance with data governance and legal rules.

2) Proximity to Customers

3) Pricing

4) Available Services

5) Based on Latency: how much band width of data is coming

Q. Edge location use?

Used for caching the data.

For example, a user in India accessing a file from US east region for the first time the file will be accessed from US East region. If any other user from India tries to access the same file, then they can directly access it from caching server in India.

Q. Maximum Limit set by aws for roles, Users in an account?

1000 roles,

5000 Users,

300 Groups,

Iam roles and users per policies can be 10,

IAM users can be member of 10 groups

You can raise a ticket to support team and ask for increasing the limit.

1. Iam

Q. IAM

- Identity and Access Management

- enables you to manage the access to AWS services and resources securely.

- IAM Service usage is FREE!

- IAM is a Global service.

Q. Features of IAM:

- Securely manage services and resources.

- Creates IAM principals such as Users, Groups and Roles.

- Allow/Deny access via IAM Policy.

- Supports Identity Federation.

- Provides Multi-Factor Authentication.

- Provides Policy Simulator.

Q. How can users access AWS?

AWS can be accessed in 3 ways:

- AWS Management Console (Password is used to login)

- AWS Command Line Interface (access keys )

- AWS Software Developer Kit (access keys)

Q. What is a Role in Iam?

Role is not linked to any person/user or service. Rather you assume the Role as which grants you permission to access that(service).

Q. What are policies in Iam?

There are 2 types of policies.

1) Managed policies (created by AWS)

2) Custom policy (Created by us and we have to manage them)

Policies are:

1. A policy is an entity in AWS that, when attached to an identity or resource defines their permissions.

2. A policy contains different Permissions, determine whether the request is allowed or denied.

3. Policies are stored in AWS as JSON documents.

4. Policy is attached to either principals (Users, Roles or Groups) or to resources (e.g., S3 bucket).

Q. Iam policy consists of?

Policy Consists of:

1) Version: It’s a policy language version, always include “2012-10-17”

2) ID: Identifier for policy (optional), (it helps in quick recognition for which policy is)

3) Statement: one or more required

Statement consists of:

1) SID: Statement Identifier (optional)

2) Effect: Whether statement allows or denies access ()

3) Principal: accounts/user/role to which this policy is applied to (ARN is used here)

4) Actions: Set of actions policy allows or denies

5) Resources: list of resources to which action applied to

6) Conditions: conditions applied when the policy is executed

Q. IAM Terminologies:

- Resources: Resources are the things on which actions can be taken. Example: EC2 instances are resources so someone who has the power to start or stop an instance, they are acting on a resource.

- Principals: Things that can take action. So, principals act and they act upon resources. It includes users, groups and roles. These are also called as Identities.

- Users: IAM users are entities created in AWS and have permissions to perform some actions.

- Group – Collection of IAM users.

- Role – An identity that grants permission.

Q. What is ARN?

ARN --Amazon Resource Names

It is an unique name for all the resources which are being assigned in AWS.

It is helpful when we write policies, rules.

It is automatically assigned for services we create.

Q. Defaualt deny in Iam?

IAM User and IAM Group by default has no permissions to access AWS resources. This is called as “Default Deny” or “Non-Explicit Deny”.

Q. How to create/import 1000 users at a time in AWS?

By Bulk importing csv file, using AWS SSO (single Sign on) we can create 1000 users at a time.

Q. What is MFA in IAM?

AWS Multi-Factor Authentication (MFA) is a simple best practice that adds an extra layer of protection on top of your user’s name and password.

Q. What are identity-based policies?

Policies that you can attach to a principal (or identity) such as an IAM user, role or group.

These policies control what actions that identity can perform, on which resources, and under what conditions.

Q. Resource based policies?

Policies that you attach to a resource, such as S3, IAM, Lambda, EC2.

Q. How policies are different form Permission boundaries?

Permission Boundaries are like add on policies on Policies.

Example:

If you have AdminrolePolicy then if access is given for DynamoDB then your area admin for DynamoDB.

Q. What is Access Analyzer in Iam?

With Access Analyzer you can analyze:

1) what kind of security your account has?

2) What services provided for your account

2. EC2

Q. EC2

- Elastic Compute Cloud.

Infrastructure as a service.

-Virtual server to run applications.

-Scalable Computing Capacity in AWS Cloud. (Increase /decrease as required)

-Scale up or down to handle changing requirements, reduce the need to forecast traffic.

Q. Types of EC2 instances?

- General purpose. (t2, t3, t4, m4, m5) -balance b/n memory, Compute, Networking

- Compute optimized. (c4, c5, c6)

- Memory optimized. (r4, r5, r6, u, x1, x2, z1)

- Storage optimized. (d2, d3, h1, i3, im, is)

- Accelerated computing (f1, g2, g3, g4…)

Q. Instance purchasing options?

Amazon EC2 provides the following purchasing options.

On-Demand Instances – Pay, by the second.

Savings Plans – commitment to a consistent usage, for a term of 1 or 3 years.

Reserved Instances – Reserve for a term of 1 or 3 years.

Spot Instances – Request unused EC2 instances,

Dedicated Hosts – Pay for a physical host

Dedicated Instances – Pay, by the hour, for instances that run on single-tenant hardware.

Capacity Reservations – Reserve capacity for your EC2 instances in a specific Availability Zone for any duration.

Q. Type of Scaling recommend to Amazon databases?

Autoscaling

Q. Diff type of Load balancers?

- Application Load Balancer,

- Network Load Balancer,

- Gateway Load balancer,

- Classic load balancer.

Q. How can you auto delete the snapshots?

AWS ops autometer service is used which handles snapshots automatically.

Q. EC2 storage volumes?

- Storage volumes for temporary data, which are deleted when you stop, hibernate, or terminate your instance, known as instance store RAM volumes.

Ex: used for testing, POC's

- Persistent storage volumes (permanently there) for your data using Amazon Elastic Block Store (Amazon EBS), known as Amazon EBS Volumes.

3. S3

Q. S3

Simple Storage Service

Q. Default storage service in S3?

Standard.

Q. Can a bucket have different objects in different storage class?

Yes

Q. Amazon S3 is which type of service?

Object

Q. For which type of data S3 offers encryption services?

- Data in flight,

- data in rest

Q. Does S3 objects are accessible from any region.

True

Q. S3 storage classes:

1. Standard

2. Standard-IA

3. Intelligent-Tiering

4. One Zone-IA

5. Glacier Instant retrieval

6. Glacier Flexible retrieval

7. Glacier Deep archive.

Q. Copy a file from Ec2 instance to s3?

- SSH EC2 instance.

>aws s3 cp abc.txt s3://my-bucket-1b

Q. Copy a file from s3 to Ec2 instance?

>aws s3 cp s3://my-bucket-1b/xyz.txt .

4. VPC

Q. VPC main components?

VPC: Networking Layer for EC2.

Subnet: A logical partition on an IP network/address into multiple segments/IP address (or range of IP addresses)

CIDR: a method for allocating IP addresses

Route Tables: Set of rules, called Routes that are used to determine where network traffic is directed.

Internet Gateway: A gateway that you attach to your VPC to enable communication between resources in your VPC and the internet

VPC Endpoints: Allows you to connect to AWS services using a private network instead of using the public Internet. (without Internet Gateway, NAT device, VPN connection, instances do not require public IP addresses to communicate with resources in the service)

Q. Host id in Ipv4

Ipv4:

172.16.47.5

172.16 -- network prefix

47.5 -- Host ID

Q. IP addresses calculation in CIDR?

10.0.0.0/24 à256 Ip addresses (10.0.0.0 – 10.0.0.255)

Every network has 32bits

32- CIDR=32-24=8

No: of Ips è 2^8 =256

Q. Ipv4 vs Ipv6

Ipv4	Ipv6
1) Format is 32bit, 4groups of up to 3 decimal digits	1) Format is 128-bit, 8 groups of 4 hexadecimal digits
2) Default and required for all VPC's	2) Option only
3) VPC CIDR block size can be from /16 to /28	3) VPC CIDR block size fixed at /56
4) Subnet CIDR block size can be from /16 to /28	4) Subnet CIDR block size fixed at /64
5) You can choose the Private ipv4 CIDR block for your VPC	5)

Q, NACL vs Security Group?

Security Group vs NACL

SG works at instance level, NACL works at subnet level.

5. Lambda

6. Glue

Q. Access S3 for glue job?

Iam role to access S3, GlueServiceRole, GlueServiceNotebookRole, GlueConsoleAccess

Q. Type (where job runs?)

Spark, Spark Streaming, Python shell

Q. Glue version?

Spark3.1, Python 3 (Glue version 3.0)

Q. Script stored at?

In S3 bucket in specific folder

Q. Temporary directory in Glue job?

Location where temporary results are saved.

Q. Incremental load implementation?

Advanced properties -- job bookmark -- enable

Q. Monitoring options?

Job metrics (Enable creation of cloud watch metrics CloudWatch metrics)

Continuous logging

Spark UI (enable Spark UI for monitoring)

Security, Script libraries, job parameters?

Q. Library path?

Specify S3 bucket folder/key path.

Q. Dependent jars path?

Specify S3 bucket folder/key path.

Q. Referenced files path?

Specify S3 bucket folder/key path.

Q. Worker type?

G.1X (memory intensive jobs Max 299 workers

G.2X (for ML transformations) Max 149 workers

Data Processing Units (or DPUs)

Q. Max number of concurrent runs allowed for this job?

Max concurrency (default 1), error is returned when threshold reached.

Q. Number of retries?

Mention

Q. Job time out in Glue job?

Job execution time limit. Default is 2880min (48hrs)

Q. Delay notification threshold (in min)?

If job runs longer than specified time, glue sends notification via CloudWatch.

Q. Catalog options?

Use Glue Catalog as Hive meta store.

Q. Choose Data source?

Data source has to be crawled and schema table should be in Glue Catalog DB.

Q. Choose data target?

Create tables in data target.

Use tables in the data Catalog and update your data target.

Data store: S3, JDBC (select connection)

Python, PySpark, Cloud, Unix, SQL

Search This Blog

555_AWS_ETL_Glue_Project

Python Branching

170 Python Other topics

162 Python Class

161 Python Errors & Exceptions

160 Python Dates

import time
print(datetime.datetime.now())
time.sleep(5) #delay by 5sec
print(datetime.datetime.now())
O/p:
2021-04-07 20:01:56.065809
2021-04-07 20:02:01.067447

159 Python File Handling

158 Python Regular Expressions

157 Python Functions

# Default arguments
def keyword_args (name1, name2='b'):
"This prints a passed info into this function"
print ("Name1: ", name1)
print ("NAme2: ", name2)
return;

# Call function
keyword_args (name1='a')

155 Python Loops

5.3 Nested loops

Nested loops is “one loop inside another loop”.

Python Decision making

IF Else Statement:

153 Python variables

3. Variables, Dynamic Typing, Dynamic binding

Dynamic Binding or Late Binding

152 Python Data Types

1) Numeric types:

set creation from list:

599_AWS Interview Questions

Total Pageviews

Followers

Search This Blog

import timeprint(datetime.datetime.now())time.sleep(5) #delay by 5secprint(datetime.datetime.now())O/p:2021-04-07 20:01:56.0658092021-04-07 20:02:01.067447

# Default argumentsdef keyword_args (name1, name2='b'): "This prints a passed info into this function" print ("Name1: ", name1) print ("NAme2: ", name2) return;# Call functionkeyword_args (name1='a')

5.3 Nested loops

Nested loops is “one loop inside another loop”.

IF Else Statement:

3. Variables, Dynamic Typing, Dynamic binding

Dynamic Binding or Late Binding

1) Numeric types:

set creation from list:

Total Pageviews

Followers

import time
print(datetime.datetime.now())
time.sleep(5) #delay by 5sec
print(datetime.datetime.now())
O/p:
2021-04-07 20:01:56.065809
2021-04-07 20:02:01.067447

# Default arguments
def keyword_args (name1, name2='b'):
"This prints a passed info into this function"
print ("Name1: ", name1)
print ("NAme2: ", name2)
return;

# Call function
keyword_args (name1='a')