Python magic gets

 

__getattribute__ and __getattr__ and __get__ and __getitem__

Lets get into these many gets.

__getattr__

if the lookup attribute does not available in instance’s __dict__, then it will call __getattr__, still if it does not get, raises an AttributeError

The below example will make you clear(in python 2.x)

In [5]: class Test(object):
   ...:     def __init__(self):
   ...:         self.a = 1
   ...:         self.b = 2
   ...:     def __getattr__(self, att):
   ...:         print self.__dict__
   ...:         print "Calling getattr"
   ...:         return 0
   ...:     

In [6]: t = Test()

In [7]: t.__dict__
Out[7]: {'a': 1, 'b': 2}

In [8]: t.a
Out[8]: 1

In [9]: t.c
{'a': 1, 'b': 2}
Calling getattr
Out[9]: 0

In [10]: t.b
Out[10]: 2

__getitem__

If you try to access any data with square bracket of the instance, directly it calls __getitem__ method

This below one is one of the beautiful example which is taken from some source(in python 2.x):

In [4]: class Test(object):
   ...:     def __getitem__(self, items):
   ...:         print '%-15s  %s' % (type(items), items)
   ...: 
   ...: t = Test()
   ...: t[1]
   ...: t['hello world']
   ...: t[1, 'b', 3.0]
   ...: t[5:200:10]
   ...: t['a':'z':3]
   ...: t[object()]

<type 'int'>     1
<type 'str'>     hello world
<type 'tuple'>   (1, 'b', 3.0)
<type 'slice'>   slice(5, 200, 10)
<type 'slice'>   slice('a', 'z', 3)
<type 'object'>  <object object at 0x7f9222469e20>

__get__

getter, setter methods are important in oops world to encapsulate the data through these methods. Python provides same accessibility via property. So, the method-attribute can behave as a data-attribute to the instance via property. This is one of the powerful feature to control the data value without much change to the outer world.

So, In the below example, the temperature is just an attribute to the outer world for the class Celsius and with help of property concept, the business logic is encapsulated over temperature without any interference to the outer world

class Celsius:
    def __init__(self, temperature = 0):
        self._temperature = temperature

    def to_fahrenheit(self):
        return (self.temperature * 1.8) + 32

    @property
    def temperature(self):
        print("Getting value")
        return self._temperature

    @temperature.setter
    def temperature(self, value):
        if value < -273:
            raise ValueError("Temperature below -273 is not possible")
        print("Setting value")
        self._temperature = value
        
In [117]: c = Celsius()

In [118]: c.temperature
Getting value
Out[118]: 0

In [119]: c.temperature = -300
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-119-6dba9f781849> in <module>()
----> 1 c.temperature = -300

<ipython-input-116-5086c5ea8861> in temperature(self, value)
     14     def temperature(self, value):
     15         if value < -273:
---> 16             raise ValueError("Temperature below -273 is not possible")
     17         print("Setting value")
     18         self._temperature = value

ValueError: Temperature below -273 is not possible

In [120]: c.temperature = 30
Setting value

In [121]: c.temperature
Getting value
Out[121]: 30

__getattribute__

This method will get called always, if you try to access any attribute, if it could not find then it will throw AttributeError, then it does call to __getattr__ method. So, if you want to keep any security access to any type of attribute this is the right method to cover it up.

Nice example in python 3.x:

In [106]: class Count:
     ...: 
     ...:     def __init__(self,mymin,mymax):
     ...:         self.mymin=mymin
     ...:         self.mymax=mymax
     ...:         self.current=None
     ...: 
     ...:     def __getattr__(self, item):
     ...:             self.__dict__[item]=0
     ...:             return 0
     ...: 
     ...:     def __getattribute__(self, item):
     ...:         print("calling", item)
     ...:         if item.startswith('cur'):
     ...:             raise AttributeError
     ...:         return object.__getattribute__(self,item)
     ...:         # or you can use ---return super().__getattribute__(item)
     ...: 
     ...: obj1 = Count(1,10)
     ...: print(obj1.mymin)
     ...: print(obj1.mymax)
     ...: print(obj1.current)
     ...: 
calling mymin
1
calling mymax
10
calling current
calling __dict__
0

It is very overview about these magic methods in python. Still, lot more to explore to get nuance details about usage.

 

Advertisements
Posted in 10min, Python | Tagged , | Leave a comment

Class variable in python

What?

class A(object):
    inst = 1

inst is the class variable

Is class variable shared across the instances?

Yes. but no.

Usually, in object oriented languages like c++, java, the class variables are shared across instances. But here something different.

Watch the below examples and output

class variables are shared across instances in some sense.

In [70]: class A(object):
    ...:     inst = 1
    ...:     

In [71]: 

In [71]: a = A()

In [72]: b = A()

In [73]: a.inst
Out[73]: 1

In [74]: b.inst
Out[74]: 1

In [75]: A
Out[75]: __main__.A

In [76]: A.inst = 22

In [77]: a.inst
Out[77]: 22

In [78]: b.inst
Out[78]: 22

in the above sample, the change A.inst = 22 reflect across all instances a,b.

But What happens here?

In [79]: class A(object):
    ...:     inst = 1
    ...:     

In [80]: a = A()

In [81]: b = A()

In [82]: a.inst = 3     # Notice here this is the culprit

In [83]: b.inst
Out[83]: 1

In [84]: A.inst = 5     # Change in class variable

In [85]: b.inst
Out[85]: 5

In [86]: a.inst
Out[86]: 3  # Why this did not get update?

line no: 82 is the important change. Our usual understanding is, the class variable can be accessed via instance or class name. So, we can change the class varialbe via instance also. This will update entirely. It is absolutely wrong. Since, when you change the class variable’s value by instance, the scope gets changed.

a.inst = 3 means, one instance specific attribute is created dynamically which is reserved for this particular instance. So, a.inst is not exactly looking for class variable if instance attribute existing.

Then, How can we access the class variable?

just try it like below

a.__class__.inst

you should be getting latest class variable value which is 5 in our case.

This lack of understanding may cost one opportunity. Lets be aware.

Posted in Academic, General, Python | Leave a comment

Understand __mro__

python-mro.md

What is mro?

method resoultion order is an algorithm to choose the right method to get executed from inherited base classes.

Where it is used?

mro will be interesting if more than one base class is used, the method you are calling is defined in more than one base class also.

please, watch every step of the below ipython-ed example

Example?

In [29]: class A(object):
    ...:     x = 10
    ...:     

In [30]: class B(A):
    ...:     pass
    ...: 

In [31]: class C(A):
    ...:     x = 15
    ...:     

In [32]: class D(B, C):
    ...:     pass
    ...: 

In [33]: d = D()

In [34]: D.__mro__
Out[34]: (__main__.D, __main__.B, __main__.C, __main__.A, object)

In [35]: d.x
Out[35]: 15   # Output of the new-style mro

In [36]: class A:
    ...:     x = 10
    ...:     

In [37]: class B(A):
    ...:     pass
    ...: 

In [38]: class C(A):
    ...:     x = 15
    ...:     

In [39]: class D(B, C):
    ...:     pass
    ...: 

In [40]: d = D()

In [41]: D.__mro__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-c7e45d56683f> in <module>()
----> 1 D.__mro__

AttributeError: class D has no attribute '__mro__'

In [42]: d.x
Out[42]: 10  # output of the legacy style mro

Explanation for the above example:

class A, B, C is the usual to understand simple classes inherited from single parent class. (till ipython line #6)

After that, we started inheriting class from multiple base classes which is class D.

New style class:

If the base class is derived from object class means, it is called new-style classes.

Old-style class:

If the base class is not derived from object class, The topmost base class will be type class. It is used in legacy python classes.

the __mro__ is defined in new-style python classes. So, it helps to understand the execution order of the methods in all self and base classes.

In ipython line# 35 is output of new style class’s mro algorithm, also, the order of the execution is also printed in the next line D, B, C, A, object, similar to the breadth first approach

ipython line# 42 is the output of legacy-style class’s mro algorithm, similar to the depth-first approach, it doesn’t have __mro__ magic method to display the execution orders.

If you are an Appetite,

read the source paper of deriving this algorithm from old style to new style approach
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.3910&rep=rep1&type=pdf

 

Posted in 10min, Python | Tagged , | Leave a comment

import struct – a glance

What is struct in python?

struct is the python module which translates binary string to python types and vice versa. if you have any binary string which comes from network stream or anywhere can be directly translated to python types.

Where it can be used?

Network streaming applications, where it supposed to receive lots of streaming binary content, it can be processed in python with help of this module.

What are the rules?

  1. It has two familiar methods. pack, unpack
  2. It takes data format as one of the input field along with the target data.
  3. Some notables

struct.pack(format_string, data)

struct.unpack(format_string, data)

format_string is used to parse the incoming data into a given format. There are various data formats are possible based on the cpu architecture of the machine(endian-ness) endian.

for example, s represents single char, f – floating point number, I – unsigned integer. There are the codes are used in format identifier. formats

Example?

In [88]: py_data = (1, 'navaneethan', 3.14)

In [89]: py_data
Out[89]: (1, 'navaneethan', 3.14)

In [90]: import struct

In [91]: struct.pack('I 11s f', *py_data)
Out[91]: '\x01\x00\x00\x00navaneethan\x00\xc3\xf5H@'

In [92]: hex_rep = struct.pack('I 11s f', *py_data)

In [93]: import binascii

In [94]: binascii.hexlify(hex_rep)
Out[94]: '010000006e6176616e65657468616e00c3f54840'

In [95]: bin_rep = binascii.hexlify(hex_rep)

In [96]: binascii.unhexlify(bin_rep)
Out[96]: '\x01\x00\x00\x00navaneethan\x00\xc3\xf5H@'

In [97]: struct.unpack('I 11s f', hex_rep)
Out[97]: (1, 'navaneethan', 3.140000104904175)

In [103]: struct.unpack('@I 11s f', hex_rep)
Out[103]: (1, 'navaneethan', 3.140000104904175)
# `@` represents native

In [102]: struct.unpack('<I 11s f', hex_rep)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-102-94050abdb5f6> in <module>()
----> 1 struct.unpack('<I 11s f', hex_rep)

# `<` represents little-endian format, so it failed 

format_string in the above example, I 11s f - ( 1 Unsigned Integer, 11 char string, 1 float value)

Still, hungry?

https://docs.python.org/3/library/struct.html

https://pymotw.com/2/struct/

 

Posted in 10min, General, Python | Tagged , , | Leave a comment

Is FB double equals to the Shakuni of Mahabharata?

The intention of this article is comparing two completely difference scenarios

  1. The Mahabharatha story
  2. Facebook net-neutrality story.

Short Narration of The Great Mahabharata.

By viewing the Mahabharata in top angle, we have few players,.

The Sri Krishna & team(+pandavas), The Dhritarashtra & his sons (of-course, mainly Duriyothana, Dusshasana), and One important protagonist is(we need big introduction music here) Shakuni.

These three team has three different perspectives on each other.

What is the Goal of each team

Dhritarashtra’s & Co

The Dhritarashtra’s team want to have the control and long kingdom ship ever. They are not driven by themselves. They are being driven by sometimes positive force(Bhisma, Durona, Vidura), sometimes negative force (Saghuni’s mentorship).

Sri Krishna & Co

The Sri Krishna & team has the simple target of giving the freedom to live for the people. They want to give freedom and happiness to people in the sake of ruling the kingdom. They have the equal rights to share the kingdom

The one and only Shakuni

The Shakuni is the person who could not rule the kingdom authoritatively, want to have the control of the kingdom in his hands, Simply to say, each and every move should be driven from his thoughts.So, Actually he want to sit on the throne, king should be his chariot, so that king will drive his chariot horse based on his direction point.

End Scene

This is the small out view of entire Mahabharata. These mixed teams various thoughts finally ended up with more pain and damage to the innocent people though the krukshetra war.

Now lets come to the current drama which happens here what Facebook wants from India?

Similarly, we have few players here too.

Of course, Important team is Facebook&co, Our Indian Government, All other Digital vendors(like Google, LinkedIn, Coursera etc..)

Now you might have related these three people with the previous story. Lets go through little deep manner.

These teams has three different objective from each other.

The Indian Govt & Co

Similar to the Dhritarashtra’s objective.
They want to rule the nation for the very long period. Sometimes they are driven by positive forces(activist, people etc..), sometimes they are driven by negative forces(The people who corrupts the govt to inject their business model).

The Digital Vendors & Co

I would say they are the decent digital vendors. Since, they are making their business out of their value of the product which reaches to the fellow people. Take example of Google Search, Coursera courses, Gitlab codes, News magazines etc..
Their competition is in healthy manner. They don’t want to imperialise others illegible way.Yeah Ofcourse! They are decent.

Next is…(again needs aloud music here) The Facebook:

It has some value which is being given to the people. It has some valuable stock options which is higher than few other digital vendors. It is lesser than the top value creators(like Google). Now The Facebook wants to sit on the throne, so the Govt can be on it’s chariot driver, so that it can direct the Govt to rule the entire community. Catch the business

“Absolutely, there is NO much wrong on these various minds as long as they are not harming others illegally, especially not harming the fellow citizens.”

But the Imperialism happens,

Similar to Mahabharatha

How Shakuni plotted a plan for the fall of pandavas illegally, hypnotizing the Dhritarashtra’s mind and started driving him, injected his venomous action (by falling apart Dharma, keeping the Adharma), involved in false actions to reach his goal.

Why Facebook is compared with Shakuni?

Facebook creates the wrong policy statement by making it’s users supporting it’s policy without explaining the proper context to them. (injecting venomous view by closing the eyes of the customer)

Trying to make the fall apart the other valuable digital vendors(if Facebook starts driving the data carrier freely for it’s revenue growth, what would happen to other digital vendors. Facebook is trying to make other digital vendors to pay the corruption fee, so that they could drive their business as they want to. If not, simply Facebook will take over their business.)

Where is Dharma? (Facebook want to change the policy for their lust, simply closing the other eye, not to see the repercussions which would happen for the people.)

Further read

We don’t want to have the similar end scene of Mahabharata even here too. Hence, I am finishing up this article by deriving

Facebook == The Shakuni of Mahabharatha.

Gratitude for your time.

Further links:

https://en.wikipedia.org/wiki/Mahabharata

Posted in General, General_Tech, Problems | Tagged , , , | Leave a comment

Avoid duplicate process copy through cron

Scenario:

You are executing a background job for every x minuts periodically.The job execution time takes approximately y mins.Sometimes y > x. So, you are endup with multiple duplicate active process running at back.

Effets:

If duplicate process runs at back, it creates multiple side effect. The one is you would surprise to see the duplicate record in your table.

So, We can write a shell script which should check whether any active same process running or not? If not, spawn one.

Trick:

This trick might help to you.

#!/bin/bash

status=`ps -efww | grep -v grep |grep -w "python manager.py send_email" | awk -vpid=$$ '$2 != pid { print $2 }'`
echo $status
if [ ! -z "$status" ]; then
    echo "Process already running"
else
    cd /home/edge/projects/venv
    source bin/activate

    cd /home/edge/projects/taleo
    python manager.py send_email

    echo "[`date`] New Process Spawned"
fi

my process python manager.py send_email will be grepped before it gets spawned.

your suggestions are welcome 🙂

Posted in 10min, Flask, Python | Tagged , , | 2 Comments

Same Log messages multiple times in Python [issue]

The Mistake happened in my python loggers

what ?

I use Python/Flask framework to develop web applications. So, For my few usecase scripts, I have to use the logger to log the flow.I have created a common handler which workes nicely as expected.I have defined the logger in seperate module called utils/__init__.py. From this module I used to import to my script to log the flow.

import logging
import logging.handlers

def get_logger(name, file_name=None):
    LOG_FILENAME = file_name if file_name else name + '.log'
    # Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    # Add the log message handler to the logger
    handler = logging.handlers.RotatingFileHandler(LOG_FILENAME,
                                                      maxBytes=10000000,
                                                      backupCount=5,
                                                      )
    formatter = logging.Formatter("[%(asctime)s] [%(filename)s:%(lineno)s - %(funcName)5s() - %(processName)s] %(levelname)s - %(message)s")
    handler.setFormatter(formatter)
    my_logger.addHandler(handler)
    return my_logger

When I import this logger object to multiple files, log statements getting printed multiple times as many imports i did.

Stackoverflow saved my time.

The reason is Each call(log = get_logger('taleo_integration')) is creating a new log handler for the logger you create as given below.

What is under the hood ?

In [13]: from hirenew.utils import get_logger

In [14]: log.handlers
Out[14]: 
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>]

In [15]: log = get_logger('taleo_integration')

In [16]: log.handlers
Out[16]: 
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>]

In [17]: log = get_logger('taleo_integration')

In [18]: log.handlers
Out[18]: 
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>,
 <logging.handlers.RotatingFileHandler at 0x5ff0810>]

Soln

Finally, I added couple of lines in get_logger.

# Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    if not my_logger.handlers:
         # Here I created handler, formatter, loglevel etc..
    return my_logger

This works fine.

 

Posted in Flask, Python | Tagged , | 2 Comments