Python magic gets


__getattribute__ and __getattr__ and __get__ and __getitem__

Lets get into these many gets.


if the lookup attribute does not available in instance’s __dict__, then it will call __getattr__, still if it does not get, raises an AttributeError

The below example will make you clear(in python 2.x)

In [5]: class Test(object):
   ...:     def __init__(self):
   ...:         self.a = 1
   ...:         self.b = 2
   ...:     def __getattr__(self, att):
   ...:         print self.__dict__
   ...:         print "Calling getattr"
   ...:         return 0

In [6]: t = Test()

In [7]: t.__dict__
Out[7]: {'a': 1, 'b': 2}

In [8]: t.a
Out[8]: 1

In [9]: t.c
{'a': 1, 'b': 2}
Calling getattr
Out[9]: 0

In [10]: t.b
Out[10]: 2


If you try to access any data with square bracket of the instance, directly it calls __getitem__ method

This below one is one of the beautiful example which is taken from some source(in python 2.x):

In [4]: class Test(object):
   ...:     def __getitem__(self, items):
   ...:         print '%-15s  %s' % (type(items), items)
   ...: t = Test()
   ...: t[1]
   ...: t['hello world']
   ...: t[1, 'b', 3.0]
   ...: t[5:200:10]
   ...: t['a':'z':3]
   ...: t[object()]

<type 'int'>     1
<type 'str'>     hello world
<type 'tuple'>   (1, 'b', 3.0)
<type 'slice'>   slice(5, 200, 10)
<type 'slice'>   slice('a', 'z', 3)
<type 'object'>  <object object at 0x7f9222469e20>


getter, setter methods are important in oops world to encapsulate the data through these methods. Python provides same accessibility via property. So, the method-attribute can behave as a data-attribute to the instance via property. This is one of the powerful feature to control the data value without much change to the outer world.

So, In the below example, the temperature is just an attribute to the outer world for the class Celsius and with help of property concept, the business logic is encapsulated over temperature without any interference to the outer world

class Celsius:
    def __init__(self, temperature = 0):
        self._temperature = temperature

    def to_fahrenheit(self):
        return (self.temperature * 1.8) + 32

    def temperature(self):
        print("Getting value")
        return self._temperature

    def temperature(self, value):
        if value < -273:
            raise ValueError("Temperature below -273 is not possible")
        print("Setting value")
        self._temperature = value
In [117]: c = Celsius()

In [118]: c.temperature
Getting value
Out[118]: 0

In [119]: c.temperature = -300
ValueError                                Traceback (most recent call last)
<ipython-input-119-6dba9f781849> in <module>()
----> 1 c.temperature = -300

<ipython-input-116-5086c5ea8861> in temperature(self, value)
     14     def temperature(self, value):
     15         if value < -273:
---> 16             raise ValueError("Temperature below -273 is not possible")
     17         print("Setting value")
     18         self._temperature = value

ValueError: Temperature below -273 is not possible

In [120]: c.temperature = 30
Setting value

In [121]: c.temperature
Getting value
Out[121]: 30


This method will get called always, if you try to access any attribute, if it could not find then it will throw AttributeError, then it does call to __getattr__ method. So, if you want to keep any security access to any type of attribute this is the right method to cover it up.

Nice example in python 3.x:

In [106]: class Count:
     ...:     def __init__(self,mymin,mymax):
     ...:         self.mymin=mymin
     ...:         self.mymax=mymax
     ...:         self.current=None
     ...:     def __getattr__(self, item):
     ...:             self.__dict__[item]=0
     ...:             return 0
     ...:     def __getattribute__(self, item):
     ...:         print("calling", item)
     ...:         if item.startswith('cur'):
     ...:             raise AttributeError
     ...:         return object.__getattribute__(self,item)
     ...:         # or you can use ---return super().__getattribute__(item)
     ...: obj1 = Count(1,10)
     ...: print(obj1.mymin)
     ...: print(obj1.mymax)
     ...: print(obj1.current)
calling mymin
calling mymax
calling current
calling __dict__

It is very overview about these magic methods in python. Still, lot more to explore to get nuance details about usage.


Posted in 10min, Python | Tagged , | Leave a comment

Class variable in python


class A(object):
    inst = 1

inst is the class variable

Is class variable shared across the instances?

Yes. but no.

Usually, in object oriented languages like c++, java, the class variables are shared across instances. But here something different.

Watch the below examples and output

class variables are shared across instances in some sense.

In [70]: class A(object):
    ...:     inst = 1

In [71]: 

In [71]: a = A()

In [72]: b = A()

In [73]: a.inst
Out[73]: 1

In [74]: b.inst
Out[74]: 1

In [75]: A
Out[75]: __main__.A

In [76]: A.inst = 22

In [77]: a.inst
Out[77]: 22

In [78]: b.inst
Out[78]: 22

in the above sample, the change A.inst = 22 reflect across all instances a,b.

But What happens here?

In [79]: class A(object):
    ...:     inst = 1

In [80]: a = A()

In [81]: b = A()

In [82]: a.inst = 3     # Notice here this is the culprit

In [83]: b.inst
Out[83]: 1

In [84]: A.inst = 5     # Change in class variable

In [85]: b.inst
Out[85]: 5

In [86]: a.inst
Out[86]: 3  # Why this did not get update?

line no: 82 is the important change. Our usual understanding is, the class variable can be accessed via instance or class name. So, we can change the class varialbe via instance also. This will update entirely. It is absolutely wrong. Since, when you change the class variable’s value by instance, the scope gets changed.

a.inst = 3 means, one instance specific attribute is created dynamically which is reserved for this particular instance. So, a.inst is not exactly looking for class variable if instance attribute existing.

Then, How can we access the class variable?

just try it like below


you should be getting latest class variable value which is 5 in our case.

This lack of understanding may cost one opportunity. Lets be aware.

Posted in Academic, General, Python | Leave a comment

Understand __mro__

What is mro?

method resoultion order is an algorithm to choose the right method to get executed from inherited base classes.

Where it is used?

mro will be interesting if more than one base class is used, the method you are calling is defined in more than one base class also.

please, watch every step of the below ipython-ed example


In [29]: class A(object):
    ...:     x = 10

In [30]: class B(A):
    ...:     pass

In [31]: class C(A):
    ...:     x = 15

In [32]: class D(B, C):
    ...:     pass

In [33]: d = D()

In [34]: D.__mro__
Out[34]: (__main__.D, __main__.B, __main__.C, __main__.A, object)

In [35]: d.x
Out[35]: 15   # Output of the new-style mro

In [36]: class A:
    ...:     x = 10

In [37]: class B(A):
    ...:     pass

In [38]: class C(A):
    ...:     x = 15

In [39]: class D(B, C):
    ...:     pass

In [40]: d = D()

In [41]: D.__mro__
AttributeError                            Traceback (most recent call last)
<ipython-input-41-c7e45d56683f> in <module>()
----> 1 D.__mro__

AttributeError: class D has no attribute '__mro__'

In [42]: d.x
Out[42]: 10  # output of the legacy style mro

Explanation for the above example:

class A, B, C is the usual to understand simple classes inherited from single parent class. (till ipython line #6)

After that, we started inheriting class from multiple base classes which is class D.

New style class:

If the base class is derived from object class means, it is called new-style classes.

Old-style class:

If the base class is not derived from object class, The topmost base class will be type class. It is used in legacy python classes.

the __mro__ is defined in new-style python classes. So, it helps to understand the execution order of the methods in all self and base classes.

In ipython line# 35 is output of new style class’s mro algorithm, also, the order of the execution is also printed in the next line D, B, C, A, object, similar to the breadth first approach

ipython line# 42 is the output of legacy-style class’s mro algorithm, similar to the depth-first approach, it doesn’t have __mro__ magic method to display the execution orders.

If you are an Appetite,

read the source paper of deriving this algorithm from old style to new style approach


Posted in 10min, Python | Tagged , | Leave a comment

import struct – a glance

What is struct in python?

struct is the python module which translates binary string to python types and vice versa. if you have any binary string which comes from network stream or anywhere can be directly translated to python types.

Where it can be used?

Network streaming applications, where it supposed to receive lots of streaming binary content, it can be processed in python with help of this module.

What are the rules?

  1. It has two familiar methods. pack, unpack
  2. It takes data format as one of the input field along with the target data.
  3. Some notables

struct.pack(format_string, data)

struct.unpack(format_string, data)

format_string is used to parse the incoming data into a given format. There are various data formats are possible based on the cpu architecture of the machine(endian-ness) endian.

for example, s represents single char, f – floating point number, I – unsigned integer. There are the codes are used in format identifier. formats


In [88]: py_data = (1, 'navaneethan', 3.14)

In [89]: py_data
Out[89]: (1, 'navaneethan', 3.14)

In [90]: import struct

In [91]: struct.pack('I 11s f', *py_data)
Out[91]: '\x01\x00\x00\x00navaneethan\x00\xc3\xf5H@'

In [92]: hex_rep = struct.pack('I 11s f', *py_data)

In [93]: import binascii

In [94]: binascii.hexlify(hex_rep)
Out[94]: '010000006e6176616e65657468616e00c3f54840'

In [95]: bin_rep = binascii.hexlify(hex_rep)

In [96]: binascii.unhexlify(bin_rep)
Out[96]: '\x01\x00\x00\x00navaneethan\x00\xc3\xf5H@'

In [97]: struct.unpack('I 11s f', hex_rep)
Out[97]: (1, 'navaneethan', 3.140000104904175)

In [103]: struct.unpack('@I 11s f', hex_rep)
Out[103]: (1, 'navaneethan', 3.140000104904175)
# `@` represents native

In [102]: struct.unpack('<I 11s f', hex_rep)
error                                     Traceback (most recent call last)
<ipython-input-102-94050abdb5f6> in <module>()
----> 1 struct.unpack('<I 11s f', hex_rep)

# `<` represents little-endian format, so it failed 

format_string in the above example, I 11s f - ( 1 Unsigned Integer, 11 char string, 1 float value)

Still, hungry?


Posted in 10min, General, Python | Tagged , , | Leave a comment

Is FB double equals to the Shakuni of Mahabharata?

The intention of this article is comparing two completely difference scenarios

  1. The Mahabharatha story
  2. Facebook net-neutrality story.

Short Narration of The Great Mahabharata.

By viewing the Mahabharata in top angle, we have few players,.

The Sri Krishna & team(+pandavas), The Dhritarashtra & his sons (of-course, mainly Duriyothana, Dusshasana), and One important protagonist is(we need big introduction music here) Shakuni.

These three team has three different perspectives on each other.

What is the Goal of each team

Dhritarashtra’s & Co

The Dhritarashtra’s team want to have the control and long kingdom ship ever. They are not driven by themselves. They are being driven by sometimes positive force(Bhisma, Durona, Vidura), sometimes negative force (Saghuni’s mentorship).

Sri Krishna & Co

The Sri Krishna & team has the simple target of giving the freedom to live for the people. They want to give freedom and happiness to people in the sake of ruling the kingdom. They have the equal rights to share the kingdom

The one and only Shakuni

The Shakuni is the person who could not rule the kingdom authoritatively, want to have the control of the kingdom in his hands, Simply to say, each and every move should be driven from his thoughts.So, Actually he want to sit on the throne, king should be his chariot, so that king will drive his chariot horse based on his direction point.

End Scene

This is the small out view of entire Mahabharata. These mixed teams various thoughts finally ended up with more pain and damage to the innocent people though the krukshetra war.

Now lets come to the current drama which happens here what Facebook wants from India?

Similarly, we have few players here too.

Of course, Important team is Facebook&co, Our Indian Government, All other Digital vendors(like Google, LinkedIn, Coursera etc..)

Now you might have related these three people with the previous story. Lets go through little deep manner.

These teams has three different objective from each other.

The Indian Govt & Co

Similar to the Dhritarashtra’s objective.
They want to rule the nation for the very long period. Sometimes they are driven by positive forces(activist, people etc..), sometimes they are driven by negative forces(The people who corrupts the govt to inject their business model).

The Digital Vendors & Co

I would say they are the decent digital vendors. Since, they are making their business out of their value of the product which reaches to the fellow people. Take example of Google Search, Coursera courses, Gitlab codes, News magazines etc..
Their competition is in healthy manner. They don’t want to imperialise others illegible way.Yeah Ofcourse! They are decent.

Next is…(again needs aloud music here) The Facebook:

It has some value which is being given to the people. It has some valuable stock options which is higher than few other digital vendors. It is lesser than the top value creators(like Google). Now The Facebook wants to sit on the throne, so the Govt can be on it’s chariot driver, so that it can direct the Govt to rule the entire community. Catch the business

“Absolutely, there is NO much wrong on these various minds as long as they are not harming others illegally, especially not harming the fellow citizens.”

But the Imperialism happens,

Similar to Mahabharatha

How Shakuni plotted a plan for the fall of pandavas illegally, hypnotizing the Dhritarashtra’s mind and started driving him, injected his venomous action (by falling apart Dharma, keeping the Adharma), involved in false actions to reach his goal.

Why Facebook is compared with Shakuni?

Facebook creates the wrong policy statement by making it’s users supporting it’s policy without explaining the proper context to them. (injecting venomous view by closing the eyes of the customer)

Trying to make the fall apart the other valuable digital vendors(if Facebook starts driving the data carrier freely for it’s revenue growth, what would happen to other digital vendors. Facebook is trying to make other digital vendors to pay the corruption fee, so that they could drive their business as they want to. If not, simply Facebook will take over their business.)

Where is Dharma? (Facebook want to change the policy for their lust, simply closing the other eye, not to see the repercussions which would happen for the people.)

Further read

We don’t want to have the similar end scene of Mahabharata even here too. Hence, I am finishing up this article by deriving

Facebook == The Shakuni of Mahabharatha.

Gratitude for your time.

Further links:

Posted in General, General_Tech, Problems | Tagged , , , | Leave a comment

Avoid duplicate process copy through cron


You are executing a background job for every x minuts periodically.The job execution time takes approximately y mins.Sometimes y > x. So, you are endup with multiple duplicate active process running at back.


If duplicate process runs at back, it creates multiple side effect. The one is you would surprise to see the duplicate record in your table.

So, We can write a shell script which should check whether any active same process running or not? If not, spawn one.


This trick might help to you.


status=`ps -efww | grep -v grep |grep -w "python send_email" | awk -vpid=$$ '$2 != pid { print $2 }'`
echo $status
if [ ! -z "$status" ]; then
    echo "Process already running"
    cd /home/edge/projects/venv
    source bin/activate

    cd /home/edge/projects/taleo
    python send_email

    echo "[`date`] New Process Spawned"

my process python send_email will be grepped before it gets spawned.

your suggestions are welcome 🙂

Posted in 10min, Flask, Python | Tagged , , | 2 Comments

Same Log messages multiple times in Python [issue]

The Mistake happened in my python loggers

what ?

I use Python/Flask framework to develop web applications. So, For my few usecase scripts, I have to use the logger to log the flow.I have created a common handler which workes nicely as expected.I have defined the logger in seperate module called utils/ From this module I used to import to my script to log the flow.

import logging
import logging.handlers

def get_logger(name, file_name=None):
    LOG_FILENAME = file_name if file_name else name + '.log'
    # Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    # Add the log message handler to the logger
    handler = logging.handlers.RotatingFileHandler(LOG_FILENAME,
    formatter = logging.Formatter("[%(asctime)s] [%(filename)s:%(lineno)s - %(funcName)5s() - %(processName)s] %(levelname)s - %(message)s")
    return my_logger

When I import this logger object to multiple files, log statements getting printed multiple times as many imports i did.

Stackoverflow saved my time.

The reason is Each call(log = get_logger('taleo_integration')) is creating a new log handler for the logger you create as given below.

What is under the hood ?

In [13]: from hirenew.utils import get_logger

In [14]: log.handlers
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>]

In [15]: log = get_logger('taleo_integration')

In [16]: log.handlers
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>]

In [17]: log = get_logger('taleo_integration')

In [18]: log.handlers
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>,
 <logging.handlers.RotatingFileHandler at 0x5ff0810>]


Finally, I added couple of lines in get_logger.

# Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    if not my_logger.handlers:
         # Here I created handler, formatter, loglevel etc..
    return my_logger

This works fine.


Posted in Flask, Python | Tagged , | 2 Comments

Weird thing to understand from mongoengine in Flask

Inital mongo collection

I am creation a mongo dument in Flask using MongoEngine Object mapper.Simple, consider the collection name is TestMe and it has two fields (name, address).
The name of the document is nava.

from mongoengine import Document, connect, StringField


class TestMe(Document):
    name = StringField(required=True)
    address = StringField(required=True)

Adding a new row

I am adding one row document in the collection using this mongoengine class which is described below.

 # 1. Initial document creation in mongo collections with only two fields(name and address)                                                        
    t = TestMe(name="Navaneethan", address="Bangalore")

So, the document is added successfully.

Verify in your mongo client

For quick confirmation, We can verify the document in mongod client.

(hire)navaneethan@navaneethLap:~/workspace/experiments/python/flask$ mongo
MongoDB shell version: 3.0.4
connecting to: test
Server has startup warnings: 
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] 
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] 
> show dbs;
employee  0.078GB
hirenew   0.078GB
local     0.078GB
nava      0.078GB
> use nava;
switched to db nava
> db.t
db.test_me          db.toLocaleString(  db.toString(        db.tojson(
> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore" }

Adding a new field in the existing collection

Now I need a field has to be added in the existing db nava, which is is_active a booleanfield.
So, the refined model is,

class TestMe(Document):
    name = StringField(required=True)
    address = StringField(required=True)
    is_active = BooleanField(default=False)

To my current scenario, I have to update the field to the selected objects which would have the default value, but it should be reflected in db.

Updating the default value to the exisiting rows for the new field

in my python shell

obj = TestMe.objects(name='Navaneethan').first()
obj.is_active = False
print "updated successfully"

I am retrieving all the fields from the db and updating to the default value.The save happens successfully.

Verifying in the mongod client

But what surprising me is,
Unable to see the new field in mongo db.

switched to db nava
> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore" }


Here I scrateched my head for almost 2 hours.

Changing the default value to some new value(false –> true)

After somepoint, tried to update the field with different value(not the default one)

obj = TestMe.objects(name='Navaneethan').first()
obj.is_active = True  # note here the default value of the field is changed
print "updated successfully"


Verifying in the mongod client

The field is updated to the existing rows.

> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore", "is_active" : true }

my mind before hitting the head against wall:

It seems little weird to me. Since it is a nosql mapper or what? I come from django experience, I expected the field has to be updated even the default value is given to be updated for the existing rows. But it doesn’t happen. So, what happens is, If the value of the new field changed from the default value(`false` here) to the some other value(`true`) then the update reflects successfully.

There is a solution for every problem.

Yes. This is an obvious statement which I realized after a while hitting my head against the wall.

if you inherit from `DynamicDocument` for mongoengine odm. You will get the clear solution. It does the job.

The learn happens after this post gets published :)

Thanks for your time.

Posted in Flask, Python | Tagged , | Leave a comment

Erlang maps basics

erl> maps: for python>>>dict() people

just watch and think series:

How to create a new map in erlang?

 47> MyNewMap = maps:new().
 48> MyNewMap
 48> .
 49> maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 50> MyNewMap.
 51> MyNewMap2 = maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 52> MyNewMap2.
 #{name => "Navaneethan"}
 53> MyNewMap.
 54> maps:put(age, 26, MyNewMap2).
 #{age => 26,name => "Navaneethan"}
 55> MyNewMap2.
 #{name => "Navaneethan"}
 56> % RETURNS NEW MAP. Not updating into old one.%

Part2 in Maps(remove, put, update, without,values,to_list ):

 56> maps:remove('name',MyNewMap).
 57> maps:remove(name,MyNewMap).
 58> MyNewMap.
 59> maps:remove(name,MyNewMap2).
 60> MyNewMap2.
 #{name => "Navaneethan"}
 61> MyNewMap2 = maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 62> MyNewMap2 = maps:put(age, "26", MyNewMap).
 ** exception error: no match of right hand side value #{age => "26"}
 63> MyNewMap2.
 #{name => "Navaneethan"}
 64> f(MyNewMap2).
 65> MyNewMap2 = maps:put(age, "26", MyNewMap).
 #{age => "26"}
 66> MyNewMap2.
 #{age => "26"}
 67> MyNewMap2 = maps:put(work, "Python", MyNewMap2).
 ** exception error: no match of right hand side value #{age => "26",work => "Python"}
 68> MyNewMap3 = maps:put(work, "Python", MyNewMap2).
 #{age => "26",work => "Python"}
 69> maps:to_list(MyNewMap3).
 70> maps:update(work,"erlang",MyNewMap3).
 #{age => "26",work => "erlang"}
 71> MyNewMap3.
 #{age => "26",work => "Python"}
 72> %Update Returns New one.%
 72> maps:values(MyNewMap3).
 73> ReqKeys = [name, 'book', age, 'age'].
 74> maps:with(ReqKeys, MyNewMap3).
 ** exception error: undefined function maps:with/2
 75> MyNewMap3.
 #{age => "26",work => "Python"}
 76> maps:with([name], MyNewMap3).
 ** exception error: undefined function maps:with/2
 77> maps:with(ReqKeys, MyNewMap3).
 ** exception error: undefined function maps:with/2
 78> maps:without(ReqKeys, MyNewMap3).
 #{work => "Python"}

Points to be noticed:

This is erlang map which is similar to the python dictionary.Here we do some mandate operations with map.

If you pass your map to in built function in maps module, it returns new map which is unbounded.(line:62 & 67)

Rest of the things are similar like other programming languages.

Posted in erlang | Tagged | Leave a comment

Erlang day1

Erlang day1:

  1. Erlang shell an introduction
  2. How to use the erlang shell?
  3. few basic data structure tricks

Erlang shell an introduction:

In order to get your erlang shell,you have to install the erlang OTP for your platform(Windows/Linux).
Once installed erlang software, you can type erl to launch the shell.
In erl you can do the experiments in erlang.

watch the screen & read the below marks:

erl day 1

Few things keep remember:

The new values can’t be assigned to the bounded varialbes.Use always a unbound variable to store the value.Varialbes starts with caps(i.e Var).
_ is always used to unregister the value.The value will be omitted if it is assigned to the underscore.

Data structures are atoms, tuple, list.

  • tuple: It is represented by {}. e.g {1,2,3,4}.
  • atom: This is just a variable which is used name the data to match the pattern.
    eg.  {person, Name} = {person, "navaneethan"}.

    in this above example two tuples are matched with help of atom person, so the variable Name will be bounded to the value navaneethan.Finally, this pattern match is successful.

  • list: []
    e.g: [1,2,3,4].

Few weird things with tuples and lists:

  • Two tuples are compared with the no of elements on each tuple.
  • Two lists are compared with Dictionary associate property.
Posted in erlang | Tagged | 1 Comment