Understand __mro__

python-mro.md

What is mro?

method resoultion order is an algorithm to choose the right method to get executed from inherited base classes.

Where it is used?

mro will be interesting if more than one base class is used, the method you are calling is defined in more than one base class also.

please, watch every step of the below ipython-ed example

Example?

In [29]: class A(object):
    ...:     x = 10
    ...:     

In [30]: class B(A):
    ...:     pass
    ...: 

In [31]: class C(A):
    ...:     x = 15
    ...:     

In [32]: class D(B, C):
    ...:     pass
    ...: 

In [33]: d = D()

In [34]: D.__mro__
Out[34]: (__main__.D, __main__.B, __main__.C, __main__.A, object)

In [35]: d.x
Out[35]: 15   # Output of the new-style mro

In [36]: class A:
    ...:     x = 10
    ...:     

In [37]: class B(A):
    ...:     pass
    ...: 

In [38]: class C(A):
    ...:     x = 15
    ...:     

In [39]: class D(B, C):
    ...:     pass
    ...: 

In [40]: d = D()

In [41]: D.__mro__
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-41-c7e45d56683f> in <module>()
----> 1 D.__mro__

AttributeError: class D has no attribute '__mro__'

In [42]: d.x
Out[42]: 10  # output of the legacy style mro

Explanation for the above example:

class A, B, C is the usual to understand simple classes inherited from single parent class. (till ipython line #6)

After that, we started inheriting class from multiple base classes which is class D.

New style class:

If the base class is derived from object class means, it is called new-style classes.

Old-style class:

If the base class is not derived from object class, The topmost base class will be type class. It is used in legacy python classes.

the __mro__ is defined in new-style python classes. So, it helps to understand the execution order of the methods in all self and base classes.

In ipython line# 35 is output of new style class’s mro algorithm, also, the order of the execution is also printed in the next line D, B, C, A, object, similar to the breadth first approach

ipython line# 42 is the output of legacy-style class’s mro algorithm, similar to the depth-first approach, it doesn’t have __mro__ magic method to display the execution orders.

If you are an Appetite,

read the source paper of deriving this algorithm from old style to new style approach
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.19.3910&rep=rep1&type=pdf

 

Posted in 10min, Python | Tagged , | Leave a comment

import struct – a glance

What is struct in python?

struct is the python module which translates binary string to python types and vice versa. if you have any binary string which comes from network stream or anywhere can be directly translated to python types.

Where it can be used?

Network streaming applications, where it supposed to receive lots of streaming binary content, it can be processed in python with help of this module.

What are the rules?

  1. It has two familiar methods. pack, unpack
  2. It takes data format as one of the input field along with the target data.
  3. Some notables

struct.pack(format_string, data)

struct.unpack(format_string, data)

format_string is used to parse the incoming data into a given format. There are various data formats are possible based on the cpu architecture of the machine(endian-ness) endian.

for example, s represents single char, f – floating point number, I – unsigned integer. There are the codes are used in format identifier. formats

Example?

In [88]: py_data = (1, 'navaneethan', 3.14)

In [89]: py_data
Out[89]: (1, 'navaneethan', 3.14)

In [90]: import struct

In [91]: struct.pack('I 11s f', *py_data)
Out[91]: '\x01\x00\x00\x00navaneethan\x00\xc3\xf5H@'

In [92]: hex_rep = struct.pack('I 11s f', *py_data)

In [93]: import binascii

In [94]: binascii.hexlify(hex_rep)
Out[94]: '010000006e6176616e65657468616e00c3f54840'

In [95]: bin_rep = binascii.hexlify(hex_rep)

In [96]: binascii.unhexlify(bin_rep)
Out[96]: '\x01\x00\x00\x00navaneethan\x00\xc3\xf5H@'

In [97]: struct.unpack('I 11s f', hex_rep)
Out[97]: (1, 'navaneethan', 3.140000104904175)

In [103]: struct.unpack('@I 11s f', hex_rep)
Out[103]: (1, 'navaneethan', 3.140000104904175)
# `@` represents native

In [102]: struct.unpack('<I 11s f', hex_rep)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
<ipython-input-102-94050abdb5f6> in <module>()
----> 1 struct.unpack('<I 11s f', hex_rep)

# `<` represents little-endian format, so it failed 

format_string in the above example, I 11s f - ( 1 Unsigned Integer, 11 char string, 1 float value)

Still, hungry?

https://docs.python.org/3/library/struct.html

https://pymotw.com/2/struct/

 

Posted in 10min, General, Python | Tagged , , | Leave a comment

Is FB double equals to the Shakuni of Mahabharata?

The intention of this article is comparing two completely difference scenarios

  1. The Mahabharatha story
  2. Facebook net-neutrality story.

Short Narration of The Great Mahabharata.

By viewing the Mahabharata in top angle, we have few players,.

The Sri Krishna & team(+pandavas), The Dhritarashtra & his sons (of-course, mainly Duriyothana, Dusshasana), and One important protagonist is(we need big introduction music here) Shakuni.

These three team has three different perspectives on each other.

What is the Goal of each team

Dhritarashtra’s & Co

The Dhritarashtra’s team want to have the control and long kingdom ship ever. They are not driven by themselves. They are being driven by sometimes positive force(Bhisma, Durona, Vidura), sometimes negative force (Saghuni’s mentorship).

Sri Krishna & Co

The Sri Krishna & team has the simple target of giving the freedom to live for the people. They want to give freedom and happiness to people in the sake of ruling the kingdom. They have the equal rights to share the kingdom

The one and only Shakuni

The Shakuni is the person who could not rule the kingdom authoritatively, want to have the control of the kingdom in his hands, Simply to say, each and every move should be driven from his thoughts.So, Actually he want to sit on the throne, king should be his chariot, so that king will drive his chariot horse based on his direction point.

End Scene

This is the small out view of entire Mahabharata. These mixed teams various thoughts finally ended up with more pain and damage to the innocent people though the krukshetra war.

Now lets come to the current drama which happens here what Facebook wants from India?

Similarly, we have few players here too.

Of course, Important team is Facebook&co, Our Indian Government, All other Digital vendors(like Google, LinkedIn, Coursera etc..)

Now you might have related these three people with the previous story. Lets go through little deep manner.

These teams has three different objective from each other.

The Indian Govt & Co

Similar to the Dhritarashtra’s objective.
They want to rule the nation for the very long period. Sometimes they are driven by positive forces(activist, people etc..), sometimes they are driven by negative forces(The people who corrupts the govt to inject their business model).

The Digital Vendors & Co

I would say they are the decent digital vendors. Since, they are making their business out of their value of the product which reaches to the fellow people. Take example of Google Search, Coursera courses, Gitlab codes, News magazines etc..
Their competition is in healthy manner. They don’t want to imperialise others illegible way.Yeah Ofcourse! They are decent.

Next is…(again needs aloud music here) The Facebook:

It has some value which is being given to the people. It has some valuable stock options which is higher than few other digital vendors. It is lesser than the top value creators(like Google). Now The Facebook wants to sit on the throne, so the Govt can be on it’s chariot driver, so that it can direct the Govt to rule the entire community. Catch the business

“Absolutely, there is NO much wrong on these various minds as long as they are not harming others illegally, especially not harming the fellow citizens.”

But the Imperialism happens,

Similar to Mahabharatha

How Shakuni plotted a plan for the fall of pandavas illegally, hypnotizing the Dhritarashtra’s mind and started driving him, injected his venomous action (by falling apart Dharma, keeping the Adharma), involved in false actions to reach his goal.

Why Facebook is compared with Shakuni?

Facebook creates the wrong policy statement by making it’s users supporting it’s policy without explaining the proper context to them. (injecting venomous view by closing the eyes of the customer)

Trying to make the fall apart the other valuable digital vendors(if Facebook starts driving the data carrier freely for it’s revenue growth, what would happen to other digital vendors. Facebook is trying to make other digital vendors to pay the corruption fee, so that they could drive their business as they want to. If not, simply Facebook will take over their business.)

Where is Dharma? (Facebook want to change the policy for their lust, simply closing the other eye, not to see the repercussions which would happen for the people.)

Further read

We don’t want to have the similar end scene of Mahabharata even here too. Hence, I am finishing up this article by deriving

Facebook == The Shakuni of Mahabharatha.

Gratitude for your time.

Further links:

https://en.wikipedia.org/wiki/Mahabharata

Posted in General, General_Tech, Problems | Tagged , , , | Leave a comment

Avoid duplicate process copy through cron

Scenario:

You are executing a background job for every x minuts periodically.The job execution time takes approximately y mins.Sometimes y > x. So, you are endup with multiple duplicate active process running at back.

Effets:

If duplicate process runs at back, it creates multiple side effect. The one is you would surprise to see the duplicate record in your table.

So, We can write a shell script which should check whether any active same process running or not? If not, spawn one.

Trick:

This trick might help to you.

#!/bin/bash

status=`ps -efww | grep -v grep |grep -w "python manager.py send_email" | awk -vpid=$$ '$2 != pid { print $2 }'`
echo $status
if [ ! -z "$status" ]; then
    echo "Process already running"
else
    cd /home/edge/projects/venv
    source bin/activate

    cd /home/edge/projects/taleo
    python manager.py send_email

    echo "[`date`] New Process Spawned"
fi

my process python manager.py send_email will be grepped before it gets spawned.

your suggestions are welcome🙂

Posted in 10min, Flask, Python | Tagged , , | 2 Comments

Same Log messages multiple times in Python [issue]

The Mistake happened in my python loggers

what ?

I use Python/Flask framework to develop web applications. So, For my few usecase scripts, I have to use the logger to log the flow.I have created a common handler which workes nicely as expected.I have defined the logger in seperate module called utils/__init__.py. From this module I used to import to my script to log the flow.

import logging
import logging.handlers

def get_logger(name, file_name=None):
    LOG_FILENAME = file_name if file_name else name + '.log'
    # Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    # Add the log message handler to the logger
    handler = logging.handlers.RotatingFileHandler(LOG_FILENAME,
                                                      maxBytes=10000000,
                                                      backupCount=5,
                                                      )
    formatter = logging.Formatter("[%(asctime)s] [%(filename)s:%(lineno)s - %(funcName)5s() - %(processName)s] %(levelname)s - %(message)s")
    handler.setFormatter(formatter)
    my_logger.addHandler(handler)
    return my_logger

When I import this logger object to multiple files, log statements getting printed multiple times as many imports i did.

Stackoverflow saved my time.

The reason is Each call(log = get_logger('taleo_integration')) is creating a new log handler for the logger you create as given below.

What is under the hood ?

In [13]: from hirenew.utils import get_logger

In [14]: log.handlers
Out[14]: 
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>]

In [15]: log = get_logger('taleo_integration')

In [16]: log.handlers
Out[16]: 
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>]

In [17]: log = get_logger('taleo_integration')

In [18]: log.handlers
Out[18]: 
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>,
 <logging.handlers.RotatingFileHandler at 0x5ff0810>]

Soln

Finally, I added couple of lines in get_logger.

# Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    if not my_logger.handlers:
         # Here I created handler, formatter, loglevel etc..
    return my_logger

This works fine.

 

Posted in Flask, Python | Tagged , | 2 Comments

Weird thing to understand from mongoengine in Flask

Inital mongo collection

I am creation a mongo dument in Flask using MongoEngine Object mapper.Simple, consider the collection name is TestMe and it has two fields (name, address).
The name of the document is nava.

from mongoengine import Document, connect, StringField

connect('nava')

class TestMe(Document):
    name = StringField(required=True)
    address = StringField(required=True)
    

Adding a new row

I am adding one row document in the collection using this mongoengine class which is described below.

 # 1. Initial document creation in mongo collections with only two fields(name and address)                                                        
    t = TestMe(name="Navaneethan", address="Bangalore")
    t.save()

So, the document is added successfully.

Verify in your mongo client

For quick confirmation, We can verify the document in mongod client.

(hire)navaneethan@navaneethLap:~/workspace/experiments/python/flask$ mongo
MongoDB shell version: 3.0.4
connecting to: test
Server has startup warnings: 
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] 
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] 
> show dbs;
employee  0.078GB
hirenew   0.078GB
local     0.078GB
nava      0.078GB
> use nava;
switched to db nava
> db.t
db.test_me          db.toLocaleString(  db.toString(        db.tojson(
> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore" }
> 

Adding a new field in the existing collection

Now I need a field has to be added in the existing db nava, which is is_active a booleanfield.
So, the refined model is,

class TestMe(Document):
    name = StringField(required=True)
    address = StringField(required=True)
    is_active = BooleanField(default=False)

To my current scenario, I have to update the field to the selected objects which would have the default value, but it should be reflected in db.

Updating the default value to the exisiting rows for the new field

in my python shell

obj = TestMe.objects(name='Navaneethan').first()
obj.is_active = False
obj.save()
print "updated successfully"

I am retrieving all the fields from the db and updating to the default value.The save happens successfully.

Verifying in the mongod client

But what surprising me is,
Unable to see the new field in mongo db.

switched to db nava
> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore" }
> 

Why?

Here I scrateched my head for almost 2 hours.

Changing the default value to some new value(false –> true)

After somepoint, tried to update the field with different value(not the default one)

obj = TestMe.objects(name='Navaneethan').first()
obj.is_active = True  # note here the default value of the field is changed
obj.save()
print "updated successfully"

Now,

Verifying in the mongod client

The field is updated to the existing rows.

> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore", "is_active" : true }
> 

my mind before hitting the head against wall:

It seems little weird to me. Since it is a nosql mapper or what? I come from django experience, I expected the field has to be updated even the default value is given to be updated for the existing rows. But it doesn’t happen. So, what happens is, If the value of the new field changed from the default value(`false` here) to the some other value(`true`) then the update reflects successfully.


There is a solution for every problem.

Yes. This is an obvious statement which I realized after a while hitting my head against the wall.

if you inherit from `DynamicDocument` for mongoengine odm. You will get the clear solution. It does the job.

The learn happens after this post gets published :)

Thanks for your time.

Posted in Flask, Python | Tagged , | Leave a comment

Erlang maps basics

erl> maps: for python>>>dict() people

just watch and think series:

How to create a new map in erlang?

 47>
 47>
 47> MyNewMap = maps:new().
 #{}
 48> MyNewMap
 48> .
 #{}
 49> maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 50> MyNewMap.
 #{}
 51> MyNewMap2 = maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 52> MyNewMap2.
 #{name => "Navaneethan"}
 53> MyNewMap.
 #{}
 54> maps:put(age, 26, MyNewMap2).
 #{age => 26,name => "Navaneethan"}
 55> MyNewMap2.
 #{name => "Navaneethan"}
 56> % RETURNS NEW MAP. Not updating into old one.%
 

Part2 in Maps(remove, put, update, without,values,to_list ):

 56> maps:remove('name',MyNewMap).
 #{}
 57> maps:remove(name,MyNewMap).
 #{}
 58> MyNewMap.
 #{}
 59> maps:remove(name,MyNewMap2).
 #{}
 60> MyNewMap2.
 #{name => "Navaneethan"}
 61> MyNewMap2 = maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 62> MyNewMap2 = maps:put(age, "26", MyNewMap).
 ** exception error: no match of right hand side value #{age => "26"}
 63> MyNewMap2.
 #{name => "Navaneethan"}
 64> f(MyNewMap2).
 ok
 65> MyNewMap2 = maps:put(age, "26", MyNewMap).
 #{age => "26"}
 66> MyNewMap2.
 #{age => "26"}
 67> MyNewMap2 = maps:put(work, "Python", MyNewMap2).
 ** exception error: no match of right hand side value #{age => "26",work => "Python"}
 68> MyNewMap3 = maps:put(work, "Python", MyNewMap2).
 #{age => "26",work => "Python"}
 69> maps:to_list(MyNewMap3).
 [{age,"26"},{work,"Python"}]
 70> maps:update(work,"erlang",MyNewMap3).
 #{age => "26",work => "erlang"}
 71> MyNewMap3.
 #{age => "26",work => "Python"}
 72> %Update Returns New one.%
 72> maps:values(MyNewMap3).
 ["26","Python"]
 73> ReqKeys = [name, 'book', age, 'age'].
 [name,book,age,age]
 74> maps:with(ReqKeys, MyNewMap3).
 ** exception error: undefined function maps:with/2
 75> MyNewMap3.
 #{age => "26",work => "Python"}
 76> maps:with([name], MyNewMap3).
 ** exception error: undefined function maps:with/2
 77> maps:with(ReqKeys, MyNewMap3).
 ** exception error: undefined function maps:with/2
 78> maps:without(ReqKeys, MyNewMap3).
 #{work => "Python"}
 79>

Points to be noticed:

This is erlang map which is similar to the python dictionary.Here we do some mandate operations with map.

If you pass your map to in built function in maps module, it returns new map which is unbounded.(line:62 & 67)

Rest of the things are similar like other programming languages.

Posted in erlang | Tagged | Leave a comment