Avoid duplicate process copy through cron


You are executing a background job for every x minuts periodically.The job execution time takes approximately y mins.Sometimes y > x. So, you are endup with multiple duplicate active process running at back.


If duplicate process runs at back, it creates multiple side effect. The one is you would surprise to see the duplicate record in your table.

So, We can write a shell script which should check whether any active same process running or not? If not, spawn one.


This trick might help to you.


status=`ps -efww | grep -v grep |grep -w "python manager.py send_email" | awk -vpid=$$ '$2 != pid { print $2 }'`
echo $status
if [ ! -z "$status" ]; then
    echo "Process already running"
    cd /home/edge/projects/venv
    source bin/activate

    cd /home/edge/projects/taleo
    python manager.py send_email

    echo "[`date`] New Process Spawned"

my process python manager.py send_email will be grepped before it gets spawned.

your suggestions are welcome :)

Tagged with: , ,
Posted in 10min, Flask, Python

Same Log messages multiple times in Python [issue]

The Mistake happened in my python loggers

what ?

I use Python/Flask framework to develop web applications. So, For my few usecase scripts, I have to use the logger to log the flow.I have created a common handler which workes nicely as expected.I have defined the logger in seperate module called utils/__init__.py. From this module I used to import to my script to log the flow.

import logging
import logging.handlers

def get_logger(name, file_name=None):
    LOG_FILENAME = file_name if file_name else name + '.log'
    # Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    # Add the log message handler to the logger
    handler = logging.handlers.RotatingFileHandler(LOG_FILENAME,
    formatter = logging.Formatter("[%(asctime)s] [%(filename)s:%(lineno)s - %(funcName)5s() - %(processName)s] %(levelname)s - %(message)s")
    return my_logger

When I import this logger object to multiple files, log statements getting printed multiple times as many imports i did.

Stackoverflow saved my time.

The reason is Each call(log = get_logger('taleo_integration')) is creating a new log handler for the logger you create as given below.

What is under the hood ?

In [13]: from hirenew.utils import get_logger

In [14]: log.handlers
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>]

In [15]: log = get_logger('taleo_integration')

In [16]: log.handlers
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>]

In [17]: log = get_logger('taleo_integration')

In [18]: log.handlers
[<logging.handlers.RotatingFileHandler at 0x62f4b50>,
 <logging.handlers.RotatingFileHandler at 0x5ff0790>,
 <logging.handlers.RotatingFileHandler at 0x5ff04d0>,
 <logging.handlers.RotatingFileHandler at 0x5ff0810>]


Finally, I added couple of lines in get_logger.

# Set up a specific logger with our desired output level
    my_logger = logging.getLogger(name)
    if not my_logger.handlers:
         # Here I created handler, formatter, loglevel etc..
    return my_logger

This works fine.


Tagged with: ,
Posted in Flask, Python

Weird thing to understand from mongoengine in Flask

Inital mongo collection

I am creation a mongo dument in Flask using MongoEngine Object mapper.Simple, consider the collection name is TestMe and it has two fields (name, address).
The name of the document is nava.

from mongoengine import Document, connect, StringField


class TestMe(Document):
    name = StringField(required=True)
    address = StringField(required=True)

Adding a new row

I am adding one row document in the collection using this mongoengine class which is described below.

 # 1. Initial document creation in mongo collections with only two fields(name and address)                                                        
    t = TestMe(name="Navaneethan", address="Bangalore")

So, the document is added successfully.

Verify in your mongo client

For quick confirmation, We can verify the document in mongod client.

(hire)navaneethan@navaneethLap:~/workspace/experiments/python/flask$ mongo
MongoDB shell version: 3.0.4
connecting to: test
Server has startup warnings: 
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] 
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-07-30T10:19:08.103+0530 I CONTROL  [initandlisten] 
> show dbs;
employee  0.078GB
hirenew   0.078GB
local     0.078GB
nava      0.078GB
> use nava;
switched to db nava
> db.t
db.test_me          db.toLocaleString(  db.toString(        db.tojson(
> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore" }

Adding a new field in the existing collection

Now I need a field has to be added in the existing db nava, which is is_active a booleanfield.
So, the refined model is,

class TestMe(Document):
    name = StringField(required=True)
    address = StringField(required=True)
    is_active = BooleanField(default=False)

To my current scenario, I have to update the field to the selected objects which would have the default value, but it should be reflected in db.

Updating the default value to the exisiting rows for the new field

in my python shell

obj = TestMe.objects(name='Navaneethan').first()
obj.is_active = False
print "updated successfully"

I am retrieving all the fields from the db and updating to the default value.The save happens successfully.

Verifying in the mongod client

But what surprising me is,
Unable to see the new field in mongo db.

switched to db nava
> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore" }


Here I scrateched my head for almost 2 hours.

Changing the default value to some new value(false –> true)

After somepoint, tried to update the field with different value(not the default one)

obj = TestMe.objects(name='Navaneethan').first()
obj.is_active = True  # note here the default value of the field is changed
print "updated successfully"


Verifying in the mongod client

The field is updated to the existing rows.

> db.test_me.find()
{ "_id" : ObjectId("55b9bf94aff2d3212702a517"), "name" : "Navaneethan", "address" : "Bangalore", "is_active" : true }

my mind before hitting the head against wall:

It seems little weird to me. Since it is a nosql mapper or what? I come from django experience, I expected the field has to be updated even the default value is given to be updated for the existing rows. But it doesn’t happen. So, what happens is, If the value of the new field changed from the default value(`false` here) to the some other value(`true`) then the update reflects successfully.

There is a solution for every problem.

Yes. This is an obvious statement which I realized after a while hitting my head against the wall.

if you inherit from `DynamicDocument` for mongoengine odm. You will get the clear solution. It does the job.

The learn happens after this post gets published :)

Thanks for your time.

Tagged with: ,
Posted in Flask, Python

Erlang maps basics

erl> maps: for python>>>dict() people

just watch and think series:

How to create a new map in erlang?

 47> MyNewMap = maps:new().
 48> MyNewMap
 48> .
 49> maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 50> MyNewMap.
 51> MyNewMap2 = maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 52> MyNewMap2.
 #{name => "Navaneethan"}
 53> MyNewMap.
 54> maps:put(age, 26, MyNewMap2).
 #{age => 26,name => "Navaneethan"}
 55> MyNewMap2.
 #{name => "Navaneethan"}
 56> % RETURNS NEW MAP. Not updating into old one.%

Part2 in Maps(remove, put, update, without,values,to_list ):

 56> maps:remove('name',MyNewMap).
 57> maps:remove(name,MyNewMap).
 58> MyNewMap.
 59> maps:remove(name,MyNewMap2).
 60> MyNewMap2.
 #{name => "Navaneethan"}
 61> MyNewMap2 = maps:put(name, "Navaneethan", MyNewMap).
 #{name => "Navaneethan"}
 62> MyNewMap2 = maps:put(age, "26", MyNewMap).
 ** exception error: no match of right hand side value #{age => "26"}
 63> MyNewMap2.
 #{name => "Navaneethan"}
 64> f(MyNewMap2).
 65> MyNewMap2 = maps:put(age, "26", MyNewMap).
 #{age => "26"}
 66> MyNewMap2.
 #{age => "26"}
 67> MyNewMap2 = maps:put(work, "Python", MyNewMap2).
 ** exception error: no match of right hand side value #{age => "26",work => "Python"}
 68> MyNewMap3 = maps:put(work, "Python", MyNewMap2).
 #{age => "26",work => "Python"}
 69> maps:to_list(MyNewMap3).
 70> maps:update(work,"erlang",MyNewMap3).
 #{age => "26",work => "erlang"}
 71> MyNewMap3.
 #{age => "26",work => "Python"}
 72> %Update Returns New one.%
 72> maps:values(MyNewMap3).
 73> ReqKeys = [name, 'book', age, 'age'].
 74> maps:with(ReqKeys, MyNewMap3).
 ** exception error: undefined function maps:with/2
 75> MyNewMap3.
 #{age => "26",work => "Python"}
 76> maps:with([name], MyNewMap3).
 ** exception error: undefined function maps:with/2
 77> maps:with(ReqKeys, MyNewMap3).
 ** exception error: undefined function maps:with/2
 78> maps:without(ReqKeys, MyNewMap3).
 #{work => "Python"}

Points to be noticed:

This is erlang map which is similar to the python dictionary.Here we do some mandate operations with map.

If you pass your map to in built function in maps module, it returns new map which is unbounded.(line:62 & 67)

Rest of the things are similar like other programming languages.

Tagged with:
Posted in erlang

Erlang day1

Erlang day1:

  1. Erlang shell an introduction
  2. How to use the erlang shell?
  3. few basic data structure tricks

Erlang shell an introduction:

In order to get your erlang shell,you have to install the erlang OTP for your platform(Windows/Linux).
Once installed erlang software, you can type erl to launch the shell.
In erl you can do the experiments in erlang.

watch the screen & read the below marks:

erl day 1

Few things keep remember:

The new values can’t be assigned to the bounded varialbes.Use always a unbound variable to store the value.Varialbes starts with caps(i.e Var).
_ is always used to unregister the value.The value will be omitted if it is assigned to the underscore.

Data structures are atoms, tuple, list.

  • tuple: It is represented by {}. e.g {1,2,3,4}.
  • atom: This is just a variable which is used name the data to match the pattern.
    eg.  {person, Name} = {person, "navaneethan"}.

    in this above example two tuples are matched with help of atom person, so the variable Name will be bounded to the value navaneethan.Finally, this pattern match is successful.

  • list: []
    e.g: [1,2,3,4].

Few weird things with tuples and lists:

  • Two tuples are compared with the no of elements on each tuple.
  • Two lists are compared with Dictionary associate property.
Tagged with:
Posted in erlang

an intro to Erlang

an intro to Erlang:

I have started Erlang last weekend.Few things will amaze you if you start following it.You would get inspired of it’s powerful features.I would like to share with you my thoughts also what I learned in this one day.

Erlang creations:

Using erlang, there are quite # of powerful tools were built for real time usage.For example,

  • Facebook chat 100million requests are being handled
  • simpledb in AWS were implemented
  • Delicious bookmarking tool
  • sms component in TMobile
  • rabbitmq
  • couchdb
  • It has been tested in all sides in ATM AXD301 machine.

What erlang gives to us?

It is the concurrent programming language. It derives many features from various type of programming language to be focused o concurrent application implementation effectively.

So, Erlang guts are

  • No shared memory
  • Functional programming language
  • immutable and pattern matching

No sharing

There is nothing about shared memory in erlang virtual machine.The process communication is handled through message passing in asynchronous way to avoid synchronization overhead.So, it is the server side programming language, where you can derive the needful specific behavior(event handler, finite state machine) from erlang library to implement your server.

immutable and pattern matching

variables can’t be reassigned.The value of the variable can’t be changed.It has the feature called pattern matching to extract the data from the clause.

Functional programming language

Dynamic type system is used in erlang. Functional programming features are the core in erlang.

useful links



Tagged with:
Posted in erlang

10min python: dis assembler

10min python: dis

what is dis?

dis is the disassembler. It shows to you how the python interpreter executes your python code in human readable format.

why do we need it?

To understand and optimise your python code to make it easier to the world.

how it works?

There is an inbuild module called dis which has the job and does it.It just reads the compiler understandable code(means byte code) and converting into mneumonics which is the cpu understandable language.

def your-function():



In [3]: import dis

In [4]: help(dis)

In [5]: def show_mybytes(a, b):
   ...:     l = []
   ...:     l.append(a)
   ...:     l.append(b)
   ...:     c = a + b
   ...:     l.append(c)
   ...:     return l

In [6]: show_mybytes(4,5)
Out[6]: [4, 5, 9]


In [8]: dis.dis(show_mybytes)
  2           0 BUILD_LIST               0
              3 STORE_FAST               2 (l)

  3           6 LOAD_FAST                2 (l)
              9 LOAD_ATTR                0 (append)
             12 LOAD_FAST                0 (a)
             15 CALL_FUNCTION            1
             18 POP_TOP             

  4          19 LOAD_FAST                2 (l)
             22 LOAD_ATTR                0 (append)
             25 LOAD_FAST                1 (b)
             28 CALL_FUNCTION            1
             31 POP_TOP             

  5          32 LOAD_FAST                0 (a)
             35 LOAD_FAST                1 (b)
             38 BINARY_ADD          
             39 STORE_FAST               3 (c)

  6          42 LOAD_FAST                2 (l)
             45 LOAD_ATTR                0 (append)
             48 LOAD_FAST                3 (c)
             51 CALL_FUNCTION            1
             54 POP_TOP             

  7          55 LOAD_FAST                2 (l)
             58 RETURN_VALUE        

In [9]: 

in this above example, first column is line no of our python code statements.Second column is the neumonic instraction address position(I think but not sure), third column is the mneumonic instructions for the cpu, last column is human understanble format mapping between python code to assembly code

Tagged with: , ,
Posted in 10min, Python

If not python, then what?

  • 31,885 hits

Get every new post delivered to your Inbox.

Join 258 other followers