Making a twitter bot using Appengine

Twitter bots are autonomous , they collect data from a source or generate data and send them out as tweets . In this article I will show how to create a twitter bot , I call Serial Notifier which will tweet out everytime there new episode of Two and Half Men comes out .

To start out , one must get a source . In this case i choose epguides . Download the csv file from the page .

class Entry(db.Model):
  """Models an individual Guestbook entry with an author, content, and date."""
  serial = db.StringProperty()
  season = db.IntegerProperty()
  episode = db.IntegerProperty()
  name = db.StringProperty()
  date = db.DateProperty()

Entry is the database model that we will use to store each episode yet to come .

class BuildDB(webapp.RequestHandler):
	def get(self):

		f = open("list.txt","r")

		d = datetime.date.today()
		month = ["","Jan","Feb","Mar","Apr","May","Jun","Jul","Aug","Sep","Oct","Nov","Dec"]

		for serial in f.readlines():
			csv = open(serial.replace(" ","").strip().lower() + ".csv","r")

			# Remove first line
			csv.readline()
			serial = serial.strip()

			# format : number,season,episode,production code,airdate,title,special?
			for line in csv.readlines():
				parts = line.split(",")
				date = parts[4].split("/")
				print 2000 + int(date[2]),month.index(date[1]), int(date[0])
				show = datetime.date(2000 + int(date[2]),month.index(date[1]), int(date[0]))
				if (show > d):
					print line
					entry = Entry()
					entry.serial = serial
					entry.episode = int(parts[2])
					entry.season = int(parts[1])
					entry.name = parts[5]
					entry.date = show
					entry.put()

Once we have the csv file we need to build the database .  There is one file called list.txt which contains the list of sitcoms i wish to index . For each entry there is a corresponding csv file in the root . For each entry in the csv file , it parses out the line and stores in db only if it occurs at a later date .

Now that the db is built , the main part of sending out tweets . There are many wrappers for twitter api . Tweepy , python twitter to mention a few . But these have dependencies . In our case we have all the needed things and we just want to send out tweets . So I have choosen mikenapp’s Appengine-OAuth Library . It is a single file and hence easier to handle .

Go to dev.twitter.com and create a new application . Note down the consumer key and consumer secret . Also generate the access token and access token secret for your application .

class Cron(webapp.RequestHandler):
	def get(self):
		# Defines for serial notifier -- Add your credentials here
		consumer_key='xxx'
		consumer_secret='xxx'
		access_token='xxx'
		access_token_secret='xxx'
		callback_url='http://geo-lite.appspot.com/'

		status_format = "Today S%2dE%2d of %s - %s"

		# oauth client
		client = oauth.TwitterClient(consumer_key, consumer_secret,
				        callback_url)

		# Prepare a query
		q = Entry.all()
		yesterday = datetime.date.today() + datetime.timedelta(-1)
		q.filter("date =",yesterday)
		results = q.fetch(20)

		for result in results:
			additional_params = {
				"status": status_format % (result.season, result.episode, result.serial, result.name)
			}
			result = client.make_request(
				"http://twitter.com/statuses/update.json",
				token=access_token,
				secret=access_token_secret,
				additional_params=additional_params,
				method=urlfetch.POST)

		print "Completed today"
application = webapp.WSGIApplication([
	('/build', BuildDB),
	('/cron',Cron),
	('/.*',Home)
], debug=False)

That almost completes the twitter bot . Every time the cron handler is called it gets all the episodes that happened yesterday and tweets them out .

The last part is setting up cron jobs . Cron is an activity that must be called in my case every 24 hours . Cron jobs can be set up using the cron.yaml file . My cron.yaml file is as below :

cron:
- description: daily tweet out
  url: /cron
  schedule: every 24 hours

There you have your very own twitter bot \m/ . Complete source for this bot can be found at github .

Weekend Miscellaneous

After a long time – I decided that every week I will add a few nice links I found while transvering the web .

For this week — well a few of my favourites : :

• A nice problem I encountered [Math Stack exchange] http://math.stackexchange.com/questions/58711/solve-by-induction-nn-en/59077
• A recursive problem -> One step math : The ballot problem using reflection principle http://webspace.ship.edu/msrenault/ballotproblem/
• Many a times while reading I found reference to a language named R — initially I thought it was a typo . Actually its a very useful language for statistical computation http://en.wikipedia.org/wiki/R_(programming_language) .
• The place where you will find many many interesting topics related to programming :: http://www.reddit.com/r/programming/ .

And then there is this song from Speedy Singh :: Shera Di Kaum ft Ludacris

Ubuntu 11.04 boot freezes with 2.6.38-8-generic-pae kernel

So you’ve just installed 32-bit Ubuntu 11.04 (Natty Narwhal), and in my case, upgraded from Ubuntu 10.10 (Maverick Meerkat).  The upgrade process seemingly went fine and then you were asked to restart your computer.  You diligently do so, only to come back to find that your computer has frozen at GRUB2 upon trying to boot the 2.6.38-8-generic-pae kernel from the GRUB list.

In such a case go to Previous Linux Version – > Select one of the available options . They should work fine . Once you get into Ubuntu then reinstall the following packages ::

linux-image-2.6.38-8-generic-pae
linux-generic-pae 
linux-headers-generic-pae 
linux-image-generic-pae 
linux-headers-2.6.38-8-generic-pae

You should now be able to restart your computer and use the PAE-based kernel. This solution was found on the Ubuntu forums – hope it helps you as well  

Working with Git behind proxy

Most corporate companies and institutes allow members to access net through proxy . Working behind a proxy can be a real pain in the ass . I had to move some code to github but the squid proxy has only port 80 open .

The most common error ::

anil@anil-Studio-1558:~/GIT Repos/code-works$ git push -v origin master
Pushing to git@github.com:anilshanbhag/code-works.git
ssh: connect to host github.com port 22: Connection timed out
fatal: The remote end hung up unexpectedly

After much searching around link after link , I finally solved the problem .

export http_proxy='http://username:password@proxy:port'
export https_proxy='http://username:password@proxy:port'

The above two use username & password for proxy . Below use github username .

git push https://username@github.com/username/repo.git

This should work well on linux / mac . On windows http_proxy would need to replaced by HTTP_PROXY  .

Through the programming world : Languages I learnt

Well many people start out learning C++ or Java and move up the chain to use many other programming languages . Mine started with C++ .

My cycle can be best represented as the following chain :: plain -=> plain -=> self:: -=> this -=> this. -=> self .

Well it started out as C++ as a part of introductory course . I was my first experience with programming and having an awesome prof [ alias for professor : our insti lingo ] made the all so much better  . Well the first language form the basics for all thats to come .

As it moved on I started learning to program flash / as3 . Flash is a magnet for any newbie  :  its easy , Flash CS5 is a very easy to use development environment and flash stuff please you the most .

Then moving towards web development towards the end of winter holidays I start learning PHP . It was easy to get started – a well documented language meant function lookups were a cake walk . Some nice tutorials I scraped off the net here and there and  a nice series at lynda.com was all it took.

During my second sem we had a course on programming paradigms and abstractions – mostly using scheme , a nice functional programming language . Well if you start out its nice – everything boiling down to recursion , some nice concepts to learn but well scheme is no longer used in any new projects coming up .

After the end of sem we have 3 months summer hols – I devoted myself to learning the all powerful client side programming Javascript and beautiful language Python which one person [ I dont remember where or who ] had quoted “Life’s better without braces”  . Python is one language which is on the web and on the native apps which makes it very desirable to learn .

In the process I made quite some nice stuff ::

• Course Project for CS101 – A Snake game in C++
• Some arbit random stuff in flash : http://homepages.iitb.ac.in/~anilshanbhag/extra.php
• A simple site in php : http://homepages.iitb.ac.in/~anilshanbhag/index.php
• Course Project for CS152 – PIPE – Pipe Is a Photo Editor built in scheme
• Pixel drawing : www.cse.iitb.ac.in/~anil/pixels – When the canvas was just introduced .
• A few chrome extensions : Voice In , Speech Recognizer ,Twitter Tweet to Speech
• An app on appengine : http://sdtimer.appspot.com
• A game made purely in js and html5 : http://chrome-copter.appspot.com
• Last and most importantly I got a chance to work with Mozilla as a part of GSoC project :: Git repo

Python Logo

Hiding data : Generating png file with zip file in it

I stumbled upon this nice piece of information while searching for adding data into png files .

So how to add some data into image file . For all the know the trick is just that image files are read top down . While zip files are read bottom up .

Hence a simple trick of concatenating the two files should be readable by both the image viewer and archive extractor .

Can be done in unix using the commands :

$cat img.png >> outfile.png

$cat zipped.zip >> outfile.png

Here is a sample file that should be openable in archive extractor .

Save the image, open in archive extractor [ like winrar , archive mounter , etc ] and enjoy .

Note : On some environments the image may need to be renamed as outfile.zip or file with .zip extension for reading .

Python Debugger [ PDB ] Cheatsheet

To start pdb from within script :

import pdb
...
pdb.set_trace()

 Fom command line :

$python -m pdb xxx.py

A list of useful functions while using pdb ::

• up,down : Move up and down along the call stack
• step : go to next step
• next : same as step but skips stopping at function calls
• args : arguments given
• until : continue till line number greater than present
• return goes till return
• list : shows the source +- 5 lines from current line
• list a,b : shows source from line no a to b
• print r : executes r but outside pdb
• pp n : pretty print
• where : shows present point in callstack
• break : shows current breakpoints
• break lineNo : adds breakpoint at lineNo
• break func : adds breakpoint at function
• continue : continue till breakpoint

Those make the most commonly used commands in PDB .

Prototypical Inheritance in Javascript

I decided to port a flash game to HTML5 . Many would have played the game ‘Helicopter’ .

One of the most essential things to know before getting started with HTML5 game development is how to write object oriented code in javascript . Javascript is different with respect to oop in comparison to other languages that it follows the protoypical inheritance . It is the only instance of a language , all other languages follow the classical inheritance pattern.

For starters in JavaScript , there is no such thing as a class exactly in javascript . All are objects are derived from objects . There are array objects , function objects ,etc . All objects were in start derived from <’Object’> object .

We can create ‘pseudo’ classes – something called pseudo classical inheritance pattern in javascript .

Consider :

function Dummy(id){
this.id = id;
}

Dummy.prototype.printId = function(){
console.log(this.id);
}

myDummy = new Dummy(5);

myDummy.printId();

I am essentially creating Dummy class and myDummy is a new instance of Dummy class . So what exactly happened :

function Dummy(id){
this.id = id;
}

Dummy.prototype.constructor = Dummy;
Dummy.prototype.printId = function(){
console.log(this.id);
}

By default when you attach a method to function’s prototype , you set the function itself to the constructor .

myDummy = new Dummy(5);

The keyword new is important : why? , because it creates a new instance . You do not want to manipulate the original function itself. If you miss the ‘new’ myDummy would get Dummy itself and bad things would happen . The instance now points to prototype members of Dummy and hence call to printId would log the id .

Inheritence
Prototypical inheritence can be used to create a inheritence between classes .
Consider creating a new class DummyMax from Dummy .

function DummyMax(name){
   this.name = name;
}

DummyMax.prototype = new Dummy();
DummyMax.prototype.constructor = DummyMax;

DummyMax.prototype.printName = function(){
   console.log(this.name);
}

Essentially I have created a new object/class [call it whatever you feel comfortable ] DummyMax

DummyMax.prototype = new Dummy();

DummyMax’s prototype members are replaced with that of Dummy .So at this point DummyMax has constructor as Dummy and has method printId .

DummyMax.prototype.constructor = DummyMax ;

I reassign DummyMax as the constructor for DummyMax.

DummyMax.prototype.printName = function(){
   console.log(this.name);
}

And lastly I add a new method onto DummyMax .

Once you understand the flow its easier and simpler to effect a code reuse into other projects .
Any corrections/suggestions , do leave me a comment .

Ip to Country : Sqlite based python project

Ip to country is a small python project which can give the country for a given ip . This is based on fact that ip blocks are assigned to countries . Hence there is a direct relation . The code for this project can be found on github : https://github.com/anilshanbhag/code-works/tree/master/python/projects/ip-to-country

 

Guide to making it yourself ::

To start off download the csv from : http://ip-to-country.webhosting.info/downloads/ip-to-country.csv.zip .

That done it is seen that querying from csv file is not easy . Csv files needs to read in each time and tables need to be created . Alternative to this it I though it is  better to convert to SQLite DB which is one time affair . After that querying becomes lot easier . Going forward create a new db and a new table , i named it as ‘iptable’ . ‘iptable’ contains 5 columns start of ip block , end of ip block ,country 2 letter code , country 3 letter code and country full name .

Some pseudo code :

sql query - create table
for line in file :
    data = line split into parts
    run sql query to insert data into table

SQLite is a good for this purpose : Why? because it is standalone . The db i a standalone file and sqlite3 module can be used to interact with them .

Then the main.py file would read in an ip address . Convert it to an ip number : 1.2.1.0 would become 1*256**3 + 2*256**2 + 1*256**1 + 0*256**0 . To make it terminal friendly you can use the sys module to read in an argument . Run a query on the database and print the output .

 

2D Graphics : Cairo Graphics Library

Cairo is a very good 2D graphics library . It is used in Firefox , Webkit [Chrome , Safari] use Cairo for all the rendering .

I happened to working on cairo for around one month now mostly because i have got a project under GSoC which involves 2D graphics. For most part of it till now i have pyCairo , the python bindings of the Cairo library .

While working in pycairo : It is like a film 1.0×1.0 on which you draw and then it is scaled up to the required width and height .

Essentially 6 main functions exist [in most graphics library] :

move_to –> To move certain point

line_to  –> Drawing line to another point

curve_to –>  An extension of line to – can draw a curve

transfrom  –> Transfrom the coordinates

stroke –> Stroke the path

fill –> Fill the area bounded by path

A combination of these is used to render the background , border . Additionally text rendering and other features exist .

If you are interested you can have a look an example here . This code simulates the dashed border drawing in browsers .

Output of code :