Python 2.7.4 and 3.3.1 had just came out, they offer quite a few performance enhancements as well as bug fixes. Also, upgrading to 2.7.4 seems like the first logical step before moving to Python 3.3.x.

I hope this post can help you upgrade your Python as painless as possible.

1. Stop relying on base Python installation

Installing a different version of Python is very easy now thanks to pythonz.

# 1. ALWAYS preview the code before piping to bash!
curl -kL https://raw.github.com/saghul/pythonz/master/pythonz-install | bash

# 2. Source pythonz paths
source $HOME/.pythonz/etc/bashrc

# 3. Install a different version of Python
pythonz install 2.7.4

2. Use virtualenv to sandbox all your Python modules

Virtualenv is here to stay, it has already been absorbed as standard library in Python 3.3. It’s a great tool to setup Python modules for your application.

This is the example on how to setup virtualenv using pythonz’s Python.

# VARIABLES
PYTHON27_VERSION='2.7.4'
PYTHON27_NAME="CPython-$PYTHON27_VERSION"
PYTHON27_BIN="~/.pythonz/pythons/$PYTHON27_NAME/bin/python"
PYTHON27_VENV_DIR="~/.pythonz/venvs/$PYTHON27_NAME"

# NOTE:
# pythonz put all the different pythons here: ~/.pythonz/pythons
# We put our venvs under ~/.pythonz/venvs to make organization simple.

PROJECT_NAME="example"

# Create directory for virtualenv
mkdir -p $PYTHON27_VENV_DIR

# Create virtualenv
virtualenv --no-site-packages -p $PYTHON27_BIN $PYTHON27_VENV_DIR/$PROJECT_NAME

Here’s another example, the snippet setup PyPy as well.

3. Activate your virtualenv to install Python modules

source ~/.pythonz/venvs/CPython-2.7.4/$PROJECT_NAME/bin/activate
pip install -r requirements.txt
deactivate

4. Don’t worry about activate() vs deactivate() when running your daemon

Just run the venv Python binary directly.

This approach is very convenient when dealing with cron or Supervisord.

~/.pythonz/venvs/CPython-2.7.4/$PROJECT_NAME/bin/python

I’ve finally gotten around to release async Stripe library as full fledged open source project. It has been in production for On the Plates for a few months now.

The cool thing about this API is that, it maps to Stripe Curl API URLs exactly one-to-one. For example:

from tornado_stripe import Stripe
stripe = Stripe('api_key', blocking=True)

stripe.charges                                  # == /v1/charges
stripe.charges.id(CHARGE_ID)                    # == /v1/charges/{CHARGE_ID}
stripe.customers                                # == /v1/customers
stripe.customers.id(CUSTOMER_ID)                # == /v1/customers/{CUSTOMER_ID}
stripe.customers.id(CUSTOMER_ID).subscription   # == /v1/customers/{CUSTOMER_ID}/subscription
stripe.invoices                                 # == /v1/invoices
stripe.invoices.id(INVOICE_ID)                  # == /v1/invoices/{INVOICE_ID}
stripe.invoiceitems                             # == /v1/invoiceitems
stripe.invoiceitems.id(INVOICEITEM_ID)          # == /v1/invoiceitems/{INVOICEITEM_ID}
stripe.tokens                                   # == /v1/tokens
stripe.tokens.id(TOKEN_ID)                      # == /v1/tokens/{TOKEN_ID}
stripe.events                                   # == /v1/events
stripe.events.id(EVENT_ID)                      # == /v1/events/{EVENT_ID}

To install:

pip install tornado-stripe

For more information, visit its GitHub page.

Finally, curiosity go the best of me. I decided to install Go language and it’s web.py like framework called web.go.

My OS Info

OS X Snow Leopard, 2.26 GHz Intel Core 2 Duo, 8 GB 1067 MHz DDR3 RAM

Prerequisites(on OS X)

• To install go, simply download the package and double click.
• To install web.go:

# You may need to restart your terminal session after installing Go language.
sudo go get github.com/hoisie/web

Hello World Page

Follow the instruction on web.go github for creating the hello world page. I copy-pasted the instruction here for convenience:

# Do all 3 steps in the same directory
# 1. Create hello.go
package main

import (
    "github.com/hoisie/web"
)

func hello(val string) string { return "hello " + val } 

func main() {
    web.Get("/(.*)", hello)
    web.Run("0.0.0.0:9999")
}

# 2. Compile hello.go
go build hello.go

# 3. Run the executable
./hello

# Visit: http://localhost:9999/world to see the page.

The compilation time is miniscule, but that doesn’t say much because our application is tiny. Once we got the hello world page up, like a true tech blogger, we have to benchmark it using AB tool.

# 100,000 requests. 210 simultaneous requests. Keep-alive is on.
ab -k -n 100000 -c 210 http://localhost:9999/world

The result is quite astonishing. On fresh boot, our little app server took only 4.1 MB of RAM and after AB is finished, the memory usage climbs to 15.3 MB. Not bad at all.

Now let’s look at the AB result:

Server Software:        web.go
Server Hostname:        localhost
Server Port:            9999

Document Path:          /world
Document Length:        11 bytes

Concurrency Level:      210
Time taken for tests:   25.870 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    100000
Total transferred:      16700000 bytes
HTML transferred:       1100000 bytes
Requests per second:    3865.54 [#/sec] (mean)
Time per request:       54.326 [ms] (mean)
Time per request:       0.259 [ms] (mean, across all concurrent requests)
Transfer rate:          630.41 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1  22.8      0    1000
Processing:     0   54  36.6     51     486
Waiting:        0   54  36.6     51     486
Total:          0   54  42.9     51    1159

Percentage of the requests served within a certain time (ms)
  50%     51
  66%     57
  75%     62
  80%     71
  90%    102
  95%    116
  98%    155
  99%    175
 100%   1159 (longest request)

3865.54 requests per second and 0 failed requests! I am very impressed, this result beats anything I’ve ever tested in this laptop. And look at that time per request: 54 ms. Wow!

The Situation

You’ve checked that your SSL certs are valid and HTTPS is still not working on your Linode instance that’s behind NodeBalancer.

The Solution

1. Login to your Linode web interface
2. Edit your port 443 NodeBalancer settings. Change protocol HTTP to TCP.
3. Profit.

…through prepackaged binary from python.org. As opposed to using virtualenv.

This post is pretty much a self-reminder post. I was upgrading Python to 2.7.2 from 2.6.1 on OS X Snow Leopard.

cd /tmp

# fix pip
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
sudo python /tmp/get-pip.py

# Fix setuptools
# Download the appropriate .egg from here: http://pypi.python.org/pypi/setuptools
curl -O http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg#md5=fe1f997bc722265116870bc7919059ea
sudo python /tmp/setuptools-0.6c11-py2.7.egg

# Fix distribute
curl -O http://python-distribute.org/distribute_setup.py
sudo python distribute_setup.py

Today is finally the day when Python package management is giving me feud. Which is a total surprise because it had always worked reliably.

OS: OS X Snow Leopard

Problem:

/usr/bin/easy_install-2.6:7: UserWarning: Module pkg_resources was already imported from /System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/pkg_resources.pyc, but /Library/Python/2.6/site-packages is being added to sys.path
    from pkg_resources import load_entry_point

The error message is so not obvious, by the way.

How to fix:

Just reinstall distribute

curl -O http://python-distribute.org/distribute_setup.py
/usr/bin/python2.6 distribute_setup.py

Today I finally had the opportunity to use Fabric (v.1.4). True to “hello world” spirit, this is my fabfile.py setup:

from fabric.api import run

def ps_aux():
    run(‘ps aux’)

And then fabric freak out and complain about this:

No handlers could be found for logger “ssh.transport”

This type of error usually occur when a module uses logging standard library. So, to make logging module stop making sad face, we need to set the log level on “ssh.transport”:

logging.basicConfig()
logging.getLogger(‘ssh.transport’).setLevel(logging.INFO)

That’s it.

What is Tornado? Tornado is a popular event-loop based web framework.

Most people who talk about Tornado usually mentioned how fast it is when hitting “Hello World” using Apache Benchmark. This post is not about that.

Background jobs

In most web application request cycle, it’s common for the request handler (or controller) to perform various tasks that are less important than page delivery (and sometimes expensive). Tasks like recording analytic, updating counts, processing uploaded files, etc.

Django applications solve this problem by creating jobs on Celery or Gearman.

Rails applications solve this problem by creating jobs on Resque or various AMQP solutions.

Tornado applications solve this problem through IOLoop.add_callback() within itself, without any external daemons:

def expensive_callback():
   pass # your expensive work

with tornado.stack_context.NullContext():
   tornado.ioloop.IOLoop.instance().add_callback(expensive_callback)

Scheduled jobs

Web applications sometimes need to perform routine work. Many people solve this problem using cron. Some people try to reinvent cron through scheduler daemon.

Meanwhile, Tornado have built-in solution to this problem:

def run_me_everyday_callback(): pass

tornado.ioloop.PeriodicCallback(run_me_everyday_callback, 24 * 60 * 60 * 1000).start()

1. node.js also have the same benefit through setInterval()
2. It’s true that the interval doesn’t use system clock, but that’s solvable by running the callback every X seconds, but check the system clock first.

External dependencies

Tornado has 0 external dependencies. It has its own HTTP parser, epoll.c, template engine, and even web server. It used to depend on curl strictly, but now Tornado shipped SimpleAsyncHTTPClient that’s 100% Python.

Having minimal dependencies is very nice. When I encountered any bugs, I can file them under GitHub issue tracking or mentioned them on the mailing list. The core developers can basically fix everything inside Tornado.

Its fast web server also makes deployment simpler. You typically only need nginx/haproxy in front of Tornado instances.

It’s a micro framework

Besides having zero external dependencies, Tornado is also small in terms of LoC. Because of its simplicity:

• Tornado can run under PyPy.
• Tornado applications can use cream of the crop Python modules (e.g. SQLAlchemy).
• It’s very easy to add/extends any parts of Tornado.
• With a careful planning, your Tornado application can use very little memory per instance. Less memory means you can spawn more instances per box.

It supports OAuth authentication to famous social networks

Thanks to the various mixins that Tornado provides, your application can connect to Google, Twitter, or Facebook in no time.

It has global Application object

Unlike Django and very similar to the design of Flask, CherryPy, web.py, or Sinatra+Rack. With this design, it’s very easy to instantiate multiple Tornado apps inside one Python runtime. It makes interactive debugging easier.

It has a familiar template engine

Tornado’s template engine is very similar to Django template or Jinja2, developers won’t have much trouble getting up to speed in it.

It’s been fun a ride. We’ve known each other since 2006. Through good times and bad times. One dotcom to another. I know your shortcuts like the back of my own hands.

But it’s time to move on. You are starting to show your age. Accidental Command+Shift+F always killed you while Command+T is too slow on TextMate 2 Alpha.

Meet my new editor, Sublime Text 2. Here are a few tips I just discovered on OS X.

How to: Setup command line executable

Follow this instruction.

How to: Open Python prompt

Press Control+`

How to: Install Package Control

Follow this instruction.

How to: Install Git package

Follow this instruction.

How to: Install any packages

How to: Use git blame on current buffer file

1. Install Git package first. See above.
2. Press Command+Shift+P to open context sensitive menu.
3. Type gitbl, it will reveal Git Blame.
4. Press enter.

How to: Select with multiple cursors

Press Command while clicking.

How to: Indent a block of text

1. Highlight the text.
2. Indent outward with Command+] or inward with Command+[

How to: go to line in opened file

Press Control+G (This one bugs me a little). It turns out, I can also do: Command+P, then : (colon)

How to: go to method in opened file

Press Command+R

How to: find and replace in opened file

Press Command+Alt+F

That said, I prefer my Command+F to do the job. So I changed the key bindings. To do that:

1. open Preferences > Key Bindings
2. reverse the following:

{ “keys”: [“super+f”], “command”: “show_panel”, “args”: {“panel”: “find”} },
{ “keys”: [“super+alt+f”], “command”: “show_panel”, “args”: {“panel”: “replace”} },

How to: Debug your Flask or Tornado application interactively

1. Open Python prompt using: Control+`
2. Import your Application object.

More resources:

• http://net.tutsplus.com/tutorials/tools-and-tips/sublime-text-2-tips-and-tricks/

The GitHub repo is here. Use this mixin to interact with Foursquare OAuth2 API.

It’s very similar to Tornado’s auth.FacebookGraphMixin. Below is sample code on how to authenticate to Foursquare:

class FoursquareLoginHandler(LoginHandler, FoursquareMixin):
    @tornado.web.asynchronous
    def get(self):
        if self.get_argument(“code”, False):
            self.get_authenticated_user(
                redirect_uri=’/auth/foursquare/connect’,
                client_id=self.settings[“foursquare_client_id”],
                client_secret=self.settings[“foursquare_client_secret”],
                code=self.get_argument(“code”),
                callback=self.async_callback(self._on_login)
            )
            return

        self.authorize_redirect(
            redirect_uri=’/auth/foursquare/connect’,
            client_id=self.settings[“foursquare_api_key”]
        )

    def _on_login(self, user):
        # Do something interesting with user here. See: user[“access_token”]
        self.finish()

Similar to Tornado’s other API setup, don’t forget to assign foursquare_client_id and foursquare_client_secret into Application constructor. See below:

import tornado.web

class Application(tornado.web.Application):
    def __init__(self):
        tornado.web.Application.__init__(self, routes(), **dict(
            foursquare_client_id    = options.foursquare_client_id,
            foursquare_client_secret= options.foursquare_client_secret,
        ))

What a dark October. John McCarthy (1927-2011), father of Lisp and inventor of garbage collection, passed away today.

$ racket
Welcome to Racket v5.0.1.
> (displayln “Good bye, John McCarthy. Thank you for Lisp and Garbage Collection”)

Man, this past weeks have been such a downer. Dennis Ritchie, the co-creator of C and Unix has passed away (1941-2011).

# include <stdio.h>

int main()

{

    printf(“Thank you so much for creating C and Unix.”);

    return 1;

}

This is the first time, ever, in my life that I felt really sad about a person I’ve never met before.

That guy I saw on TV, around 95-96. Young, rich, good looking & does “computer”. The same guy who drilled “American Dream” in me for more than a decade, is no longer with us.

Rest in peace, Mr. Jobs (1955-2011).

The problem that prompted this blog post is this AmazonSES error:

RequestExpired, it must be within 300secs/of server time

Let’s check my system clock:

date

Sat Aug  6 09:36:38 PDT 2011

Now let’s point my ntp daemon to do different server: pool.ntp.org

sudo ntpdate pool.ntp.org

Now, let’s do date again:

date

Sat Aug  6 09:30:31 PDT 2011

Yup. My system clock was 6 minutes too fast. And sure enough that fixed the AmazonSES error.

The title says it all, this blog post will demonstrate on how to run Tornado application inside PyPy environment.

Prerequisites:

• The OS we’ll use is OS X Snow Leopard.
• system-wide pip.
• Obviously, default python installation (2.6.1).

Virtualenv with PyPy in a nutshell:

• Install virtualenv

sudo pip install virtualenv

• Download your binary of choice from http://pypy.org/download.html
• Extract the compressed file to a location, for example: /tmp. You will see directory that look like this: pypy-c-jit-43780-b590cf6de419-osx64
• Go to the desired location of your pypy environment, for example: ~/www

cd ~/www

• Create a new virtualenv, let’s call it pypy-env. It is best to not include site-packages because we don’t really know if pypy has support on your existing eggs.

virtualenv —no-site-packages -p /tmp/pypy-c-jit-43780-b590cf6de419-osx64/bin/pypy pypy-env

• Activate your virtualenv

source ~/www/pypy-env/bin/activate

• Start installing eggs. All your eggs will be located inside ~/www/pypy-env/site-packages

pip install MySQLdb-python

pip install tornado

Something a little different (my own approach):

In this section, I will suggest my style of setting up pypy environment for each Tornado app. It is heavily influenced by Isolate in Ruby land. If you don’t like what you read here, that’s cool. Just follow the nutshell above.

I like how Isolate puts all ruby gems under $RAILS_ROOT/tmp/isolate. It makes auditing dependencies easy. So, we’ll emulate that by putting pypy-env inside our application directory (Example: ~/www/my-tornado-app/pypy-env). You can automate the whole process by using this bash script.

Don’t forget to exclude the pypy-env and pypy binary directories on your source control (.gitignore or .hgignore).

Stumbling blocks:

• Unfortunately, not everything is rosy. MySQLdb-python still doesn’t work quite right in Tornado in PyPy (See: stacktrace). It seems like copy.deepcopy doesn’t like something inside MySQLdb.converters.conversions. To side-step this problem, I added support for pymysql inside tornado.database (See: this commit). Once you apply this patch, simply do: pip install pymysql
• Another stumbling block is intermittent SSL error when using tornado.auth library. It is already documented and fixed by other people. Since the fix is recent(2011-05-15), you need to compile PyPy from source to get it. The error was this:

AttributeError: ‘SSLObject’ object has no attribute ‘peer_certificate’

Conclusion:

Besides those two issues, Tornado runs like a champ inside PyPy. Unsurprisingly, the app use a lot more RAM when running inside PyPy (without: 15mb vs with: 45mb).