Pagination with App Engine cursors in django-nonrel

First published on adamalton.co.uk on 21st August 2011.

A much more extensive version of this post is available on the Potato London website.

The stack:

• django (non-rel)
• Google AppEngine
• djangooappengine

The task:

• Using Datastore cursors to paginate through a metric bucket load of objects.

Note: if you're using the webapp framework rather than Django with djangoappengine then this blog post is not for you.

So you've got  a model with thousands objects, and you want to display them on a page a few at a time with some lovely pagination.  Django's standard paginator (django.core.paginator) is a bit dumb, in that it fetches all of the objects and then paginates through them, which other than defeating half the point of paginating, means that this doesn't really work on App Engine because by default MyModel.objects.all() will only return the first 1000 objects.

So we need to write some custom pagination code.  On AppEngine, MyModel.objects.all()[1001:2000] will work, and will correctly return you objects 1001 to 2000, so we could just use that.  But the beauty of AppEngine is the way that it scales, so we should think big.  When you get to MyModel.objects.all()[100000:100500] the Datastore is going to have to trawl its way through the first 100000 Entities, and then give you the next 500, and that is going to be kind of slow.  This is where Datastore cursors come in.

What are cursors?

A Datastore cursor is just a marker to a starting point in query.  It's just like giving an OFFSET to a SQL query, but it's much more efficient because it allows the datastore to jump to the starting point, instead of having to trawl its way through the first <OFFSET> objects before it starts.
Remember: cursors are an AppEngine Datastore thing.  Models are a Django thing.  But djangoappengine has glued the two together for us with 2 functions:

• get_cursor - give it a Django queryset and it returns the Datastore cursor which marks the position of the end of that query.
• set_cursor - give it a Django queryset and a cursor and it returns a new queryset which will return its results starting at the object marked by the cursor.

Let's write some code

#views.py
from djangoappengine.db.utils import get_cursor, set_cursor

def my_page(request):
    """ View some objects. """
    results_per_page = 100
    queryset = MyModel.objects.all()
    #Note, it hasn't called the datastore yet as querysets are lazy
    cursor = request.GET.get('cursor')
    if cursor:
        queryset = set_cursor(queryset, cursor)
    results =queryset[0:results_per_page] #starts at the offset marked by the cursor
    cursor_for_next_page = get_cursor(results)
    template_vars = {
        "results": results,
        "cursor_for_next_page": cursor_for_next_page,
    }
    return render_to_response("template.html", template_vars)

#template.html
<h1>Page 1</h1>
<a href="/myview/?cursor={{cursor_for_next_page}}">Next page</a>
{% for object in page1 %}
    {{object}} {#whatever you're displaying #}
{% endfor %}

This is fine when you just want a 'next' link, but what if you want to be able to skip to page 5 or page 10? Then it gets a bit more tricky.
It's time for me to eat some dinner, but if this post gets some comments then I'll expand on this stuff a bit more and reveal ways of making pagination links to other pages (including the previous page).