#googelappengine

[eatdev]

Handling user ratings on App Engine

So far I've been running Django-nonrel on Google's App Engine.  This was due to familiarity with Django, and unfamiliarity with App Engine's Datastore.  I was able to go through about 2 months of work, with a fairly simple database, working pretty well with this setup.  Because the database was simple, I could move between App Engine and a SQL backend.

In the past week, I've refactored some code and added more advanced functionality.  I'm starting to run into Django limitations now.  First issue is pretty common.  App Engine's datastore doesn't support SQL JOINs.  I've worked around it with Django, but it's not a performance optimal version.  I haven't done a perfomance analysis, but maybe I'll do a writeup when I get to that.

I've run into a second issue with App Engine's write frequency limitation.  The proper way to get around it is to shard the item you're writing to.  There's an example from Google for updating a counter frequently.  I'm building a rating system, so I need to show an average value instead of a counter.  You only need to use add_to_average(name, value) on each new rating that's added, where name is a name of the counter (so you can use multiple independent counters), and value is the value you want to add to the average. You can retrieve the average value with get_average(name).

# Copyright ProjectEAT 2011 # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Adapted from Google generalcounter.py # http://code.google.com/p/google-app-engine-samples/source/browse/trunk/sharded-counters/generalcounter.py from google.appengine.api import memcache from google.appengine.ext import db import random import logging class GeneralAverageShardConfig(db.Model): """Tracks the number of shards for each named average.""" name = db.StringProperty(required=True) num_shards = db.IntegerProperty(required=True, default=20) class GeneralAverageShard(db.Model): """Shards for each named average""" name = db.StringProperty(required=True) count = db.IntegerProperty(required=True, default=0) average = db.FloatProperty(required=True, default=0.0) def get_average(name): """Retrieve the value for a given sharded average. Parameters: name - The name of the counter """ average = memcache.get(name+"a") if average is None: sum = 0 divisor = 0 for subavg in GeneralAverageShard.all().filter('name = ', name): sum += subavg.average * subavg.count divisor += subavg.count average = sum / divisor logging.error("get_average from DB got average=" + str(average) + " divisor=" + str(divisor)) memcache.set_multi( { name+"a" : average, name+"c" : divisor }, time=60) return average def add_to_average(name, value): """Update and recalculate the value for a given sharded average. Parameters: name - The name of the average """ config = GeneralAverageShardConfig.get_or_insert(name, name=name) def txn(): index = random.randint(0, config.num_shards - 1) shard_name = name + str(index) average = GeneralAverageShard.get_by_key_name(shard_name) if average is None: average = GeneralAverageShard(key_name=shard_name, name=name) avgsum = average.average * average.count average.count += 1 average.average = (avgsum + float(value)) / average.count average.put() db.run_in_transaction(txn) # does nothing if the key does not exist client = memcache.Client() while True: # Retry loop mcavg = client.get_multi(["a","c"],key_prefix=name,for_cas=True) if len(mcavg) == 0: #Uninitialized average break logging.error("add_to_average from memcache got average=" + str(mcavg["a"]) + " divisor=" + str(mcavg["c"])) avgsum = float(mcavg["a"]) * float(mcavg["c"]) mcavg["c"] += 1 mcavg["a"] = (avgsum + float(value)) / mcavg["c"] retval = client.cas_multi(mcavg,key_prefix=name) if len(retval) == 0: break; def increase_shards(name, num): """Increase the number of shards for a given sharded counter. Will never decrease the number of shards. Parameters: name - The name of the counter num - How many shards to use """ config = GeneralAverageShardConfig.get_or_insert(name, name=name) def txn(): if config.num_shards < num: config.num_shards = num config.put() db.run_in_transaction(txn)