tothinkornottothink-blog - Tumblr blog

tothinkornottothink-blog · 9 years ago

Text

React Native navigation evolution and thoughts on it

When I started coding in React Native, Android support was fresh off the bench and navigation at that time was simple but limited. There were two choices: Navigator and NavigatorIOS. The choice for new comers was clear, NavigatorIOS was limited to iOS as the name implies, it had several bugs and there was an official statement in the docs recommending to use Navigator at that time. However, after some time using Navigator, you could really feel that it was the black sheep of the React Native family. First, its API was imperative, it was quite basic handling nested routes, it was limited in many ways, you had to pass navigator as a prop everywhere, etc.

React Native core team, started soon working on a different approach, that they called NavigatorExperimental. They haven't fully documented it and they warned everyone that the API could change drastically with every new release of RN. Today, you can see how it is being discussed when NavigatorExperimental will turn into maybe simply Navigation and join the stable API lines, deprecating the old fashioned but popular in many projects Navigator.

If you've been following the latest RN releases, you've seen components pop and dissapear (like NavigationAnimatedView while this new navigation system was being refined. It all started with a navigation-rfc that wanted to solve the pains mentioned above. It stablished that this new navigation would be laid out in different components using a flux like architecture:

This way navigation state can be saved to disk

You can use actions/reducers to affect your state

Manage animation between scenes

To name a few from the full list.

But as far as I know, a big part of the community these days is using third party apps built on top of these new "experimental" components. Sure you could use experimental navigation directly, but I guess it's nice to have some syntactic sugar on top that handles the most common situations and doesn't feel like reinventing the wheel. Quoting the README of one of these projects ex-navigation:

NavigationExperimental ships with React Native, it is powerful and flexible, and that comes at the cost of exposing some internals to the app developer. ExNavigation is built on top of NavigationExperimental with the aim of providing a more feature-rich out of the box experience.

One of the most popular projects is react-native-router-flux, which currently has more than 2000 stars in Github. I've been using this project for my latest app, because I had had enough Navigator for a while with my previous experience. The documentation of react-native-router-flux is a little scarce and when you try to do something a little uncommon it's likely that you will find yourself digging within Github issues and the code itself. But it's got some nice features, like having an overview of your scenes and how they nest, having an Actions object that you can import and use anywhere to move between scenes, this object has function names that match your scene keys, so moving between scenes is quite semantic, you can also easily save your navigation stack, or dynamic route things based on your state... although bare in mind many of these are only simple mentions in the docs without code examples.

When I realized how often the NavigatorExperimental API was changing, renaming components, their properties and functioning, I wasn't sure how a project like this could cope with those many changes, that's when I realized that Pavel Aksonov had creeated a clone of ExperimentalNavigation from RN 0.26 in react-native-experimental-navigation which react-native-router-flux sets as a dependency and I'm guessing he is waiting for NavigatorExperimental to settle down to move back into upstream components.

Sure we could talk about many other navigation projects like react-native-router, react-native-navigation (based on react-native-controllers for iOS), ex-navigation (also powered by NavigationExperimental and that has superseded ex-navigator). They all have different approaches, some are implemented natively and others are JS based, but that's another flame.

It's worth mentioning that the people behind Exponent are making the effort to keep ex-navigator up to date with RN upstream and that they are using react-redux under the hood, which is very well known in the RN community.

In my humble opinion, navigation in React Native is still painful, many considered it its weakest part, but it is clear that the core team cares about it and has been long working on fixing this issue and sooner than later we will move onto other itches. Finally, it looks like component navigation with flux like architectures has come to stay, considering all its benefits, so if you haven't you should read a little about it.t

#react-native #react

0 notes

tothinkornottothink-blog · 12 years ago

Text

Raspberry Pi + SimpleCV + OpenCV + Raspicam CSI camera

In this Codemotion Madrid 2013, I'm giving a talk about Computer Vision. It's going to be a beginner's talk, an introduction to high level concepts of artificial vision using SimpleCV, OpenCV and Python. For making this talk more interesting I'm going to develop a software prototype, whose internals I will explain at the end of the talk, putting in play some of the concepts introduced during the talk, so people see a real application of things learned.

In the abstract of the talk, I stated I was going to run this prototype in a Raspberry Pi. I have some experience with the Raspi using USB cameras and compact cameras handled with libgphoto2, but many of these hog Pi's CPU, their framerate is too slow, the camera is too expensive or too big.

We've all heard of Raspberry CSI camera board or Raspicam. It's specs are great and one of its great strengths is that is uses ~2% CPU. That's crucial as I'm going to need every CPU cycle available for analyzing frames looking for patterns. The problem is that the camera is not USB, so I knew it wasn't going to be an easy fit.

I couldn't find a single tutorial on how to make Raspicam work with SimpleCV, only questions about it online. Actually, I only found one for OpenCV, a seven part tutorial by Pierre Raufast, the guy behind Magic Mirror project. The idea of this tutorial is to modify source code of raspivid and raspistill to use buffer memory of the camera to feed OpenCV image objects. This was a great step in making it work, but not an easy one, everyone can follow.

Then I searched for a driver that made things easier, one that made possible recognizing the camera as if it were a USB camera, a V4L2 (Video4Linux2) driver. I knew that if I didn't find one, I may have to code my own and that was going to be a tough task. Luckily I found UV4L (UserVideo4Linux) project, with a recent release announcement for a CSI video driver, great! The author of this driver seems to be nicknamed RpiName, if someone can confirm it I will update the article.

So here is a detailed how-to explaining how to put all the pieces together. We will be using Raspbian, compiling OpenCV and using latest SimpleCV version, it's been a long time since its last PyPi release to be honest. Beware that I've tried to make this the easiest to follow and some hard parts are automated through nice scripts.

We'll need to download Raspbian and record it in our SD, I'm using version released 2013-07-26. I will take for granted you know how to do this step, if not you should check online, there's plenty of tutorials and documentation.

After first boot and going through raspi config program, log in and then run:

sudo apt-get install python-pygame python-scipy python-numpy sudo apt-get install python-pip

After that you can upgrade pip doing:

sudo pip install --upgrade pip

Now logut and then login again (Otherwise pip won't be found in your path, as bash is most likely caching paths). Then run:

sudo pip install virtualenvwrapper

Now edit your .bashrc file using your favorite editor:

vim ~/.bashrc

Add this line to the file:

source /usr/local/bin/virtualenvwrapper.sh

Reload .bashrc file with:

. .bashrc

Now, we are ready to create a virtual environment. We will create one that uses system site packages, because some of the tools that SimpleCV needs, are hard to install or compile and we will be using apt versions installed.

mkvirtualenv --system-site-packages hal

The name of my prototype is "Red Light Hal", it's still a secret what it is about :) come to the talk and if you can't, don't worry, there will be a post about it in the future.

Now we need to download a little script by Jay Rambhia, an active developer of SimpleCV, that publishes great tutorials online. This script we'll do all the heavy lifting for preparing your system for download and compiling OpenCV 2.4.5 in your Raspbian, piece of cake. This is going to take easily 5 hours, so plan ahead:

wget https://raw.github.com/jayrambhia/Install-OpenCV/master/Ubuntu/2.4/opencv2_4_5.sh chmod +x opencv2_4_5.sh ./opencv2_4_5.sh

Now let's install SimpleCV latest version, we'll need git for this:

sudo apt-get install git git clone git://github.com/sightmachine/SimpleCV.git cd SimpleCV python setup.py install

If we now run a simplecv shell and do:

camera = Camera()

We will get this error:

/bin/sh: 1: lsof: not found [0] WARNING: caught exception: SystemError("Cannot identify '/dev/video0': 2, No such file or directory",) WARNING: SimpleCV can't seem to find a camera on your system, or the drivers do not work with SimpleCV.

Now we need to install UV4L CSI driver, there are detailed intructions in this thread, so I won't copy those here.

Somewhere in the internet you will read this driver doesn't support video, well that was when it was released, if you read this Raspberry Pi forum thread, you will see the author state:

The new driver release from the repository also supports H264 video encoding.

From the last release on, the driver allows to control almost all the image properties on the fly, while the camera is in use, without having to reload the driver with new settings. These properties include image size, encoding, hue, saturation, vertical/orizontal flip, iso, etc...

These are great news! After following the steps described in installation instructions, we can try again with a simplecv shell:

SimpleCV:1> camera = Camera() /bin/sh: 1: lsof: not found /dev/video0 does not support memory mapping munmap: Invalid argument munmap: Invalid argument munmap: Invalid argument munmap: Invalid argument Unable to stop the stream.: Bad file descriptor munmap: Invalid argument munmap: Invalid argument munmap: Invalid argument munmap: Invalid argument

Don't worry about those errors, a camera.getImage() we'll work, and that's it! Now you have latest OpenCV and SimpleCV. Your raspicam working and all your CPU power available for Artificial Vision.

Now it's your choice to use OpenCV, SimpleCV or both together, enjoy!

#raspberry #simplecv #opencv #raspicam #python

14 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Django Forms I: Custom fields and widgets in detail

On Thursday September 8th I copresented with Daniel Greenfeld my first talk "Advanced Django Form Usage" at DjangoCon.us 2011. It was my first talk for many reasons. I had never spoken in front of such a large audience before, less in English and this was also my first Python talk. If you haven't seen the video of the talk and the slides, you probably want to, because this post will try to extend, clarify and recap it somehow. Also this will be my official erratum for things that were not exactly true or right that appeared in the slides

Some people really loved the talk and came after it to comment it with me and ask some questions. By that time, I was already thinking that the topic did deserve its own series of posts. Gabriel Hurley, Django committer and documentation specialist, stated:

Apparently some core developers were asking to use the code examples and a transcript from the talk as a starting point for this task. Well, I wasn't sure if the transcript would be good enough, many things were left to say, but I agreed with Gabriel that it's a good starting point to base docs on, which is better than starting from scratch. I still feel this necessity of writing these series of posts, which will hopefully be a better starting point than the transcript. I've been extremely busy since I gave the talk, but now I happen to have some spare time, so let's start.

This first post will start with some advanced things: creating custom form fields, custom widgets and how to get the most out of them. Beware that you need to have a good grasp of Django forms to grok these concepts.

Built-in fields and widgets

Here is a list of available built-in Django form fields and Django widgets. I will cover in the next sections how to create your custom ones, but you should be familiar with these lists, so that you don't end up reinventing the wheel.

Creating custom form fields

In the different Django projects I have come acrossed through my work experience, I haven't seen many that implemented custom form fields. This is probably beacuse docs don't explain this in detail. Django docs have a section called Creating custom fields that claims:

> If the built-in Field classes don't meet your needs, you can easily create custom Field classes. To do this, just create a subclass of django.forms.Field. Its only requirements are that it implement a clean() method and that its init() method accept the core arguments mentioned above (required, label, initial, widget, help_text).

Well, the truth is that you don't need to implement a clean() method to create your custom field and the core arguments at the moment are: required, widget, label, initial, help_text, error_messages, show_hidden_initial, validators and localize.

But before writing your first form field, you should fully understand form validation as it plays an important role on how you write your own fields. The docs have a well documented section for this Form and field validation that you should read before going any farther. However, the first time I got to Django and read it, even though I reread it several times, I didn't quite get the big picture. Not sure why, but I had this feeling that some pieces were missing in the puzzle. Thus, I created for the talk this diagram:

I wanted to show the order of execution of the different methods and add the role that widget plays in this schema. Widgets are part of forms, however they are only mentioned once in the previous section:

> to_python() [...] This method accepts the raw value from the widget and returns the converted value.

I honestly think saying "raw value" here is not completely right, because some widgets: MultipleHiddenInput, CheckboxInput, NullBooleanSelect, SelectMultiple, CheckboxSelectMultiple, MultiWidget, SplitDateTimeWidget and SplitHiddenDateTimeWidget don't return the raw value. They process it and return Python lists or booleans.

Every widget has to have a method named value_from_datadict(data, files, name). data is in fact request.GET/request.POST, files is request.FILES and name is the name of the field to which the widget is attached to. This method usually grabs from data and files what it needs using the name as a key and returns a Python value.

You've probably wondered before what type of value clean(value) or to_python(value) gets. Well, the answer is that it depends on the widget. It can get a string, a list or any Python object. Widgets are the first part of building a custom field, so you need to have this in mind.

The advanced reader is probably thinking, so if the widget can do this, what is to_python supposed to do. Remember that to_python converts a "raw value" into a Python value. Well, borders are sometimes thin, I would recommend you to do your main conversion in to_python and code your widgets as data agnostic as possible. As a rule of thumb, you shouldn't use any non-standard Python datatype in a widget.

Subclassing Field

Subclassing Field has some implications that you should be aware of. Field has the following class-level variables that you should know about:

widget: Default widget to use for rendering the field, defaults to TextInput.

hidden_widget: Default widget to use when rendering this as "hidden", defaults to HiddenInput.

default_validators: Default list of validators to be run, defaults to an empty list.

default_error_messages: Default error messages. Defaults to {'required': _(u'This field is required.'), 'invalid': _(u'Enter a valid value.')}

As you probably imagine, class-level variables are used to define defaults in a field.

First custom field: AMarkField

Your custom fields should live in a file named fields.py. Let's create a form field, that only accepts an uppercase "A". We will be subclassing forms.Field, this gets us most of the dirty work done.

class AMarkField(forms.Field): default_error_messages = { 'not_an_a': _(u'you can only input A here! damn!'), } def to_python(self, value): if value in validators.EMPTY_VALUES: return None if value != 'A': raise ValidationError(self.error_messages['not_an_a']) return value

If you look carefully we are overwriting the class-level variable default_error_messages. This means we don't have required or invalid error messages. We are not handling extra parameters in our AMarkField so there is no need to overwrite the constructor. As we didn't specified anything different, by default we are using TextInput as the widget, so we get a string as value. If the string is not an uppercase we raise a ValidationError using self.error_messages['not_an_a']. self.error_messages is a dictionary that contains a merge of self.default_error_messages and error_messages, an optional dictionary that is used to override the default messages that the field will raise. An example would make this clearer:

class ExampleForm(forms.Form): mark = AMarkField(error_messages = {'not_an_a': "Only A please"})

This a basic form that uses AMarkField and is using error_messages to pass an overriden message for not_an_a. That's why we use self.error_messages, to display overriden messages if there are. The creation of this dictionary is done in the constructor of forms.Field so you don't have to worry about that. If you happen to overwrite the constructor don't forget to call the parent constructor too.

Some picky readers will have noticed that I'm doing validation in to_python. The only time you should raise a ValidationError in to_python is when you are coercing the value and an exception occurs. So the right way to do AMarkField would be:

class AMarkField(forms.Field): default_error_messages = { 'not_an_a': _(u'you can only input A here! damn!'), } def to_python(self, value): if value in validators.EMPTY_VALUES: return None return value def validate(self, value): if value != 'A': raise ValidationError(self.error_messages['not_an_a'])

Note that EMPTY_VALUES is basically the tuple (None, '', [], (), {}).

What a custom field means

In a custom field you can wrap validation and Python coercion of any kind. This is the best practice if you are handling the same unit of information in several forms, what you shouldn't be doing is this:

class MyModelForm(forms.ModelForm): mark = CharField() def clean_mark(self): mark_value = self.cleaned_data['mark'] if mark_value is not None and mark_value != 'A': raise ValidationError(_(u'you can only input A here! damn!')) return mark_value class ExampleForm(forms.Form): mark = CharField() def clean_mark(self): mark_value = self.cleaned_data['mark'] if mark_value is not None and mark_value != 'A': raise ValidationError(_(u'you can only input A here! damn!')) return mark_value

clean_ should be used for validation of a field in the specific context of that form. In the example above we are repeating validation logic, violating the famous Django DRY principle. For sure there are ways to make this look more DRY, like wrapping that logic in a function that gets called. But you will agree with me, that these are mere hacks.

If you construct your fields the right way, you will rarely find a use case in which you need to write a clean_. Normally it is used to avoid creating a form field for validation that is not trivially expressed using validators.

Another example: JSONField

Imagine we want to have a form field that returns a JSON validated list. Here it is:

class JSONField(forms.Field): default_error_messages = { 'invalid': 'This is not valid JSON string' } def to_python(self, value): if value in validators.EMPTY_VALUES: return [] try: json = simplejson.loads(value) except ValueError: raise ValidationError(self.error_messages['invalid']) return json

This time we are raising a ValidationError in the right method, because we are handling an exception while doing the coercion of the value.

Adding widgets to the equation

The same we have custom fields, we can have custom widgets and both concepts together turn Django forms into a powerful weapon. Your custom widgets should live in a widgets.py file in one of you apps that compose your project.

In widgets you will usually subclass Widget, Input or MultiWidget. Widget is the baseclass for the rest of the built-in widgets available in Django. Therefore Input and MultiWidget are already subclasses of Widget.

Subclassing Widget

You will rarely will find yourself in a situation in which you need to subclass Widget itself. There are other baseclasses that inherit from it, that will do the job.

Anyway, what you need to know when you subclass Widget is that you will have to overwrite the render(name, value, attrs=None) method, which is in charge of returning a string that represents the widget rendered as HTML. Also, if you overwrite the constructor in your subclass you should handle attrs parameter and pass it to the parent.

The class-level variables that you should know that are attributes of Widget are:

is_hidden = False: Determines whether this corresponds to an .

needs_multipart_form = False: Determines if the widget needs multipart-encrypted form

is_localized = False: Determines if the widget is localized.

is_required = False: Dertermines if the widget is required.

Subclassing Input

If you think it carefully most widgets should render <input />, textarea or similar things. If you need to render a customized <input />, you should know that Input is already a subclass of Widget so you get more dirty work done. Input defines a class-level variable named input_type that is used for specifying the type of input <input type="BLABLA" />.

When subclassing Input we get already a render method that will suffice most of the times. Sometimes you will want to add some logic, before calling the parent class render method. The next example is quite basic, but you will probably get the idea of how to subclass Input:

class HTML5Input(Input): def __init__(self, type, attrs): self.input_type = type super(HTML5Input, self).__init__(attrs)

What we are doing here is adding a parameter type that will overwrite class-level variable input_type. This way I'm creating some sort of universal configurable input, therefore now we can use this and do things like:

class ExampleForm(forms.Form): user_email = forms.EmailField( widget=HTML5Input(type='email', attrs={'class': "emailFields"}) )

This way we will get a fancy HTML5 <input type="email" class="emailFields" /> for our EmailField. Also note that I'm passing an attrs dictionary, that is used for rendering the widget. We will see more on this later. But the dictionary key class turns into the class of the input. This is a typical beginners questions too, how do you set a custom CSS class for your widget?

Of course we could have created our own specific EmailInput widget doing:

class EmailInput(Input): input_type = 'email'

Anyway, If you are looking for a set of HTML5 widgets that rock, you should check django-floppyforms, a Django app by Bruno Renié.

Subclassing MultiWidget

MultiWidget is a baseclass for creating composed widgets. First thing you need to understandard is that there is not a 1:1 relation between fields and widgets. You can have a form field, that uses a MultiWidget subclass as its widget. Hopefully you will fully understand what this means by the end of the next example.

Imagine you create an AddressField and you want that field to get the values of several inputs in your form. You need to create a widget that renders several inputs, for example: street, number and zipcode. This is done subclassing MultiWidget.

MultiWidget is constructed using a tuple/list of other widgets. Imagine we call the widget for our field, AddressWidget. AddressWidget could be built using (TextInput, TextInput, TextInput) or (StreetInput, TextInput, ZipCodeInput), if you happen to have those custom fields StreetInput and ZipCodeInput in your app.

When subclassing MultiWidget, there are two main things you need to do:

Create a constructor that passes a widgets parameter to the parent's class constructor. You can name your variable what you want, I use widgets in my example, but remember it needs to be a tuple or a list.

You have to implement decompress(value) method.

What's decompress for?

Every widget in the list of widgets used for building AddressWidget expects a value. Those values are obtained using a list that maps 1:1 to the list of widgets. decompress(value) method is called when value is a string, so it should return a list that maps values to widgets. Why this would happen? Well, when you create a form instance, using initial or instance.

form = ExampleForm(request.POST, initial={'address': "Imaginary Road,2,2039"})

This mean we have a form called ExampleForm which has a form field named address and suppose it's using AddressWidget, so it gets a value "Imaginary Road,2,2039". Suppose we are using commas to handle the separation of fields in the string. Then decompress should parse this string and return ['Imaginary Road', '2', '2039']. MultiWiddget baseclass code will handle the assignment of those values to the different widgets:

StreetInput

Handling render in MultiWidget

In the previous output you probably imagined something like this, right:

Well, for that we currently have three options. I recommend you read the three of them in order:

1. The hard one but customizable

You will have to overwrite render method. Forget about using parent class render, because it returns a string and that string is generated using inline Python. That would mean calling parent render and injecting our labels in the string. Trust me that is not worth the time and effort. I'm showing the code of a render method base on MultiField.render that adds some html labels. This is the code block I crafted for DjangoCon.us presentation. You should not do it this way, unless you are sure about what you are doing.

Please don't panic, I'll explain it in detail right away and I will show you a couple easier ways to do this :)

def render(self, name, value, attrs=None): # HTML to be added to the output widget_labels = [ '<label for="id_%s">Address: </label>', '<label for="id_%s">Number: </label>', '<label for="id_%s">ZipCode: </label>' ] if self.is_localized: for widget in self.widgets: widget.is_localized = self.is_localized # value is a list of values, each corresponding to a widget in self.widgets if not isinstance(value, list): value = self.decompress(value) output = [] final_attrs = self.build_attrs(attrs) id_ = final_attrs.get('id', None) for i, widget in enumerate(self.widgets): try: widget_value = value[i] except IndexError: widget_value = None if id_: final_attrs = dict(final_attrs, id='%s_%s' % (id_, i)) output.append(widget_labels[i] % ('%s_%s' % (name, i))) output.append(widget.render(name + '_%s' % i, widget_value, final_attrs)) return mark_safe(self.format_output(output))

I have added a widget_labels list that contains the labels we want to add in front of the each input. The next three lines set the is_localized attribute of the widgets that compose the AddressWidget if AddressWidget.is_localized is set. If value is not a list, we decompress it to get a list. build_attrs is a Widget method that builds a dictionary out of two: the local attrs and self.attrs (the one passed when instantiating the widget). Using that, I iterate over self.widgets, picking their corresponding value and adding to output pairs of labels and the html rendered by widget. After iterating over all the widgets, I return the output marked as safe and formatted. I will explain what is format_output for later.

Easy? Well, probably not, depends on how familiar you are with Python/Django. Sure this is tedious, not DRY and not elegant. This is probably not what you expected from Django, right? Well, that's because this is not the right way to do it. The reason I've explained this in so little detail is that I needed to state this very clear.

2. Easier but more rigid

If you looked at the previous code MultiWidget has a method named format_output that is called when returning output. output is a list of rendered widgets html as strings. So you could overwrite format_output and inject your labels in that list. Problem is that those labels have to be quite fixed or you need to do some parsing out of the corresponding rendered widgets to extract ids and so on.

def format_output(self, rendered_widgets): """ Given a list of rendered widgets (as strings), returns a Unicode string representing the HTML for the whole lot. This hook allows you to format the HTML design of the widgets, if needed. """ rendered_widgets.insert(0, "<label for="id_address_field_0">Street:</label>) rendered_widgets.insert(2, "<label for="id_address_field_1">Number:</label>") rendered_widgets.insert(4, "<label for="id_address_field_2">Zip Code:</label>") return u''.join(rendered_widgets)

I think the code is easy to follow. You probably agree with me, this is the right way to do it even though you might have to add some parsing to insert customizable strings in rendered_widgets.

3. Third one, using MultiWidgetLayout

As you may know, I'm the lead developer of django-uni-form, so I decided to borrow some of its concepts, creating an alternative to MultiWidget, named MultiWidgetLayout.

The difference is that you can control the rendering of your widget using a simple layout, which is a list of strings and widgets. So you don't have to specify widgets aside. Using it AddressWidget code turns into this:

from django.utils.datastructures import SortedDict class AddressWidget(MultiWidgetLayout): def __init__(self, attrs=None): layout = [ "<label for="%(id)s">Street:</label>", TextInput(), "<label for="%(id)s">Number:</label>", TextInput(), "<label for="%(id)s">Zip Code:</label>", TextInput() ] super(AddressWidget, self).__init__(layout, attrs) def decompress(self, value): if value: return value.split(",") return [None, None, None]

This will get you the exact same results. Where can you get MultiWidgetLayout? Well, I've created a ticket in Django (#16959) to see if the concept is accepted into Django itself, meanwhile you can grab the code from django-multiwidgetlayout.

Custom fields and widgets both together

If you didn't get lost in this path you may want to know how do you play both together. Remember the diagram I showed you way above? Widgets return Python values and fields have a to_python function that can turn those into different ones. Want to see an example of what this means?

1. AddressField with AddressWidget

This time we play together a custom field and a custom widget and this is what happens:

fields.py

class AddressField(forms.Field): self.widget = AddressWidget def to_python(self, value): # Already gets a Python list return value

forms.py

from fields import AddressField class ExampleForm(forms.Form): address = AddressField()

Our cleaned_data for this field would be a Python list: ["SW Sixth Avenue", "921", "Portland"]

2. AddressWidget with a built-in field

If we for example use built-in field CharField with AddressWidget:

forms.py

class ExampleForm(object): address = forms.CharField(widget=AddressWidget)

Our cleaned_data for this field would be a Python string: "['SW Sixth Avenue', '921', 'Portland']" Note that this is a string that represents a list, but still a string.

A Field for MultiWidget: MultiValueField

Django developers thought that probably you would like to have the same concept of MultiWidget for fields, creating the baseclass MultiValueField. MultiValueField is a field composed of several form fields. So now imagine you have your AddressWidget that returns the previous list ["SW Sixth Avenue", "921", "Portland"] and you want each value of the list to be validated using fields you have available. It's easier than you may think, you would do:

class AddressField(forms.MultiValueField): widget = AddressWidget def __init__(self, *args, **kwargs): fields = (forms.CharField(), forms.IntegerField(), forms.CharField()) super(AlternativeAddressField, self).__init__(fields, *args, **kwargs) def compress(self, data_list): return data_list

This means that "SW Sixth Avenue" would be validated as a CharField. But maybe you have bigger plans for this. You could create a StreetField that geolocalized the data in validate() and raised errors if found or returned an instance of a class Spot otherwise. Now you have plenty of possibilities and aces up your sleeve. "921" would be validated as an IntegerField and so forth.

One thing you have to do when subclassing MultiValueField is implementing compress. This method is called before returning cleaned_data in clean(). The comments in this method say:

> Returns a single value for the given list of values. The values can be assumed to be valid. > For example, if this MultiValueField was instantiated with fields=(DateField(), TimeField()), this might return a datetime object created by combining the date and time in data_list.

Well, this is not completely true in my humble opinion. You might want to return the list as it is and then handle it as you want later or maybe you do want to compress it. In the example above I just simply return the list as it is: ["SW Sixth Avenue", 921, 'Portland'].

Note that we not only validated the inputs but we cleaned and coerced them using form fields, so now we have a list of [string, integer, string], neat right?

More fields and widgets

Django core developer Carl Meyer put together a Django app named django-form-utils that has some nice extra form fields and widgets that you might be interested in.

Conclusions

Hopefully if you have read the whole article, I will have clarified some of the stuff we went over in the talk quite fast. I cannot promise when next post will come out as I'm busy lately, but I will try to get it out soon.

If you find any bugs, misexplained things or anything else, please comment it below and I will fix it asap. Also feel free to post your opinion or comments.

If you like what I write or work on, you can follow me on Twitter or Github.

#django #forms #django-uni-form #django-floppyforms #python #fields #widgets #MultiWidget #MultiWidgetLayout #JSONField

51 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Django-uni-form 0.9.0 is out [security fix]

Version 0.9.0 is out and you should update because of security reasons. An XSS bug has been fixed, thanks to Charlie Denton for reporting it. If you are using django-uni-form and a form field that renders the input of a field as part of the error message without sanitizing it, such as ChoiceField, you are vulnerable to it. This is because errors are rendered using |safe filter.

This has been addressed as fast as possible. Fixes for previous versions might come in the soon future. PyPi package has been update so you can do:

pip install --upgrade django-uni-form

This is the commit that introduced the security bug, as you can see it affects version 0.7.0 and forward versions.

I want to thank every contributor and user that has made this version rock even more than previous ones.

Bug fixes

It's got support for show_hidden_initial, Fieldset's legends internationalization, error_css_class and required_css_class. MultiField layout object has fixed markup and has been optimized.

Performance boost

The project has seen several performance tune ups, related to templates handling, that make it run blazing fast. You can run now django-uni-form without template caching only one second slower than the cached version.

Customizable templates

All layout objects have now an attribute named template that points to a template file. This template is in charge of how the rendering of the layout object. Using you custom templates for layout objects is not a hassle anymore.

Improved docs

The docs are on their way to be easier to grok for new comers. New sections have been added that explain some of the new 0.9.0 features and some of the most hidden features of django-uni-form. It's got now more examples and use cases, that will hopefully resolve some of the most seen questions. For better readability we are now using kenneth Reitz kr sphinx themes, based on Flask themes, for read the docs.

Layout inheritance

If you have a layout shared by many forms, you will be able to do some neat layout inheritance with it. That will help you being more DRY.

Better organization

helpers.py file has been divided into three different files: helper.py, layout.py and utils.py. You can still import everything using helpers, but a deprecation warning has been added, so that you can start moving to the new importing structure. helper holds FormHelper class, layout holds the Layout class and layout objects, while utils holds render_field, an internal function used for rendering your django-uni-form layouts.

More tests and better test coverage

This is the only way we will be able to mantain the quality of this project and keep up the pace of faster release cycles.

#django #python #django-uni-form #0.9.0 #uni-form #forms #layout #DRY #layouts

3 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

New kung-fu in django-uni-form 0.8.0

This won't be a new official release announcement as Daniel already did the honors. I'm now the co-lead developer of django-uni-form and in this post I will try to go over some examples that teach you new kung-fu techniques. But before doing so, let me emphasize how cool this project is and how many new nice features are packed in the new version 0.8.0. If you like DRY principles and don't think your Django forms are DRY enough using built-in filters, you should check it! if you haven't yet.

To explain some of the new features, let's use a real form and a real example, so it's easier to follow how django-uni-form works and what has been enhanced.

The following form will be used from now onwards to exemplify the new features in django-uni-form. I'm using different types of form fields to demonstrate how django-uni-form can work with any built-in widget or form field or any other custom field that you have defined. That is, django-uni-form plugs well with other form apps.

from django import forms class ArticleFeedbackForm(forms.Form): like_example = forms.TypedChoiceField( label = _("¿Do you like this article?"), choices = ((1, "Yes"), (0, "No")), coerce = lambda x: bool(int(x)), widget = forms.RadioSelect, initial = '1', required = True, ) reason = forms.CharField( label = "Why did you like it?", max_length = 80, required = True, ) rating = forms.IntegerField( label = "Rating (from 0 to 10)", min_value = 0, max_value = 10, required = False, ) notes = forms.CharField( label = "Additional notes or feedback", max_length = 200, widget = forms.Textarea, required = False, )

To load django-uni-form template tags in our template we will have to add:

{% load uni_form_tags %}

Django-uni-form still has |as_uni_form filter, which is used exactly the same way. The only difference is that it now supports rendering formsets. This is how you would use it, supposing we have the form in the template context as example_feedback_form:

<form ...> {{ example_feedback_form|as_uni_form }} </form>

In my opinion, the real power in django-uni-form comes from its {% uni_form %} tag and being able to set layouts. But let's go step by step to avoid confusion.

Django-uni-form implements a class called FormHelper which is attached to the form. This class defines the form rendering behavior.

Let's add to our form an instance-level form helper variable:

from django import forms from uni_form.helpers import FormHelper class ExampleFeedbackForm(forms.Form): like_example = forms.TypedChoiceField( label = _("¿Do you like this example?"), choices = ((1, "Yes"), (0, "No")), coerce = lambda x: bool(int(x)), widget = forms.RadioSelect, initial = '1', required = True, ) reason = forms.CharField( label = "Why did you like it?", max_length = 80, required = True, ) rating = forms.IntegerField( label = "Rating (from 0 to 10)", min_value = 0, max_value = 10, required = False, ) notes = forms.CharField( label = "Additional notes or feedback", max_length = 200, widget = forms.Textarea, required = False, ) def __init__(self, *args, **kwargs): self.helper = FormHelper() self.helper.form_id = 'example_feedback_form' return super(ExampleFeedbackForm, self).__init__(*args, **kwargs)

Now that we have a helper, we can use helper methods to set how the form should render. In this case I've just set the form DOM id to 'example_feedback_form'.

New FormHelper attributes

Version 0.8.0 introduces new attributes that can be set in a helper:

form_error_title: If you are rendering a form using {% uni_form %} tag and it has non_field_errors to display, they are rendered in a div. You can set the title of the div with this attribute. Example: “Form Errors”.

formset_error_title: If you are rendering a formset using {% uni_form %} tag and it has non_form_errors to display, they are rendered in a div. You can set the title of the div with this attribute. Example: “Formset Errors”.

form_style: If you are using uni-form CSS, it has two different form styles built-in. You can choose which one to use, setting this variable to "default" or "inline".

Creating a Layout

Django-uni-form defines another powerful class called Layout. You can create your Layout to define how the form fields should be rendered. With "how" I mean things like the order of the fields, wrap them in divs or other html structures with ids or classes set to whatever you want, add things in between, etc. And all that without writing a custom template, rather fully reusable without writing it twice.

A Layout is constructed by what I like to call layout objects, which can be thought of as form components. You assemble your layout using those. For the time being, your choices are: Button, ButtonHolder, Column, Fieldset, HTML, Hidden, MultiField, Reset, Row and Submit.

All these components are explained in helper API docs. What you need to know now about them is that every component renders a different template. Let's write a couple of different layouts for our form, continuing with our form class example (note that I'm not showing the full form again):

from uni_form.helpers import Fieldset, Submit (...) def __init__(self, *args, **kwargs): self.helper = FormHelper() self.helper.form_id = 'example_feedback_form' layout = Layout( Fieldset('The first arg is the legend of the fieldset', 'like_example', 'reason', 'rating', 'notes', ) ) self.helper.add_layout(layout) self.helper.add_input(Submit('register', 'Register')) return super(ExampleFeedbackForm, self).__init__(*args, **kwargs)

Now if we want to render our form example_feedback_form using what we've set in the helper and our layout, we would do in the template:

{% uni_form example_feedback_form example_feedback_form.helper %}

This is exactly the form that we would get rendered using the previous tag:

<form action="." class="uniForm" method="post" id="example_feedback_form"> <fieldset> <legend>The first arg is the legend of the fieldset</legend> <div id="div_id_like_example" class="ctrlHolder "> <label for="id_like_example" class="requiredField"> ¿Do you like this article?<span class="asteriskField">*</span> </label> <ul> <li><label for="id_like_example_0"><input checked="checked" name="like_example" value="1" id="id_like_example_0" type="radio" class="radioselect" /> Yes</label></li> <li><label for="id_like_example_1"><input value="0" type="radio" class="radioselect" name="like_example" id="id_like_example_1" /> No</label></li> </ul> </div> <div id="div_id_reason" class="ctrlHolder "> <label for="id_reason" class="requiredField"> Why did you like it?<span class="asteriskField">*</span> </label> <input id="id_reason" class="textinput textInput" type="text" name="reason" maxlength="80" /> </div> <div id="div_id_rating" class="ctrlHolder "> <label for="id_rating" > Rating (from 0 to 10) </label> <input id="id_rating" type="text" name="rating" class="textinput textInput" /> </div> <div id="div_id_notes" class="ctrlHolder "> <label for="id_notes" > Additional notes or feedback </label> <textarea id="id_notes" class="textarea" rows="10" cols="40" name="notes"></textarea> </div> </fieldset> <div class="buttonHolder"> <input type="submit" name="register" value="Register" class="submit submitButton" id="submit-id-register"/> </div> </form>

Note that every field is wrapped by default in a div with an id. This makes it very easy to style any field using CSS.

Buttons are now Layout objects

Now, I will show another layout example that would go in the same place shown before:

from uni_form.helpers import Fieldset, Submit, Row, ButtonHolder, HTML (...) Layout( Row( HTML("""<p>Did you know that some layout objects can hold others within?</p>""") 'like_example', 'reason', ), MultiField("Additional information", 'rating', 'notes', ), ButtonHolder( HTML(""" {% if message %} <span class="notification">{{ message }}</span> {% endif %} """), Submit('save', 'Save'), ) )

You can wrap fields in divs, like Row. MultiField is a layout object that comes from a uni-form adaptation: instead of rendering errors surrounding inputs in boxes, it renders them in a list and boxes all inputs affected. Sometimes the best way to familiarize yourself with these layout objects is getting your hands dirty writing and rendering some layouts to see what they do. They are harder to explain than to understand once you see them working.

Note that HTML is a layout object that can be used to input arbitrary HTML code in the layout.

In this version of django-uni-form you can have buttons (Button, Hidden, Reset and Submit) as layout objects. The recommended way of doing so is having them wrapped in a ButtonHolder which renders to a <div class="buttonHolder">, that is, a div that uni-form CSS magically knows how to position.

Django-uni-form is more context-aware

In the last example, you've seen how HTML can now be part of a Django template while being fully context-aware, which means that it's got access to the full context of the template in which it's being rendered. This gives you some really nice extra power that I find very useful.

However, not only HTML is context-aware. Fieldset legends behave the same way:

Fieldset("Please fill in your data {{ user.username }}", 'like_example', [...] )

Setting DOM ids and classes in Layout objects

Now you can set DOM ids and classes in every layout object, using css_id and css_class. Let's see a couple of examples:

Submit('Save', 'Guardar', css_class='button white') Fieldset("legend", 'field1', 'field2', css_id="iLoveThisFieldset", css_class="fieldsets_that_rock" )

Also, the ids and classes are left untouched; they are not slugified anymore so you can use camelCase or whatever you want.

Formsets support

Now you can render formsets using django-uni-form. Let's see how:

from django.forms.models import formset_factory ExampleFeedbackFormSet = formset_factory(ExampleFeedbackForm, extra = 3) formset = ExampleFeedbackFormSet()

You can use {% uni_form %} tag the same way as with a form:

{% uni_form formset formset.form.helper %}

Note that you can still use a helper (in this case we are using the helper of the form used for building the formset). The main difference here is that helper attributes are applied to the form structure, while the layout is applied to the formset's forms. Rendering formsets injects some extra context in the layout rendering so that you can do things like:

HTML("{% if forloop.first %}Message displayed only in the first form of a formset forms list{% endif %}", Fielset("Item {{ forloop.counter }}", 'field-1', [...])

Django-uni-form fails silently

When django-uni-form encounters errors it failed in a very intrusive way. Now, by default, it will log errors and continue working if possible. A settings variable called UNIFORM_FAIL_SILENTLY has been added so that you can control this behavior. If you want django-uni-form to raise errors instead of logging, telling you what's going on when you are developing in debug mode, you can set it to:

UNIFORM_FAIL_SILENTLY = not DEBUG

Help texts are wrapped in divs

This means that in your form fields you can set your help_text to things like:

help_text = "<ul><li>First help note</li><li>Second help note<li></ul>"

Layouts and Layout objects are now dynamic

The first steps have been taken to make adapting a Layout or a Layout object on the go possible, so that django-uni-form plays nice with dynamic form generation. Version 0.8.1 will hopefully come with an API that makes handling this a breeze.

Conclusions

Django-uni-form is in active development. We are aiming to build the DRY-est form building solution for Django. It comes with better docs in RTD, better error messages, a new full testing suite with high coverage and new code carefully cleaned and optimized for performance.

We are already working on version 0.8.1 which will be released in less than a month from now, packed with new killer features.

Special thanks go to Javier Maestro for reviewing my writing in this article.

#django #django-uni-form #uni-form #python #0.8.0 #DRY #forms #layout

2 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Be careful how you use static variables in forms

Working on django-uni-form I came across with a weird situation and it took me some time to figure out what was going on. I wouldn't say it was a bug, it was more a misuse or a documentation problem. After that, I realized other coders make the same mistake that was affecting us, so I thought about writing an article for warning Python/Django programmers.

If you are building a Django application in which you add attributes to a form, that are not form fields, like in this example:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) type_of_form = 'dangerous'

You should know that every attribute that is not a form field, such as type_of_form, is static or a class-wide variable. Let's see what this means and please stay with me till the end, don't switch channel. If you handle a form instance in one of your views and change type_of_form value:

form = ExampleForm(request.POST) form.type_of_form = "secure"

Then type_of_form turned into an instance-level variable. Which means only that specific form instance has a type_of_form set to 'secure'. You can still access the class wide variable, using the class name:

>>> ExampleForm.type_of_form 'dangerous'

Things get complicated if we turn type_of_form into a list. For example:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) type_of_form = []

If we work with the list without instantiating a new one, the list behaves like a Singleton, there is only one instance of that object. Now imagine someone posts the form once and we do:

form = ExampleForm(request.POST) >>> id(form.type_of_form) 140281019424640 form.type_of_form.append(1)

type_of_form is now the list [1] for this and every future object we instantiate. So if a second person posts to the same view:

form = ExampleForm(request.POST) >>> id(form.type_of_form) 140281019424640 >>> form.type_of_form [1]

If you see type_of_form is not an empty list, it's not instantiated again, we share the same object than before in memory, it persists. If we manually create a new list:

form.type_of_form = []

In that case we have created a class-level variable, but the class wide variable still exists and hasn't changed:

>>> ExampleForm.type_of_form [1]

Remember that next time we instantiate a form, unless we manually change the variable for a new object, type_of_form will be [1], It persists

form = ExampleForm(request.POST) >>> id(form.type_of_form) 140281019424640 >>> form.type_of_form [1]

Some of you, might be thinking, but everything in Python is an object, true. The only difference here is that if type_of_form is a string, as strings are immutable in Python, every time we manipulate it we create a new string, a new instance-level variable. But with a list or any other object, it's very easy to get tempted to manipulate the class-wide variable, without instantiating a new one, if done without knowing what we do, it will provoke side effects.

Of course, this not a Django bug, this is how Python classes like Django forms work. The problem is many coders use Django forms this way without being aware of what they do. Therefore when doing this in your open source application, try to be clear in your docs and use static variables where due.

Real example

This is the way Django-uni-form advised to add helpers to a form:

class ExampleForm(forms.Form): [...] helper = FormHelper() helper.form_id = 'publishTransportSecondStep'

That made it easy for someone to manipulate the helper the wrong way, doing:

def view(request): # Create the form form = MyForm() helper.add_input(Reset('reset','reset button'))

See the problem? Now we have added to helper, which behaves like a Singleton, a new button. That button will be shown in every form of MyForm type we create. Every time we go into the view, a new button will be added, so the form will end up having tons of buttons. Thus we changed the advised way of adding helpers to forms.

This way, if we manipulate the helper in a form, we do it for that specific form helper. But sometimes people don't manipulate helpers in views, or if they do, they do it the right way, thus, some people might want to have helpers as static variables, for improving performance and memory footprint.

Solution

Conclusion is that you should use static variables carefully and in the situations they should be used, always understanding what you do. If you don't want a variable to be static, wrap it in the constructor and turn it into an instance-level variable:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) def __init__(self, *args, **kwargs): self.type_of_form = [] super(ExampleForm, self).__init__(*args, **kwargs)

Note that turning type_of_form into a property will not solve the problem:

class ExampleForm(forms.Form): description = forms.CharField( label = u"Short description", max_length = 40, ) @property def type_of_form(self): return []

If you do something like this:

form.helper.append(1) form.helper.append(2) >>> form.helper []

In this example, every time you call form.helper you get a new list, a new object in memory, so changes don't persist. So wrapping a variable in a property should not be used as an alternative to instance-level variables.

Thanks to Jonathan Stoppani for his corrections and help.

#django #forms #python #static

16 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Using jQuery templating ICanHaz.js with Django

When working on a Django project with an important part of the user experience based on AJAX, you better use a frontend templating language. The same way Django has a templating language that we use to populate values easing your coder's life, jQuery has client side alternatives. One of these plugins and my personal favorite is ICanHaz.js created by @HenrikJoreteg. It's only 4.4 Kb.

ICanHaz.js syntax will be very familiar to Django devs, because a variable is defined using double curly brackets {{ variable }}. But I will not enter into details, the project has very easy to follow setup instructions and code examples.

When people start using ICanHaz.js in their Django templates, they have a very hard first time trying to get it to work. The problem is simple, Django and ICanHaz.js share the same variable definition, so Django interprets the script templates, breaking them.

When I found this out, I looked over the Internet until I found a fix by Eric Florenzano (@ericflo). He coded a templatetag called verbatim, that avoids that Django interprets all content within the block tag. He published it as a gist in Github. You simply have to put this within one of your app's templatetags directory. The way you use it in a template:

{% verbatim %} <script id="user" type="text/html"> <li> <p class="name">Hello I'm {{ name }}</p> <p><a href="http://twitter.com/{{ twitter }}">@{{ twitter }}</a></p> </li> </script> {% endverbatim %}

simple, right? Well, I recently work on an enhancement for this gist. Eric Florenzano's version of the verbatim tag escapes everything, which means that you can't use tags like {% url %} {% csrf_token %} or others. These are tags and not variables, so I didn't see any reason for not having them around. Therefore I forked Ericflo's gist and added tags support. Now you can do:

{% verbatim %} <script id="user" type="text/html"> <a href="{% url addItem %}">Add Item</a> </script> {% endverbatim %}

Working on this wasn't as easy as other tags I have worked on. I had to go inside the guts of django.template.defaulttags and django.template.base (former __init__.py in Django 1.2) to understand how the Lexer, Token, Parser, Filter/Tag, Node, NodeList, Template structure in Django works. I couldn't just simply parse the Tokens and build a NodeList, I had to write my own template parser and I based my code on Parser.parse. Pretty much I extracted the bits I needed from there.

#django #jquery #template #icanhaz.js

2 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Django PositiveNormalizedDecimalField

Some weeks ago I was working on a Django project that was using some FloatFields where they should not be used, the database backend was Postgres 8.4. If you are not aware of it, FloatFields are turned into double precission types, that means 64-bit floating-point numbers. Those fields were storing values that didn't need such precission. For example I don't think you need them to store items' weights, do you?

I migrated those fields to DecimalFields using an average (max_digits=7, decimal_places = 2), which means a total of 7 digits, with 2 decimals. Of course I had to migrate the database, doing some manual alters, as Django-south was not being used. By default DecimalFields render the number with all its decimal digits, which was not what I was looking for. I needed a decimal field that:

By default render decimal values normalized using non-scientific notation. See Decimal("10.400").normalize()

Only allowed positive values

I could have used floatformat built-in filter, but that would have supposed to change hundreds of different templates. Even though grep and sed are a big allies, the process would be quite error prone. I simply needed a centralized solution. Obviously It was time to create a custom model field, I decided to call it PositiveNormalizedDecimalField. I started reading Writing custom model fields, which gave me a good idea of how things work.

I decided to start subclassing DecimalField and rendering values normalized. I started overwritting to_python method. From the docs I understood to_python was in charge of turning the value stored in the database into a python object. But things were not working. This is the code I had at that time

class PositiveNormalizedDecimalField(models.DecimalField): def to_python(self, value): if value is None: return value try: # we want to return a normalized non-scientific notation decimal return decimal.Decimal('{0:f}'.format(decimal.Decimal(str(value)).normalize()))

However, to_python wasn't working as django docs prayed. So I thought that maybe Django had a bug, honestly I was pretty much sure I was doing something wrong. So I got into Django's code and put some breakpoints using pdb, don't do this in your production server. Then I found out that to_python was not even being called, damn! So I filled a ticket for documentation improvement quite angry after loosing some hours on this.

Thanks to Gabriel Hurley I found out that I needed to use the metaclass SubFieldBase to have to_python being called. That finally made the trick and I got it working. Here is the final code:

class PositiveNormalizedDecimalField(models.DecimalField): __metaclass__ = models.SubfieldBase default_error_messages = { 'positive': _('Introduce a positive number.'), } def to_python(self, value): if value is None: return value try: # we want to return a normalized non-scientific notation decimal number = decimal.Decimal('{0:f}'.format(decimal.Decimal(str(value)).normalize())) # This ensures no negative number can get to the model (model validation) if number < decimal.Decimal(0): raise exceptions.ValidationError(self.error_messages['positive']) return number except decimal.InvalidOperation: raise exceptions.ValidationError(self.error_messages['invalid'])

After talking to some developers in the django IRC channel I realized adding a field like this into the core was not going to make it. So instead I'm publishing it here and I might be releasing a Django-app in the future with some more model fields.

Some Built-in fields such as PositiveIntegerField have a database CHECK constraint to allow only positive numbers, that constraint is located at django/db/backends/*.py This could be another option to check positive values, however modyfing Django's code without using official accepted patches is not a recommended option.

This is also a good example in any case you need to redefine default rendering behavior of a model field. Bare in mind, that Django use of to_python is a little bit inconsistent. So when doing aggregate queries to get minimums or maximums, you will need to filter output manually.

#decimalfield #django #field #model #positivenormalizeddecimalfield #to_python

1 note · View note

tothinkornottothink-blog · 14 years ago

Text

Last hints on Erlang

Soon I will start writing some posts on Django, my current passion, in which I work fulltime at the moment. But before, I really want you to understand how Erlang real applications are built. If you look at Wikipedia's Erlang article section: Project's using Erlang, is simple to understand Erlang is used for what is was built, concurrency. These projects have a common denominator. Most need to be real-time, they need to be efficient and distributed. Erlang excels at this.

How do I build my GUI?

When people start learning Erlang, they feel the necessity of building some sort of GUI (Graphical User Interface). Since Erlang's R13B release, Erlang has a GUI built-in library called wxErlang, which can be a good option building a desktop application that needs multicore usage or controls a distributed cluster.

However, most of the times Erlang applications are built-in using a server-client architecture. We code a web server in Erlang that expects requests coming to be in a specific protocol. For example ejabberd expects requests coming in XMPP protocol (Extensible Messaging and Presence Protocol), which is based on XML. Other Erlang applications are RESTful or use SOAP. So many times, we have a frontend application, that draws the GUI, that communicates with the Erlang application using RPC or raw sockets. So you can build GUIs using java libraries, C#, Python, etc. or you can create web services.

Why so I do this effort?

Of course not everyone needs their applications to be distributed or concurrent. I would say everyone would love to have fault tolerant applications. So if you got here, after reading all these articles and you still wonder why should I write a server, then a frontend GUI or web service and so on. That's because you didn't pay attention. If you need what Erlang offers, you should pick Erlang.

There are two Erlang's features that really impressed me when I started:

One was the ability to hot load code without stopping the application.

The other was being able to restart little portions of an application, if something failed you didn't need to restart everything, just that part. And if it continued failing you could have the rest working until you came up with a hotfix.

Open Source Erlang projects I can look into

The bulk of Erlang projects and Erlang hackers have moved to Github and I can understand why, but my Github love can wait for another post. These are 5 Erlang projects that you should know:

Erlang/OTP itself

Nitrogen: A web framework in Erlang. There are some demos to see it working

CouchDB the DB that is getting very known

RabbitMQ is a great messaging system

Ejjaberd A Jabber instant messaging server famous because Facebook used it to built its chat application

But you didn't talk about...

Well, there are many many topics I didn't cover or mention, remember this was not going to be a book on Erlang, just a teaser. Hopefully I left a path so you can do it and you understand Erlang's phylosophy and when to use it. Some important topics I haven't mentioned are:

Records data types

ETS and DETS large date storage mechanisms

The great database Mnesia, which can be replicated through distributed nodes easily, or can run in RAM for performance.

How to officialy documment the code

How to interface with other programming languages using sockets

Much more on OTP and Erlang's shell

Error handling and supervisions techniques

Security and cookies

The last pointers

An online Erlang interpreter that comes with a hands-on tutorial

Erlang community website

Erlang Programming Another good book, that covers some topics not seen in other books

#couchdb #dets #ejjaberd #erlang #ets #mnesia #nitrogen #otp #rabbitmq #rpc

7 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Real Erlang means OTP

After this series of posts about Erlang: 1, 2 & 3, there is too much left to say. Erlang is a wonderful language built in for distributed fault tolerant apps. But Erlang's magic needs your help. Erlang won't do anything of this if you don't use it carefuly and smart.

Erlang starts to be handy when you really know Erlang. It comes packed with a framework called OTP (Open Telecom Platform), thus Erlang is sometimes referred as Erlang/OTP. You can find a great explanation about it in What is OTP? which is part of a great online tutorial called learnYouSomeErlang that I highly recommend for new comers.

So when real Erlang hackers code Erlang, they most of the time code Erlang using OTP, they code behaviours. Behaviours are not an Erlang concept, if you know other programming languages, you will find them easy to follow. As a Djangonaut I would say behaviours are like swappable backends, you need to implement some defined functions that will get called when you change the backend module. A more general explanation, would be they are interfaces used through callbacks. Of course the best way to fully understand them is seeing a good example. You can find one in OTP design principles Behaviours overview

The most important behaviours are supervisors, gen_server and gen_fsm. Supervisors are one of the most important resources to turn your Erlang fault tolerant as the propaganda prays. The next two are interfaces to create server-client processes and finite state machines.

Erlang online docs are more than decent and they should be your first reference point. One thing I come across when I talk to someone with little knowledge of Erlang, is that they try to think and lay out and architecture for an Erlang system based on former knowledge. Don't forget Erlang is not ADA, is not LISP or Caml. You need to know Erlang in deep to really appreciate its complexity and its concurrent nature.

#OTP #behaviours #desing #erlang #gen_fsm #gen_server #principles #superviros

2 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Getting into some basic concurrency in Erlang

Raising the bets

In last article, first baby steps in Erlang. We wrote our first module, a simple add function and our first recursive function count_list_items. The main power Erlang has comes from the concurrent part of it, so we will get into creating processes and assingning them some tasks.

Don't forget these series of articles are just an Erlang teaser, so you get to see some of Erlang's power and start reading more about it.

Erlang is structured in modules. Every module can contain public (exported) and private functions, as we have seen. Besides this we can create processes that run functions. Open Erlang's shell and create your first process, here is how:

erl > Pid = spawn(io,format,["Hello world! ~n", []]). Hello world!

spawn function is in charge of creating a process that runs the function passed. spawn/3 expects these parameters: Module, function, list_of_parameters. So what we are doing here is creating a process that prints a "Hello world!" message and then dies. Note that spawn returns the pid (process identifier).

Let's do something more complicated

You might think that Erlang has some nice features but you don't like the trade of dropping imperative languages and using locks for concurrency. Well, if you think that, I find it normal but with time you will realize how Erlang is much more powerful and useful for real solutions.

Now we are told to code a function that looks for a pattern within a string, returns atom match if it finds it or atom no_match otherwise. We will call this function find and as you can deduce it only needs 2 arguments: String and Pattern. So it's got an arity of 2. Here is a possible solution to this problem with a hidden bug (see if you can spot it).

-module(prueba). % This exports all functions so they can be used out of the module -compile(export_all). % Returns match if Pattern is part of the String, false otherwise % the find/2 is a wrapper that makes sure lists are passed % and calls find/3 which has some sort of accumulator parameter find(String, Pattern) when is_list(String), is_list(Pattern) -> find(String, Pattern, Pattern). find(String, [], []) -> match; find(String, [], Pattern) -> match; find([], [], Pattern) -> match; find([], Pattern, Pattern) -> no_match; find([H|String], [H|TPattern], Pattern) -> find(String, TPattern, Pattern); find([H|String], [PH|TPattern], Pattern) -> find(String, Pattern, Pattern).

If you are a newbie in Erlang, take some time reading the code, hopefully you will see how it works. Basically there are 3 parameters, the String, a dynamic Pattern and a static Pattern. The dynamic Pattern checks if the head of the string is equal to its head:

find([H|String], [H|TPattern], Pattern) -> find(String, TPattern, Pattern);

If heads are equal we could have for example:

find(["a"|"bcdef"], ["a"|"h"], "ah") -> find("bcdef", "h", "ah")

So next step will fall into:

find(["b"|"cdef"], ["h"|], "ah")

These times the heads are not equal, so the pattern matching falls into:

find([H|String], [PH|TPattern], Pattern) -> find(String, Pattern, Pattern).

Which kind of restarts the process:

find(["b"|"cdef"], ["h"|], "ah") -> find("cdef", "ah, "ah")

If you are an avid reader you have probably spotted a bug in the code. What happens if the String was "aah" and the Pattern "ah"?

erl > c(prueba). {ok,prueba} > prueba:find("aah", "ah") no_match

Right, the code will say there is no match. We have restarted the Pattern, but we should be looking in the whole string, not only its tail. If you are thinking about changing the clause to something like:

find([H|String], [PH|TPattern], Pattern) -> find([H|String], Pattern, Pattern).

I have to warn you that this will put you in a shoot yourself in the foot situation, an infinite loop, as Pattern never pops items, nor the String. So the solution is to add a clause and be more fine granular:

find(String, [], []) -> match; find(String, [], Pattern) -> match; find([], [], Pattern) -> match; find([], Pattern, Pattern) -> no_match; % first letter of the String matches first of the accumulator pattern % Pop bot letters and continue checking find([H|String], [H|TPattern], Pattern) -> find(String, TPattern, Pattern); % The first letter doesn't match and the acc pattern is equal to the pattern find([H|String], [PH|TPattern], [PH|TPattern]) -> find(String, [PH|TPattern], [PH|TPattern]); % The letters don't match but the acc patern does not match the pattern % This is the clause we were missing, we pass the whole string and restart find([H|String], [PH|TPattern], [PatternH|TailPattern]) -> find([H|String], [PatternH|TailPattern], [PatternH|TailPattern]).

The order of the clauses here is also very important, think about it. Of course understanding functional recursion is quite hard in the beginning, but if you think it carefuly this way of programming is a lot more close to the way humans think. Only our corrupted by imperative languages minds find it hard.

Add some concurrency, spice it up

Now we are going to create a loop function. This is used to handle a concurrent request and keep the process alive. We add to our module prueba these lines of code

% This function spawns a process and sends it a request % using rpc function find_concurrent(String, Pattern) -> Pid = start(), rpc(Pid, {find, String, Pattern}). % Creates a process running function string_worker start() -> spawn(prueba, string_worker, []). % Remote procedure call % sends a request and listens for an answer from that process rpc(Pid, Request) -> Pid ! {self(), Request}, receive {Pid, Response} -> Response end. % Encapsulates the find function in a loop string_worker() -> receive {From, {find, String, Pattern}} -> io:format("Received. String: ~p Pattern: ~p~n", [String, Pattern]), From ! {self(),find(String, Pattern)}, string_worker(); % Anything different than the first clause stops the process Any -> void end.

A lot of new code appears in these lines. But if you are in Erlang, it's because you probably know some other programming language well, so you will probably understand most of it. Even though Erlang's syntaxis looks like Prolog and it's ugly, you get used to it. The ! operator is the send message operator. It is used as:

pid_destination ! message

When we call start function, it spawns a process running string_worker, which listens for messages to arrive using receive ... end. directives. The rpc function sends this spawned new process a request conformed by a tuple of tuples {pid_sending_message, {find, String, Pattern}}. self() built-in function (BIF) returns the PID of the process that calls it. After sending the request it starts listening for a response from that single process, not any other process. Let's run it in the shell:

> c(prueba). > prueba:find_concurrent("asdfaah","sed"). Received. String: "asdfaah" Pattern: "sed" no_match > prueba:find_concurrent("asdfaah","ah"). Received. String: "asdfaah" Pattern: "ah" match

It is left for the reader to read more on Erlang's concurrency. But if you are focused, you probably have noticed, that the process spawned never dies. In general, in Erlang we will want to create just on process of every different type

#ada #caml #concurrency #erlang #haskel #lisp #recursion #recursive #tail

5 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

First baby steps in Erlang

Erlang is distributed by itself

Sure a statement like this echoes in many heads. Erlang has location transparency, which means that no matter how you setup your erlang environment, it will run. When you do concurrency using locks, even if you get the locking right, you still will run into trouble if a thread is moved out of the same machine, as the communication scheme has changed a lot. That involves refactoring concurrent code that will probably take years to refine and weed out of bugs.

Erlang code can be run the same way in a laptop than in a cluster. That's because of the actor model I talked about. Processes don't need to know where they are, where others are to run. Scalability becomes a problem so much easier to attack. Things like how I split my program into parts that can be distributed or replicated, how should they communicate and many others are resolved in Erlang built-in.

Let's start with the real deal

If you read the books recommended in my refreshing erlang article and you fully understand and master them, you will know more than me :) and you will become quite a good Erlang hacker. But mastery comes from practice. I would love to show advanced code, but we need to start with some simple examples, so people don't get scared too fast.

Tail recursion

As I said before, recursion is the path of the master. But there are two kinds of recursive functions and understanding the difference is tremendously important:

sum(N) -> sum(N-1) + N. do_sum(0, Total) -> Total. do_sum(N, Total) -> do_sum(N-1, Total+N).

First one sum is a non-tail recursive function. Second one do_sum is a tail recursive function. Tail functions do "tail calls", these ones are easy to spot by a compiler. If you know how recursion works, you are probably aware of the stack that builds up with more and more memory used, the bookeeping problem, keeping track of what to do later and what goes next. This steals efficiency and memory. If Erlang code is going to run forever, the system cannot run out of memory

Erlang does not run out of memory because it does tail call optimization which means that the compiler only keeps in the stack the last current information, as it does not need anything else, so the stack doesn't grow. This makes recursive tail functions extremely efficient. You should always rely on tail call optimization

Show me the code

Let's create our first module in which we will put some functions. The module name will be "prueba", so the file needs to be named prueba.erl

% This a comment in Erlang. % Next line defines this file as an Erlang module -module(prueba). % Next line makes these functions public, available out of the module % It's a list with the [function_names/arity] -export([add/2]). % Our first function, which adds 2 numbers, dead simple add(A, B) -> A + B.

Now let's get into Erlang's shell, which is more than a typical shell. It's also the debugger and much more. We need to run it in the same path as our module:

erl > % comments don't make sense in the shell :) > Let's compile and load our module > c(prueba). {ok,prueba} > % ok means all went right, no typos or syntax errors > % Let's call our add function module:fun(...) > prueba:add(4,5). 9 > prueba:add(2,2) - prueba:add(1,0). 3

It's time to build our first recursive tail function. We can start coding one that counts items in a list (This code would be added after add function):

% Counts items of a list % Beware the first function ends with ";" % This is a clause and it's part of the same function count_items([], Total) -> Total; count_items([H|T], Total) -> count_items(T, Total+1).

count_items is made out of two clauses separated by a semicolon. When you call the function it does a top down pattern matching, returning the first match. So we have included a base case (when the list is empty) in charge of ending recursion. The [H|T] functional operator does the rest, this is the functional magic in place.

A typical beginner mistake would be to put T (the tail) in between brackets [T]:

count_items([], Total) -> Total; count_items([H|T], Total) -> count_items([T], Total+1).

As tail is already a list, what you are really doing is [[T]] thus creating an infinite loop, boom! There is room for improvement in my code. We are assigning H the head of the list and not using it. This will prompt:

./prueba.erl:11: Warning: variable 'H' is unused

So we should be using an anonymous match operator "_":

count_items([_|T], Total) -> count_items([T], Total+1).

Try count_items function. Total is an accumulator parameter. It stores the value accumulated that in the end will be the result. Of course it needs to start being zero:

> % recompile module > c(prueba). {ok,prueba} > prueba:count_items([1,2,3,4], 0) 4

Let's wrap it up, as we only want to pass one argument: the list. We don't need to know anything about what the accumulator parameter is expected to be. So we wrap count_items in another function called count_list_items which has a guard that checks the parameter passed is a list:

% Counts items of a list % Beware that count_list_items ends with a dot % is_list(...) is a guard. There are many more: is_atom(...), is_integer(...), is_number(...) etc. count_list_items(L) when is_list(L) -> count_items(L, 0). count_items([], Total) -> Total; count_items([H|T], Total) -> count_items(T, Total+1).

Let's export our new function:

-export([add/2, count_list_items/1]).

count_items is now a private function within the module, as there is no need to export it. See how it works:

> prueba:count_list_items(5). ** exception error: no function clause matching prueba:count_list_items(5) > prueba:count_list_items({5,4}). ** exception error: no function clause matching prueba:count_list_items({5,4}) > prueba:count_list_items([1,2,3,4]). 4

#concurrency #concurrent #distributed #erlang #recursion

4 notes · View notes

tothinkornottothink-blog · 14 years ago

Text

Refreshing Erlang

Erlang is a functional concurrent programming language. It is very different to what you probably use in your day to day as a programming language. Back in San Diego I took an Erlang programming course and I found it very interesting. Erlang programmers are like gnomes hiding in the deep of the forest, like magic unicorns, some people say they exist.

If you haven't heard of it, this is probably a good time. If you have coded concurrent languages like ADA or functional languages like Caml, Haskel or LISP, you know a mixture of the two paradigms has to be a hard mental challenge, it would be hard to make a post on how to code Erlang from scratch. But I will cover some basics, so you can know some more about this interesting language.

Why Erlang rocks

You can do fault-tolerant applications that can be modified without taking them out of service. It supports hot-swapping

It's been in use in real large-scale industrial products, like telecommunications.

Wikipedia's erlang article

Why is Erlang different?

Variables are all Uppercase and they are not really variables, stay with me and you will know why.

Every statement ends with a dot. Syntax is weird and if you love Python's syntax, you won't find it here.

Variables are like math variables in an equation. X = X + 1 you can't do this in Erlang. When you assign a value, you can't change it. So variables are not variables, and = operator is almost an assignment operator. They are called single assignment variables, they can be bound once and they are created unbound. Of course this is done in purpose, so forget imperative languages prejudices. A variable is kind of a pointer to a value that cannot change. If you know concurrent programming, you know how hard can be to debug a program, making variables static, makes your debugging pain more bearable. That's why Erlang is known as a nonmutable state language. You might be wondering, how can I code in Erlang with this sort of variables? Answer could be "be recursive my friend, find new names for your variables and control scope"

Erlang has something called atoms, atoms are non-numerical global constant values. Atoms are all lowercase, so forget using beautiful camelcase code style.

Erlang comes geared with tuples (unfortunately they use curly brackets, so Python coders like me suffer extra pain) P = {15, 20}

The = operator is considered a pattern matching operator. This is very important, because this is the way we extract values from tuples or other data structures. Code example:

> Point = {20, 10}. {20,10}. > X, Y = Point. {20, 10} > X. 20 > Y. 10

The underscore _ is called anonymous variable. It means we are not interested in this value. Code example:

>{_, {_, Who}} = {23, {name, miguel}}. {23, {name, miguel}} > Who. miguel

Lists are created with square brackets. The lists are the salsa of every functional language. [1, 2, 3, 4] the first item of a list is called head, that would be 1. The tail is the rest of the list [2, 3, 4]. If T is a list then [H|T] is also a list. H is the head and T is the tail.

> [X,Y|Z]=[a,b,c,d,e,f]. > X. a > Y. b > Z. [c, d, e, f]

Strings in Erlang are lists of integers. Besides this, Erlang is used in software like ejabberd

Of course this is only the peak of the iceberg, Erlang has many idiosyncrasies.

Concurrency made simple

It is made simple, simple as it can be made. Concurrency has never been easy and something in our brains crack when doing it. When I started learning concurrency, I was coding ADA and ADA took some time to be digestible and thank god the little book of semaphores was there to make digestion easier.

However Erlang approach to concurrency is very different and a lot simpler. It uses something called the actor model. Erlang doesn't let you create threads that share memory among them. Instead it creates ligthweight processes that communicate with each other passing messages. This is similar to how people interact in real life, they talk to each other. So you will not have to do locks or semaphores for managing concurrency.

Installing Erlang

In ubuntu you just need to do:

sudo apt-get install erlang

Then you can run the shell with erl command and the compiler with erlc. Let's do a hello world (it creates a module and defines and exports 2 functions). First create a greeting.erl file, then paste this code:

-module(greeting). -export([hello/0, print_hello/0]). hello() -> "Hello World". print_hello() -> io:format("~w ~n", [hello()]).

The way you run Erlang code is doing erl -run Module function. So in this case we would do:

erl -run greeting print_hello [72,101,108,108,111,32,87,111,114,108,100]

Good books to continue learning

Programming Erlang: Software for a concurrent world by Joe Armstrong

Erlang and OTP in action by Martin Logan, Eric Merritt, Richard Carlsson

Hopefully I will be posting some code examples in the next few days, showing some Erlang's goodies.

#ada #caml #code #concurrency #erlang #functional #haskel #lisp

2 notes · View notes