smhk

Migrating from Tastypie to Django REST Framework

All notes in this series:

Here are some tips from my experience of migrating an existing Django application from using Tastypie for its REST API to instead using Django REST Framework.

These notes assume the following versions:

django==2.2.3
django-tastypie==0.14.2
djangorestframework==3.9.4
django-filter==2.1.0

Separate v1 and v2 APIs §

Tastypie hosted its API at /api/v1/. To ensure a smooth transition, we configured Django REST Framework to be exposed at /api/v2/. The plan was to build the new v2 API in a manner that is backwards-compatible with the old v1 API. Clients to the REST API could then switch between either using the v1 or v2 API with a simple configuration change.

Here is how to run both Tastypie at /api/v1/ and DRF at /api/v2/:


# Tastypie API
v1_api = Api(api_name="v1")
v1_api.register(UserResource())
v1_api.register(GroupResource())
# etc...

# DRF API
v2_api = routers.DefaultRouter()
v2_api.register(r"user", UserViewSet)
v2_api.register(r"group", GroupViewSet)
# etc...

# URLs
urlpatterns = [
    # ...
    
    # Tastypie v1 API
    url(r"^api/", include(v1_api.urls)),
    
    # DRF v2 API
    path("api/v2/", include(v2_api.urls)),
    
    # DRF v2 API authentication (optional)
    path("api/v2/api-auth/", include("rest_framework.urls")),
]

Filtering §

Out of the box, Django REST Framework does not support the same filtering (and ordering) that Tastypie does. Use django-filter to add support for this.

INSTALLED_APPS = [
    # ...
    "django_filters",
]
REST_FRAMEWORK = {
    # ...
    "DEFAULT_FILTER_BACKENDS": ["django_filters.rest_framework.DjangoFilterBackend"],
}

Then create a filters.py and place a filter in there for each view set:

from django_filters import rest_framework as filters

class FooBarFilter(filters.FilterSet):
    timestamp = filters.IsoDateTimeFilter(field_name="timestamp")

Then in your views.py update the view set to use that filter:

from rest_framework import viewsets
from my_app.serializers import FooBarSerializer
from my_app.filters import FooBarFilter
from my_app.models import FooBar

class FooBarViewSet(viewsets.ModelViewSet):
    queryset = FooBar.objects.all().order_by("id")
    serializer_class = FooBarSerializer
    filterset_class = FooBarFilter  # <-- Add this line

Filtering expressions (lt, lte, gt, gte, etc.) §

Note however that by default this only adds exact matching. To add support for other filtering expressions, you must add an additional filter which uses the lookup_expr keyword argument for each additional expression you wish to support. It feels like there should be a simpler way to do this, but this is the only way I’ve found.

For example, to build upon our previous FooBarFilter and add support for querying timestamps less than (or equal to) or greater than (or equal to):

from django_filters import rest_framework as filters

class FooBarFilter(filters.FilterSet):
    timestamp = filters.IsoDateTimeFilter(field_name="timestamp")
    timestamp__lt = filters.IsoDateTimeFilter(field_name="timestamp", lookup_expr="lt")
    timestamp__lte = filters.IsoDateTimeFilter(field_name="timestamp", lookup_expr="lte")
    timestamp__gt = filters.IsoDateTimeFilter(field_name="timestamp", lookup_expr="gt")
    timestamp__gte = filters.IsoDateTimeFilter(field_name="timestamp", lookup_expr="gte")

Filtering with choice fields §

Some extra work is required if you wish to have a choices field whereby the integer values are used internally (in the database) but the string values are used externally (for REST API parameters). Say you have the following Django model:

CHOICES = (
    (0, "Foo"),
    (1, "Bar"),
    (2, "Baz"),
)

class Widget(models.Model):
    status = models.IntegerField(
        default=0
        choices=CHOICES,
    )

You might then expect to be able to use django-filter to allow filtering like so:

class WidgetFilter(filters.FilterSet):
    status = filters.ChoiceFilter(choices=CHOICES)

That will work, but the REST API parameter will have to be the integer value rather than a string, e.g.:

/api/v1/widget?status=0

We want to use the string, e.g.:

/api/v1/widget?status=Foo

The simplest way I found to do this is by overriding the filter’s filter method to map the string to the integer before the filtering takes place.

First, we must reverse the mapping of the CHOICES. This is necessary because the left-hand item in the key, value pairs is expected by django-filter in the URL (e.g. ?status=xxx), and we want it to expect the string rather than the integer.

REVERSE_CHOICES = tuple((v, k) for (k, v) in CHOICES)

Next we subclass filters.ChoiceFilter to swap the string for the integer before the filtering takes place. We can use the REVERSE_CHOICES as a dictionary for this, since it contains a mapping of strings to integers.

class MappingChoiceFilter(filters.ChoiceFilter):
    def filter(self, qs, value):
        value = dict(REVERSE_CHOICES)[value]  # <-- The swap happens here!
        return super().filter(qs, value)

Finally, we simply update the WidgetFilter to use our MappingChoiceFilter. Note that because fitlers.ChoiceFilter (which MappingChoiceFitler inherits from) validates that the incoming string is a key from the REVERSE_CHOICES dictionary, we do not need to expect any KeyError when we do the swapping via the dictionary lookup in MappingChoiceFilter.filter, since the key is guaranteed to exist.

class WidgetFilter(filters.FilterSet):
    status = MappingChoiceFilter(choices=REVERSE_CHOICES)

At last this allows us to query the choice through the API with the string, whilst using the integer internally:

/api/v1/widget?status=Foo

Ordering §

Tastypie style ordering is built in to Django REST Framework, but requires a small configuration change to enable and another change to be compatible.

First you must add "rest_framework.filters.OrderingFilter" to the DEFAULT_FILTER_BACKENDS to enable filtering by default. (Alternatively you can enable it on a case-by-case basis for each view).

Note however that while Tastypie used the order_by parameter, Django REST Framework uses ordering. To override this, use the ORDERING_PARAM setting:

REST_FRAMEWORK = {
    # ...
    # Enable filtering and ordering
    "DEFAULT_FILTER_BACKENDS": [
        "django_filters.rest_framework.DjangoFilterBackend",
        "rest_framework.filters.OrderingFilter",
    ],
    # Use `order_by` rather than `ordering` for the order paramater
    "ORDERING_PARAM": "order_by",
}

Pagination §

By default, Tastypie used limit and offset ordering, while Django REST Framework uses page ordering.

Fortunately it is trivial to switch Django REST Framework to use limit and offset ordering with the built in LimitOffsetPagination paginator.

Pagination Format §

Further, the response returned by Django REST Framework’s paginator is slightly different to the default response given by Tastypie. While Tastypie returns something like:

{
    "meta": {
        "limit": 20,
        "next": null,
        "offset": 0,
        "previous": null,
        "total_count": 4
    },
    "objects": {
        # ...
    }
}

Django REST Framework returns something like:

{
    "meta": {
        "next": null,
        "previous": null,
    },
    "count": 4,
    "results": {
        # ...
    }
}

Fortunately it is easy to configure Django REST Framework to use a custom paginator, specifically if you override the get_paginated_response class you can configure the JSON exactly as you like.

Hydrate and dehydrate §

In Tastypie, the hydrate methods are used to add extra functionality when converting incoming JSON to a Python object, and likewise the dehydrate methods are used to add extra functionality when converting an outgoing Python object to JSON. This is extremely useful for cases such as adding in extra fields to the JSON response, renaming fields, or converting values (e.g. to handle enumerations).

The equivalent of this in Django REST Framework is to override the to_representation (aka dehydrate) and to_internal_value (aka hydrate) methods.

For example:

class FooSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = Foo
        fields = (
                "name",
                "frobnigate",
            )

    def to_representation(self, instance):
        ret = super().to_representation(instance)
        ret["frobnigate"] = "Enabled" if ret["frobnigate"] is True else "Disabled"
        ret["new_field"] = "whoop"
        return ret

    def to_internal_value(self, data):
        # Simplified example - you might want to perform some validation
        data["frobnigate"] = True if data["frobnigate"] == "Enabled" else False
        return super().to_internal_value(data)

Validation §

If you just want to validate a field or group of fields in the serializer, then use the validate_<field> or validate methods respectively.

However, if you want to validate and convert modify it, then you must perform the validation in the to_internal_value. This is necessary because validate (or validate_<field>) is run after to_internal_value. (Also, the validation order is not officially documented, so it is safer not to tie ourselves in to relying upon the current execution order).

Note that there is a subtle difference in how you should raise a validation error in validate_<field> compared to validate and to_internal_value.

In validate_<field> you can simply raise the exception with a string:

def validate_foo(self, value):
    if value not in (1, 2, 3):
        raise serializers.ValidationError("A valid value is required")
    return value

However in to_internal_value (or validate) you must pass in a dictionary which maps the fields to the errors:

def to_internal_value(self, data):
    if data["foo"] not in (1, 2, 3):
        raise serializers.ValidationError({"foo": "A valid value is required"})
    return data

Both of these would then return an error such as the following:

HTTP 400 Bad Request
Allow: GET, POST, PATCH
Content-Type: application/json
Vary: Accept

{
    "foo": "A valid value is required"
}

This subtle difference is mentioned in the DRF serializers documentation (emphasis mine):

If any of the validation fails, then the method should raise a serializers.ValidationError(errors). The errors argument should be a dictionary mapping field names (or settings.NON_FIELD_ERRORS_KEY) to a list of error messages. If you don’t need to alter deserialization behavior and instead want to provide object-level validation, it’s recommended that you instead override the .validate() method.

If you fail to pass in a dictionary when necessary, you will get a traceback like the following:

Traceback (most recent call last):
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/django/core/handlers/exception.py", line 34, in inner
    response = get_response(request)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 115, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/django/core/handlers/base.py", line 113, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/viewsets.py", line 116, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/views.py", line 495, in dispatch
    response = self.handle_exception(exc)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/views.py", line 455, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/views.py", line 492, in dispatch
    response = handler(request, *args, **kwargs)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/mixins.py", line 84, in partial_update
    return self.update(request, *args, **kwargs)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/mixins.py", line 69, in update
    serializer.is_valid(raise_exception=True)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/serializers.py", line 244, in is_valid
    raise ValidationError(self.errors)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/serializers.py", line 574, in errors
    return ReturnDict(ret, serializer=self)
  File "/opt/myapp/myenv/lib64/python3.6/site-packages/rest_framework/utils/serializer_helpers.py", line 20, in __init__
    super(ReturnDict, self).__init__(*args, **kwargs)
ValueError: too many values to unpack (expected 2)

Supporting PATCH §

All viewsets.ModelViewSet view sets support PATCH by default under Django REST Framework. Note however that if you have overridden to_internal_value as we did above, you will need to be careful not to break this behaviour!

Let’s look again at the code from above:

def to_internal_value(self, data):
    if data["foo"] not in (1, 2, 3):
        raise serializers.ValidationError({"foo": "A valid value is required"})
    return data

The problem with this code is that if the field "foo" is not supplied during a PATCH request - which is entirely valid, since a PATCH should not require all fields - then the validation will fail. To fix that, we can make the validation optional dependant upon whether the field exists:

def to_internal_value(self, data):
    if "foo" in data:
        if data["foo"] not in (1, 2, 3):
            raise serializers.ValidationError({"foo": "A valid value is required"})
    return data

Non-ORM Endpoints §

Tastypie supported non-ORM endpoints through overriding various methods in the Resource class, such as obj_get, obj_get_list, get_object_list. Fortunately this is simpler in Django REST Framework. For read-only non-ORM endpoints, you simply create a view set which inherits from ViewSet (rather than ModelViewSet, which is used for ORM data) and then override the list method to return a Response object of your choice.

Add resource_uri to each endpoint §

While in Tastypie each resource has a resource_uri included by default, in Django REST Framework you must explicitly add this field to each serializer. Further, by default Django REST Framework calls this field url.

First update the settings.py to set default value for URL_FIELD_NAME to resource_uri:

REST_FRAMEWORK = {
    # ...
    "URL_FIELD_NAME": "resource_uri",
}

Then update all of your serializers to add the "resource_uri" field:

class UserSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = User
        fields = ("resource_uri", "username", "email", "groups")


class GroupSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = Group
        fields = ("resource_uri", "name")

Using non-PK for URL §

By default Django REST Framework HyperlinkedModelSerializer uses the pk field of the Django model for the URL. If you wish to use a different (unique) field for the URL, the steps are as follows.

For example, say we have a Person model which has a unique username which we would like to use as the URL, instead of the pk field.

In the view set, specify the field to use in the lookup_field attribute:

class PersonViewSet(viewsets.ModelViewSet):
    queryset = Person.objects.all().order_by("id")
    serializer_class = PersonSerializer
    lookup_field = "username"

In the serializer Meta, use the extra_kwargs to specify the lookup_field which the resource_uri should use.

class PersonSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = Person
        fields = (
                # ...
            )
        extra_kwargs = {
            "resource_uri": {"lookup_field": "username"}
        }

Note that we are using resource_uri rather than url because we changed URL_FIELD_NAME in the previous section! If you fail to set it as described above you will get the following error:

django.core.exceptions.ImproperlyConfigured: Could not resolve URL for hyperlinked relationship using view name "person-detail". You may have failed to include the related model in your API, or incorrectly configured the `lookup_field` attribute on this field.

Note that any other serializers which have foreign keys to the Person model will also need the extra_kwargs to be set. For example:

class DogSerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = Dog
        fields = (
                # ...
                "owner",  # Foreign key to Person who owns this Dog
            )
        extra_kwargs = {
            "owner": {"lookup_field": "username"}
        }

It was thanks to the DRF 3.0 change notes that I was made aware of the extra_kwargs field which is necessary for these changes!

Infer the non-PK field from the URL §

A problem with the above application of lookup_field is that it makes update requests cumbersome: you have to specify the PK in both the URL and the payload. For example, in the Person model from above, if you wanted to PUT or PATCH to /api/v1/person/denvercoder9/ the payload would also have to include the username field, e.g.:

{
    "username": "denvercoder9",
    # ...
}

To avoid this, we can modify the to_internal_value method to infer the non-PK field from the URL, by extracting it from the current context:

    def to_internal_value(self, data):
        try:
            data["username"] = self.context["request"].parser_context["kwargs"]["username"]
        except KeyError:
            pass
        return super().to_internal_value(data)

This automatically tries to add the data["username"] field to the incoming request based upon the PK used in the URL, which means it is no longer necessary for the client to pass it through the payload.

Use PUT to create §

As of DRF v3.0, doing an HTTP PUT to a detail endpoint returns a 404 error by default:

{
    "detail": "Not found."
}

To allow using PUT to create, there is a mixin provided, which I’ve included below with the necessary imports (thanks to Natim and feliesp):

from django.http import Http404

from rest_framework.response import Response
from rest_framework.request import clone_request
from rest_framework import status


class AllowPUTAsCreateMixin(object):
    """
    The following mixin class may be used in order to support PUT-as-create
    behavior for incoming requests.
    """
    def update(self, request, *args, **kwargs):
        partial = kwargs.pop('partial', False)
        instance = self.get_object_or_none()
        serializer = self.get_serializer(instance, data=request.data, partial=partial)
        serializer.is_valid(raise_exception=True)

        if instance is None:
            lookup_url_kwarg = self.lookup_url_kwarg or self.lookup_field
            lookup_value = self.kwargs[lookup_url_kwarg]
            extra_kwargs = {self.lookup_field: lookup_value}
            serializer.save(**extra_kwargs)
            return Response(serializer.data, status=status.HTTP_201_CREATED)

        serializer.save()
        return Response(serializer.data)

    def partial_update(self, request, *args, **kwargs):
        kwargs['partial'] = True
        return self.update(request, *args, **kwargs)

    def get_object_or_none(self):
        try:
            return self.get_object()
        except Http404:
            if self.request.method == 'PUT':
                # For PUT-as-create operation, we need to ensure that we have
                # relevant permissions, as if this was a POST request.  This
                # will either raise a PermissionDenied exception, or simply
                # return None.
                self.check_permissions(clone_request(self.request, 'POST'))
            else:
                # PATCH requests where the object does not exist should still
                # return a 404 response.
                raise

To use the AllowPUTAsCreateMixin mixin, simply inherit from it in the relevant view. However, make sure to get the inheritance order correct! Due to the Python Method Resolution Order (MRO), the mixins must come before the ModelViewSet, else they will have no effect. For example, this is correct:

class ThingViewSet(AllowPUTAsCreateMixin, viewsets.ModelViewSet):
    # ...

It will then be possible to PUT to create.

All notes in this series: