Testing with Unmanaged Databases in Django

Originally published on svbtle on AUGUST 23, 2016

I’m testing some API endpoints in a Django app using two different databases. One of them is a read-only legacy SQL Server database (another topic in itself) with multiple schemas, while another is Postgres and stores things like authentication and sessions. Django isn’t great with SQL Server to begin with, but having it be unmanaged made things slightly trickier. Here is how I was able to get things working.

OK, this is a sample class:

class Foo(models.Model):  
    id = models.PositiveIntegerField(
        primary_key=True, 
        db_column='FooId')
    bar = models.CharField(
        max_length=256, 
        db_column='BarEntr')

    class Meta:
        managed = False
        db_table = 'UNREADABLEENTERPRISENAME'

This is a pretty standard Django model. The differences are

- You have to define the database columns that the model attributes are mapping to (the db_column argument) - You have to explicitly define the ID field - The Meta class tells the application that it isn’t managing the table - everything is already being taken care of.

And then, since my database is being hosted somewhere not locally, I have to change my test settings so my colleagues don’t see a surprise database whenever I run my tests.

# settings/test.py

DATABASES['default'] = {  
    'ENGINE': 'django.db.backends.sqlite3',
    'NAME': 'test_database.db'
}

DATABASES['master_data'] = {  
    'ENGINE': 'django.db.backends.sqlite3',
    'NAME': 'test_master_data.db'
}

:boom: easy, right?

No, wait. That ended up returning something like ValueError table does not exist: ‘FooTableENTERPRISESUFFIX’.

We’ve made one step, but still need to do more in order for everything to work. Since the database isn’t managed by Django in production, it won’t be created when I’m running tests, meaning that we can’t test anything that uses those models.

I Googled around and found this article, which was interesting and a little out of date, and didn’t quite work. I’m going to explain this part even if it isn’t part of the solution…

The article suggests creating a test runner for unmanaged tables. So basically it would parse every model and see whether it had the managed meta attribute set to false, if so, then temporarily set it to true. In the end I didn’t have to do that (it didn’t work anyways). It made me realize that I have to look for something outside of the model to change, but it took a little longer before the realization hit.

Anyways, I keep changing stuff and running the tests, and I had totally forgotten about migrations. I create them and look at them, and everything seems fine. So I run the tests again and they don’t work, so I finally take a look at the migrations…

Hmmm…

Oh yeah, so the migration itself stores the managed setting. Meaning that, if we want to test this model, we have to be able to change the attribute in both the model and the migration in order for the table to be created. So, since I didn’t feel like making a migration parser to run inside the test runner, and since I only have a couple of models for now, I just created a setting in my settings files:

# settings/test.py
IS_TESTING = True  

# settings/production.py
IS_TESTING = False  

You might also be able to use the DEBUG setting, but I’m not populating a local database with test data, so I want those to be different. This probably isn’t the ideal solution - migrations have been slightly painful, for instance - but I’m working alone and it isn’t too bad yet. If it gets worse, I can always spend the time to find a better solution, so this works for now.

Now go to your migration(s) and change the managed attribute to something like…

operations = [  
    migrations.CreateModel(
        name='Foo',
        fields=[
            # fields…
        ],
        options={
            'db_table': 'UNREADABLEENTERPRISENAME',
            'managed': settings.IS_TESTING,  # HERE!!
        },
    ),
]

Eh? Just set the managed attribute in options to be more dynamic and then it works.

Admittedly not ideal, but hopefully it will get your tests at least running and you pointed in the right direction.

The only issue I’ve had so far is with migrations. Changes in my model aren’t registered, so if I add a field, I get an error saying the column doesn’t exist. To workaround that, I’ve just been manually adding the columns to the migrations file or resetting them. So: ./manage.py migrate myapp zero.

I don’t know how workable this set up would be in a serious app with multiple developers, but it’s been smooth for me so far.

Happy testing!