Getting field key error for given_name for detail people view

[fgregg]

On my local machine:

// 20150521104713
// http://10.42.2.102/ocd-person/076e5dae-ca9a-460a-bcae-7ff925b620d8/

{
  "traceback": "Traceback (most recent call last):\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 94, in get_fields\n    ret = {x: fwrap(root[x]) for x in concrete}\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 94, in <dictcomp>\n    ret = {x: fwrap(root[x]) for x in concrete}\nKeyError: 'given_name'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/projects/api.opencivicdata.org/virt/lib/python3.4/site-packages/restless/views.py\", line 106, in dispatch\n    response = super(Endpoint, self).dispatch(request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/virt/lib/python3.4/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 137, in _\n    return fn(self, request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 122, in _\n    return fn(self, request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 381, in get\n    related, config = get_fields(self.serialize_config, fields=fields)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 96, in get_fields\n    raise FieldKeyError(*e.args)\nimago.helpers.FieldKeyError: <FieldKeyError: given_name>\n",
  "error": "<FieldKeyError: given_name>"
}

[paultag]

I can't reproduce this - didn't you say you tracked this down to an unrun migration?

[fgregg]

Seems like pupa only ran the first migration:

09:51 AM|fgregg-ThinkPad-X201|~/public/opencivicdata/repos/scrapers-us-municipal>pupa dbinit us
no pupa_settings on path, using defaults
Operations to perform:
  Apply all migrations: opencivicdata, pupa
Running migrations:
  Applying pupa.0001_initial... OK
193137 divisions loaded from CSV and  193137 already in DB
no work to be done!

How should I tell pupa to run the second migration here: https://github.com/opencivicdata/python-opencivicdata-django/tree/02dd8120456ba43c1c2f5db80cf79ca9db138063/opencivicdata/migrations

[paultag]

Django should run them all; do you have an old version of Pupa?

[fgregg]

Nope. All this work is from the opencivicdata meta repo I cloned two days ago.

[paultag]

Sure, but what version of Pupa are you running?

Does pydoc pupa | grep __init__ output a path with /src/ in it, or is
it your dev path?

On Thu, May 21, 2015 at 1:04 PM, Forest Gregg notifications@github.com
wrote:

Nope. All this work is from the opencivicdata meta repo I cloned two days
ago.

—
Reply to this email directly or view it on GitHub
#53 (comment).

Paul Tagliamonte
Software Developer | Sunlight Foundation

[fgregg]

'pupa==0.4.1'

[paultag]

github issues cut off my second half

Does `pydoc pupa | grep __init__` output a path with `/src/` in it, or is it your dev path?

[paultag]

Oh, also that migration isn't even in Pupa, it's in python-opencivicdata; what's that at? (same question as above)

[fgregg]

> pydoc pupa | grep __init__
/home/fgregg/public/opencivicdata/repos/pupa/pupa/__init__.py

> pydoc opencivicdata | grep __init__

returns nothing

> pydoc opencivicdata 
Help on package opencivicdata:

NAME
    opencivicdata

PACKAGE CONTENTS
    admin (package)
    api (package)
    apps
    common
    divisions (package)
    management (package)
    migrations (package)
    models (package)
    tests (package)

FILE
    (built-in)

[fgregg]

Python 3.4.0 (default, Apr 11 2014, 13:05:11) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import opencivicdata
>>> opencivicdata.__path__
_NamespacePath(['/home/fgregg/public/opencivicdata/repos/python-opencivicdata-divisions/opencivicdata', '/home/fgregg/public/opencivicdata/.env/src/opencivicdata-django/opencivicdata', '/home/fgregg/public/opencivicdata/.env/src/opencivicdata.api/opencivicdata'])

[paultag]

/home/fgregg/public/opencivicdata/.env/src/opencivicdata-django/opencivicdata
looks pip installed. cd into opencivicdata-django and python setup.py
develop it in your virtualenv

On Thu, May 21, 2015 at 1:22 PM, Forest Gregg notifications@github.com
wrote:

Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.

import opencivicdata
opencivicdata.path
_NamespacePath(['/home/fgregg/public/opencivicdata/repos/python-opencivicdata-divisions/opencivicdata', '/home/fgregg/public/opencivicdata/.env/src/opencivicdata-django/opencivicdata', '/home/fgregg/public/opencivicdata/.env/src/opencivicdata.api/opencivicdata'])

—
Reply to this email directly or view it on GitHub
#53 (comment).

Paul Tagliamonte
Software Developer | Sunlight Foundation

[fgregg]

Done:

>>> import opencivicdata
>>> opencivicdata.__path__
_NamespacePath(['/home/fgregg/public/opencivicdata/repos/python-opencivicdata-divisions/opencivicdata', '/home/fgregg/public/opencivicdata/.env/src/opencivicdata.api/opencivicdata', '/home/fgregg/public/opencivicdata/repos/python-opencivicdata/opencivicdata'])
>>> import opencivicdata.migrations as m
>>> m.__file__
'/home/fgregg/public/opencivicdata/repos/python-opencivicdata/opencivicdata/migrations/__init__.py'

Running pupa dbinit us does not run any migrations.

[fgregg]

but pupa dbinit --reset us DID!

[fgregg]

argh! but now

Traceback (most recent call last):
  File "/home/fgregg/public/opencivicdata/.env/bin/pupa", line 9, in <module>
    load_entry_point('pupa==0.4.1', 'console_scripts', 'pupa')()
  File "/home/fgregg/public/opencivicdata/repos/pupa/pupa/cli/__main__.py", line 71, in main
    subcommands[args.subcommand].handle(args, other)
  File "/home/fgregg/public/opencivicdata/repos/pupa/pupa/cli/commands/dbinit.py", line 60, in handle
    call_command('loaddivisions', country)
  File "/home/fgregg/public/opencivicdata/.env/lib/python3.4/site-packages/django/core/management/__init__.py", line 109, in call_command
    defaults = parser.parse_args(args=args)
  File "/home/fgregg/public/opencivicdata/.env/lib/python3.4/site-packages/django/core/management/base.py", line 64, in parse_args
    return super(CommandParser, self).parse_args(args, namespace)
  File "/usr/lib/python3.4/argparse.py", line 1729, in parse_args
    self.error(msg % ' '.join(argv))
  File "/home/fgregg/public/opencivicdata/.env/lib/python3.4/site-packages/django/core/management/base.py", line 70, in error
    raise CommandError("Error: %s" % message)

[fgregg]

Okay, that last part was becuase I upgraded pupa, which upgrades django to 1.8 but opencivicdata-django needs 1.7 sorted that out. Hoewever, it didn't seem to fix the problem.

>pupa dbinit --reset us
no pupa_settings on path, using defaults
dropping table opencivicdata_bill
dropping table opencivicdata_billabstract
...
dropping table opencivicdata_votesource
Operations to perform:
  Apply all migrations: pupa, opencivicdata
Running migrations:
  Applying opencivicdata.0001_initial... OK
  Applying opencivicdata.0002_auto_20150131_1021... OK
  Applying opencivicdata.0003_auto_20150507_1554... OK
193137 divisions loaded from CSV and  0 already in DB
193137 divisions created
> pupa update chicago
...
import jurisdictions...
import organizations...
import people...
import posts...
import memberships...
import bills...
import events...
import votes...
chicago (scrape, import, report)
  events: {}
  people: {}
  bills: {}
bills scrape:
  duration:  0:00:16.437688
  objects:
    bill: 1
events scrape:
  duration:  0:00:03.458544
  objects:
    event: 1
jurisdiction scrape:
  duration:  0:00:00.062029
  objects:
    jurisdiction: 1
    organization: 2
    post: 52
people scrape:
  duration:  0:00:06.270001
  objects:
    membership: 7
    organization: 6
    person: 1
import:
  bill: 1 new 0 updated 0 noop
  event: 1 new 0 updated 0 noop
  jurisdiction: 1 new 0 updated 0 noop
  membership: 7 new 0 updated 0 noop
  organization: 8 new 0 updated 0 noop
  person: 1 new 0 updated 0 noop
  post: 52 new 0 updated 0 noop
>./scripts/ocd-dev.sh 
[   ] clone imago skippedd
[ ✓ ] pull imago
[   ] clone opencivicdata.org skipped
[ ✓ ] pull opencivicdata.org
[   ] clone api.opencivicdata.org skipped
[ ✓ ] pull api.opencivicdata.org
[   ] clone vagrant-opencivicdata.org skipped
[ ✓ ] pull vagrant-opencivicdata.org
[ . ] vagrant up'ing db
[ ✓ ] halting db
[ ✓ ] bringing db online
[ ✓ ] provisioning db
[ . ] vagrant up'ing site
[ ✓ ] halting site
[ ✓ ] bringing site online
[ ✓ ] provisioning site
[ . ] vagrant up'ing api
[ ✓ ] halting api
[ ✓ ] bringing api online
[ ✓ ] provisioning api

Go to server, turn on Debugging:

// 20150521141601
// http://10.42.2.102/ocd-person/b899b75b-3178-4c88-bb24-585e726a439e/

{
  "traceback": "Traceback (most recent call last):\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 94, in get_fields\n    ret = {x: fwrap(root[x]) for x in concrete}\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 94, in <dictcomp>\n    ret = {x: fwrap(root[x]) for x in concrete}\nKeyError: 'family_name'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/projects/api.opencivicdata.org/virt/lib/python3.4/site-packages/restless/views.py\", line 106, in dispatch\n    response = super(Endpoint, self).dispatch(request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/virt/lib/python3.4/site-packages/django/views/generic/base.py\", line 87, in dispatch\n    return handler(request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 137, in _\n    return fn(self, request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 122, in _\n    return fn(self, request, *args, **kwargs)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 381, in get\n    related, config = get_fields(self.serialize_config, fields=fields)\n  File \"/projects/api.opencivicdata.org/src/imago/imago/helpers.py\", line 96, in get_fields\n    raise FieldKeyError(*e.args)\nimago.helpers.FieldKeyError: <FieldKeyError: family_name>\n",
  "error": "<FieldKeyError: family_name>"
}