Experiments in recommending text for composition tasks.
(TODO: write up)
Demos:
- Deploy code to production:
make deploy. Test on server. - Post HIT (HITs/2018-04-09.html).
make get-datamake get-completed-participantsmake data/analyzed/trial_cueX.csv
- Download MTurk metadata:
scripts/backup_mturk.py. - Run script on downloaded data:
scripts/assign_qualification_to_workers.py.
TODO: update this for the cue writing tasks.
- Make a new branch from the frontend commit (git_rev in the "init" request):
git branch tmp $frontend_commit - Make the hotfix there (perhaps by cherry-picking or copying from master).
- Note the current hash.
- Checkout master, edit analysis_util.py to add an entry to
rev_overridesmapping frontend commit to the hash just recorded in #3. - Incorporate the hotfix commit into the git history:
git merge -s ours tmp.
You can use a worktree to make this easier;
git worktree add tmp-worktere -b tmp $frontend_commit- then when you're done:
git worktree remove tmp-worktree.
Random assignment to condition ordering does not suffice to ensure adequate counterbalancing. So when a new participant connects, we assign them the condition ordering with the lowest expected number of completed trials given the assignments and completions so far. This expectation is a sum of indicators: 1 for any already-comple trial, and the probability of completion for any assigned-but-incomplete trial. We take the probability of completion of an incomplete trial to be 0.5.
This algorithm requires knowing, for each participant within the batch, what ordering they were assigned and whether or not they completed. We take the simplest possible approach for both of these: we read the ordering out of the login entry (the first line) of each log, and we create a new file ('{participant_id}.completed') for a participant who completed.
Project based on the cookiecutter data science project template. #cookiecutterdatascience