Rate limit with ip doesn't work on Django 3/Nginx/Gunicorn

[bzimor]

Ratelimit based on user ip doesn't work completely on Django 3 app. Initially, it raised an error due to the absence of REMOTE_ADDR and this error was fixed with a new patch on main branch. Although, it can access to the user ip, it doesn't block any over limit on production. However, it works when I run the project using runserver 0.0.0.0:8000.
I deployed my django app with the following configurations:
nginx:

upstream my_app_server {
    server unix:/home/projects/my_app/src/gunicorn.sock fail_timeout=0;
}

server {
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_redirect off;
...

gunicorn:

#!/bin/bash

NAME="my_app"
DJANGODIR=/home/projects/my_app/src
SOCKFILE=/home/projects/my_app/src/gunicorn.sock 
NUM_WORKERS=5 
DJANGO_SETTINGS_MODULE=my_app.settings 
DJANGO_WSGI_MODULE=my_app.wsgi
TIMEOUT=300

echo "Starting $NAME as `whoami`"

cd $DJANGODIR
source ../env/bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

exec ../env/bin/gunicorn ${DJANGO_WSGI_MODULE}:application \\
--name $NAME \
--workers $NUM_WORKERS \
--timeout $TIMEOUT \
--bind=unix:$SOCKFILE \
--log-level=debug \
--log-file=-

I added this setting:

RATELIMIT_IP_META_KEY = 'HTTP_X_REAL_IP'

Also, I checked with PyMemcacheCache and LocMemCache, even I tried with custom middleware to change REMOTE_ADDR header, but no luck.
Any ideas?
Django version is 3.1.5.

[jsocol]

Do limits based on other fields such as user ID work?

[bzimor]

I checked, it doesn't work either.

[jsocol]

Thanks for checking. Can you share your cache settings?

You mentioned LocMemCache That definitely shouldn’t be used in production since it’s far too volatile and not shared across processes or hosts.

[bzimor]

I think I fixed my issue, before I used PyMemecacheCache with this setting:

CACHES = {
    'default': {
        'BACKEND': 'djpymemcache.backend.PyMemcacheCache',
        'LOCATION': [
           '127.0.0.1:11211',
        ],
    },

Then I found one Stackoverflow question about that multiple gunicorn workers can have their own cache dict and because of it they cannot share the saved values. So, I changed my Memcached config to unix socket with following setting:

'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': 'unix:/tmp/memcached.sock',
    },

Then it worked. Thanks for your attention

UPD. Sorry, actually it didn't fix my issue anyway.

This is my memcached configuration:

-m 256
-u memcache
-P /var/run/memcached/memcached.pid
-s /tmp/memcached.sock
-a 0766

[SimonHurst]

Silly question @bzimor : but are you setting block=True in the decorator.

I only ask as I found your issue when trying to see why the rate limiter didn't work for me. Not sure why True in this instance isn't the default as otherwise it appears to just do nothing and I'm not sure of the point! If you've already set that then, ignore my comment and I'll let someone with more knowledge go ;)

[bzimor]

Thanks for your advice. Strange, if I use block=True, it ignores the limit and it always shows 403 error whenever I access that url.

[SimonHurst]

Can you show what you’re setting for the view?

…

On Sun, 31 Jan 2021 at 13:04, Boburmirzo Hamraqulov < ***@***.***> wrote: Thanks for your advice. Strange, if I use block=True, it ignores the limit and it always shows 403 error whenever I access that url. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIDNATHVRDI4KXDTAYT623S4VIPTANCNFSM4WTAJWMQ> .

[jsocol]

Huh wow I’m going to be honest, I haven’t been working in Django lately and the default has been block=False for so long that I’d completely forgotten about it.

Digging way back into my memory, I believe the intent was twofold

1. if you block on the first rate in a stack of @ratelimits, the subsequent ones don’t get checked or counted. That can be surprising.
2. if you block on the wrong type of field, like key='post:username' on a login field (which is a good way to limit login attempts) you can create a denial of service vector where I can lock someone else out of their account. These are good use cases for a captcha.

But thinking about it now, with eyes that haven’t seen it in a while, I feel like block=False is more surprising as a default and from what I can tell the login form use case is not the 80% use case.

Obviously this would be a major API change and require a major version bump, but we may need to do v4 anyway to namespace the package.

[bzimor]

@SimonHurst , this is my view:

@ratelimit(key='ip', rate='10/m', block=True)
def test_view(request):
    s = request.META.get('HTTP_X_REAL_IP')
    return HttpResponse('your ip is ' + s)

[SimonHurst]

You might want to set a value for m. '200/<n>m' n being the number of minutes.

…

On Sun, 31 Jan 2021 at 16:25, Boburmirzo Hamraqulov < ***@***.***> wrote: @SimonHurst <https://github.com/SimonHurst> , this is my view: @ratelimit(key='ip', rate='200/m', block=True) def test_view(request): s = request.META.get('HTTP_X_REAL_IP') return HttpResponse('your ip is ' + s) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIDNAVBZELEH3DI7PCUSQLS4WAAHANCNFSM4WTAJWMQ> .

[bzimor]

I logged ratelimit variables on core.py and I got this:

user_ip: 162.158.183.130
method: my_app.views.test_view
cache_name: default
rate: 10/m
cache_key: rl:22f71a140f65c22dfdf73b15b85c4f2c
period: 60
limit: 10
count: None

I found that I got blocked because count is None. It means that cache cannot be found by key...

[SimonHurst]

Sorry deleted that last comment, didn't read it properly.

I would still be explicit:

@ratelimit(key='ip', rate='10/1m', block=True)

Only because it's a little more obvious if someone else has tor read it later.

My next question is what is your caching policy on that view?

[jsocol]

To keep issues cleaner but still help with debugging, I’m going to move this to Discussions (it’ll be the very first one!)