The server's rate limiting does not trigger the proper exception.

[mjstahlberg]

Using get_latest_match_seq_num, instead of a dota2api.src.exceptions.APITimeoutError, a JSON decoding exception is raised whenever the server denies one of your requests due to rate limiting.

[joshuaduffy]

Thanks for the report @Viech. We will get this fixed in master

[xunlao]

Is this the reason that I got this? it seems that after getting ~1100 match, I got an ValueError: No JSON object could be decoded

By the way, how much is the rate limit? (Or I'd better just figure it out myself)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-46-98018d218792> in <module>()
      4 while len(all_matches) < 10000:
      5     last_seq += 1
----> 6     result = api.get_match_history_by_seq_num(start_at_match_seq_num = last_seq, matches_requested = 100)
      7     result = result['matches']
      8 #     print ' getting result.. len =', len(result)

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dota2api/__init__.pyc in get_match_history_by_seq_num(self, start_at_match_seq_num, **kwargs)
    100             self.logger.info('URL: {0}'.format(url))
    101         if not self.__check_http_err(req.status_code):
--> 102             return response.build(req, url)
    103 
    104     def get_match_details(self, match_id=None, **kwargs):

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/dota2api/src/response.pyc in build(req, url)
     13 
     14 def build(req, url):
---> 15     req_resp = req.json()
     16     if 'result' in req_resp:
     17         if 'error' in req_resp['result']:

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/requests/models.pyc in json(self, **kwargs)
    806                     # used.
    807                     pass
--> 808         return complexjson.loads(self.text, **kwargs)
    809 
    810     @property

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    336             parse_int is None and parse_float is None and
    337             parse_constant is None and object_pairs_hook is None and not kw):
--> 338         return _default_decoder.decode(s)
    339     if cls is None:
    340         cls = JSONDecoder

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.pyc in decode(self, s, _w)
    364 
    365         """
--> 366         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    367         end = _w(s, end).end()
    368         if end != len(s):

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.pyc in raw_decode(self, s, idx)
    382             obj, end = self.scan_once(s, idx)
    383         except StopIteration:
--> 384             raise ValueError("No JSON object could be decoded")
    385         return obj, end

ValueError: No JSON object could be decoded

[mjstahlberg]

@xunlao I have not seen such a ValueError as far as I can remember, so this might be a different error. My rate-limited exception is a JSON decoding excpetion (I don't know the exact type name anymore) with the description Expecting value: line 1 column 1 (char 0).

With respect to the wait time: I built an adpative rate limiter that increases it after a failed attampt multiplicatively by 20% up to a maximum of 10s and reduces it additive by 0.1s after every successful request. The resulting average wait time after a request is 6.2 seconds, currently, though this may include periods of downtime where every request was a failure. You could do the same or try wait times between 4 and 7 seconds and see which one gives you the greatest efficiency.

[xunlao]

@Viech Thanks for the suggestion.. I think I should just encode a logic for this.

Currently I think my code (should) works well, which looks like:

sleep_time = 10
while True:
    try:
        result = api.get_match_history_by_seq_num(start_at_match_seq_num = last_seq, matches_requested = 100)['matches']
        # then export data to some json or somewhere; or keep in memory and collecting then export, but more complex code
    except:
        time.sleep(sleep_time)

I would run this overnight.. See what happens.