Undefined method `text` for NilClass (Broken scrapers)

[fabianrbz]

While running the example code - with other apps happens too-:

# Download/parse the app.
app = MarketBot::Play::App.new('com.facebook.katana')
app.update

I get the following error quite frequently -Added the verbose output from typhoeus-:

From what I've seen, it's because of the html that's returned. Sometimes it matches the one this gem expects, but sometimes it doesn't.

Expected html:

And the one that generates the error:

Is anyone else getting this? I guess we would need to add another parser and select one depending on the html. I couldn't find a way to get the same response consistently though.
If anyone else is getting this, I might find some time to implement it

[sunboshan]

This is a issue with Google Play store. If you do curl couple of times

$ curl -v https://play.google.com/store/apps/details?id=com.facebook.katana

You will get two htmls for the same page. One is the old format, with all divs has fixed property, like itemprop="contentRating", so that we can parse it easily.

Another html is using new format, with all divs has random class name, like <span class="htlgb">. In this case it's very hard to parse meaningful content.

The first time I saw this is in end of Feb, and I'm seeing this lot more often now. I believe they are doing an update on their server gradually. Some server returning the old format, some returning the new format.

[fabianrbz]

@sunboshan exactly, that's what I'm seeing. What do you think about having another parser for the new format? I guess they are moving towards that one and soon the current implementation will be obsolete.

[sunboshan]

I think the purpose for Google is pretty obvious, which is they don't want others to parse the content of their play store. I don't have a good idea on how to parse div/span with random class name. Yes soon the current implementation will be obsolete when Google full roll out their random html page for play store.

[chadrem]

@lightalloy I believed mentioned this problem to me too. So definitely sounds like it’s going to be a major problem. Hopefully it’s possible to have a second parser to resolve this. I don’t have time to implement a new parser so hopefully the community can work on it. I’m happy to help coordinate as I’m sure this will be useful to a lot of people.

[fabianrbz]

@sunboshan do you think it has random classes? I'm getting the same ones, but yeah, it could be that they are planning to make them random.

[sunboshan]

As far as I can see, the metadata part is using <span class="htlgb">. Here's a simple parse on <span class="htlgb"> on fb app(in Erlang terms)

[
  {"span", [{"class", "htlgb"}], ["March 19, 2018"]},
  {"span", [{"class", "htlgb"}], ["Varies with device"]},
  {"span", [{"class", "htlgb"}], ["1,000,000,000+"]},
  {"span", [{"class", "htlgb"}], ["Varies with device"]},
  {"span", [{"class", "htlgb"}], []},
  {"span", [{"class", "htlgb"}],
...

So the parser needs to pretty much assume we are looking for span on class htlgb, then the first one is Update time, the second one is binary size, the third one is download number, etc. Who knows if Google will change the order, or change the class name, it will break again.

[lightalloy]

It seems like today Google changed the apps' page design (and HTML) fully, though yesterday it was mostly ok. The new parser needs to be written.
I know that @chadrem doesn't have time for that. I think I'll do the task, but I first need to discuss the issue with my colleagues.

[chadrem]

@lightalloy I hope your colleagues agree to it! PRs welcome!

[chadrem]

I've updated all the test data by running ./bin/update_test_data and pushed the commit to master. This results in most of the tests now failing. If anyone works on new scrapers then please make sure these tests pass.

[chadrem]

@refaelos submitted PR #74 that may fix the scrapers, but decreases code coverage. Is anyone else working on fixing the scrapers? If so, it would be helpful to coordinate here. I'd like to see any PR that fixes this problem maintain 100% test coverage. Lets communicate any fixes with scrapers in this issue so that we can all work together to keep market bot working.

[lightalloy]

As for the pull-request and the feature overall, I think that the code related to the old HTML should be removed, cause, as I see, Google doesn't serve the old version anymore.
Also, the tests in the PR remain the same as they were, so they still rely on the old HTML version which is in spec/market_bot/play/data which is incorrect. The downloaded files should be updated (replaced) and the tests should be reworked accordingly (if needed).
I will look more precisely and decide if it's possible to rework the PR, or I'll just make another one.

[lightalloy]

@chadrem I see you've updated the test data in the master branch 👍