Failure to recognize >50 item blasts

  • 9
  • Problem
  • Updated 7 years ago
I use Newsblur to keep up with some scientific journals' "Current issue" feeds, which sometimes dump 60-100 items in with the same timestamp. I have noticed that Newsblur consistently only recognizes 50 items. Two feeds where I have seen this behavior are http://feeds.feedburner.com/pnas/SMZM (today) and http://feeds.feedburner.com/jbc/SUcv (for the past few weeks) - it may also be relevant that they are both feedburner feeds. The missing articles never show up, as far as I can tell.
Photo of Michael Clarkson

Michael Clarkson

  • 2 Posts
  • 0 Reply Likes

Posted 7 years ago

  • 9
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
Yeah, if NewsBlur doesn't see the story in time, it won't show up. No way around that unless more people subscribe to the feed and it fetches more often. You can see by right-clicking on the feed and opening Statistics.
Photo of Michael Clarkson

Michael Clarkson

  • 2 Posts
  • 0 Reply Likes
So, the Proceedings feed is followed by 155 people and checked every 10-15 minutes according to the statistics. JBC is only followed by 9 but for some reason is set to real-time. I also saw this issue again today with the Nature Current Issue feed (http://feeds.nature.com/nature/rss/cu...) which is followed by 734 people, and has stories arriving in real time supported by a supplementary check every 2-2.5 hours. So, it seems to me that based on the stated fetch frequency I should be seeing all the items from these events in a timely manner.

Even were this not the case, I find your answer pretty confusing. I mean, regardless of how often it's being checked, it isn't like the tail ends of those blasts are just disappearing from the feed. They're still there; when I manually check the feed I can see them. If they're not recognized in the first check they should be picked up in the second, whenever that happens. So, if Proceedings publishes 75 articles to its feed within a few minutes on a Tuesday and only 50 show up on the first check, then the other 25 should show up 10 minutes later at the next check, right? But they never do. Those 25 items go into limbo and never appear, period.
Photo of Antti Honkela

Antti Honkela

  • 1 Post
  • 2 Reply Likes
I think this "feature" is making NewsBlur pretty much useless for following scientific journal feeds.

As far as I can tell, these feeds work by pushing the entire contents of one journal issue in one go. In case of the above mentioned Nature Current Issue feed, the latest issue seems to contain 73 stories. Very annoyingly NewsBlur only ever fetches the first 50, and the last 23 are lost forever.

Increasing the update frequency will not help because the publication of the 73 stories appears to be atomic. Instead the arbitrary limit of 50 stories per update should be raised much higher.
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
This seems to be one of the main issues with the dx.com feed also. (It has usually 3-500 items in each update, but only 50 is show'n, but most oftenly Newsblur times out when it's fetched)
Photo of Toussanter

Toussanter

  • 7 Posts
  • 1 Reply Like
Ok, yet another odd NewsBlur restriction.. this service is starting to look really bad. Sorry for saying but at first NewsBlur looked really great but constantly I'm finding out these restrictions (which do not go away even if I'd pay) which are NOT DOCUMENTED.. I'm starting to be really disappointed and it seems that I'll have to find another service.. damn you google for quitting reader.. it seems that there's really no good alternative around.

Here's one of my feeds that are affected by this max 50 items stupidity: http://www.levykauppax.fi/feed/coming/

Google Reader worked just fine with this feed.
Photo of ojiikun

ojiikun

  • 475 Posts
  • 62 Reply Likes
GReader suffered from this problem to a degree, too. Sadly, it is a fundamental problem with RSS feeds that don't use pubsub. Feed owners will complain if the feed is hit too often, but then update an XML doc that only holds 10 stories at a time at a rate such that you'd need to be polling it constantly.

I've had this problem with a dozen feeds over the last decade and every time, contacting the feed owner and asking them to increase the number of items returned or limit the burst-iness of updates solves it nicely.
Photo of Toussanter

Toussanter

  • 7 Posts
  • 1 Reply Like
For the past few days I've been testing NewsBlur, Feedly & g2reader side by side and testing DealExtreme feed (among other things too) and this max 50 items problem with NewsBlur.

NewsBlur: DX feed returns only 50 items. (today about 300 new items in the actual RSS feed)

g2reader: seems to fetch all ~300 items but all are marked as "read" automatically because the site puts the feed in to "passive mode" because it contains hundreds of new items every day.

Feedly: all ~300 items are fetched AND all are marked as unread!!!

So long NewsBlur, for now.. Feedly is a great service (it doesn't have many other restrictions that NewsBlur has). It's not perfect, it has some minor things to fix/develop, but it really seems to be the only choice to replace Google Reader (and I've tested over 10 alternatives).

Good luck NewsBlur, keep developing your service, currently it's "number 2" in my list, and I might check back someday. Thanks for the hard work.
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
Samuel, Can you please respond to this or one of the other two tickets that report the issue with DX.com only recieving 50 items each time the feed is updated with 300+ items?
As a paying member I must say that the lack of response on this is really bad.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
Markus, this is the third thread you've posted in for the same issue. Please stop posting the same issue, especially as I answered your question already. Not only that, but I answered this thread above, as well.

As a paying member, you're entitled to a refund if NewsBlur isn't fulfilling your needs. Email me and I'd be happy to give you one.
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
No, you haven't touched on the issue above.
The timeout's you commented on 6 month's ago is not the same thing as the limitation that makes the DX and other feeds stop at 50 items.

The reason I did this mass-post was that you didn't respond to anyone before. It seems to have worked.
I would prefer if I could use NewsBlur, but if this isn't something that you plan to fix, please let me know.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
OK, I bumped the limit from 50 to 100 at a time. I hope it doesn't have any negative effect on my feed fetchers performance, because if it does, that limit, which was put in there for good reason, is going right back in. But I have a feeling I'll be able to absorb it just fine.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
Does that mean that we will get all 300 new items, but at 100 items at a time (and that NewsBlur will loop through the feed more than one time) or does that mean that feeds that update more than 100 items at a time will keep loosing them?

You claim that the limit is 500 items per feed in this ticket:
https://getsatisfaction.com/newsblur/...
But is that only possible if the site that creates the feed trickles the items at max 100 items at a time?
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
?
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
The max is 100 stories at a time.
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
So, any feed that has more than 100 updates at a time will just be truncated and the additional items will be lost forever without any notice to the user?!?
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
?
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
?
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
!
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
...
Photo of lgladdy

lgladdy

  • 5 Posts
  • 0 Reply Likes
Markus,

It's pretty clear that's exactly what this means.

I think you're aiming to get support for a severe edge case of usage here, one that would severely impact all the other users if it were to be implemented.

I'm pretty sure i'm correct in saying that if you wanted too, you could just grab the newsblur source from github, make the required change, and run it yourself somewhere if you want to do unsupported things.
Photo of Markus Reimer

Markus Reimer

  • 12 Posts
  • 0 Reply Likes
You think it's an edge case to have a feed that updates with more than 100 items at a time? (Or rather, between each poll from NewsBlur)
As you can see here it's many different feeds that have this problem, and I guess that there are 10-50 feeds with annoyed users for each feed that is mentioned here.

And *every* other rss reader solution handles this without any problem.

Sure, I do understand that you might have a limit on the number of items that you parse on each pass, and I do understand why you want to have a limit on the total number of items in a feed (even if the current limit of 500 is a tad on the low). But I don't see how it can be acceptable to drop items at random in a feed just because it was to much to read at one pass.
At least there should be some indication to warn the user that you don't have a complete feed so that you can go to the source and see what you missed!

Just silently dropping items in a feed isn't an acceptable practice.

And "code it yourself" isn't a good answer to a bug in a system with paying users.
Photo of lgladdy

lgladdy

  • 5 Posts
  • 0 Reply Likes
It certainly isn't a bug though is it?

A bug is a unintended defect in code.

This is an imposed limit for the benefit of the speed and sustainability of the service...

Samuel's already offered you a refund if this is a showstopper for you, and if other services do it, why not just move to one of them? I never really understand the argument of "well other service does it" when you have the option to switch...

In my experience of trying all the other services before settling on NewsBlur, they all had significant disadvantages too!
Photo of Marcus Brayman

Marcus Brayman

  • 35 Posts
  • 2 Reply Likes
Markus,

Newsblur simply works like this, just as Samuel explained clearly in this thread. Almost no one has a problem with that, including me. He is *not* going to change that any further.

If you don't like that then please ask for a refund, as Samuel proposed.
Photo of Toussanter

Toussanter

  • 7 Posts
  • 1 Reply Like
Markus, it's pretty obvious that Mr. S. Clay is not willing to openly discuss or document all these restrictions he has put in his product. I don't like it at all, a service that is trying to hide several limitations.. good riddance I say.

Take the offered money back and switch to something else that actually works. E.g. Feedly. I'm very happy with it.