Significant Delay for updates or no updates for some feeds

  • 21
  • Problem
  • Updated 6 years ago
Yesterday someone answered a question I asked about feeds being real time under normal circumstances. I am sympathetic to the current challenge supporting an mass influx of users. The answer was that most of the performance problems have been resolved as of Sunday PM and that all feeds are updated in real time.

I created my account yesterday and have yet to see 75% of new articles appear in NewsBlur unless I manually update each feed. I currently have 59 sites. I am not seeing updates for nearly all of the less popular feeds. Some of the more popular feeds are updating very slow - many hours behind.

Are things operating normally overall or do I have a problem with my specific account?

Thanks
Photo of rfc1918

rfc1918

  • 8 Posts
  • 1 Reply Like

Posted 7 years ago

  • 21
Photo of Silver Leaf

Silver Leaf

  • 1 Post
  • 0 Reply Likes
Same problem here, with both newsblur.com and dev.newsblur.com. At best, all my feeds are several hours behind. Even manual updates don't seem to load new articles. Currently there are no new articles showing in NewsBlur for my local news feed. That same feed in Google Reader has at least 15-20 new articles over the past 6-7 hours.
Photo of raticus

raticus

  • 1 Post
  • 0 Reply Likes
I'm also having this issue with several of my news feeds, but it appears that manually updating them does the trick.

Edit: There is a feed that even manually updating it, doesn't show an update that Google Reader does show.
Photo of Carol H Tucker

Carol H Tucker

  • 17 Posts
  • 1 Reply Like
I cleared everything out of Google Reader and NewsBlur two hours ago. Google Reader is showing 36 new items; NewsBlur 4. Some of the feeds haven't been updated in Newsblur since I opened a premium account.

Hopefully things will even out before July.....
Photo of Sufo

Sufo

  • 14 Posts
  • 2 Reply Likes
I have to say this is disconcerting, If I don't double check with the old Google Reader, I'm missing articles sometimes by hours. It's taking twice as long with having to manually update some of my feeds to make sure I get what's current. Having said that, it's my only issue and I do hope it gets resolved soon otherwise I'm not sure it was worth paying for a premium account at this stage.
Photo of Carol H Tucker

Carol H Tucker

  • 17 Posts
  • 1 Reply Like
I could put up with a delay -- I don't like it but I could tolerate it. What has me worried is that some feeds aren't showing up at all
Photo of Andrew Auletta

Andrew Auletta

  • 23 Posts
  • 1 Reply Like
I'm having the same problem, and share your concern. Sometimes, when you compare the articles in newsblur side-by-side with the same feed in Google, there are Google updates that don't show up in newsblur until hours later. Until you scale newsblur to replicate the consistency of google, I'm extremely concerned.
Photo of Marvin K Mooney

Marvin K Mooney

  • 33 Posts
  • 2 Reply Likes
I can accept a delay of a few hours, but not several days as I see on some feeds.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
The scaling that I need to do to the backend is about a week away from shipping. It's a very involved process that requires me to do a lot of prep work and testing. Once that's in place, I will crank up the feed fetching schedule to what it once was, which is every feed held by a premium user is updated at the minimum of once an hour. Popular feeds more often, and if the feed has published less than an average of 1 story over the past month, then it gets updated just a few times a day.

But right now I have had to turn feed fetching speed down. I turn it up every day, and you can monitor my progress by watching the Feeds Fetched number on the dashboard go up. It's at 1MM now, but I expect it to touch up to 5MM a day.
Photo of Kádár Tamás

Kádár Tamás

  • 53 Posts
  • 6 Reply Likes
Thanks for the transparency. much appreciated.
Photo of Sufo

Sufo

  • 14 Posts
  • 2 Reply Likes
Thanks! Your work here is appreciated despite the issues some have experienced.
Photo of bill.waldron

bill.waldron

  • 29 Posts
  • 11 Reply Likes
Yes, thanks. After paying for premium I was also seeing some of my less popular feeds updating VERY slowly -- but had thought that they'd be checked hourly, and was starting to wonder...
Photo of cjrecord

cjrecord

  • 32 Posts
  • 1 Reply Like
As important as all the hard coding you're doing is taking the five here and there to let us know what's going on!
Photo of rfc1918

rfc1918

  • 8 Posts
  • 1 Reply Like
This clears up everything. It makes sense you had to lower the interval for awhile, even for premium members. I will keep monitoring. Let us know how the new scaling efforts go.
Photo of Andrew Auletta

Andrew Auletta

  • 23 Posts
  • 1 Reply Like
The immediate example of my issue with feed updating is my USAToday feed. The last article has a time stamp of 8:26 a.m. this morning. Google newsreader shows the lastest article with a time stamp of 8:22 p.m. today. Why?
Photo of rfc1918

rfc1918

  • 8 Posts
  • 1 Reply Like
After last night's upgrade things are improving quite considerably. My feeds are updating in Newsblur just about as they update in Google Reader.

I think things are looking up.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
That's awesome! Yes, I have spent the last two days significantly speeding up the queuing system and fetching system. And I have yet to introduce a huge change that will work 10-20x faster for queuing feeds (which was becoming my limiting reagent in the feed fetch equation). Now I just need to finish the sharding of mongo, which is now considerably easier to do after having completing a 2 day data migration that re-establishes a non-sparse unique index on stories, so I can set it as my shard key and not have to use an index prefix. That will allow me to spin up more task fetchers which will be needed as usage increases. Hurray!
Photo of Sufo

Sufo

  • 14 Posts
  • 2 Reply Likes
Vast improvement in such a short space of time! Well done and thanks for all your hard work
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
Oh, and once this is all done I'm working on shipping the re-design, then fixing all the damn bugs that thousands of people now have their eyes on. And THEN new features!
Photo of Janky99

Janky99

  • 3 Posts
  • 0 Reply Likes
Good to hear Samuel! ;-9 Nice work. Let us know once done..
Photo of Diana

Diana

  • 43 Posts
  • 6 Reply Likes
I'm still experiencing this as of this afternoon (4/1). Several feeds show no new items, but instafetching reveals one or more unread items, some 3 or 4 days old. I'm content to be patient if this is still part of NewsBlur's growing pains, but wanted to confirm. Thanks!
Photo of Derrick Tennant

Derrick Tennant

  • 8 Posts
  • 0 Reply Likes
The feed for my local newspaper doesn't seem to update: http://www.gcdailyworld.com/feed/rss/...

I've tried deleting and re-adding it a few times, but after everything is read... it just stays empty.
Photo of Chan Kai Tan

Chan Kai Tan

  • 1 Post
  • 0 Reply Likes
The feed here @ http://www.the-magicbox.com/rssparser... does not seem to be correct. I'm only seeing articles up till "Arc System Works announced Guilty Gear Xrd: Sign....". Re-adding the feed doesn't help at all.
Photo of Will Cicola

Will Cicola

  • 1 Post
  • 0 Reply Likes
Most of my feeds are working perfectly, but this one:

http://www.balloon-juice.com/feed/

hasn't updated for nearly two weeks. I tried removing and re-adding it, but NewsBlur just hangs after I click "Add".
Photo of CM

CM

  • 1 Post
  • 0 Reply Likes
+1 re: balloon-juice. Newsblur is also adding a new [Untitled] site for b-j every time I try to parse the feed (which is not updating with the website, the last post showing in the newblur feed is a month ago)
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
I think instead of resurrecting a thread like this, you should create a new one. As for Balloon Juice, try going to the feed: https://www.balloon-juice.com/feed/. It doesn't work.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 6514 Posts
  • 1474 Reply Likes
There, I fixed it. It was declared as a https, but http worked, so I switched it over. You could do the same by right-clicking on the feed and opening Site Settings. You can enter http://www.balloon-juice.com/feed/ there. But it should now work.
Photo of Ilkka Salminen

Ilkka Salminen

  • 0 Posts
  • 0 Reply Likes
Also, Newblur hasn't updated What If feed (https://www.newsblur.com/site/1272495...) for a while now. Site settings show:
2013-07-17 08:00:32 Not modified (304)

The feed itself says it was last updated on July 16th:


<updated>July 16, 2013</updated>
<entry>
<title>Drain the Oceans: Part II</title>
<link href="http://what-if.xkcd.com" />
<id>http://what-if.xkcd.com</id>
<published>2013-07-16T00:00:00Z</published>
Photo of cjrecord

cjrecord

  • 32 Posts
  • 1 Reply Like
For what it's worth for diagnostics, off my copy of it:

FEED FETCH
2013-07-17 08:02:44
Not modified (304)
2013-07-17 07:02:08
Not modified (304)
2013-07-17 06:06:10
Not modified (304)
2013-07-17 05:13:04
OK (200)
2013-07-17 04:13:59
Not modified (304)

PAGE FETCH
2013-07-17 07:02:08
OK (200)
2013-07-17 05:13:04
OK (200)
2013-07-17 02:22:28
OK (200)
2013-07-16 20:27:38
OK (200)
2013-07-16 18:35:18
OK (200)

I usually recall that it shows up by end-of-Wednesday for What If posts on Tuesday, so....
Photo of Angela Jones

Angela Jones

  • 0 Posts
  • 0 Reply Likes
It has been awhile since I've caught up on my news feed, and I noticed that NewsBlur has dropped some of the older posts that I hadn't seen and marked them as read.
Photo of Knic Pfost

Knic Pfost

  • 5 Posts
  • 3 Reply Likes
Google Reader did the same thing (after 30 days). Newsblur leaves things unread for 2 weeks i believe. According to Samuel on another thread, maintaining unread status across this many accounts for a significant amount of time requires an unbelievable amount of space.

https://getsatisfaction.com/newsblur/...
Photo of Tim Lavoie

Tim Lavoie

  • 4 Posts
  • 0 Reply Likes
I've just tried adding a new feed (http://feeds.feedburner.com/TeefuryDa...), and it shows within Newsblur as having been untouched for months. A quick manual check with my browser or curl does show that it is in fact updated daily. Since their product is time-limited, regular updates are pretty vital on their end.
Photo of Marvin K Mooney

Marvin K Mooney

  • 33 Posts
  • 2 Reply Likes
How is it working now? Any improvement?
Photo of Andrew Auletta

Andrew Auletta

  • 23 Posts
  • 1 Reply Like
I gave up on the USA Today feed. With a timestamp between 8 a.m. and 8:30 a.m. each date, it doesn't update again until late in the day or early evening.
Photo of Tim Lavoie

Tim Lavoie

  • 4 Posts
  • 0 Reply Likes
Brick House: No, no change to the Teefury feed, still shows April 8 2013 as the last update.
Photo of Marvin K Mooney

Marvin K Mooney

  • 33 Posts
  • 2 Reply Likes
Wonder if Newsblur staff are even reading this thread, as they don't seem to be responding. April 8 was 4 months ago.
Photo of Marvin K Mooney

Marvin K Mooney

  • 33 Posts
  • 2 Reply Likes
In another thread I was told that feeds are retrieved "regularly". Is every 4 months "regularly"?
Photo of Marvin K Mooney

Marvin K Mooney

  • 33 Posts
  • 2 Reply Likes
Have you tried looking at it outside newsblur to see if there is actually any content to fetch?

It's one thing if newsblur isn't attempting to fetch. But, if there simply isn't any content, that's not really a newsblur issue to solve.
Photo of Tim Lavoie

Tim Lavoie

  • 4 Posts
  • 0 Reply Likes
Yes, the URL I posted earlier (http://feeds.feedburner.com/TeefuryDa...) is updated daily, checked last night and again just now.
Photo of Pascal Groell

Pascal Groell

  • 17 Posts
  • 2 Reply Likes
When subscribing to the URL you show it only pulls one item dated 09 April 2013.
I tried the XML subscription option shown on the page but could not get other items.
I tried it in Thunderbird and it worked but pulled only one item dated yesterday.