The Verge RSS feed not Updating

  • Problem
  • Updated 3 years ago
When I add http://www.theverge.com/rss/index.xml to NewsBlur it doesn't get any new items after Oct. 30th (when the site switched from being ThisIsMyNext to The Verge.) even though they're in the RSS.

If it helps, my username is "justjohn"
Photo of John Roepke

John Roepke

  • 4 Posts
  • 0 Reply Likes
  • sad

Posted 4 years ago

  • 2
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 5467 Posts
  • 1225 Reply Likes
A number of users have mentioned that The Verge's RSS feed doesn't work. I checked on it and confirmed that something is going on. I have e-mailed them to let them know.

Here's what's going on, for those interested (and can read Python):

>>> import feedfinder
>>> feedfinder.feed('http://theverge.com')
'http://theverge.com/rss/index.xml'

>>> import feedparser
>>> fp = feedparser.parse('http://theverge.com/rss/index.xml')
>>> fp
{'feed': {}, 'status': 200, 'bozo': 1, 'headers': {'status': '200 OK', 'content-length': '19383', 'via': '1.1 sbnation.com', 'content-encoding': 'deflate', 'vary': 'Accept-Encoding', 'x-runtime': '578', 'connection': 'Keep-Alive', 'etag': '"fd2f3a5b33a1a706fea1dec62ad25df3"', 'cache-control': 'private, max-age=0, must-revalidate, private, max-age=0, must-revalidate', 'date': 'Fri, 04 Nov 2011 00:06:59 GMT', 'p3p': 'CP="CAO DSP COR CURa ADMa DEVa PSAa PSDa CONi OUR IND PHY ONL UNI COM NAV INT CNT STA"', 'content-type': 'application/xml; charset=utf-8'}, 'etag': u'"fd2f3a5b33a1a706fea1dec62ad25df3"', 'href': u'http://www.theverge.com/rss/index.xml', 'entries': [], 'bozo_exception': error('Error -3 while decompressing data: incorrect header check',)}
>>> fp.entries
[]
>>> fp.bozo
1
>>> fp.bozo_exception
error('Error -3 while decompressing data: incorrect header check',)
Photo of John Roepke

John Roepke

  • 4 Posts
  • 0 Reply Likes
I just tried the same thing you posted on my machine (Ubuntu 11.10) and I get the correct feed, so their feed isn't completely broken (at least for me):

import feedfinder
import feedparser

url = feedfinder.feed("http://theverge.com")
fp = feedparser.parse(url)
fp
{'feed': {'updated': u'2011-11-05T20:14:06Z', 'subtitle': u'', 'updated_parsed': time.struct_time(tm_year=2011, tm_mon=11, tm_mday=5, tm_hour=20, tm_min=14, tm_sec=6, tm_wday=5, tm_yday=309, tm_isdst=0), 'language': u'en', 'title': u'The Verge - All Posts', ...lots more entries...

I tried with python 2.7.2 & feedparser 5.0.1
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 5467 Posts
  • 1225 Reply Likes
No dice. Trying on my local Mac (Python 2.7.1):

>>> import requests
>>> r = requests.get('http://www.theverge.com/rss/index.xml')
>>> r.headers
{'status': '200 OK', 'content-length': '5972', 'via': '1.1 sbnation.com', 'content-encoding': 'deflate', 'vary': 'Accept-Encoding', 'x-runtime': '137', 'connection': 'Keep-Alive', 'etag': '"365ff5ec199a2c8d5b50ccbe2a5c3dda"', 'cache-control': 'private, max-age=0, must-revalidate, private, max-age=0, must-revalidate', 'date': 'Sat, 05 Nov 2011 23:06:05 GMT', 'p3p': 'CP="CAO DSP COR CURa ADMa DEVa PSAa PSDa CONi OUR IND PHY ONL UNI COM NAV INT CNT STA"', 'content-type': 'application/xml; charset=utf-8'}
>>> import zlib
>>> zlib.decompress(r.content)
Traceback (most recent call last):
File "", line 1, in
error: Error -3 while decompressing data: incorrect header check
>>>
Photo of John Roepke

John Roepke

  • 4 Posts
  • 0 Reply Likes
I don't know if you need to use zlib to decompress the data from the request. You should be able to access r.content directly.

I was able to get content (albeit with some interesting unicode at the beginning) from the request on my Mac (Python 2.7.1) so it sounds to me like there's some proxy or out of date cache in the way if you're not getting any new content.

--
Here's the headers/content I got

>>> r.headers

{'status': '200 OK', 'content-length': '19519', 'via': '1.1 sbnation.com', 'content-encoding': 'deflate', 'vary': 'Accept-Encoding', 'x-runtime': '108', 'connection': 'Keep-Alive', 'etag': '"9b5325d4194bc6c6dd3c2c29a24c36a7"', 'cache-control': 'private, max-age=0, must-revalidate, private, max-age=0, must-revalidate', 'date': 'Sun, 06 Nov 2011 04:12:49 GMT', 'p3p': 'CP="CAO DSP COR CURa ADMa DEVa PSAa PSDa CONi OUR IND PHY ONL UNI COM NAV INT CNT STA"', 'content-type': 'application/xml; charset=utf-8'}

>>> r.content
 

u'\x02\x00\x00\x00\ufffd\ufffd\x00\ufffd\x1f\x04\ufffd<?xml version="1.0" encoding="UTF-8"?>\n<feed xml:lang="en" xmlns="http://www.w3.org/2005/Atom">\n <title>The Verge - All Posts</title>\n <subtitle></subtitle>\n <updated>2011-11-06T03:54:02Z</updated>\n <id>http://www.theverge.com/rss/index.xml</id>\n <link type="text/html" rel="alternate" href="http://www.theverge.com/"/>\n <entry>\n <published>2011-11-06T03:54:02Z</published>\n <updated>2011-11-06T03:54:02Z</updated>\n <title>US Cellular
Photo of John Roepke

John Roepke

  • 4 Posts
  • 0 Reply Likes
For anyone interested, I've decided to just work around this issue until it's resolved by proxying the feed through a server I control. If anyone else wants to do the same here's the PHP script I'm using:



<?php
print(file_get_contents('http://www.theverge.com/rss/index.xml'));
?>
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 5467 Posts
  • 1225 Reply Likes
I just emailed them again yesterday. They don't see a problem, but I see the issue on both my machine at home and the servers.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 5467 Posts
  • 1225 Reply Likes
And fixed. Dear lord, I had to track down the problem to an out-of-date feedparser error. Keep 'em coming! (Also, this means that you might want to check any of your other feeds that are throwing 500s)
Photo of Matt

Matt

  • 22 Posts
  • 4 Reply Likes
I know this is dragging up an old one, but The Verge doesn't appear to be updating for me now. Is anyone else seeing this? I've not had a new story show up in there since March 7.
Photo of Robert Wawrzyniak

Robert Wawrzyniak

  • 28 Posts
  • 0 Reply Likes
I'm seeing this as well but because I have some more feeds like The Verge in my list, I didn't recognize it...

Seems to be a nasty bug though.
Photo of Samuel Clay

Samuel Clay, Official Rep

  • 5467 Posts
  • 1225 Reply Likes