Loading Profile...
Twitter,
Facebook, or email.
I have noticed that Google is indexing listings on my site that are
way outside of my area. I have no idea how these listing pages would be generated to begin with, because they are not in the zip codes, tracts or communities that I have designated either on my XML sitemap or via links or shortcode anywhere else on my site. Is this a known issue, or did I do something wrong?
I’m
frustrated
Listing pages generated outside of my areas
Official
Response
-
Hey all, I'm going to say this, and then solve this issue, but feel free to continue to discuss:
We can guess all day long how or why google indexed this or that city/zip/listing/etc. but honestly the only way to know is if you had someone sift through your logs to see how googlebot ended up on those pages.
Robert mentioned some ways google could have ended up there, but there are more. Such as: googlebot now submits forms on occasion. It's "possible" (though at this time unlikely) that they put their own value into the city dropdown. Anyway, that's probably not how they got to those pages, but it's meant to help show that there's no really GREAT way of telling you how. Unless you had someone look through your raw server logs, or you had an analytics app go through them (Google Analytics does not track googlebot FYI).
The bottom line is: googlebot is meant to index the web, and google is REALLY good at helping googlebot find all the pages on your site. Clearly it's an issue because google only indexes a limited # of pages on your site, and you want google to see the ones you care about. Theoretically you could setup your robots.txt to allow/disallow this or that url, but that's not entirely manageable (though possible).
With that said, this is something we've thrown around at the office, and we're considering a solution for the future, but I don't have any news or info I can give you now. But we're not ignoring this problem.
EMPLOYEE
Create a customer community for your own organization
Plans starting at $19/month
-
EMPLOYEE
I’m
trying to help
Can you please provide a URL sample or two of pages that have been indexed by Google that you did not intend to generate?Comment -
-
I’m
lost
Here are a couple:
http://listings.intownelite.com/idx/m...
http://listings.intownelite.com/idx/m...
Incidentally, these are no longer indexed in Google. (I'll never figure out the mystery that is Google.) Still, the pages are there and shouldn't be.
I have a theory on the subject: In Atlanta's FMLS, we have a field called "MLS Areas" which DS has mapped as "Communities." These MLS areas have a name and a number, e.g. "Atlanta North (21)". When I generate my XML site map with Communities, the parenthetical characters end up looking like this: /atlanta-north-%2821%29/
I know the link renders properly when clicked on, but maybe somehow the jumbled characters are causing the system to generate listings in other Communities outside of the intended area? I was thinking about deleting the Communities from my XML map anyway, since it's somewhat redundant with the zip codes I already have there. -
-
EMPLOYEE
I’m
confident
welllll.... the jumbled characters you see are standard replacements for non letters and numbers in URLs, so Google recognizes these and would not be looking in other areas or communities other than the indicated community.
As far as listing pages go... right now every listing in the IDX feed for your MLS is available on your site, regardless of whether you are linking to them or not. As an example, the listing below is 3 states away in Arkansas... but because it's in your IDX feed, it is a valid page that Google or other bots (or visitors) can view:
http://listings.intownelite.com/idx/m...
What makes the difference is:
If you never use shortcode, links, allow searching, or otherwise provide a link that would somehow someway get a visitor to the above listing... then no one would ever see it (or index it).
If you had listing pages that were indexed previously but are no longer being indexed, my guess is you had a link at some point somewhere that could have landed visitors onto those listings. Maybe when you first were experimenting with the plugin? -
-
I’m
not so confident
Nope. I don't know how those listings outside of my area could have possibly been generated. I did my "experimenting" on another domain entirely. On this site I don't have, and never have had, any widgets, shortcode, links or anything that could have pointed to those listings.
Incidentally, the other day I had around 100 listings indexed in Google. Today I have 10. All those listings are still on my site, why is Google no longer indexing them? I guess that's a topic for another thread. -
-
I’m
unsure
I see Virginia listings indexed on my site and have never set up any links or shortcodes to those areas. http://bethesdabuzz.com is my site and when I look at the site: I can see not only the VA listings but counties in MD that I've also never set up pages for.
Do I understand correctly that at some point every listing in my local MLS could potentially be indexed here as a result of using the shortcodes in static pages that I'm setting up for certain zip codes and communities? -
-
Hey all, I'm going to say this, and then solve this issue, but feel free to continue to discuss:
We can guess all day long how or why google indexed this or that city/zip/listing/etc. but honestly the only way to know is if you had someone sift through your logs to see how googlebot ended up on those pages.
Robert mentioned some ways google could have ended up there, but there are more. Such as: googlebot now submits forms on occasion. It's "possible" (though at this time unlikely) that they put their own value into the city dropdown. Anyway, that's probably not how they got to those pages, but it's meant to help show that there's no really GREAT way of telling you how. Unless you had someone look through your raw server logs, or you had an analytics app go through them (Google Analytics does not track googlebot FYI).
The bottom line is: googlebot is meant to index the web, and google is REALLY good at helping googlebot find all the pages on your site. Clearly it's an issue because google only indexes a limited # of pages on your site, and you want google to see the ones you care about. Theoretically you could setup your robots.txt to allow/disallow this or that url, but that's not entirely manageable (though possible).
With that said, this is something we've thrown around at the office, and we're considering a solution for the future, but I don't have any news or info I can give you now. But we're not ignoring this problem.- view 1 more comment
-
-
This is on our "todo" list, it hasn't been forgotten.
-
-
Cool, thanks!
-
-
-
-
Mike - I get out of area leads from some of my web sites too - this is a great opportunity to earn referrals... I have a couple agents I've established in areas that I don't serve that take leads from me and send me a nice referral fee back in return. They are happy, the client is happy, and I make some money.
When I send the lead to that agent, the agent contacts the consumer and says that they are a partner of mine that knows the area better. No buyer complaints as of yet.-
Aaron, that's a excellent tactic. However, as Jon said above, "...google only indexes a limited # of pages on your site, and you want google to see the ones you care about." I'd prefer the opportunity to make full commission over the referral fee. But you're right, that's a great idea to handle the out-of-area leads in the mean time.
-
-
-
-
-
You bet. Users can search beyond what is set up to index though too so this is something that may pay you dividends for a long time to come.
-
-
I’m
happy
Looks like a fix has been implemented!
Changelog
1.1.9
Prevent requests to /idx/ from working since the results on that page were showing a lot of properties outside the area and were diluting indexing results.
Thanks again guys.-
FYI, even after this change, Google is still indexing listings outside of my service area. I know DS doesn't control Google, but just letting you know.
-
-
-
-
-
Could you put in some wildcard masks in your robots.txt file? See here: http://www.google.com/support/webmast...
Blocking the URLs with the cities you don't want indexed might work?-
Thanks Aaron, I'll give that a shot.
-
-
-
-
-
I’m
ok
FYI, this is still happening, despite the version 1.1.9 fix. I have no idea how Google is finding these pages. I have a sitemap generator which crawls my entire site like a search engine spider/bot, and it does not find these out-of-area listings. They are simply not linked from anywhere in my site. Also, my drop-down lists only contain the city and zips that I'm targeting.- view 4 more comments
-
-
-
-
-
the /idx/page-x will stop working in the next dsIDXpress update. coming this week sometime.
-
-
I see that the update is already ready! I just updated and activated. Thanks again!
-
-
-
-
This is another thing that could be solved by some criteria masks that only allow listings with certain criteria be accessed through the plugin.
-
