[Web] [Development] Search engine indexing of qt-project.org
jgastal at profusion.mobi
Wed Jun 12 15:14:45 CEST 2013
On Tue, Jun 11, 2013 at 1:30 PM, Jonas Gastal <jgastal at profusion.mobi>wrote:
> Interestingly, doing wget I get the same as Giuseppe, using the
> browser(chromium) I get the same as Robin.
> On further analysis the reason I was getting the disallow everything
robots.txt in chromium was because of https everywhere.
I don't see much of a reason to block https crawling, on the other hand I
doubt that's what's causing bad results for google searches.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Web