[Dspace-general] Check that your DSpace instance can be indexed

[Robert Tansley]

All,

We're noticing that many DSpace instances still have misconfigured
robots.txt files which are preventing them from being indexed by search
engines.

In short -- please ensure that *NEITHER* of the following lines are present
in your robots.txt file at http://your.dspace.url/robots.txt:

Disallow: /browse
Disallow: /browse-date

If either line is present (as accidentally shipped with DSpace 1.5 and
1.5.1), search engines may be prevented from indexing any content in your
DSpace instance.

For more information on how to ensure your DSpace is indexed correctly,
including making use of Sitemap support, see:

http://wiki.dspace.org/index.php/Ensuring_your_instance_is_indexed

Thanks,

Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/pipermail/dspace-general/attachments/20090511/28753a13/attachment.htm