Facebook  Twitter 


+- +-


+- User Information

Welcome, Guest.
Please login or register.
Forgot your password?

+- Forum Stats

Total Members: 12373
Latest: smurfs
New This Month: 0
New This Week: 1
New Today: 0
Total Posts: 40210
Total Topics: 7081
Most Online Today: 95
Most Online Ever: 2482
(April 09, 2011, 07:02:45 pm)
Users Online
Members: 0
Guests: 32
Total: 32

Author Topic: Archiver Mod info and robot.txt  (Read 4490 times)

0 Members and 1 Guest are viewing this topic.

Offline GameSocket

  • Jr. Member
  • **
  • Posts: 79
  • NZ Made
    • View Profile
    • GameSocket
Archiver Mod info and robot.txt
« on: June 22, 2006, 04:06:56 am »
I guess I have seen an advantage of installing this mod, as today I was crawled by "ia_archiver".
On Investigating this is what I have found.

The crawler is Alexa crawler (robot), which identifies itself as ia_archiver.
Whenever ia_archiver lands on the top level of a Web site, it looks for a file called "robots.txt". Robots.txt is a file website administrators can place at the top level of a site to direct the behavior of web crawling robots.

A crawler will always pick up a copy of the robots.txt file prior to its crawl of the site.

To exclude all robots, the robots.txt file should look like this:

User-agent: *
Disallow: /
To exclude just one directory (and its subdirectories), say, the /images/ directory, the file should look like this:

User-agent: *
Disallow: /images/

Web site administrators can allow or disallow specific robots from visiting part or all of their site. Alexa's crawler identifies itself as ia_archiver, and so to allow ia_archiver to visit (while preventing all others), your robots.txt file should look like this:

User-agent: ia_archiver
To prevent ia_archiver from visiting (while allowing all others), your robots.txt file should look like this:

User-agent: ia_archiver
Disallow: /

For more information regarding robots, crawling, and robots.txt visit the Web Robots Pages at http://www.robotstxt.org, an excellent source for the latest information on the Standard for Robots Exclusion.

In any event, simply by visiting your site with the Alexa Toolbar open, Alexa will learn of your site and add it to our list of sites to visit, thus ensuring your inclusion in the Alexa service and in the Alexa archive.
If you are the type of person who won't be satisfied until you get to click a button that says "Crawl My Site," then Alexa have just the form for you. 


I have not been crawled by Alexa before installing this mod.

(O.o )   *If You need help, best not to ask me*
(> < )

Offline SMFHacks

  • Administrator
  • Hero Member
  • *****
  • Posts: 15105
    • View Profile
Re: Archiver Mod info and robot.txt
« Reply #1 on: June 22, 2006, 06:42:26 am »
Good news. I think it is better since it allows the search engines to find the boards and threads easier without going though all the other links they find.

Get your Forum Ranked! at https://www.forumrankings.net - find out how your forum compares with others!

Like What I do? Support me at https://www.patreon.com/vbgamer45/


Related Topics

  Subject / Started by Replies Last post
1 Replies
Last post January 29, 2009, 09:30:15 pm
by SMFHacks
1 Replies
Last post July 11, 2009, 02:49:38 pm
by SMFHacks
0 Replies
Last post March 14, 2010, 12:11:07 am
by nin79
0 Replies
Last post December 23, 2010, 07:02:57 am
by morokat
3 Replies
Last post July 10, 2011, 06:19:11 am
by cosmicx

+- Recent Topics

Download System Lite by Rock Lee
June 03, 2020, 07:34:24 pm

Font question by SMFHacks
May 27, 2020, 08:15:26 am

Error message with latest SMF 2.1 Github build by Hatshepsut
May 25, 2020, 01:43:26 am

smfblog not working on 2.0.17 by tech9
May 20, 2020, 01:44:34 pm

Copyright removal by stbc
May 18, 2020, 01:27:57 am

Mod Verified User i can't square the image by Rock Lee
May 07, 2020, 07:56:10 pm

SMF4Mobile 2.0 released for SMF 2.0.x by SMFHacks
May 06, 2020, 12:13:14 pm

SMF Social Login Pro - Discontinued? by Nicole
May 02, 2020, 05:47:04 pm

Likes by SMFHacks
April 30, 2020, 09:50:35 pm

Upgrade Issue - by SMFHacks
April 28, 2020, 12:40:13 pm

Powered by EzPortal