Inside Google Sitemaps

New name better reflects our commitment to communicate with you

2006-08-04T15:46:00.000-07:00

We're surrounded by moving boxes and construction dust. To better reflect our goal to provide tools and information to keep you, the webmasters, informed and help you increase your crawl coverage and visibility on Google, we're renaming Google Sitemaps to Google webmaster tools. The Sitemaps protocol remains unchanged and Sitemaps submission mechanisms and reporting is still available from the Sitemaps tab. We're also renaming our blog and Google Group to be more broadly focused on webmaster topics.

You can find handy links to all the tools, the new blog, Google Group, help center and more from our newly launched Google webmaster central. Join us at the blog's new location at googlewebmastercentral.blogspot.com where you'll learn about all the exciting things we're planning. Oh, and please don't forget to update your bookmarks and feed!

BlogHer Tips

2006-08-03T18:16:00.000-07:00

Yesterday's post was about how BlogHer attendees (and anyone else) can ensure that their web pages get indexed in Google. Here are more tips:

If you want to keep Google from indexing some pages of your site, you can use a robots.txt file, a simple text file that you place in the root of your site. We provide detailed information about creating a robots.txt file. You can also use the robots.txt analysis tool in Google Sitemaps to make sure that your file blocks and allows the pages you intend.

If Google has indexed content that you don't want to appear in search results, you have two options:

If you want to keep the content on your site, you can block that page with a robots.txt file and then request that we remove the page from our index;
If you remove the content from your site (for instance, you decide that you revealed information that is too personal and you edit your blog post), you can request a cache removal.

Good content keeps visitors coming back and compels other sites to link to you. In addition to blog posts, you can provide othe types of content, such as video. A number of video hosting sites are available including YouTube, Yahoo! Video, and our very own Google Video. At the conference, some attendees were asking about copyright, and to answer that question, you retain all rights to any content you upload to Google Video.

If you're interested in running ads on your site, take a look at AdSense. The ads are related to what you're talking about that day on your blog -- and you can control what ads display.

Time to verify

2006-08-03T17:25:00.000-07:00

Very soon, we'll be doing one of our periodic checks of verified sites. As our documentation notes, we do this to make sure that the verification HTML file or meta tag still exists. If we don't find the file or tag during this check, the account will no longer show that site as verified, and you'll no longer have access to diagnostics and statistics. If you find that a site that previously verified no longer is, simply click the "Verify" link on the My Sites page and upload the file or meta tag again.

Back from BlogHer

2006-08-02T09:38:00.000-07:00

I just got back from BlogHer, a conference primarily for women about the technical and community aspects of blogging. As a woman who blogs, I had a wonderful time. As a woman who blogs about topics of interest to site owners, I gained some new perspectives.

These bloggers tend to already put into practice a lot of the things we tell site owners who ask how to get more of their context indexed and make it easier to find through Google searches. As group, they:

Provide unique perspectives and content on topics
Think about their visitors, making sure the sites meet their visitors' needs
Give visitors reasons to come back, and other sites reasons to link to them (they update content regularly, and most offer in-depth information, such as tutorials, reviews, or in-depth explanations).

The panelists understood visitor awareness. They told the crowd to look closely at their sites to determine how unique they were from other sites out there. They talked about getting visitors to care. This thoughtfulness leads to great sites, which in turn can lead to great search results.

Given all of the dedication these bloggers put into their sites, they are of course interested in attracting visitors. Some want the joy of sharing; others are interested in making money from their writing. Here are a few tips to help make your site easier to find, whatever your motivation:

1) High-quality links from other sites help bring visitors to your site and can help your site get indexed and well-ranked in Google. The number one question asked at this conference was "What is your site?" People want to know so they can go read it, and if they like it, link to it. But the number one answer to this question was the name of the site, not the URL. Bloggers had T-shirts of their sites available, and many of those didn't have URLs. Tell people your URL so they can find you, read you, and link to you!

2) Make sure Google can crawl your site. We use an automated system (called "Googlebot") to visit pages on the web, determine their contents, and index them. Sometimes, Googlebot isn't able to view pages of a site, and therefore can't index those pages. There are two primary things you can do to check your site.

a. Read our webmaster guidelines to learn about how we crawl sites and what can make that easier.

b. Sign up for a Google Sitemaps account to see a list of errors we encountered when we tried to crawl your site. This way you can find out if we can't reach your site, and why. Once you sign up for a Google Sitemaps account and add your site URL, you need to verify site ownership before you can see diagnostic information. To do this, simply click the Verify link and follow the steps outlined on the Verify page.

You can verify site ownership in one of two ways:

Upload an HTML file with a specific name to your site
Add a <meta> tag to your site's home page

Once you verify site ownership, you can see other details about your site in addition to errors. For instance, you can see what visitors are searching for when they click in the Google search results. You can see this for visitors searching our web index as well as blogsearch. You can also see what searches visitors do to find your images.

You can also see what words other sites use to link to you (which helps explain why your site might show up for searches that you think are unrelated to your site).

3) Submit an RSS feed of your site to quickly tell us about all of your pages, so we know to crawl and index them.

Support for Polish

2006-07-14T15:13:00.000-07:00

Google Sitemaps has added support for Polish. If you already use Google in Polish, you should see the Sitemaps user interface in Polish automatically. Otherwise, you can click the Preferences link from the Google home page and choose Polish from the interface list.

The webmaster help center is also now available in Polish. This includes all the content in the help center, including our webmaster guidelines. Simply choose "Polish" from the Change Language menu. We also have a Polish Google Group for discussing Sitemaps and other webmaster issues.

More control over titles too

2006-07-14T10:37:00.000-07:00

Yesterday we told you that you can use a meta tag to ask us not to use descriptions of your site from the Open Directory Project (ODP) when we generate snippets. Some of you have asked if this meta tag prevents us from using the title from the ODP as well. Yes, this meta tag does apply to both the title and description from the ODP.

Also, as noted in our webmaster help center, you can combine parameters in the meta tag. So, for instance, you could use the following meta tag:

<meta="robots" content="noodp, noarchive">

More control over page snippets

2006-07-13T10:56:00.000-07:00

The way we generate the descriptions (snippets) that appear under a page in the search results is completely automated. The process uses both the content on a page as well as references to it that appear on other sites.

One source we use to generate snippets is the Open Directory Project, or ODP. Some site owners want to be to able to request not using the ODP for generating snippets, and we're happy to let you all know we've added support for this. All you have to do is add a meta tag to your pages.

To direct all search engines that support the meta tag not to use ODP information for the page's description, use the following:

<META NAME="ROBOTS" CONTENT="NOODP">

Note that not all search engines may support this meta tag, so check with each for more information.

To direct Google specifically from using this information to describe a page, use the following:

<META NAME="GOOGLEBOT" CONTENT="NOODP">

For more information, visit the webmaster help center.

Once you add this meta tag to your pages, it may take some time for changes to your snippets to appear. Once we've recrawled your pages and refreshed our index, you should see updated snippets.

Tips for Non-U.S. Sites

2006-07-07T15:38:00.000-07:00

If you're planning to be at Search Engine Strategies Miami next Monday and Tuesday, stop by and say hi at the Google booth. I'll be there answering any questions you may have about Sitemaps. This event is all about the Latin American market, so here's some information you might find helpful if your site is in Latin America or elsewhere.

If you want your site to show up for country-restricted searches, make sure it uses a country-specific domain (such as www.example.com.br). If you use a domain that isn't country specific (such as .com), make sure that the IP address of the site is located in that country.

If you want to know what visitors from different countries are searching for, take a look at the query stats in Sitemaps. This lets you see the difference in searches for each location, as well as what languages visitors use to type in their queries.

Yahoo! Merchants get Sitemaps

2006-06-28T15:34:00.000-07:00

Yahoo! has announced that it is integrating Sitemaps into Y! Merchant stores. Starting today, Yahoo! merchants can automatically create a Sitemap file for their stores and share information about their site with Google and other search engines to improve their visibility online. With this integration, Yahoo! merchants can also easily verify ownership of their stores -- so they can create a Google Sitemaps account and benefit from the additional reports and tools that show them Google's view of their site.

Get more from the latest release

2006-06-21T12:05:00.000-07:00

In response to your requests, our latest release expands some of the features of Google Sitemaps. Here’s a roundup of what’s new.

Increased crawl errors
Previously, we showed you up to 10 URLs for each error type. We now show all URLs we’ve had trouble crawling. We’ve also put 404 (not found) errors in a separate table from other HTTP errors.

Just choose an error type and either browse the table using the Next and Previous links or download the entire table as a CSV file.

Expanded query stats
Query stats show you the top 20 search queries that brought up your site in the Google search results (both when users clicked on your site in the results and when they didn’t), along with the average top position of your site for that query. Previously, you could view aggregate data across all properties and countries, as well as mobile-specific queries.

Now, you can view data for individual properties and countries as well. For instance, you can see the search queries from users searching Google Images in Germany that returned your site in the results. You’ll only see properties and countries for which your site has data.

Site owners can also view aggregate information for all properties and languages. Properties include Images, Froogle, Groups, Blog search, Base, and Local. More than 100 countries are available.

Previously, query stats were available for sites that were located at the top-level domain (for instance, http://www.example.com/). These stats are now also available for sites located in a subfolder (for instance, http://www.example.com/mysite/).

Increased number of common words
On the Page analysis page, we’ve expanded the list of words we show in the report of common words on your site and in external links to your site from 20 to 75 and we've removed http and www from the words we list.

Increased limit of sites and Sitemaps that can be added to an account
In response to requests, we’ve raised the number of sites and Sitemaps that site owners can add to a Google Sitemaps account from 200 to 500 — a direct result of a request from a Google Group member.

robots.txt analysis tool addition
Our robots.txt analysis tool is a great way to ensure that the robots.txt file on the site blocks and allows only what’s intended. We’ve added the ability to test against the new Adsbot-Google user agent, which crawls AdWords landing pages for quality evaluation. We only use this bot if you use Google AdWords to advertise your site. You can find out more about this user agent in the AdWords help center.

We want to know what you think
We are constantly looking to improve Google Sitemaps and appreciate the feedback we get from our Google Group, other places online, and at conferences. But we know that we don’t get to hear from everyone that way. And so, to gather more feedback, we’ve added a rating tool to each feature in Sitemaps. Tell us if you love the feature, would like us to improve it, or if you don’t find it useful. Simply click your choice beside each feature.

Webmaster help center updates

2006-06-13T15:46:00.000-07:00

We recently updated our webmaster help center . Two new sections that you may find particularly useful are:

Using a robots.txt file
Understanding HTTP status codes

Using a robots.txt file
We've added a new section of help topics in the How Google crawls my site section. These topics include information on:

Understanding HTTP status codes
This section explains HTTP status codes that your server might return when we request a page of your site. We display HTTP status codes in several places in Google Sitemaps (such as on the robots.txt analysis page and on the crawl errors page) and some site owners have asked us to provide more information about what these mean.

The Sitemaps Google Group

2006-06-11T19:09:00.000-07:00

If you use Sitemaps, you should check our our Google Group. The Sitemaps team monitors this community to learn about any issues Sitemaps regulars may be having, as well as about feature requests and suggestions for enhancements. In addition, the members are great about helping each other out. If you do post a question or issue to the Group, please include your site URL (and Sitemap URL if your question is about a Sitemap) and any error message you received. This enables other Group members to help you more quickly and helps the Sitemaps team troubleshoot issues. We greatly appreciate the Group discussions and use the feedback to make Sitemaps better.

Thanks for your participation and input!

Another update on the site: operator

2006-06-09T10:41:00.000-07:00

We've fixed the issue with the trailing slash in site: operator queries. Queries for site:www.example.com and site:www.example.com/ should now return the same set of results.

An update on the site: operator

2006-06-02T19:28:00.000-07:00

We've fixed the issue with site: queries for domains with punctuation in them. We are still working on site: operator queries for domains that include a trailing slash at the end (such as site:www.example.com/ ), so you may get better results for now by omitting the trailing slash in your queries. The Index Stats page of Google Sitemaps no longer uses the trailing slash for its queries, so you should see correct results when using this page.

Thanks for your feedback and patience.

Live in our hometown

2006-05-28T15:41:00.000-07:00

We had a great time at Search Engine Watch Live Seattle last week, answering questions and getting feedback. We even got to meet one of our Google Group members! We wanted to share some of the questions we answered for those who couldn't be there.

When I do a link: search, the results don't include all the links to my site. How can I tell you about the other links?
A search using the link: operator returns only a sampling of pages that link to a site. It doesn't include the full list we know about. We find links to your site through our regular crawling mechanisms, so there's no need to tell us separately. Keep in mind that our algorithms can distinguish natural links from unnatural links.

Natural links are links to your site that develop as part of the dynamic nature of the web when other sites find your content valuable and think it would be helpful for their visitors. Unnatural links are links to your site placed there specifically to make your site look more popular to search engines. Some of these types of links are covered in our webmaster guidelines:

Don't participate in link schemes designed to increase your site's ranking or PageRank. In particular, avoid links to web spammers or "bad neighborhoods" on the web, as your own ranking may be affected adversely by those links.
Avoid "doorway" pages created just for search engines.

In general, linking to web spammers and "bad neighborhoods" can harm your site's indexing and ranking. And while links from these sites won't harm your site, they won't help your indexing or ranking. Only natural links add value and are helpful for indexing and ranking your site.

My site participates in an affiliate program. What tips can you provide?
Google's goal is to provide relevant and useful results to searchers. Make sure that your site provides unique content that adds value beyond an affiliate link or the content provided as part of the program. We talk about this in our webmaster guidelines as well:

Avoid "cookie cutter" approaches such as affiliate programs with little or no original content.
If your site participates in an affiliate program, make sure that your site adds value. Provide unique and relevant content that gives users a reason to visit your site first.

Look at your site and determine what you can offer that will make searchers want to visit and what can distinguish it from other sites in the same affiliate program. And while we expanded on the information in our guidelines specifically because so many people asked us about their affiliate sites, this information is true for all sites. If there's no added value to users, then it's unlikely that search engines will find added value either.

A few questions from our Google Group

2006-05-24T14:54:00.000-07:00

In this post, we thought we'd answer a couple of the questions we've been seeing in the Google Group lately.

Why do I have to add my Sitemap file to my Google Sitemaps account? Can't I just link to it from my site?
There are several reasons we ask you to add the Sitemap file. Here are a couple of them:

If you aren't yet indexed, submitting a Sitemap file lets us know about your site--you can proactively tell us about it rather than wait for us to find it.
When you add your Sitemap file to your Google Sitemaps account, we can let you know if the file has any errors, and then you can resubmit the file once you've fixed the errors.

My site is indexed using both the www and non-www versions of the domain. How can I fix this?
This can happen when both versions of the domain (for instance, http://www.example.com/ and http://example.com/) point to the same physical location, and links to your site use both versions of the URL. To tell us which version you want the content indexed under, we recommend you do a 301 redirect from one version to the other. If your site runs on an Apache server, you can do this using an .htaccess file. You can also use a script. Do a Google search for [301 redirect] for more information on how to set this up for your site. Note that once you implement the 301 redirect, it may take some time for Googlebot to recrawl the pages, follow the redirects, and adjust the index.

If your pages are listed under both versions of the domain, don't use our URL removal tool to remove one version of the pages. Since the pages are at the same physical location for both versions of the domain, using the URL removal tool will remove both versions from the index.

We also suggest you link to other pages of your site using absolute, rather than relative, links with the version of the domain you want to be indexed under. For instance, from your home page, rather than link to products.html, link to http://www.example.com/products.html . And whenever possible, make sure that other sites are linking to you using the version of the domain name that you prefer.

Issues with the site: operator query

2006-05-19T16:19:00.000-07:00

We'd like to give you all a quick update on some of the issues you have been seeing when you do a site: search to see how many pages of your site are in the index. We've been refreshing our supplemental results (you can read more about that in Matt Cutts' blog post) and this refresh has involved some serious changes under the hood. Unfortunately, with change of this scale, there sometimes are bugs. In this case, we found a few bugs that affected the site: operator. Some particular ones you may have noticed are that the following types of queries don't return the correct number of results:

site: queries where you type in a trailing slash (such as site:www.example.com/)
site: queries for a domain with punctuation (such as site: www.example-site.com)

We've got fixes for all of these rolling out in the next few days. They didn't come out sooner because we've been testing them thoroughly, making sure you don't get any unexpected surprises.

This bug doesn't involve any pages being dropped from the index. It's the site: operator that isn't working properly. We're freezing all refreshes of the supplemental results until these issues are fixed, and things should be back to normal in a few days. We'll keep you posted when all fixes have been made.

In the meantime, site: queries without the trailing slash may provide a better result (such as site:www.example.com). If you are checking your site using the Index Stats page of Google Sitemaps, note that it uses the trailing slash in the query, so you may see incorrect results until this bug is fixed.

Thanks for your patience as we resolve this issue.

https verification

2006-05-17T17:23:00.000-07:00

Recently, we told you that we were working on making verification of site ownership available for https sites. This is now available, so if you have an https site, you can now verify site ownership and see statistics and diagnostic information.

More about meta tag verification

2006-05-02T15:12:00.000-07:00

As we mentioned last week, we've added a new option for verifying site ownership. This method requires that you place a specific <meta> tag in the source code of your home page. Many features are available only to site owners and we want as many webmasters as possible to have access. Most site owners who can't upload files or specify names for files should be able to use this new method to verify. For instance, if you use Blogger, you can verify using this option.

To verify using the <meta> tag, simply click the Verify link for your site, choose Add a meta tag as the verification option, and then copy the tag provided to the <head> section of your home page.

This tag looks like this:

<meta name="verify-v1" content="unique-string">

You must place this meta tag:

On the home page of your site (sometimes called the index page or root page).

In the source code for that page.

In the first <head> section of the page, before the first <body> section.

You can't place this tag inside of another tag in the head section.

For instance, this is correct:

<html>
<head>
<meta name="verify-v1" content="unique-string">
<title>Title</title>
<style>
<!-- style info here -->
</style>
</head>
<body>
<!-- body of the page here -->
</body>
</html>

The meta tag inside the style section is incorrect:

<head>
<title>Title</title>
<style>
<meta name="verify-v1" content="unique-string">
</style>
</head>
<body>
...

The meta tag inside the body section is incorrect:

<html>
<head>
<title>title</title>
</head>
<body>
<meta name="verify-v1" content="unique-string">
</body>
</html>

The meta tag on a page with no head section is incorrect:

</html>
<meta name="verify-v1" content="unique-string">
...
<html>

Below are some questions you might have about verification.

I have a blog, and anyone can post comments. If they post this meta tag int a comment, can they can claim ownership of my site?
No. We look for this meta tag in the home page of your site, only in the first section and before the first . If your home page is editable (for instance, your site is a wiki-like site or a blog with comments or has a guest book), someone can add this meta tag to the editable section of your page, but cannot claim that they own it.

So, how do you generate these cryptic tags anyway?
The unique string is generated by base-64 encoding the SHA256 hash of a string that is composed of the email address of the proposed owner of the site (for instance, admin@example.com) and the domain name of the site (for instance, example.com).

From this unique string, can someone determine my email address or identity?
Short answer, no. Long answer, we use a hashing scheme to compute the contents of the meta tag. Hashes cannot be "decrypted" back into the message.

Can the meta tag contents be cracked through a dictionary attack?
To reduce the risk of dictionary attacks, we use a random sequence of bytes (called salt) as a seed to the hash function. This makes dictionary attacks much more difficult.

Can someone determine if the same webmaster own multiple sites?
We use the domain name of your site (for instance, example.com ) to compute the unique string. Based on the contents of the tag, someone can determine if a webmaster owns different sites on the same domain, but not if the webmaster owns sites on different domains. For instance, someone can determine if the same webmaster owns http://www.example.com/ and http://subdomain.example.com/, but can't determine if the same webmaster owns http://www.example.com/ and http://www.google.com/.

What if my home page is not HTML content?
This method may not work for you. You must have a <head> section in order to be able to verify using this approach. Instead, try uploading a verification file to verify ownership of your site.

I've added my tag, but Google Sitemaps says it couldn't find it. Why?
Make sure your tag is in the first <head> section, and before the <body> section. Also ensure that it's not within the <style> tags or other specialized tags. The easiest way to make sure that the placement is one we can recognize is placing it right after your opening <head> tag, as follows:

<head>
<meta name="verify-v1" content="unique-string">

Updated robots.txt status

2006-04-26T15:37:00.000-07:00

Thanks to our users for alerting us to an issue with incorrectly reporting that sites and Sitemaps were being blocked by robots.txt files. We have resolved this issue. If you were unable to add a site or Sitemap because of this issue, you should now be able to add them.

If Sitemaps was reporting that your home page was blocked by robots.txt, you should soon see an updated status. Thanks for your patience as we refresh the display of this data.

If we were incorrectly reporting that your robots.txt file was blocking your Sitemap, you will see an updated status the next time we download your Sitemap.

Thanks as always for your valuable feedback.

A whole new look and a lot more

2006-04-26T10:30:00.000-07:00

If you log in to Sitemaps today, you'll notice some changes. We've revamped the interface based on your feedback to make your account easier to use. There's more below on the facelift, but first, here are details about the new features we've added.

Since the Google Sitemaps program is built on the idea of two-way communication between Google and webmasters we hope this update gives you as much information as possible to help you debug your site and help ensure it is crawled and indexed as effectively as possible.

New verification method
Many features of Sitemaps are available only to site owners. Some of you aren't able to use our existing verification method, so we asked for your feedback on an alternate method of verification that uses a meta tag on the root page of your site. The response was overwhelmingly positive, so we've added this method as an option.

To verify ownership of your site using this method, simply click the Verify link for your site, choose Add a META tag as the verification option, and then copy the tag provided to the <head> section of your home page. Once you've done that, select the checkbox and click Verify. We'll post something soon with more details about this.

Indexing snapshot
The new Summary page provides a quick snapshot about the state of your site, including:

If site is in the index
When Googlebot last accessed the home page
If some pages of the site are partially indexed
If the home page is currently inaccessible
If Googlebot has encountered a large number of errors when trying to crawl the site

Notification of violations of the webmaster guidelines
We may use the summary page to tell you if a site has violated the webmaster guidelines so you can correct the problems and request reinclusion.

Reinclusion request form
The reinclusion request form is available once you sign in to Sitemaps. Simply sign in, click the link, and fill out the form. This form is also available from the Summary page for sites that show violations. Remember, if your site hasn't violated the webmaster guidelines, there's no need to submit a reinclusion request.

Spam report
The spam report form is available once you sign in to Sitemaps. This form is already available outside of Sitemaps, but we wanted to make it available inside as well for two reasons: so that all tools you need for the Google index are available in one place, and because reports that come from within Sitemaps are from users who are signed in, so your report may receive more in-depth consideration.

New webmaster help center
We've launched a new help center for webmasters as a central place for comprehensive information. It includes the webmaster guidelines and details on Googlebot, crawling, indexing, and ranking--plus information about using Sitemaps. It includes all of the information that was previously located at www.google.com/webmasters, as well as much of the information that was located in the Sitemaps documentation. Over time, we'll be adding more information to the help center to answer any questions you might have.

Throughout your Sitemaps account, you'll notice a [?] link in places where additional information is available. Simply click the link to access the help center.

More about our new look
We want Google Sitemaps to be easy to use for everyone. You may notice that a few things have moved around. For instance:

Adding a Sitemap
To add a Sitemap, first add the site. Then, click the Add a Sitemap link beside the site on the My Sites page. To add more Sitemaps for that site, just access the Sitemaps tab for the site and click Add a Sitemap there.

Navigating the tabs
You'll notice that the tabs have changed a bit. When you access a site, you'll see the Summary page of the Diagnostic tab. All the information that used to be under the Errors tab and the robots.txt tab is available from the Diagnostic tab as well. The Statistics tab contains all the pages that were under the Stats tab and the Sitemaps tab has remained the same. You won't see a Verify tab, but the Verify link is available from the My Sites page and the Diagnostic and Statistics tabs.

Back from Pubcon

2006-04-21T14:55:00.000-07:00

We had a great time meeting with webmasters and talking about Sitemaps at Pubcon in Boston. We value feedback and suggestions, and appreciate our Google Group posters. But it's great to be able to talk to webmasters in person too. For those of you who couldn't be there for in-person conversation, here's a recap of the top three questions we were asked.

What's the best way to go about changing domain names?
We've done two blog posts about this that you might find helpful:

Do you have any advice for using a robots.txt file?
We have lots of advice! And if you are thinking of creating a robots.txt file for your site or modifying your existing one, be sure to check out the Sitemaps robots.txt analysis tool so you can test how Googlebot will see your file.

Can you look at my site and tell me how to get better ranking and indexing?
I sat in on the Organic Site Reviews panel, where the panelists were giving out lots of great advice about this. Check out my guest post in Matt Cutts' blog for the highlights.

Thanks to everyone who came up to say hello. And thanks for all the great questions and feedback.

Join us for lunch!

2006-04-17T08:00:00.000-07:00

If you're planning to be at PubCon in Boston this week, join us for lunch with Googlers on Tuesday at 12:50. We'll be talking about how to get the most from your Sitemaps account and answering your questions. Hope to see you there!

More third-party tools

2006-04-16T15:09:00.000-07:00

We've just updated our list of third-party tools that support Sitemaps. Check it out if you're looking to create a Sitemap.

We appreciate the support of the Sitemaps community and all the hard work that has gone into development of these tools.

Using the lastmod attribute

2006-04-14T15:09:00.000-07:00

With our recent infrastructure changes, we've made some minor changes in how we process the lastmod attribute. If you omit the time portion, it defaults to midnight UTC (00:00:00Z). If you specify a time, but omit the timezone, you'll get an invalid date error. You'll also get an invalid date error if you specify an invalid date or time (like February 80th) or the date isn't in the correct format. You'll no longer see errors associated with future dates.

Dates must use W3C Datetime encoding, although you can omit the time portion. For instance, the following are both valid:

2005-02-21
2005-02-21T18:00:15+00:00

https verification

2006-04-12T10:16:00.000-07:00

Some of you have had trouble verifying ownership of https sites (for instance, https://www.example.com/). We are working on this issue and will have it resolved as quickly as possible. We'll let you know as soon as you are able to verify https sites.

Resolving issues listed in the Errors tab

2006-04-11T06:58:00.000-07:00

Another question from our mailbag:

Q: Under the Errors tab, I see a URL listed with a 404 error. That page doesn't exist on my site and I don't link to it. Why does this show up?

A: The Errors tab lists both errors we encountered following links in your Sitemap and errors we had following links during our regular discovery crawl. If we tried to follow the link from the Sitemap, the URL lists "Sitemap" beside it. If it doesn't list "Sitemap", then we tried to follow that link from a web page (either on your site or another site). You might double-check your site and make sure that none of the internal links are pointing to that URL. If they aren't, then this URL is probably linked from an external page.

Adding new content

2006-04-10T15:30:00.000-07:00

This week, we thought we'd answer some of the questions we've gotten recently. If you have questions, just head on over to our Google Group. The group members are really helpful and we'll answer questions here as we can.

Q: I'm adding a new section to my site, which will add quite a bit of content. Should I create a Sitemap with all of my content and submit that, or should I create several Sitemaps and submit them over a period of a few weeks?

A: A single Sitemap and multiple Sitemaps are handled in exactly the same way, so it really comes down to what's easiest for you. If you have fewer than 50,000 URLs, you can simply create one Sitemap with everything in it and submit that. You could also create a separate Sitemap with the new content if that is easier for your to manage. However, you don't need to stagger the submission dates. Whether you choose to create one Sitemap or multiple Sitemaps, you can submit all your pages at one time.

Robots.txt tab maintenance

2006-04-04T14:11:00.000-07:00

Today, we'll be performing maintenance on the infrastructure that displays the information for the robots.txt tab. During this time, you may see a status of "Not found" for your robots.txt file. This maintenance affects only the display of the information, and will not impact Googlebot's processing of your file. Googlebot will continue to read and follow your file correctly during this maintenance period.

We will finish this maintenance as quickly as possible and you should see the correct display of data on the robots.txt tab later today. Thanks for your patience.

More information on the new "unsupported file format" error for Sitemaps

2006-03-20T12:39:00.000-08:00

As we told you a couple of days ago, we've recently enhanced the infrastructure that processes Sitemaps. We've begun processing Sitemaps against stricter guidelines because we are committed to interoperability with other tools that are using this protocol. Because of this, some of you now see an "unsupported file format" error for Sitemaps that previously had an "OK" status.

You'll see this error if the parser doesn't recognize a valid Sitemap file. Here are a few things to check if you see this error:

Confirm that the file uses the correct header. For a Sitemap file, the header can look like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.google.com/schemas/sitemap/0.84">

For a Sitemap index file, the header can look like this:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.google.com/schemas/sitemap/0.84">

Ensure the namespace in the header is "http://www.google.com/schemas/sitemap/0.84". Note that this must end in 0.84. If it ends in .84, you'll see an error.

Make sure each XML attribute is enclosed in either single quotes (') or double quotes (") and that those quotes are straight, not curly. If you use a word processing program, such as Microsoft Word, you may find that it inserts curly quotes.

Improving things behind the scenes

2006-03-15T15:00:00.000-08:00

We've just made some enhancements to the infrastructure that processes Sitemaps. You shouldn't notice many changes, although you may see new and more detailed error messages for your Sitemaps. If you see a different error message than you've seen before, you can click on it to view more information about it.

This change may cause some Sitemaps that used to have a status of OK to display an error message. This is because we've enhanced our reporting to provide error information that wasn't available to you before. For instance, if you see an "Invalid file format" error, make sure that you've declared the namespace in your Sitemap file correctly and that the header matches the examples we provide in our documentation.

As part of this change, you can no longer list Sitemap index files within Sitemap index files. Each Sitemap index file can list only Sitemaps. Remember that you can list up to 1,000 Sitemaps in each Sitemap index file. If you have more than 1,000 Sitemaps for a site, you can submit multiple Sitemap index files.

We've also changed the verification error message we talked about in a recent blog post. If you request verification and we receive a status other than 200 or 404 when we try to access a non-existent file on your site, you'll now see an "HTTP error".

We expect this change to be a smooth one, but please let us know in our Google Group if you experience any trouble.

If you see a "we couldn't find your verification file" error when you try to verify

2006-03-09T17:49:00.000-08:00

When you verify site ownership, we check to see if the verification file exists on your webserver. We also make sure that your server returns a status of 404 (not found) when we request a file that doesn't exist. We do this to make sure that when we check to see if the verification file exists, we're getting the right response because it does exist and not because the server is misconfigured.

Some of you have gotten the following message when trying to verify your site: "We couldn't find your verification file. Make sure it is named correctly and is uploaded to the correct location." But when you check the file in a browser, it does exist.

We have looked into this and have found that we are displaying the incorrect error message. What is actually happening in this case is that when we request a file that doesn't exist, we are getting a response other than 404 or 200. (If we get a response of 200, we display a different error message.)

We are working to display the correct response for these cases. In the meantime, if you see this message when you try to verify and your verification does exist in the correct location, check your webserver configuration and make sure that it returns a status of 404 when a request is made for a non-existent page.

We've fixed an issue with verification files that included leading zeros

2006-03-03T12:30:00.000-08:00

Thank you to our Google Group members for finding a bug in the verification file logic, which stripped out leading zeros when we accessed the verification file. We've fixed this. This should have only affected site owners with verification files that had the pattern google0<unique_string>.html. If your verification file has this pattern and you've had trouble verifiying, please request verification again.

More new features

2006-03-01T13:48:00.000-08:00

We've just released a few new features.

Query stats: average top position
We already show you the top search queries that both returned your site in the results and those that searchers clicked on. Now you can see the top position for your site in the search results for both sets of queries. This position is the average over the last three weeks.

Top mobile search queries
We now also show you the top search queries from mobile devices that returned your site in the results. You'll see both mobile searches of all sites , and mobile searches of mobile-specific sites. You'll only see these results if they are available for your site. Note that at this time, we show only top search queries and not the clicks for those queries.

Downloading Sitemap details, stats, and errors
You can now download information from your Sitemaps account to a comma separated value file, which you can open in any text editor or spreadsheet program. Download this information per table, or download Sitemap details, errors, or stats, for all sites in your account in one file.

Come by and say hi

2006-02-26T15:21:00.000-08:00

This week, some of the Sitemaps team will be at Search Engine Strategies NYC. If you're planning to be there on Monday, come have lunch with us and Matt Cutts. We'll be around throughout the conference and we'd love to talk to you. Please come over and say hi if you see us.

Also, last week, I talked all things Sitemaps with GoodROI on his show GoodKarma. Check out the podcast on Webmaster Radio.

Using a robots.txt file

2006-02-24T12:44:00.000-08:00

A couple of weeks ago, we launched a robots.txt analysis tool. This tool gives you information about how Googlebot interprets your robots.txt file. You can read more about the robots.txt Robots Exclusion Standard, but we thought we'd answer some common questions here.

What is a robots.txt file?

A robots.txt file provides restrictions to search engine robots (known as "bots") that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a robots.txt file exists that prevents them from accessing certain pages.

Does my site need a robots.txt file?

Only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one).

Where should the robots.txt file be located?

The robots.txt file must reside in the root of the domain. A robots.txt file located in a subdirectory isn't valid, as bots only check for this file in the root of the domain. For instance, http://www.example.com/robots.txt is a valid location. But, http://www.example.com/mysite/robots.txt is not. If you
don't have access to the root of a domain, you can restrict access using the Robots META tag.

How do I create a robots.txt file?

You can create this file in any text editor. It should be an ASCII-encoded text file, not an HTML file. The filename should be lowercase.

What should the syntax of my robots.txt file be?

The simplest robots.txt file uses two rules:

User-Agent: the robot the following rule applies to

Disallow: the pages you want to block

These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines in one entry.>

User-Agent

A user-agent is a specific search engine robot. The Web Robots Database lists many common bots. You can set an entry to apply to a specific bot (by listing the name) or you can set it to apply to all bots (by listing an asterisk). An entry that applies to all bots looks like this:

User-Agent: *

Disallow

The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).

To block the entire site, use a forward slash.
```
Disallow: /
```

To block a directory, follow the directory name with a forward slash.
```
Disallow: /private_directory/
```

To block a page, list the page.
```
Disallow: /private_file.html
```

URLs are case-sensitive. For instance, Disallow: /private_file.html would block http://www.example.com/private_file.html, but would allow http://www.example.com/Private_File.html.

How do I block Googlebot?

Google uses several user-agents. You can block access to any of them by including the bot name on the User-Agent line of an entry.

Googlebot: crawl pages from our web index

Googlebot-Mobile: crawls pages for our mobile index

Googlebot-Image: crawls pages for our image index

Mediapartners-Google: crawls pages to determine AdSense content (used only if you show AdSense ads on your site).

Can I allow pages?

Yes, Googlebot recognizes an extension to the robots.txt standard called Allow. This extension may not be recognized by all other search engine bots, so check with other search engines you're interested in to find out. The Allow line works exactly like the Disallow line. Simply list a directory or page you want to allow.

You may want to use Disallow and Allow together. For instance, to block access to all pages in a subdirectory except one, you could use the following entries:

User-Agent: Googlebot
Disallow: /folder1/

User-Agent: Googlebot
Allow: /folder1/myfile.html

Those entries would block all pages inside the folder1 directory except for myfile.html.

I don't want certain pages of my site to be indexed, but I want to show AdSense ads on those pages. Can I do that?

Yes, you can Disallow all bots other than Mediapartners-Google from those pages. This keeps the pages from being indexed, but lets Googlebot-MediaPartners bot analyze the pages to determine the ads to show. Googlebot-MediaPartners bot doesn't share pages
with the other Google user-agents. For instance, you could use the following entries:a

User-Agent: *
Disallow: /folder1/

User-Agent: MediaPartners-Google
Allow: /folder1/

I don't want to list every file that I want to block. Can I use pattern matching?

Yes, Googlebot interprets some pattern matching. This is an extension of the standard, so not all bots may follow it.

Matching a sequence of characters

You can use an asterisk (*) to match a sequence of characters. For instance, to block access to all subdirectories that begin with private, you could use the following entry:

User-Agent: Googlebot
Disallow: /private*/

How can I make sure that my file blocks and allows what I want it to?

You can use our robots.txt analysis tool to:

Check specific URLs to see if your robots.txt file allows or blocks them.

See if Googlebot had trouble parsing any lines in your robots.txt file.

Test changes to your robots.txt file.

Also, if you don't currently use a robots.txt file, you can create one and then test it with the tool before you upload it to your site.

If I change my robots.txt file or upload a new one, how soon will it take affect?

We generally download robots.txt files about once a day. You can see the last time we downloaded your file by accessing the robots.txt tab in your Sitemaps account and checking the Last downloaded date and time.

We'd like your feedback on a potential new verification process

2006-02-22T08:45:00.000-08:00

Before we show you site stats, we verify site ownership by asking you to upload a uniquely named file. Some webmasters can't do this (for instance, they can't upload files or they can't choose filenames). We are considering adding an alternate verification option and would like your feedback. This option would be an additional choice. It would not replace the current verification process. If you have already verified your site, you won't need to do anything new.

This alternate verification process would require that you place a META tag in the <HEAD> section of your home page. This META tag would contain a text string unique to your Sitemaps account and site. We would provide you with this tag, which would look something like this:

<meta name="owner-v1" contents="unique_string">

We would check for this META tag in the first <HEAD> section of the page, before the first <BODY> section. We would do this so that if your home page is editable (for instance, is a wiki-type page or a blog with comments), someone could not add this META tag to the editable section of your page and claim ownership of your site.

The unique string would be a base-64 encoded SHA-256 hash of a string that is composed of the email address of the owner of the site (for instance, admin@google.com) and the domain name of the site (for instance, google.com).

We'd like your comments on this proposal. For those of you who can't verify now, would this be a method you could use to verify? Do you see any potential problems with this method? Let us know what you think in our Google Group.

We've fixed a few things

2006-02-13T15:45:00.000-08:00

We've just fixed a few issues that our Google Group members brought to our attention.

Capitalization in robots.txt lines
Our robots.txt analysis tool didn't correctly interpret lines that include capitalized letters. This has been fixed. All results with URLs that include capital letters are being processed correctly. Note that in a robots.txt file, the capitalization of the rules don't matter. For instance, Disallow: and disallow: are interpreted in the same way. However, capitalization of URLs does matter. So, for the following robots.txt file:

User-agent: *
Disallow: /Myfile.html

http://www.example.com/Myfile.html is blocked. But http://www.example.com/myfile.html is not blocked.

Google User-agents other than Googlebot
Our robots.txt analysis tool didn't correctly process robots.txt files for Google user-agents other than Googlebot. This has been fixed. The Google user-agents we provide analysis for are:

Googlebot-Mobile: crawls pages for our mobile index
Googlebot-Image: crawls pages for our image index
Googlebot-MediaPartners: crawls pages to determine AdSense content (used only if you show AdSense ads on your site)

robots.txt files with extra initial characters
Some robots.txt files have extra characters before the start of the first rule. Some text editors place these characters in the file, but you can't see them with the editor. When the tool processed these characters, it reported a syntax error. The tool now mimics Googlebot's behavior and ignores these extra characters.

Capitalization in site URLs
When you add a site, we now convert all the letters in the domain portion of the URL to lowercase, regardless of how you entered the letters. For instance, if you enter http://www.Example.com/, we convert that to http://www.example.com/ in your account. This applies only to the domain, so for instance, if you add http://www.Example.com/MySite/, we will convert it to http://www.example.com/MySite/. If you added sites to your account using capitalized letters, you'll notice the domain portions have been converted to lowercase. We made this minor change as part of our efforts to ensure you see all available stats for your site.

Improving your site's indexing and ranking

2006-02-13T09:18:00.000-08:00

You've submitted a Sitemap for your site. As we explain in our docs, a Sitemap can help us learn more quickly and comprehensively about the pages of your site than our other methods of crawling, but it doesn't guarantee indexing and has no impact on ranking.

What other things can you do to increase your site's indexing and ranking?

Make sure your site is full of unique, high-quality content.
Google's automated crawling, indexing, and ranking processes are focused on providing quality search results. Is your site a high-quality result for the queries you want to rank highly for? Look at your home page. Does it provide information or does it consist primarily of links? If it is mostly links, where do those links go? Do they lead visitors to good information on your site or simply to more links? Look at your site as a searcher would. If you did a search, would you be happy with your site as a result?

Does your site follow the webmaster guidelines?
Take a close look at your site and our webmaster guidelines. Remember that your site should be meant for visitors, not search engines. It's a good idea to read these guidelines and evaluate your site to make sure it meets them. If it doesn't, your site probably won't be indexed, even if you submit a Sitemap. Here are a few things to check.

Does your site use hidden text?
Hidden text is generally not visible to visitors and is meant to give web-crawling robots, such as Googlebot, different content. For instance, a site might add text in a very small font that is the same color as the page's background. Webmasters sometimes do this because they want to provide more information to the web-crawling robots, and this hidden text is often a list of keywords that the webmaster would like the site to be ranked highly for. Don't use hidden text on your site. Since Google's automated processes are focused in giving searchers high quality results, our guidelines are clear that sites should show Googlebot the same thing they show visitors so our processes can accurately evaluate them.

Does your site use keyword stuffing?
Webmasters sometimes use keyword stuffing much the same way as hidden text. They want to give Googlebot a list of terms that they want their site to rank highly for. But Google's automated processes analyze the contents of a site based on what visitors see, not on a laundry list of keywords. If you want your site to rank highly for particular keywords, make sure your site includes unique, high-quality content related to those keywords.

Does your site buy links from other sites or participate in link exchange programs that don't add value for visitors?
You want other sites to link to you. So, guidelines about links may seem confusing. You want genuine links: another site owner thinks your content is useful and relevant and links to your site. You don't want links that are intended only for Googlebot. For instance, you don't want to pay for a program that spams your link all over the Internet. You don't want to participate in link schemes that require you to link to a bunch of sites you know nothing about in exchange for links on those sites. Do you have hidden links on your site? These are links that visitors can't see and are almost always intended only for search engine web-crawling robots. Think about links in terms of visitors: are the links meant to help them find more good content or are they only meant to attract Googlebot?

Do you use search engine optimization?
If you use a search engine optimization company (SEO), you should also read through our SEO information to make sure that you aren't using one who is unfairly trying to manipulate search engine results.

If your site isn't indexed at all (you can check this by using the site: operator or by logging into your Sitemaps account, accessing the Index stats tab and then clicking the site: link) and you violate these guidelines, you can request reinclusion once you modify your site.

If your site isn't in the index, but doesn't violate these guidelines, there is no need to request reinclusion. Focus on ensuring that your site provides unique, high-quality content for users and submit a Sitemap. Creating a content-rich site is the best way to ensure your site is a high-quality result for search queries you care about and will cause others to link to you naturally.

Analyzing a robots.txt file

2006-02-10T14:03:00.000-08:00

Earlier this week, we told you about a feature we made available through the Sitemaps program that analyzes the robots.txt file for a site. Here are more details about that feature.

What the analysis means
The Sitemaps robots.txt tool reads the robots.txt file in the same way Googlebot does. If the tool interprets a line as a syntax error, Googlebot doesn't understand that line. If the tool shows that a URL is allowed, Googlebot interprets that URL as allowed.

This tool provides results only for Google user-agents (such as Googlebot). Other bots may not interpret the robots.txt file in the same way. For instance, Googlebot supports an extended definition of the standard. It understands Allow: lines, as well as * and $. So while the tool shows lines that include these extensions as understood, remember that this applies only to Googlebot and not necessarily to other bots that may crawl your site.

Subdirectory sites
A robots.txt file is valid only when it's located in the root of a site. So, if you are looking at a site in your account that is located in a subdirectory (such as http://www.example.com/mysite/), we show you information on the robots.txt file at the root (http://www.example.com/robots.txt). You may not have access to this file, but we show it to you because the robots.txt file can impact crawling of your subdirectory site and you may want to make sure it's allowing URLs as you expect.

Testing access to directories
If you test a URL that resolves to a file (such as http://www.example.com/myfile.html), this tool can determine if the robots.txt file allows or blocks that file. If you test a URL that resolves to a directory (such as http://www.example.com/folder1/), this tool can determine if the robots.txt file allows or blocks access to that URL, but it can't tell you about access to the files inside that folder. The robots.txt file may have set restrictions on URLs inside the folder that are different than the URL of the folder itself.

Consider this robots.txt file:

User-Agent: *
Disallow: /folder1/

User-Agent: *
Allow: /folder1/myfile.html

If you test http://www.example.com/folder1/, the tool will say that it's blocked. But if you test http://www.example.com/folder1/myfile.html, you'll see that it's not blocked even though it's located inside of folder1.

Syntax not understood
You might see a "syntax not understood" error for a few different reasons. The most common one is that Googlebot couldn't parse the line. However, some other potential reasons are:

The site doesn't have a robots.txt file, but the server returns a status of 200 for pages that aren't found. If the server is configured this way, then when Googlebot requests the robots.txt file, the server returns a page. However, this page isn't actually a robots.txt file, so Googlebot can't process it.
The robots.txt file isn't a valid robots.txt file. If Googlebot requests a robots.txt file and receives a different type of file (for instance, an HTML file), this tool won't show a syntax error for every line in the file. Rather, it shows one error for the entire file.
The robots.txt file containes a rule that Googlebot doesn't follow. Some user-agents obey rules other than the robots.txt standard. If Googlebot encounters one of the more common additional rules, the tool lists them syntax errors.

Known issues
We are working on a few known issues with the tool, including the way the tool processes capitalization and the analysis with Google user-agents other than Googlebot. We'll keep you posted as we get these issues resolved.

From the field

2006-02-10T12:41:00.000-08:00

Curious what webmasters in the community are saying about our recent release ? Check out Matt Cutts' blog where he discusses the new robots.txt tool and answers some of your questions.

Unexpected Common Words

2006-02-07T09:14:00.000-08:00

The new features released yesterday include a list of common words in your site's content and in external links to your site. In some cases, these common words may not match what you expect from your current content. The common words are calculated based on the results of the Googlebot crawler. This can affect the data in a number of ways:

Googlebot hasn't crawled all pages on your site. If words on particular pages are missing, make sure that those pages are being successfully crawled. If those pages are not yet in your sitemap, adding them will help guide Googlebot to those portions of your site. Also, make sure that you link to the pages of your site from within your site (for instance, by using an HTML site map).
Your site has changed since we last crawled it. If you have just redesigned your site, made significant content changes to an existing site, or purchased an existing domain and changed the contents, the data will not be updated until Googlebot has successfully crawled the new or changed pages.
Googlebot is unable to re-crawl modified pages. Googlebot may be unable to access your server due to network error, or may encounter a server error when trying to load your pages. Make sure that your server is responding properly to incoming requests.
Your site is not being crawled. New sites may take some time to be fully crawled. Read through our information for webmasters for more information about our crawling processes. Also, make sure your site doesn't violate the Webmaster Guidelines.

If you are seeing unexpected data and none of these apply to you, please let us know by posting in our Google Group. We always appreciate user feedback and are working to improve Google Sitemaps.

More stats and analysis of robots.txt files

2006-02-06T14:00:00.000-08:00

Today, we released new features for Sitemaps.

robots.txt analysis
If the site has a robots.txt file, the new robots.txt tab provides Googlebot's view of that file, including when Googlebot last accessed it, the status it returns, and if it blocks access to your home page. This tab also lists any syntax errors in the file.

You can enter a list of URLs to see if the robots.txt file allows or blocks them.

You can also test changes to your robots.txt file by entering them here and then testing them against the Googlebot user-agent, other Google user agents, or the Robots Standard. This lets you experiment with changes to see how they would impact the crawl of your site, as well as make sure there are no errors in the file, before making changes to the file on your site.

If you don't have a robots.txt file, you can use this page to test a potential robots.txt file before you add it to your site.

More stats
Crawl stats now include the page on your site that had the highest PageRank, by month, for the last three months.

Page analysis now includes a list of the most common words in your site's content and in external links to your site. This gives you additional information about why your site might come up for particular search queries.

A chat with the Sitemaps team

2006-02-02T15:51:00.000-08:00

Some members of the Sitemaps team recently took some time to answer some questions about Sitemaps. We hope we've given some insight on questions you may have wondered about.

Giving others access to Sitemaps account information

2006-02-01T09:54:00.000-08:00

Once you add a site or Sitemap to your account and verify ownership, we show you statistics and errors about the site. Now someone else on your team also wants to view information about it. Or perhaps you set up a Sitemap for a client site and now they wants to control the Sitemap submission and view site information.

Anyone who wants to view site information can simply create a Sitemaps Account, add the site (or Sitemap), and verify ownership. The site isn't penalized in any way if it is added to multiple accounts.

Some questions you may have:

I have Sitemaps for multiple clients in my account. I don't want one client to see everything that's in my account. How do I prevent this?
The client should create their own Sitemaps account and verify site ownership. The client won't see anything that's in your account (even though a site or Sitemap may be listed in both accounts).

I've already verified site ownership and the verification file is still on my server. Does the client also have to verify site ownership?
Yes, each account holder must verify site ownership separately. We ask for a different verification file for each account.

What happens if I submit a Sitemap and then someone else in my company submits that same Sitemap using a different Sitemaps account? Do you see that as duplicate entries and penalize the site?
No, we don't penalize the site in any way. No matter the number of accounts that list the Sitemap, we see it as one Sitemap and process it accordingly.

I want the client to be able to see all the stats for this site right away. I don't want them to have to wait for stats to populate. Can I transfer my account information to the client?
Stats and errors are computed for a site, not for an account. There's no need to transfer your account information because anyone who adds a site and verifies site ownership will see all information that we available to show for that site right away.

But I don't want to list the site in my account anymore. Are you sure I can't just transfer the information to the other person's account?
If you don't want the site or Sitemap in your account any longer, you can simply delete it from your account. This won't penalize the site in any way.

More about changing domain names

2006-01-27T09:27:00.000-08:00

Recently, someone asked me about moving from one domain to another. He had read that Google recommends using a 301 redirect to let Googlebot know about the move, but he wasn't sure if he should do that. He wondered if Googlebot would follow the 301 to the new site, see that it contained the same content as the pages already indexed from the old site, and think it was duplicate content (and therefore not index it). He wondered if a 302 redirect would be a better option.

I told him that a 301 redirect was exactly what he should do. A 302 redirect tells Googlebot that the move is temporary and that Google should continue to index the old domain. A 301 redirect tells Googlebot that the move is permanent and that Google should start indexing the new domain instead. Googlebot won't see the new site as duplicate content, but as moved content. And that's exactly what someone who is changing domains wants.

He also wondered how long it would take for the new site to show up in Google search results. He thought that a new site could take longer to index than new pages of an existing site. I told him that if he noticed that it took a while for a new site to be indexed, it was generally because it took Googlebot a while to learn about the new site. Googlebot learns about new pages to crawl by following links from other pages and from Sitemaps. If Googlebot already knows about a site, it generally finds out about new pages on that site quickly, since the site links to the new pages.

I told him that by using a 301 to redirect Googlebot from the old domain to the new one and by submitting a Sitemap for the new domain, Googlebot could much more quickly learn about the new domain than it might otherwise. He could also let other sites that link to him know about the domain change so they could update their links.

The crawling and indexing processes are completely automated, so I couldn't tell him exactly when the domain would start showing up in results. But letting Googlebot know about the site (using a 301 redirect and a Sitemap) is an important first step in that process.

You can find out more about submitting a Sitemap in our documentation and you can find out more about how to use a 301 redirect by doing a Google search for [301 redirect].

Answers to Friday questions

2006-01-13T16:58:00.000-08:00

Today, we wanted to review a few things that we get a lot of questions about.

I tried to verify and I got a message that said "pending verification". Why?
If we can't process the verification request right away, we'll add it to a queue and process it as soon as possible. While your request is in this queue, the status will show as pending. Once we successfully process your request, your site will show as verified. If the request is unsuccessful, you'll see a status of "not verified". You can read more about reasons the request might not be successful or take a look at our documentation.

I submitted my Sitemap and it now as a status of "Timed out downloading robots.txt". I don't have a robots.txt file on my site. What happened?
Any time we access your site, we first check to see if you have a robots.txt file that restricts our access. If we get a timeout from your server when we do this check, you'll see this message. This is probably a temporary problem. Once your server is accessible again, we will be able to see that you don't have a robots.txt file, and will then access your Sitemap. If you continue to see this error, make sure that your webserver is up and responding to requests.

I submitted my sitemap at http://www.example.com/mysite/ and verified it. But now you are asking me to verify at http://www.example.com/. I can't verify at that location because I don't have access. Why are you asking me to do this?
Some stats we can only show if you verify at the root level. If you don't have access to this location, we will show you all the information we are able to (such as errors we had crawling your site). Verification doesn't impact Sitemap submission or indexing in any way, and there is no penalty if you can't verify at the root level.

I've submitted my Sitemap, verified my site, and you've downloaded my Sitemap, but many of my stats show "Data is not available at the time time". Why?
We show you stats about what we know about your site. If we haven't yet crawled and indexed much of your site, we may not have many stats to show you. As we crawl more of your site and learn more about it, you'll start to see more stats.

More language support

2006-01-10T10:20:00.000-08:00

We've added support for four more languages and a Sitemaps Google Group for each one:

Danish
Finnish
Norwegian
Swedish

As with the other languages we support, if you already use Google in one of these languages, you should see the user interface and documentation automatically. Otherwise, you can click the Preferences link from the Google home page and choose the language from the interface list.

Happy New Year!

2006-01-04T10:39:00.000-08:00

We hope you all had a happy holiday and New Year's celebration. Over the past few weeks the team has been doing some planning for 2006 and reviewing your feedback from 2005. We are excited about the product improvements we have in store for the new year and hope that the new features and functionality will help you all reap the rewards of Sitemaps more easily and quickly.

We appreciate your continued enthusiasm and look forward to reading more of your feedback on the Google Sitemaps Group as we embark on the new year.

Wishing you all a very prosperous 2006,

The Google Sitemaps Team

www vs non-www versions of a site

2005-12-21T13:47:00.000-08:00

Two URLs to a site -- one that is prefaced with www and one that is not (for instance, http://www.example.com/ and http://example.com/) -- often point to the same location on a server. But depending on the server configuration, they may point to different locations, so search engines can't assume they are the same.

This post provides tips for viewing stats if the www and non-www version of your site point to the same location.

If you have added your site without prefacing the domain with www (for instance, http://example.com/), and the www version of this domain points to the same location, try adding the www version of the domain (for instance, http://www.example.com/) to your account. You may see a wider variety of stats for the www version of the domain.

You can add a site by:

Clicking the Add tab.
Scrolling to the "If you don't have a Sitemap" area and entering the site URL.
Clicking Add Site.

Similarly, if you have added the www version of the domain and the non-www version points to the same location, you can add the non- www version to see stats for that version.

If the verification file still exists in the root of your site and both versions of the domain point to the same location, you can verify the second version simply by accessing the Verify tab and clicking the Verify button.

Note that having both versions of the site's URL listed in your account won't affect the indexing of your site as long as you have submitted a Sitemap for only one version - the version you want to be indexed. Don't submit a Sitemap for both versions if the location and content are the same.

If both domains point to the same location and you have pages indexed under both versions, see our Google Help Center for more information on consolidating the listings under one domain.

We hear requests for help with this often, so we'll be looking at ways to improve this issue in the coming months.

Verifying a site located in a subdirectory

2005-12-19T16:51:00.000-08:00

If your site is located in a subdirectory, rather than at the root level of a domain (for instance, at http://www.example.com/site/), you are given two choices when you verify. You can either verify at the subdirectory level or at the root level.

If you verify at the root level (http://www.example.com/), we can show you a greater variety of statistics.

If you are unable to upload a file at the root level, you can still view error information for the subdirectory at which you've verified, as well as information about your Sitemaps. Verification does not affect your Sitemap submission or the crawling or indexing of your site.

If you verify at the subdirectory level, some statistics are not available. Instead, some stats pages provide a link for verifying at the root level.

If you have root-level access, simply click the verify link, place the requested verification file in the root directory and then click Verify.

Some questions you may have about verifying at the subdirectory level:

I verified successfully at http://www.example.com/site/. But when I access my stats, I see a message asking me to verify again. Why is it asking me this since I already verified?
That message is providing a link so you can verify at the root level. Rest assured that your site is still verified at the subdirectory level and you can access all information that we have available at this time for subdirectories (index stats, Sitemap details, and site errors).

I verified successfully at http://www.example.com/site/. But I can still see the Verify tab. When I access that tab and click "Verify", I get a message that Google couldn't find my verification file and that my site isn't verified. The original verification file is still present at http://www.example.com/site/, so why am I getting this message?
If you are successfully verified at the subdirectory level, the Verify tab will still appear so that you can later go back and verify at the root level. When you are already verified at the subdirectory level, we look for the verification file in the root of the domain when you try to verify again. You'll see an error message if we don't find it there.

I can't verify at the root. Will this affect my listings in Google?
No. Verification at the root lets you see a greater variety of statistics for your site. It does not affect how we use your Sitemap, how we crawl your site, your PageRank, or any other factor.

A previous version let me verify at the subdirectory level. Why did you change it in this version?
You can still verify at the subdirectory level as you could before and see everything you could before (everything listed under the Sitemaps tab and Errors tab). We've added the option of verifying at the root, which lets you see root-level stats.

More query stats; verification enhancements

2005-12-13T17:20:00.000-08:00

We've made a few improvements that we wanted to let you know about.

Expanded query stats
We've expanded the number of query stats that we show you (both top search queries and top search query clicks). Note that the exact number of stats you see depends on how often your site comes up in search results. A large site that has been around for a while will generally have more stats than a new, smaller site.

Verification enhancements
We've also been working on the verification process. Last month, we posted to the Group that we were looking for your input on how to improve this process. Yesterday, we told you that we added support for lowercase verification filenames. Now, we've added another improvement. Some of you have received one of the following messages when you try to verify:

Our system has experienced a temporary problem.

Or,

The system is currently busy. Please try again in a few minutes.

In the past, you've had to manually try again later. Now, instead of seeing one of those messages, you'll see this message:

The system is currently busy. We will process this verification as soon as possible. Please check back later for an updated status.

We'll add your verification request to a queue and try it for you later, so that you don't have to keep trying manually.

If you receive another error that might be due to a temporary issue (for instance, if we can't reach your server due to a DNS timeout), we'll show you the error, but we'll also add your verification request to the queue.

While your request is in the queue, you'll see "PENDING" as the status in the Verify tab.
Once we've processed the request successfully, we will display your site as verified and you can access your stats.
If we aren't able to successfully process the request, you'll see "NOT VERIFIED" as the status in the Verify tab. If you see this, check that your verification file is in the correct location, is named exactly as we ask, that your robots.txt file doesn't block access, and that your server is up and responding to requests. Then, try your verification request again.

Lowercase verification filenames

2005-12-12T17:15:00.000-08:00

We've been looking into ways to enhance our verification process and noticed that some members of our Group have been unable to upload case-sensitive filenames to their webservers. To help them, we now check for a lowercase verification filename if we don't find the case-sensitive one. So now you can upload a verification file with the lowercase filename. (If you have already verified, you don't need to do anything new.)

Thanks to our Groups members for their input -- and keep that feedback coming!

New version of Sitemap Generator

2005-12-07T08:25:00.000-08:00

We recently uploaded a new version (v1.4) of the the Sitemap Generator tool.

This version has the same features as the last one, but fixes a subtle bug in writing GZip compressed Sitemap files. The old version stored more path information than it needed to when it created GZip files, and this was a point of concern for some webmasters.

The bug was found, and the bugfix suggested, by members of the Sitemaps community. Thanks for bringing it to our attention.

Trouble with verification

2005-11-30T17:11:00.000-08:00

Some of you have had one of the following responses when trying to verify:

"Our system has experienced a temporary problem."

or,

" The system is currently busy. Please try again in a few minutes."

However, when you do try again later, you continue to get one of these messages. This is a known issue and we are working to resolve it as quickly as possible. Thanks for your patience.

If you don't see the full range of stats

2005-11-30T16:45:00.000-08:00

The stats we show you are all about what Google knows about your site. If some stats aren't available, it's probably because we don't know a whole lot about your site yet. As we learn more about it (from your Sitemap and our crawling mechanisms), we'll have more information to show you.

If your site is new to our index, it doesn't yet have a lot of pages indexed, or if not many searches have brought up results to your site, we may not have a lot of stats available yet. You can easily get an idea of your indexed pages by accessing the Index stats page and clicking the link in the Indexed pages in your site row of the table. Over time, our knowledge about your site will increase and we can share that with you.

Third-party programs

2005-11-23T13:09:00.000-08:00

We've just updated the list of third-party programs that you can use to create Sitemaps. You can check out the current list on code.google.com. That page also lists the Sitemaps Google Group for each language we support.

If you've written a tool that supports Google Sitemaps, let us know at code+sitemaps@google.com.

The Sitemaps team is thankful for our great user community and appreciates all the work that's gone into these tools.

Sitemaps in Japanese

2005-11-22T10:07:00.000-08:00

Last week, we added support for Japanese. As with the other languages we support, if you already use Google in Japanese, you should see the user interface and documentation in Japanese automatically. Otherwise, you can click the Preferences link from the Google home page and choose Japanese from the interface list.

We have also added a Japanese Sitemaps Google Group.

Site Verification

2005-11-18T13:29:00.000-08:00

This morning we learned of an issue with the Google Sitemaps tool that may have temporarily enabled users to view statistics about sites they do not own. We acted quickly and fixed the issue. To ensure the security of all sites using the Google Sitemaps tool, we will re-verify all sites added in the last 48 hours.

When we first started showing statistics a couple of months ago, we put a system in place to prevent anyone other than site owners from seeing stats for a site. We ask each site owner to place a unique file on the site and then we check to see if that file exists. When we do that check, we first make sure that the server isn't misconfigured to return a valid page when a request is made for a page that doesn't exist. We only verify sites that are configured correctly. You can read more about that process in our documentation.

Unfortunately, with our latest release, a bug prevented this process from working correctly. We fixed this as soon as we found out about the problem. We take your privacy very seriously and are currently investigating other approaches to further enhance security.

More stats!

2005-11-17T12:18:00.000-08:00

We've just launched some new features. The biggest change for those of you already using Sitemaps: if you've verified your site, you'll see substantially more stats and error details.

The biggest change for new users: you can now add a site to your account even before create a Sitemap for it. Once you've added and verified your site, you can see all these new stats and errors.

New stats: how the Googlebot views your site
The new stats we're showing you are all about letting you know how we see your site.

With query stats, we show you the top Google search queries that return pages to your site as well as the top queries that caused users to click on your site in the search results.

With crawl stats, you can see how we view crawled pages. You can see a distribution of the pages successfully crawled and the pages with errors as well as a distribution of PageRank for the pages in your site.

Page analysis shows you what we detect about the content and encoding of your pages.

Index stats provide an easy way for you to use our advanced search operators to return results about how we see the indexed pages of your site.

Mobile stats
You can now verify your mobile sites and see stats for them.

More detailed errors
Now you’ll have more details about problems we had crawling your site. We report on 40 different types of errors in 5 categories.

Adding a site
Even before you have a Sitemap for your site, you can still take advantage of all the statistics and error information we have available. Simply create a Sitemaps account and add a site to it. Once you verify site ownership, we provide the full range of statistics and error details. Of course, we encourage you to add a Sitemap so we can learn more about your site.

Changing domains

2005-11-11T16:54:00.000-08:00

From our Google Group:

I've moved my site to a new domain. Can I submit a Sitemap to tell you to index the new site rather than the old site?

Submitting a Sitemap for the new site is a great first step, because that helps us learn about the new pages right away. Make sure you place the new Sitemap in the root directory of the new site as the Sitemap must be located on the same domain as the site URLs contained in it.

Another important thing to do is redirect visitors from the old site to the new one. Put a 301 (permanent) redirect on every page of the old site to point to the corresponding page on the new site.

You can find out more about 301 HTTP redirects from RFC-2616 and you can learn more about how to make the site move a smooth one from our Google Help Center.

URLs with HTTP errors

2005-11-08T13:47:00.000-08:00

What can you do about URLs that we tried to crawl but couldn't because we received a 404 (not found) error? (You can see these once you've verified your site by clicking the stats link beside the Sitemap name on the My Sitemaps page.)

You don't have to do anything about them. We'll continue to crawl and index your site and will simply skip pages that return a 404. But here are some things you can do.

If we found the URLs from your Sitemap, the fix is simple. Just modify your Sitemap to list the correct pages and resubmit it.

If we found the URLs by following links, the fix isn't quite as easy. In fact, in some cases, there may be no fix. A webmaster may have liked your site and tried to link to it, but mistyped the URL. You can look for sites that link to your pages and ask webmasters to fix any broken links, but if that sounds like a lot of work, you can instead just focus on your own site.

Check the links in your site
You may not be able to control inbound links from other sites, but you can control internal links. Make sure that none of these broken links are coming from your site. You can generally check your webserver logs to see what visitors clicked on in your site that returned 404 errors.

If the links are outdated
It could be that a link points to a non-existent page because that page used to exist, but no longer does. In that case, you can:

Make sure that your site doesn't link to any outdated pages
Check to see if any of these outdated pages are in the Google index

If the page used to exist, but no longer does, it might still be listed in the Google index. (You can check this by doing a Google search for the URL.) If it is, you can either wait for subsequent crawls (if we continue to get 404 errors when we try to crawl it, it will eventually be removed), or you can request its removal using our automatic URL removal system.

In order to use this system, the outdated page must return a 404 (and if the URL is showing up on your Sitemaps Stats page, it already does). Log in and then choose the Remove an outdated link option. Type in the URL,choose anything associated with this URL and click Remove outdated link. The link will show up in a status area as pending. The page should be removed from the index within 3-5 days and the status will be updated.

What to do when your Sitemap status is "Denied URLs"

2005-11-01T17:14:00.000-08:00

If your Sitemap status is "Denied URLs" and the error listed is "URL not under Sitemap path", here are some things to check.

Make sure the URL root matches
If you submit your Sitemap using the path http://example.com/sitemap.xml, then the URLs in your Sitemap should begin with example.com. Any URLs that begin with www.example.com aren't considered to be under the Sitemap path. Along those lines, if you Submit your Sitemap using the path http://www.example.com/sitemap.xml, the URLs in that Sitemap should begin with www.example.com.

To fix this problem, you can either edit the URLs listed in your Sitemap file to match the submitted path, or you can delete the Sitemap and then submit it again using the path that matches the URLs listed in it.

Make sure the Sitemap is at the highest-level directory
If you submit your Sitemap using the path http://www.example.com/sample_folder/sitemap.xml, then all URLs in that Sitemap must begin with that path. This means that http://www.example.com/sample.html wouldn't be considered a valid URL in the Sitemap.

If all possible, place your Sitemap at the root location of your site to avoid these types of problems. If you can't place the Sitemap at the root, then list only URLs from the Sitemap location and lower.

See the Sitemaps documentation for more details.

When your site changes

2005-10-26T08:00:00.000-07:00

Here's another question we've gotten. If you have a question about Sitemaps, let us know by posting in the Sitemaps Google Group.

I've submitted a Sitemap for my site. What should I do when my site changes?

If you add pages to your site, you can let us know in several different ways:

Resubmit your Sitemap: You can add the URLs to your existing Sitemap and then resubmit it either by using your Google Account or by sending us a ping (HTTP request).

Create a new Sitemap: You can create a new Sitemap with only the new URLs and submit that separately using your Google Account.

Use a Sitemap index file: You can create a new Sitemap with only the new URLs and then list both the original Sitemap and the new Sitemap in a Sitemap index file. Then, you can simply submit the Sitemap index file. If you often add pages to your site, you can periodically create a new Sitemap with new URLs, add the the new Sitemap to the Sitemap index file, and resubmit the Sitemap index file.

Note that if you ping us to let us know about changes to your Sitemap, make sure that you submit that Sitemap using your Google Account at least once. The Last Submitted date shown in your Google Sitemaps Account lists the date you last submitted the Sitemap manually, but if you receive a status OK (200) from the ping, we received it successfully.

If you've deleted pages from your site, you can delete those pages from your Sitemap and then resubmit it (either manually through your Google Account or by pinging us).

Including site pages in a Sitemap

2005-10-24T15:09:00.000-07:00

From time to time, we use this blog to answer some common questions. Here's one:

Do I have to include every URL from my site in my Sitemap? If I don't include some of them, will they be excluded from the Google index?

Your Sitemap provides us with an additional way to learn about your site. We still use all of our other methods, such as following links from your site's HTML sitemap and from pages that link to you. We discover URLs that you don't include in your Sitemap through these regular crawling processes --it just may take us longer, and we won't have any extra information that you can provide in a Sitemap (such as priority, last modification date, and change frequency).

We won't exclude URLs that you don't list in your Sitemap from the Google index.

Searching what Google knows about your site

2005-10-12T09:47:00.000-07:00

So, you've submitted your Sitemap. How can you tell what Google knows about your site?

You can use Google’s advanced search features to get a list of:

search results from your site
pages that link to your site
pages that refer to your site’s URL

You can also see information that we have about your site.

You can do many of these advanced searches (amazingly enough) by clicking the Advanced Search link on www.google.com. You can also use our advanced search operators in your query. If you are using operators, remember that there should not be a space between the operator and the URL. Note that we use brackets to indicate the words in the search box. The query itself should not include the brackets.

Results from your site (site: operator)
To find pages from your site, use the site: operator. For instance, [site:www.google.com]. You can also type the URL in the Domain field of the Advanced Search page.

You can also use this feature to search through any site. Simply enter the search query followed by the site: operator and the site you want to search through. For instance, to search for admission information on the Stanford University web site, type [admission site:www.stanford.edu]. And, you can use this feature to search through sites from specific top-level domains. For instance, to search for information about Zürich on Swiss sites in the .ch domain, type [Zürich site:.ch].

Pages that link to your site (link: operator)
To find pages that link to your site, use the link: operator. For instance, [link:www.google.com]. You can also type the URL in the Links field of the Advanced Search page.

Pages that refer to your site’s URL (allinurl: operator)
To find pages that include your site’s URL, use the allinurl: operator. For instance, [allinurl:www.google.com].

Information Google has about your site (info: operator)
To see information that Google knows about your site, use the info: operator. For instance, [info:www.google.com]. That query results in the following:

Some questions you may have about these results...

I submitted my Sitemap but I don't see all the pages listed. When will they be indexed?
We can't guarantee if or when we'll index pages we receive from Sitemaps submissions. We use Sitemaps as another view into your site to augment our regular crawling methods. Also, we don't want the Googlebot to overwhelm your bandwidth, so we may not crawl it all at one time.

Some of my results are labeled "Supplemental". What does that mean?
That means that pages are part of our auxiliary index. You can read more about that in our webmaster guidelines.

Verifying your site

2005-10-04T14:00:00.000-07:00

Verification
Once you verify your site, we show you additional statistics. We require verification to make sure that we only show these stats to site owners.

When you verify, we ask you upload a unique text file to a particular directory on your webserver. Periodically, we check to see if this file still exists. We do this to make sure you still own the site. If we can’t find this file when we recheck, we ask you to verify again.

Questions you might have about verification:

Once I’ve verified my site, can I delete the verification text file?
You can delete it once you’ve verified, but when we do our periodic check for it, we’ll ask you to upload it again.

Someone who used to have write-access to my site no longer does. How can I make sure this person can no longer see the stats for my site?
Simply delete the verification file. When we do our next periodic check, if that person tries to see stats for the site, we’ll ask for the file again. Since that person no longer has write access to the site, we won’t find the file and won’t show the additional statistics any longer.

But I still want to see stats for the site. How can I do that if my site is no longer verified?
We ask for a different verification file for each Google Account. You can log in with your Google Account and still see statistics.

The person who uploaded the Sitemaps for my site no longer has write access. I don’t have access to that person’s Google Account. How can I see information about my site?
Simply log in with your own Google Account and submit your Sitemaps using that account. We’ll ask you to upload a verification file that is unique to your Google Account so that you can see additional stats for your site.

All new!

2005-09-30T23:27:00.000-07:00

We’ve added some new features to Google Sitemaps.

Date stamps for statistics
For information we provide once you’ve verified your site, we now let you know when we tried to crawl the URLs we tell you about.

Enhanced support for special characters in URLs
Note that the Sitemap URL must be encoded for readability by the webserver on which it is located. In addition, it can contain only ASCII characters. It can't contain upper ASCII characters, certain control codes, or special characters such as * and {}. If your Sitemap URL contains out-of-range characters, escape them when you submit the URL. Otherwise, you'll receive an error when you try to submit it. You can find more information on escaping out-of-range characters by doing a Google search for [html escape codes]. All URLs must follow the RFC-3986 standard for URIs and the RFC-3987 standard for IRIs.

Documentation updates
We’ve updated the documentation for these new features, as well as added information about the latest version of the Sitemap Generator script and about OAI-PMH submissions (both of which we talked about in earlier blog posts). We’ve also provided some information about errors you might come across when you submit a Sitemap. All we’ve made these updates in every language for which we provide documentation.

Resolved issues
And we’ve resolved two issues with this release that you brought to our attention in the Google Group.

Some of you experienced trouble when clicking on the verify link on the My Sitemaps page. Instead of being asked to verify, the My Sitemaps page simply reloaded. We've fixed this.
Some of you had trouble adding Sitemaps from Internet Explorer. This happened when you pressed Enter rather than clicked the Submit URL link. You saw a message that said your Sitemap was submitted, but the Sitemap wasn’t added to your My Sitemaps page. We've fixed this too.

If either happen to you, or if you experience any other trouble, please let us know by posting in the Google Group.

Several of these features were a direct result of your feedback. Once again, we appreciate your input during our beta period.

We show you more

2005-09-29T17:59:00.000-07:00

Just about a month ago, Google Sitemaps added new statistics about problems Google encountered crawling your pages. This stats page showed you up to three URLs we had trouble accessing for each type of error. You asked for more. So we’re giving you more.

Now, once you verify your site, we’ll show you up to 10 URLs we’ve had trouble accessing for each type of error, for a maximum total of 60 URLs.

Keep posting your suggestions to our Google Group and we'll keep listening. Thanks for your participation during our beta period.

How is a Google Sitemap different from an HTML sitemap?

2005-09-26T15:24:00.000-07:00

A Google Sitemap is an XML file that uses the Sitemap protocol. This file lists URLs in your site, along with optional descriptive information about those URLs (such as when they were last updated and how often you modify them). You can create this XML file using our Sitemap Generator or a third-party tool. Google Sitemaps are intended for processing by the Google Sitemaps program.

An HTML sitemap is intended for users of your site. Generally, this type of sitemap provides links to the pages in your site, and may provide descriptions of those pages. We encourage the use of HTML sitemaps. They make it easier for users to navigate your site. Also, as we talk about in our webmaster guidelines, a clear hierarchy with text and links helps us index your site.

You can’t submit an HTML sitemap to the Google Sitemaps program. However, if you are unable to create or generate a Google Sitemap file in the Sitemap protocol format, you can submit a text file that lists URLs in your site.

Using OAI-PMH with Google Sitemaps

2005-09-20T10:53:00.000-07:00

If your site uses the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) 2.0 protocol, an application-independent interoperability framework based on metadata harvesting, you can use your OAI repository as your Sitemap.

Simply submit the baseURL of your OAI repository (for instance, http://www.example.com/oaiserver). When we query the baseURL, we automatically add query parameters (such as ?verb=Identify or ?verb=ListRecords), so you can simply submit the baseURL itself. When we extract the URLs for your site, we expect the records in the repository to be formatted using Dublin Core, with the URLs embedded in <dc:identifier> tags. Below is a sample record that includes the <dc:identifier> tag in bold. The URL listed in that tag is what we extract.

<oai_dc:dc
 xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/"
 xmlns:dc="http://purl.org/dc/elements/1.1/"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/
 http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
<dc:title xml:lang="en">A title of extraordinary things</dc:title>
<dc:creator>McCormack, Michael</dc:creator>
<dc:subject>LCSH:Ausdehnungslehre; LCCN QA205.H99; Greatness:Amanda</dc:subject>
<dc:publisher>J. Wiley & Sons</dc:publisher>
<dc:date>Created: 1906; Available: 1991</dc:date>
<dc:type>text</dc:type>
<dc:identifier>http://example.com/physics/1796949</dc:identifier>
<dc:language>english</dc:language>
<dc:rights xml:lang="en">Public Domain</dc:rights>
</oai_dc:dc>

As with other Sitemaps, the URLs must be within the same site and at the same directory location or lower than the baseURL. For instance, if you submit http://www.example.com/oaiserver as the baseURL, the following URLs would be valid:

http://www.example.com/
http://www.example.com/samples.html
http://www.example.com/images/

However, if you submit http://www.example.com/dataprovider/oaiserver, then none of those URLs would be valid.

Combining Sitemaps into one larger Sitemap

2005-09-16T10:51:00.000-07:00

Do you have several small Sitemaps that you would like to combine into one larger one? With version 1.3 of the Sitemap Generator, which we told you about yesterday, you can do just that.

This version includes a new input method: sitemap, which lets you point to existing Sitemaps that you created with the Sitemap Generator. The Sitemap Generator will create a single Sitemap that includes the URLs contained in each Sitemap you list.

To use this input method, locate the sitemap section of the config file and modify it as needed. You should include one <sitemap> entry for each Sitemap you want to include. Each entry must contain the path parameter, whose value should be the path and filename of an existing Sitemap file.

The example_config.xml file included with the Sitemap Generator download includes a sample <sitemap> section:

<-- ** MODIFY or DELETE **
"sitemap" nodes tell the script to scan other Sitemap files. This can
be useful to aggregate the results of multiple runs of this script into
a single Sitemap.

Required attributes:  path - path to the file -->

<sitemap path="/var/www/docroot/subpath/sitemap.xml">

This section gives one example. You should replace this example and include an entry for each Sitemap you want to include. Ensure that the path value is the complete path and filename on your web server. You can list gzipped Sitemaps as well, as long as they have a .gz extension. Rather than list each Sitemap, you can use wildcards. For instance, the following entry would include any Sitemaps that begin with the word "sitemap" and have an .xml extension:

<sitemap path="/var/www/docroot/subpath/sitemap*.xml">

The Sitemap Generator extracts all URLs and the optional data listed for each URL for every Sitemap you list and creates one Sitemap with this information. At this time, we can't guarantee that this method will work for Sitemaps created with tools other than the Sitemap Generator.

Announcing Sitemap Generator version 1.3: Improved encoding support

2005-09-15T10:49:00.000-07:00

The Sitemap Generator version 1.3 is now available and provides improved encoding support. If your webserver uses an encoding other than UTF-8 or if your domain name or some the URLs in your site use non-ASCII characters, and you plan to use the Sitemap Generator to create your Sitemap, you should download this latest version.

Generally, non-ASCII URLs should be encoded using UTF-8 before being percent-escaped. However, some webservers respond correctly only if URLs are encoded specifically for the webserver's configuration. All URLs within your Sitemap, as well as the URL of the Sitemap itself, must be encoded for readability by the web server on which they are located.

If you are using the Sitemap Generator, you can specify the encoding of the URLs contained in the Sitemap from within the config.xml file. Within the site definition section of that config file, use the optional default_encoding attribute to specify the encoding used by your webserver. If you don't use this attribute and your webserver uses an encoding other than UTF-8, the Sitemap Generator can't know which encoding to use, although it does attempt to determine the correct encoding. If the generated Sitemap doesn't list the URLs correctly, you should explicitly indicate the encoding with the default_encoding attribute and run the Sitemap Generator again.

If your URLs contain non-ASCII characters, we recommend that you run the Sitemap Generator script using Python 2.3 or higher. This version of Python has increased non-ASCII support. If your domain name contains non-ASCII characters, you must use Python 2.3 or later, as Internationalizing Domain Names in Applications (IDNA) support wasn't added until this version. Without IDNA support, the Sitemap Generator can't correctly encode a non-ASCII domain name.

Google Sitemaps in your language

2005-09-08T16:31:00.000-07:00

We’ve just made our Sitemaps user interface and documentation available in ten additional languages. We have also set up Google Groups for each one. The languages available are:

Brazilian Portuguese
Dutch
French
German
Italian
Korean
Russian
Simplified Chinese
Spanish
Traditional Chinese
UK English
US English

If you already use Google in one of these languages, you should see the change automatically. Otherwise, you can click the Preferences link from the Google home page and choose one of these languages from the interface list.

As always, you can submit a Sitemap for sites with content in any language.

Verifying your site: trouble with 404 pages

2005-09-07T13:11:00.000-07:00

You want to verify your site so you can view additional statistics. You click the verify link beside the site on the My Sitemaps page, create the file we ask for, upload it to your server, and click the Check Status button. And then you see this error message:

We've detected that your 404 (file not found) error page returns a status of 200 (OK) in the header.

What should you do?

This error means that we've detected that your server returns a status of OK when the requested file is not found. This is the same status that the server returns when the file exists. When we look for the verification file, we can't tell if your server is returning a status of OK because it finds the file, or because it can't find the file. This means we are unable to verify your site.

Modify your web server configuration to return a status of 404 (file not found) in the header of 404 pages. If your site is hosted, ask your hosting company to do this.

Make sure that if your server returns a custom error page when a requested file is not found, that page returns a 404 status in the header. And make sure that the server doesn't redirect requests that return "file not found" to a valid page of your site, such as your home page. This configuration returns a redirect status code (such as 301 or 302) rather than the correct 404 status code.

You can read more about http status codes here. If you don't have a mechanism for checking the headers that your server returns, you can do a search for terms such as [check server header tool] to find online tools that will check this for you.

Once your web server is configured correctly, try to verify your site again and we'll check the configuration.

Submitting mobile Sitemaps

2005-08-31T17:17:00.000-07:00

If you've created and submitted Sitemaps for your non-mobile pages, or just want to submit a mobile Sitemap for the first time, here are a few helpful tips to help you get started:

Identify your mobile Sitemaps content

If you have content other than XHTML mobile profile, WML, and cHTML, you should create a separate non-mobile Sitemap for those and submit it using the Add a Sitemap page.

If your site serves multiple mobile markup languages, you should create a separate mobile Sitemap for each markup language.

If you have URLs that serve multiple markup languages, you should include those URLs in each mobile Sitemap that applies. For instance, if you have a URL that serves both XHTML and WML content, include that URL in two mobile Sitemaps, one for each markup language.

If you have a large site and use Sitemap index files to manage a large number of mobile Sitemaps, make sure that each Sitemap index file only includes mobile Sitemaps for one markup language.

Create your mobile Sitemaps

You can create mobile Sitemaps in the same way as other Sitemaps. The only format not supported is OAI-PMH.

If you are submitting a syndication feed, and the URLs listed in that feed serve multiple markup languages, decide which markup language fits best. You can’t submit the syndication feed multiple times, each for a different markup language, since each Sitemap (within the same directory) must have a unique name.

Submit your mobile Sitemaps

Once you’ve created your Sitemap, log in to your Google Account and submit it using the Add a Mobile Sitemap page.

Specify the markup language that the URLs listed in the mobile Sitemap serve.

Once you’ve submitted the mobile Sitemap it will be listed on your My Sitemaps page as a “Mobile” type.

Mobile pages and new statistics

2005-08-30T07:00:00.000-07:00

We just launched two new features.

Mobile Sitemaps
You can already use Google Mobile Web Search on your mobile phone to search through sites that have been specifically designed for mobile phones, PDAs, and other handheld devices. We add new sites to our mobile web index regularly.

You can help users find your mobile webpages by letting us know about those pages. Google Mobile Sitemaps lets you submit Sitemaps for URLs that serve mobile content. You create and submit a mobile Sitemap much in the same way you do other Sitemaps: with the Sitemap Generator, the Sitemap protocol, or via a syndication feed or text file.

The biggest difference with a mobile Sitemap is that you have to submit a separate one for each markup language. Right now we support:

XHTML mobile profile (WAP 2.0)

WML (WAP 1.2)

cHTML (iMode)

When you submit your mobile Sitemap, let us know which markup language the URLs in the Sitemap serve.

New Site Statistics
Now you can see information for the URLs in your site that we've had trouble accessing – both for URLs from your Sitemap and those we've discovered during a regular crawl. We won't show you these additional statistics until you verify your site, which is a very simple process. Click the verify link next to the site on your My Sitemap page, create an empty file using the name we specify, and upload it to the folder where your Sitemap is located. We'll check to see that the file is there, which tells us that you have permission to upload files to that site, and then we'll show you the information.

What URLs should a Sitemap include?

2005-08-26T15:50:00.000-07:00

So, what URLs from your site should you put in your Sitemap?

Put in everything! List the URLs that contain your content, images, media, and anything else in your site.

If you want to include only a subset of items, you can, but we’d like as much information about your site as you can give us.

Remember that we respect robots.txt, so if you include any URLs in your Sitemap that are restricted in robots.txt, we won’t crawl those.

What's in a name?

2005-08-25T10:10:00.000-07:00

How should you name your Sitemap? What extension should you give it?

The short answer is that you can name your Sitemap anything you want. You can use any extension. Just submit the URL to us, and we’ll go pick it up.

The better answer is a little longer. We recommend that you give your Sitemap an extension that identifies the file type. For instance, if you create a simple text file that lists URLs, we suggest giving that Sitemap a .txt extension.

If you create an XML Sitemap that uses our Sitemap protocol, give it an .xml extension. If you compress that file using gzip, give it an .xml.gz extension.

If you use our Sitemap Generator to create a Sitemap, you specify the resulting Sitemap name in the config.xml file. The default name is sitemap.xml.gz. If you keep the .gz extension, the resulting Sitemap file will be compressed. If you change this name to have an .xml extension, the resulting file will not be compressed. We suggest you compress the file so that your webserver will take less of a bandwidth hit when we download it.

You can submit the URL of a script that dynamically generates an XML Sitemap when we download it. That script might have an extension such as .asp or .php (depending on the script type). The extension of the file isn’t a problem, but if your script takes a long time to run, the delay will look like a server timeout and we’ll try again later. If you have trouble getting this type of Sitemap submitted, make sure your script is responsive. Also ensure that your webserver doesn’t automatically add things (such as HTML headers and footers) to the generated files, since that would cause the resulting XML file to have parsing errors.

One more thing about naming. You can name your Sitemap anything you want… almost. You can’t name it robots.txt. And if you use a robots.txt file for your site, make sure that it doesn’t restrict our access to your Sitemap file.

Using Sitemap Index Files

2005-08-18T14:55:00.000-07:00

Several of you have asked us, Should I submit Sitemaps or Sitemap indexes for my site?

If you have a small site, you probably don't need to use a Sitemap index file -- you can just list all of your URLs in one Sitemap.

If you have a larger site, you may want or need to have multiple Sitemaps for your site. In that case, you can make submitting and tracking easier by listing the Sitemaps in a Sitemap index file.

You must use multiple Sitemaps for your site when:

You have more than 50,000 URLs to list. That's the maximum that one Sitemap can include.

A single Sitemap would be larger than 10MB, uncompressed. That's the maximum size for a Sitemap.

You may want to use multiple Sitemaps for other reasons, to make organization easier. For instance if:

You want to store all of your archived URLs in one Sitemap and all of your URLs that change frequently in another. This way, when you add new URLs to the Sitemap, you'll have a smaller and more manageable file to work with.

You manage multiple sites that are in subfolders. You might want to create a Sitemap for each site and then create a Sitemap index file that lists them in the root. Remember, this method works only with subfolders. For example, this Sitemap index file:
```
www.example.com/sitemap_index.xml
```
could list the following Sitemaps:
```
www.example.com/site1/sitemap.xml
www.example.com/sitemap2/sitemap.xml
```
However, that Sitemap index file could not list the following Sitemaps:
```
site1.example.com/sitemap.xml
site2.example.com/sitemap.xml
```

Creating an index of Sitemap index files
You can also have an index of Sitemap index files. A Sitemap index file can be a maximum of 10MB as well, so if you have a really large site, you may have to use this additional organization step to keep the file sizes to a manageable level. We have a size limitation for Sitemaps and Sitemap indexes so that when we download the files, we don't overwhelm your bandwidth.

Compressing your Sitemap index file
Speaking of being considerate of your bandwidth, if you can, you should compress your Sitemaps and your Sitemap indexes using gzip. If you're not familiar with gzip, keep watching this blog. We're putting together some helpful instructions.

If you compress your Sitemap index file, you'll probably want to give it an .xml.gz extension. If you don't compress your Sitemap index file, you'll probably want to give it an .xml extension.

Submitting your Sitemap index file
So, you've got some individual Sitemaps that you've listed in a Sitemap index file. What now? Just Sign into Google Sitemaps and submit the Sitemap index file. You don't need to submit individual Sitemaps that are included in the index. Once we've processed your Sitemap index file, we'll let you know if we found errors in the Sitemap index itself, or in any of the individual Sitemaps.

If you make changes to a Sitemap included in a Sitemap index file you've submitted, just change the lastmod date for that Sitemap in your index. During this beta period, feel free to resubmit the Sitemap index file.

Just getting started...

2005-08-08T15:10:00.000-07:00

We are stoked that so many of you are trying out Sitemaps while it's in beta. When we read on the Sitemaps Google Group that you'd like an easy way to find out about new features and where we're headed with this thing, we realized we could put our very own Blogger to good use.

And since the Sitemaps team is far-flung in Mountain View, Kirkland, and Zurich, this will be a good way to keep everyone posted about new features and developments. Subscribe to our feed to keep up with the latest. We'll also e-mail each blog post to the Sitemaps Google Group, so if you get your news from there, you won't miss out on a thing.

In addition to telling you about new features here, we'll also do our best to address some of the frequently asked questions from the Google Group.

Lately, some of you have wondered if submitting a Sitemap could actually reduce the number of your site pages we have indexed.

No, submitting a Sitemap will not reduce the number of indexed pages for your site.

As we note in our information for Webmasters, each time we update our database of webpages, our index shifts: we find new sites, we lose some sites, and sites' rankings change. If your site was dropped from Google and you haven't made major changes to it, we'll likely pick it up again soon. If your site's ranking changes, ensure you are following our guidelines.

When you submit your Sitemap, you help us learn more about the contents of your site. Participation in this program will not affect your pages' rankings or cause your pages to be removed from our index.