Quantifying the Impact of Google's Keyword Referral Data Shutdown

On Tuesday, October 18th, Google announced they'd be hiding search referral data for logged-in Google searchers. When questioned by Danny Sullivan of SearchEngineLand, Google provided the following estimate on the impact to search referral data:

"Google software engineer Matt Cutts, who’s been involved with the privacy changes, wouldn’t give an exact figure but told me he estimated even at full roll-out, this would still be in the single-digit percentages of all Google searchers on Google.com"

Tragically, it appears that Cutts was either misinformed or gave misleading information, as "(not provided)" became a major referrer for many websites, climbing into double digits in early November. Now, that percentage has risen even higher, into the 20%+ range on many sites. Hubspot's Brian Whalley reported that the average website using their analytics lost 11.36% of keyword referral data and 423 sites lost more than 20% (15 unlucky souls lost 50%+, which seems almost crazy).

In an attempt to better quantify the impact, we ran a small survey last week, asking fellow marketers to supply information about the impact to their sites.

Here's a visualization of 60 sites' analytics data, showing the self-reported percent of their Google search traffic that used keyword "(not provided)":

Our average in the 6 days from Nov. 4-10 almost exactly matches the average of the several thousand Hubspot customers (11.36% vs. 12.02%), and thus makes me feel pretty good about that data from the survey-takers.

A little more about these 60 respondents:

• We collected 66 finished surveys, but scrubbed 6 that had missing, suspicious or improperly filled-out information
• The types of sites reporting data included a wide variety, as illustrated below:
  
• The sites included in the survey also included a solid variety of traffic numbers. The distribution below reports visits from Google organic search recorded in October, 2011:
  
• We asked the respondents what level of impact this change had on their content and marketing efforts, and received the following distribution of replies:
  

Approximately 1/5th of those surveyed reported no impact on their content/marketing efforts, which likely suggests those folks don't typically use keyword-level data to help them improve OR the change hasn't cost them enough data to have a negative impact. Another 1/5th claimed a strong impact, which is likely how I'd describe this change for our internal efforts. Granted, we don't actively use this data every week, but we've relied on it heavily for reporting and in the past for audits around content optimization and the generation of new content (or updating/refreshing of old material).

Here's numbers and a visualization of the referrer encryption data specifically for SEOmoz.org:

From Oct. 19th - 30th, Google sent 163,909 visits from organic search to our website. 3,762 of those visits, or 2.3%, were via keyword "(not provided)". We didn't sweat this too much. As per Matt Cutts' promise, it was in single digits and, while frustrating, had a very tiny impact on our analytics, marketing and content optimization efforts.

But from Oct. 31st to November 13th, Google sent 191,726 visits and 35,168 of these came via keyword "(not provided)," 18.34%. This has a serious impact on our ability to make our website better for visitors (in particular, identifying keywords that are sending traffic but potentially not having a great experience that we should be making new blog posts, videos, updates, etc. to help).

To me, that's the most tragic part of this change. The underselling of the change as being "single digits" was lame. The hypocrisy around keyword privacy sucks. And their motivations are questionable at best. But the crummiest part is the impact the change will have. It won't put any black hats out of business, won't stop any malware or hacking, and won't add a shred of value to the Internet. But it will make it harder for marketers and site builders to measure, understand and improve for their audience. The net impact will be a slightly worse web, and Google's claim of privacy will only protect them from criticism because it's a far easier explanation than the truth.

Sometimes, it sucks living in an ecosystem with an 800-pound gorilla.

p.s. Google's Matt Cutts responded to this post on Twitter today. I've included his comments and my replies below:

I remain somewhat skeptical that all the sites in Hubspot's data and ours would be outliers, but perhaps, at the least, this suggest the referral data disappearance won't get massively worse. Here's to hoping.