New Experiments Question the Power of Social Proof on the Web





In a lot of my presentations and research, I’ve talked about social proof, and I’ve hypothesized that it has an effect on social and viral behavior online, but I had never actually proven it. So a few weeks ago, I began a series of experiments designed to test the assumption that the effects of social proof and social conformity can be exploited on the web.

In the first two experiments, I split tested ReTweet buttons with different ReTweet counts shown on blog posts. First I compared “0 Tweets” with “776 Tweets.” The results were exactly the opposite of what I expected. After 36 hours, the button showing no Tweets had been clicked more than double the times the other button had. The sample size and variation performance are statistically significant, and the results show a 96% confidence level.

While discussing these results with Alison, she suggested that they may have been due to a “first post” effect, where people want to be the first to share a piece of content. So I tested a button showing “15 Tweets” against one showing “776 Tweets.”

While the post I used for this test was more popular, the results of the experiment showed a far less significant difference between the two buttons. The “15 Tweets” button performed marginally better, but the low confidence value means there is probably no meaningful difference between the two buttons.

The results of the first two tests had me questioning whether or not social proof has the effect online I thought it did. My next step was to test the Feedburner subscriber count RSS button, which I believed was perhaps more likely to exhibit traditional social conformity effects.

I began by testing a button displaying “12 Subscribers” against one that displayed “62172 Subscribers.” The higher variation was clicked on a slight .13% more, and again this experiment’s confidence interval is too low to really be significant.

Finally, I decided to test the “first post” effect on the RSS button, by comparing a “0 Subscribers” button against the “62172 Subscribers” button. Again, the 62172 version did a little better, but failed to reach a statistically significant level.

In spite of the insignificant results I found in 3 of 4 tests, I believe my findings are interesting for a few reasons. First and perhaps most importantly, they represent a first step towards “contagiousness testing” which would allow marketers to apply split and multivariate testing methods to content virality.

An elongated test may reveal that higher showing a higher subscriber count on an RSS button, does lead to a small, but significant click-through increase.

These 4 experiments also suggest that there may be a powerful “first post” effect that marketers can leverage in certain situations. I plan to do more research into this in the future.

If you liked this post, don't forget to subscribe to my RSS feed or my email newsletter so you never miss the science.

{ 23 comments }

Jakub Pajer August 2, 2010 at 10:25 am

in other words, Allison was right about retweets aka “first post” effect :) I found it very surprising but I belive her explanation.

Markus Roder August 2, 2010 at 11:14 am

Hi Dan,

I have an interesting hypothesis, coming from the psychological perspective :)

If the “first post effect” (more clicks on low-count bbuttons) or the “herd drive” (more clicks on high-count buttons) is stronger might depend on the context of the post!

Does it appeal more to the “neophile segment” of the population, whose social status depends on being trendsetters? For them, an oft-clicked share button and many retweets might be kryptonite.

But for grassroots movements (“herd trigger”), a high-click-count button would represent a strong herd and therefore an attractive “social joining opportunity”.

When repeating your experiment, you should definitely set up to scenarios with a clear diving line concerning that psychological marker.

On the other hand,

Emanuel August 2, 2010 at 1:34 pm

It might matter a lot on what site the experiment is done. If I was to see a post in this site with 0 tweets on your blog, I would rush to read it. I know it will be very high quality content and being the first one to tweet the blog post would be an… honor?

Anyway, if I was to see a 776 tweets button on a site that usually has 2 or 5 tweets I suppose it would stand out for me.

About the RSS readers count; I honestly do believe only a hand full of people actually read the numbers. Not only is the design not very engaging, but also a majority of all users online, do not have any idea what RSS feeds are. And the ones who do know what RSS feeds are, most probably also know that the feedburner stats have a higher “volatility” than the BP stock currently exhibits.

But those are all just guesses, I am very much looking forward to find out what you will discover in your coming research on Social Proof.

Geno Prussakov August 2, 2010 at 2:11 pm

Very interesting results, Dan.

Question: where exactly did the RT button appear (to generate the clicks)? Just the blog itself?

Don Lafferty August 2, 2010 at 2:20 pm

I think what you've discovered perhaps is that people are becoming overwhelmed and numb to content no matter who else likes it.

I also think the market space, target demographics and marketing channels used to promote a post have a great deal to do with the success of experiments like yours.

For instance you mention in the first paragraph that you measured results “after 36 hours.”

After 36 hours of what? Tweeting to your normal community? Posting a link to a LinkedIn group and/or appropriate Facebook communities? Other promotional activities?

I find the recipe behind social proof has a lot to do with the visitor's perspective, that is, their experience in the environment, the relevance of the content and the proximity of supportive content that gives the average visitor the 30 second impression that you know what you're talking about and can be considered a trusted source.

Ileane Smith August 2, 2010 at 2:45 pm

Dan, I find it amusing the way you conducted these tests and how people might be so easily influenced by numbers which may or may not have a thing to do with the actual content of a post.

Typically I ignore subscriber counts on FeedBurner widgets. Primarily because I don't subscribe to blogs in a reader. But I realize that I am an exception to the rule. If I visit a blog like yours for the first time and saw that you had 0 subscribers I would assume that there's something wrong with your feed count (not your content) because it's obvious that your blog wasn't hatched yesterday.

I'm much more of a Twitter addict so in the case of the retweet button it wouldn't matter what the count was on the button, if I like the content and I think my followers would find it interesting I will retweet it. However, seeing a low retweet count would influence me to see how many comments the post had and if the number was small I would leave a comment too. I would also be inclined to Digg, or Stumble if I felt the post needed additional exposure.

Bottom line – if I didn't like the content I don't care how many retweets, subscribers or comments there are.

Thanks for doing these tests and continuing your examination of social “proof”.

Schneider August 2, 2010 at 4:33 pm

Amazingly interesting! But the confidence rates are way too low, don't you think so?
Anyway does that means anything about *real* social proof, on the street, or is it just about web and virtual stuff?

Ike Pigott August 2, 2010 at 4:47 pm

Are people numb to content, or to meta-content?

Davezilla August 2, 2010 at 4:59 pm

I am of the same mind and to be honest, don't feel this experiment proves or disproves much at all. It certainly doesn't disprove social proof. It's hard to judge the content of a post. Also, there are other factors: Where both posts at the same time of the day? Same day of the week? Equally good content?

My point is, there's probably too much qual in content analysis to make this a valid quant test, but darn it all Dan, keep trying! You're one of the few that is trying to validate this industry.

Andy Smith August 2, 2010 at 5:05 pm

If I understand your methodology properly, you did these tests with your own posts, right? If that's the case, there's likely a large overshadowing effect of your own brand that is influencing the outcomes.

Second, I'm interested in the significant increase of CTR on the second test (.19% to .33) suggesting that something else is going on here.

I don't have the answer, but I think your effort is interesting and deserves some more testing. Observing my own behavior, I find myself less “first post” driven as much as I won't retweet something that I perceive to be fully out there already. I have some threshold number of retweets where I think that tossing in another is more likely self promotion and adding to the noise rather than the signal of the conversation.

Dan Zarrella August 2, 2010 at 5:08 pm

I don't think you should compare CTRs between experiments, only variation to variation. That way I'm controlling for post contagiousness, day of week, etc.

As far as my brand interferes, the tests shouldn't been seen to say that a button with 0 Tweets gets x CTR, but that it gets X% more clicks that another post, again controlling for my brand.

John Paul August 2, 2010 at 8:05 pm

No surprise to the subs example.. but the Twitter button example was crazy.

Since you would think people would interact more with the post that has been shared, thinking that “hey 700 ppl liked this, so will I” but that's a big no..lol

I think I missed it, but did you use the same post? did you post/tweet it out at the same times of the day same day of the week?

Sean SEO Marketer August 2, 2010 at 9:15 pm

Ultimately the test results would be scoring high to the post that is highly popular. Popularity that is based on the user's interest. So I think in that case “first post” effect will also depends on the interest of your audience.

Sean SEO Marketer August 2, 2010 at 9:16 pm

Ultimately the test results would be scoring high to the post that is highly popular. Popularity that is based on the user's interest. So I think in that case “first post” effect will also depends on the interest of your audience.

ptamaro August 2, 2010 at 10:11 pm

This is interesting, and it seems like a great start — but I would imagine there are several other factors that also contribute to the overall click-through rates. For example, I wonder how placement of the elements tested effects their usage?

Great thought provoking stuff (as usual), thanks Dan. I look forward to seeing more of your research on this topic…

Sean Weigold Ferguson August 2, 2010 at 11:10 pm

I have a few questions about your statistics.

I'm assuming you used a chi-square test of independence. Did you report the one-tailed p values? Based on the numbers you've included, I'm seeing a two-tailed p of 0.047. However, applying Yates' correction yields a p value of 0.073. Because the latter is not significant, and the former is on the border of significance, I would suggest that this study should be repeated to verify your results.

Captico August 3, 2010 at 3:09 pm

Really interesting!! Thanks for sharing!

Xavi izaguirre August 4, 2010 at 4:02 pm

Ok,

I don't doubt you have controlled the situation well so it is equal content, equal time etc…

For one, Markus is right, some people would be attracted by being first and others to not tweet something weird. Most people though would asses the content themselves ignoring the metadata and social proof or lack thereof, such as Ileane or dazevilla.

I reckon social proof comes only in scenarios where someone is stressed with uncertainty and insecurity. Twitter is not such a place….

Test different scenarios and consider coupling your research with in-depth interviews!!

'Irawan Wibisono' August 5, 2010 at 10:10 pm

This post implies the essential worth of social media as a very productive stream in both marketing and advertising services. The services that is provided by this social media transpires to the vision of delivering a very effective and convenient way that goes along with the current evolution of technological advantage. http://www.socialnetupdate.com

Joakim Ditlev August 9, 2010 at 8:48 am

Interesting test, Dan. Those kind of tests really adds to my understanding of how social media works.

The only thing I really miss here is some data on the share of first time visitors. If you visit a blog for the first time the numbers like no of retweets and subscribers plays an important role when judging the credibility and validity of the blog (In addition, I always screen the Alexa and PageRank when first-time visiting blogs).
Once you are a subscriber or returning reader you have some trust to the source, which makes the number of retweets and subscribers less important and you are more likely to share from a trusted source.

At least, that is my assumption. Perhaps something worth digging deeper into at some point.

Suzanne Lainson August 18, 2010 at 4:43 am

For what it is worth, I do read retweet numbers before retweeting something. If I want to retweet a post, but it has been heavily retweeted already, I probably won’t retweet on the assumption that my followers have already seen the post.

RSS numbers I don’t pay any attention to.

Anonymous September 3, 2010 at 11:59 pm

I started a youtube channel recently and purchased 10,000 Youtube subscribers from http://www.socialkik.com for our Youtube channel and the results were amazing! I passed all my competitors in the number of Youtube subscribers… some competitors have had their channels for over 3 years, but I now have 20 times more subscribers than them, which made it seem that I’ve been in business much longer than them !

znmeb September 30, 2010 at 4:42 pm

How did I miss this back in August? ;-) I think if you look at the dynamics of tweets and retweets within the tweet stream you’ll find something like this:

A person discovers something cool and wants to share it. He goes to bit.ly, since the tracking data are there for the link and discovers that it’s been clicked 15 times or 4700 times.

15 times: Yeah, I’ll send that out to my followers. They may not share my interest, but it’s likely to be something they haven’t seen.

4700 times: Sheesh – old news – they’re probably sick of it by now and I’ll look like a copycat.

{ 11 trackbacks }