Psychological Profiling Via Twitter





This weekend I was playing with a bunch of different linguistic analysis methods to better understand ReTweets, and while I uncovered a ton of cool new data which I’ll be sharing a little later this week, I also came upon an idea I think is pretty awesome, probably groundbreaking, and definitely worth Twittering about.

Communication is a window into a person’s mind, and the way a person talks can tell you a lot about how they think. Linguists have developed two methods to decoding the written word into a meaningful profile of a person’s cognitive processes.

One method is called the Regressive Imagery Dictionary (RID). This coding scheme is designed to measure the amount and type of three categories of content: primordial (the unconscious way you think, like in dreams), conceptual (logical and rational though) and emotional.

Significantly more primordial content has been found in the poetry of poets who exhibit signs of psychopathology than in that of poets who exhibit no such signs (Martindale, 1975). There is also more primordial content in the fantasy stories of creative as opposed to uncreative subjects (Martindale & Dailey, 1996), in psychoanalytic sessions marked by therapeutic “work” as opposed to those marked by resistance and defensiveness (Reynes, Martindale & Dahl, 1984), and in sentences containing verbal tics as opposed to asymptomatic sentences (Martindale, 1977). A cross-cultural study of folktales from forty-five preliterate societies revealed, as predicted from the “primitive mentality” hypothesis of Lévy-Bruhl (1910) and Werner (1948), that amount of primary process content in folktales is negatively related to the degree of sociocultural complexity of the societies that produced them (Martindale, 1976). Martindale and Fischer (1977) found that psilocybin (a drug that has about the same effect as LSD) increases the amount of primordial content in written stories. Marijuana has a similar effect (West et al., 1983). Research has also revealed more primordial content in verbal productions of younger children as compared with older children (West, Martindale, & Sutton-Smith, 1985) and of schizophrenic subjects as compared with control subjects (West & Martindale, 1988).

The other method is Linguistic Inquiry and Word Count (LIWC). In development for over 15 years, the LIWC measures the cognitive and emotional properties of a person based on the words they use.

In order to provide an efficient and effective method for studying the various emotional, cognitive, and structural components present in individuals’ verbal and written speech samples, we originally developed a text analysis application called Linguistic Inquiry and Word Count, or LIWC.

I’ve combined these two systems with a Porter stemming algorithm and my own Twitter analysis infrastructure to create TweetPsych.com.

TweetPsych uses the LIWC and RID to build a psychological profile of a person based on the content of their Tweets. It compares the content of a user’s Tweets to a baseline reading I’ve built by analyzing an ever-expanding group of over 1.5 million random Tweets, then highlighting areas where the user stands out.

The service analyzes your last 1000 Tweets; as such, it works best on users who have posted more than 1000 updates. It is also better suited for running analyses on accounts that are operated by a single user and use Twitter in a conversational manner, rather than simply a content distribution platform. It takes a few moments to analyze an account the first time, but subsequent views of a profile will load faster.

I’ve tried to translate the codes that come from the two linguistic systems into more meaningful explanations, but I may have missed a few. I will continue to expand these definitions, while also refining the system and algorithm to better analyze Twitter-specific content.

I think the possibilities of a system like this are enormous, from matching like-minded users to identifying users that exhibit certain useful or desirable traits. I’d love to hear your thoughts on where this could be improved or where I could take this technology next.

If you liked this post, don't forget to subscribe to my RSS feed or my email newsletter so you never miss the science.

{ 85 comments }

Jacob Friis Saxberg July 22, 2009 at 9:19 am

Very cool!

You should visualize the results.
Maybe via: http://code.google.com/apis/visualization/

Jesse Baer September 1, 2009 at 3:51 am

“Imagine if you could run advertising on via social tools knowing that certain people will interact before they do.” Funny, usually when I hear people say stuff like this, it's in fear, despair, or jest. The fact that it excites you seems creepy.

Jesse Baer September 1, 2009 at 3:51 am

“Imagine if you could run advertising on via social tools knowing that certain people will interact before they do.” Funny, usually when I hear people say stuff like this, it's in fear, despair, or jest. The fact that it excites you seems creepy.

Jesse Baer September 1, 2009 at 3:51 am

“Imagine if you could run advertising on via social tools knowing that certain people will interact before they do.” Funny, usually when I hear people say stuff like this, it's in fear, despair, or jest. The fact that it excites you seems creepy.

Jesse Baer September 1, 2009 at 3:55 am

Word. I was fascinated when I got an @reply that someone had analyzed me, but this thread is weirding me out. The fact that I didn't initiate my “analysis” makes it even sketchier. I wouldn't take it much more seriously than a mood ring, except that everyone else is.

Jesse Baer September 1, 2009 at 3:55 am

Word. I was fascinated when I got an @reply that someone had analyzed me, but this thread is weirding me out. The fact that I didn't initiate my “analysis” makes it even sketchier. I wouldn't take it much more seriously than a mood ring, except that everyone else is.

Jesse Baer September 1, 2009 at 3:55 am

Word. I was fascinated when I got an @reply that someone had analyzed me, but this thread is weirding me out. The fact that I didn't initiate my “analysis” makes it even sketchier. I wouldn't take it much more seriously than a mood ring, except that everyone else is.

Stephen Waddington September 1, 2009 at 10:36 am

Hi Dan – Congratulations. This is a very neat application and very innovative. You should commercialise this and sell it to recruiters and HR departments. I've blogged about it here: http://bit.ly/1jCZIU

Jesse Baer September 1, 2009 at 10:51 am

“Imagine if you could run advertising on via social tools knowing that certain people will interact before they do.” Funny, usually when I hear people say stuff like this, it's in fear, despair, or jest. The fact that it excites you seems creepy.

Jesse Baer September 1, 2009 at 10:55 am

Word. I was fascinated when I got an @reply that someone had analyzed me, but this thread is weirding me out. The fact that I didn't initiate my “analysis” makes it even sketchier. I wouldn't take it much more seriously than a mood ring, except that everyone else is.

fionaboyd September 1, 2009 at 7:59 pm

Hi Dan, I love this work you're doing. My partner and I set up a social networking site in 2003 called http://www.folklikeme.com which only ever had around 8000 members and we couldn't find the way to grow outwards and keep finding relevant folk to match up with others. I think Twitpsych would have helped enormously. I want to have another shot at FLM later this year, so would be really interested in where you're at with this, then.

fionaboyd September 2, 2009 at 2:59 am

Hi Dan, I love this work you're doing. My partner and I set up a social networking site in 2003 called http://www.folklikeme.com which only ever had around 8000 members and we couldn't find the way to grow outwards and keep finding relevant folk to match up with others. I think Twitpsych would have helped enormously. I want to have another shot at FLM later this year, so would be really interested in where you're at with this, then.

Genevieve Hinson September 2, 2009 at 2:37 pm

Interesting, but no frame of reference where the numbers are concerned for us non-educated tweeters.

Genevieve Hinson September 2, 2009 at 9:37 pm

Interesting, but no frame of reference where the numbers are concerned for us non-educated tweeters.

Tom September 3, 2009 at 10:57 am

Very cool, Dan, but it would help immensely to know what those scores relate to. I got a 150 on abstract thought, but a 9 on methphor. Does that mean I totally suck on metaphor, despite being a metaphor junkie, and the comment that “Many of your Tweets contain metaphors”? Lots of fun, though, thanks!

@TomYHowe

Tom September 3, 2009 at 5:57 pm

Very cool, Dan, but it would help immensely to know what those scores relate to. I got a 150 on abstract thought, but a 9 on methphor. Does that mean I totally suck on metaphor, despite being a metaphor junkie, and the comment that “Many of your Tweets contain metaphors”? Lots of fun, though, thanks!

@TomYHowe

Warren Davies September 5, 2009 at 3:38 pm

Just came across this! Very good idea, great feat of engineering! Personally, I'd prefer if it wasn't promoted as being psychological, as people might get the idea that this has to do with academic psychology, which it isn't. I put further thoughts here:

http://generallythinking.com/blog/index.php/200

It would be really cool if you could run some studies, testing your content against established psychological measures. Then you might have something that's compelling to people, and actually gives valid feedback, too. It'd be a shame if this became solely a marketing tool, or a “personality” test on a par with the ones you get in women's magazines, when there's potential there.

But either way, you've made something pretty special there, well done.

Warren Davies September 5, 2009 at 10:38 pm

Just came across this! Very good idea, great feat of engineering! Personally, I'd prefer if it wasn't promoted as being psychological, as people might get the idea that this has to do with academic psychology, which it isn't. I put further thoughts here:

http://generallythinking.com/blog/index.php/200

It would be really cool if you could run some studies, testing your content against established psychological measures. Then you might have something that's compelling to people, and actually gives valid feedback, too. It'd be a shame if this became solely a marketing tool, or a “personality” test on a par with the ones you get in women's magazines, when there's potential there.

But either way, you've made something pretty special there, well done.

Batman October 12, 2009 at 8:52 am

Do you think you can predict my next tweet? That would be a great application… :)

In other news, you've now told me how I tweet, and you've given me some numbers, and you haven't told me what they mean. Now that you know how I tweet, maybe you can explain it to me, so, that we'll both know. Would also like more than five users that I'm similar to. Thanks!!!

Batman October 12, 2009 at 3:52 pm

Do you think you can predict my next tweet? That would be a great application… :)

In other news, you've now told me how I tweet, and you've given me some numbers, and you haven't told me what they mean. Now that you know how I tweet, maybe you can explain it to me, so, that we'll both know. Would also like more than five users that I'm similar to. Thanks!!!

pixoxo October 26, 2009 at 11:06 am

Does tweetpsych works for portuguese-speakers too?

afithk October 26, 2009 at 2:18 pm

hi guys, we have a same interest. i write social media with psychological approach. maybe we can share this. thnks

pixoxo October 26, 2009 at 6:06 pm

Does tweetpsych works for portuguese-speakers too?

andywise October 27, 2009 at 2:33 am

Great stuff, do you have any plans to publish an API ? I would love to use your analysis as part of a project I'm working on at the moment.
@andrew_wise

andywise October 27, 2009 at 9:33 am

Great stuff, do you have any plans to publish an API ? I would love to use your analysis as part of a project I'm working on at the moment.
@andrew_wise

ghostexecutive October 27, 2009 at 7:30 pm

How exciting. Couldn't wait until someone did this. Thank you, Dan : )

ghostexecutive October 28, 2009 at 2:30 am

How exciting. Couldn't wait until someone did this. Thank you, Dan : )

raverants October 31, 2009 at 3:37 pm

Fascinating! Thanks for sharing.

raverants October 31, 2009 at 10:37 pm

Fascinating! Thanks for sharing.

nycteris November 20, 2009 at 12:46 pm

That's fascinating right there! I didn't know anyone was working with NLP these days at all. NLP and Twitter… what a mind-blowing concept!

nycteris November 20, 2009 at 8:46 pm

That's fascinating right there! I didn't know anyone was working with NLP these days at all. NLP and Twitter… what a mind-blowing concept!

Cory H December 1, 2009 at 12:58 am

This is such an absolutely genius concept, and with further exploration and development I am Sure it will prove itself to be quite useful and even possibly revolutionary. I'll be sure to get the word out in whatever capacities I may and to follow you for any updates. Fantastic work

Cory H December 1, 2009 at 8:58 am

This is such an absolutely genius concept, and with further exploration and development I am Sure it will prove itself to be quite useful and even possibly revolutionary. I'll be sure to get the word out in whatever capacities I may and to follow you for any updates. Fantastic work

Cory H December 1, 2009 at 8:58 am

This is such an absolutely genius concept, and with further exploration and development I am Sure it will prove itself to be quite useful and even possibly revolutionary. I'll be sure to get the word out in whatever capacities I may and to follow you for any updates. Fantastic work

codemyconcept February 3, 2010 at 10:03 pm

This is a great tool for marketing! Makes using twitter for advertisement a lot more useful.

{ 9 trackbacks }