The official blog of Taveo

All things related to click tracking, digital marketing and web analytics

Taveo Insights 1: Bot Traffic on Twitter

Authors Note: This is the first in what will hopefully be a series of posts dedicated to interesting and useful things we (and our users) have learned while using Taveo.

 

While browsing the interwebs recently I came across an interesting article about some issues surrounding URL shorteners and social media (specifically Twitter). His basic argument is that you must be careful when using URL shorteners, because they will show up as “Direct / None” traffic inside Google Analytics. To quote Mark:

Bit.ly shows 47 clicks coming from the shared link. Granted, some differences will occur between bit.ly and Google Analytics (the GA view shown below filters out sessions from some IP addresses, and bit.ly is less restrictive about tracking bot clicks) but, it’s safe to say most of the 44 direct sessions recorded have been misattributed and are a result of users clicking on the link via Twitter, not visiting the content directly.

I found this statement in particular very interesting for multiple reasons, but mostly because it is incorrect.

I know this because I have seen what happens to URLs in tweets. What actually happened is that 44 or so web scrapers “clicked” on the link after it was shared on Twitter. This actually highlights Taveo’s value-add very nicely. Taveo would have filtered the “bot clicks” out and given you a more accurate click count (as well as a bunch of fancy graphics with geographical breakdown, referrer info ,etc). We also give you access to the raw click data, so you can perform your own analysis.  Secondly, it shows that even seasoned web veterans don’t actually understand that amount of “Bot” traffic that Twitter attracts.

So how much bot traffic do Tweets attract? Read on for some actual data.

 

Robot Link Clicks on Twitter – A case study

Taveo stats showing lots of Twitter Bots

As part of our marketing efforts, we have a Twitter account. From time to time we tweet and include Taveo enable links. The first time we did this, I honestly thought we had done something wrong.  On March 9th at 4:44 PM EST we tweeted this. Within the first 60 seconds we had 12 “clicks”, 100% of which were some type of scraping bot. Only one of those even sent a Referer Header. Over the next hour, we recorded 52 clicks, 42 of which were identified by Taveo as bots! (the actual number of bots is higher, they are just more sneaky about it, see the Raw Data below).

Twitter Bot Traffic

Looking at the Referrer data paints an even bleaker picture, most of these clicks don’t even have one. Why is that? Because web scraping robots don’t usually send Referrer data! Again, if you look at the raw click data it’s pretty clear that the majority of these clicks are in fact some type of robot.

So whats the take away from all of this?

Twitter is a very popular social platform and it’s understandable that companies want to make sense of what goes on there. Hopefully the companies scraping this data are doing it for good reasons (and with Twitters approval). On the other hand, companies using Twitter need to understand this is going on and take steps to factor this into their analysis of ROI’s, etc (“Look Boss, 47 clicks in 1 minute. Were killing it!”). The original article incorrectly identifies the URL shortening platform as the problem, diving deeper we understand that the problem is more complex and requires creative solutions ( Hmm..I don’t know…maybe Taveo! ).


 
 

Raw Data exported from Taveo

This is the raw data as captured by Taveo. IP addresses have been anonymized.

Date IP (Anonymized) Country Language Type User-Agent Referer
2015-03-09 23:44:05 10.0.0.1 US Robot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2015-03-09 23:44:05 10.0.0.2 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:44:05 10.0.0.3 US English (United States) UNK InAGist URL Resolver (http://inagist.com) api.twitter.com
2015-03-09 23:44:05 10.0.0.4 US Robot Twitterbot/1.0
2015-03-09 23:44:05 10.0.0.5 US Robot Twitterbot/1.0
2015-03-09 23:44:06 10.0.0.6 US Robot Google-HTTP-Java-Client/1.17.0-rc (gzip)
2015-03-09 23:44:06 10.0.0.6 US Robot Google-HTTP-Java-Client/1.17.0-rc (gzip)
2015-03-09 23:44:06 10.0.0.7 US Robot help@dataminr.com
2015-03-09 23:44:06 10.0.0.8 US Robot Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
2015-03-09 23:44:06 10.0.0.8 US Robot Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
2015-03-09 23:44:06 10.0.0.8 US Robot Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
2015-03-09 23:44:10 10.0.0.9 US Desktop Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
2015-03-09 23:45:17 10.0.0.10 FR Robot Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)
2015-03-09 23:45:24 10.0.0.11 US English (United States) Robot Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)
2015-03-09 23:46:10 10.0.0.12 US English (United States) Desktop Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.115 Safari/537.36 t.co
2015-03-09 23:46:19 10.0.0.13 JP Desktop Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0)
2015-03-09 23:47:36 10.0.0.14 US Robot Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
2015-03-09 23:50:32 10.0.0.15 Desktop Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36
2015-03-09 23:50:33 10.0.0.16 US Desktop Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36
2015-03-09 23:50:33 10.0.0.17 GB UNK node.js t.co
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.18 GB Robot Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0
2015-03-09 23:50:33 10.0.0.19 US Robot help@dataminr.com
2015-03-09 23:50:33 10.0.0.20 FR English (United States) Robot LivelapBot/0.2 (http://site.livelap.com/crawler) tav.so
2015-03-09 23:50:33 10.0.0.21 CA English (United States) UNK Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
2015-03-09 23:50:33 10.0.0.22 US Robot MetaURI API/2.0 +metauri.com
2015-03-09 23:50:33 10.0.0.23 US Robot MetaURI API/2.0 +metauri.com
2015-03-09 23:50:33 10.0.0.24 US Desktop Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
2015-03-09 23:50:33 10.0.0.25 US Desktop Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Getter/0.1)
2015-03-09 23:50:33 10.0.0.26 DE UNK
2015-03-09 23:50:33 10.0.0.27 US UNK grokkit-crawler (pdsupport@purediscovery.com)
2015-03-09 23:50:34 10.0.0.28 KR UNK http://twitter.com
2015-03-09 23:50:34 10.0.0.21 CA English (United States) UNK Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)
2015-03-09 23:50:34 10.0.0.25 US Desktop Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Getter/0.1)
2015-03-09 23:50:38 10.0.0.29 NL UNK Ruby
2015-03-09 23:50:39 10.0.0.30 DK Desktop Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0
2015-03-09 23:50:53 10.0.0.31 JP Robot Crowsnest/0.5 (+http://www.crowsnest.tv/)
2015-03-09 23:50:53 10.0.0.32 US UNK Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)
2015-03-09 23:50:59 10.0.0.33 AU Robot MetaURI API/2.0 +metauri.com
2015-03-09 23:52:09 10.0.0.34 US UNK Ruby
2015-03-09 23:52:49 10.0.0.35 JP Robot Crowsnest/0.5 (+http://www.crowsnest.tv/)
2015-03-09 23:53:20 10.0.0.36 FR Robot Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)
Email this to someoneTweet about this on TwitterShare on FacebookShare on RedditPin on PinterestDigg thisShare on Google+Share on StumbleUpon