1. Welcome to DNTrade. If you want to find out about the latest domain name industry news or talk, share, learn, buy, sell, trade or develop domain names - then you've come to the right place. It's a diverse and active community, with domain investors, web developers and online marketers - and it's free! Click here to join now.
    Dismiss Notice

Blocking Semalt Referrer

Discussion in 'Hosting' started by chris, Mar 28, 2014.

  1. chris

    chris Administrator

    Joined:
    Mar 7, 2010
    Messages:
    2,253
    Likes Received:
    829
    auDA Member:
    Yes
    Hi all,

    After getting a few questions from clients on what the referrer "semalt" is I thought I'd put together a short article about it and was wondering if anyone else has noticed this in their analytics, seen any impact on their stats and if or what steps you've taken to block it etc.

    Personally I've been blocking it using .htaccess, but wondered what others are doing.

    Thanks in advance, I'll post the article here when it's done.

    Cheers,
    Chris
     
  2. helloworld

    helloworld Membership: VIP

    Joined:
    Apr 21, 2012
    Messages:
    1,116
    Likes Received:
    165
    Is it just this?

    Code:
    http://semalt.com/project_crawler.php
    Seems like a good reverse marketing plot. Webmasters posting links and mentions?
     
  3. chris

    chris Administrator

    Joined:
    Mar 7, 2010
    Messages:
    2,253
    Likes Received:
    829
    auDA Member:
    Yes
    Yes, that's the one...
     
  4. petermeadit

    petermeadit Membership: VIP

    Joined:
    Jul 13, 2012
    Messages:
    893
    Likes Received:
    153
    auDA Member:
    Yes
    Yeah it is blackhat, or maybe malicious. Their full intentions are not yet clear, aside from referrer spam.

    Had a lot of sites hit by them. Crawling the entire site about four times a day. Chewing up bandwidth and jacking the bounce rate through the roof.

    Because they don't identify as a crawler, they come in as visits which bounce 100%. If you have hundreds or thousands of pages on your site, then that is a lot of pages with a 100% bounce rate.

    Will be interesting to see what their full intentions are.

    Anyone else had these guys hit their site?
     
  5. nina

    nina Membership: VIP

    Joined:
    Jul 26, 2011
    Messages:
    199
    Likes Received:
    10
    auDA Member:
    Yes
    Actually I signed up for it - the free one and every day I get an email that states where I am placed in google with up to 50 names, for free - I have since then deleted my account as I was unable to see the value of it to be honest.

    What shits me though is that I can't hide my own site - eg.. they don't give you a javascript file to put into your site so you can open an account and put any url in there and track them - that is the part I don't like about it.
     
  6. chris

    chris Administrator

    Joined:
    Mar 7, 2010
    Messages:
    2,253
    Likes Received:
    829
    auDA Member:
    Yes
    I ended up adding this to .htaccess on a few sites and the bounce rate and time-on-site is back to normal.

    Code:
    # block semalt.com
    RewriteEngine on
    RewriteCond %{HTTP_REFERER} semalt\.com [NC]
    RewriteRule .* - [F]
     
  7. petermeadit

    petermeadit Membership: VIP

    Joined:
    Jul 13, 2012
    Messages:
    893
    Likes Received:
    153
    auDA Member:
    Yes
    Update on the semalt.com referrer issue.

    Of course the script crawler.php runs not only from the main domain but also from a whole bunch of hosts, and sends referral traffic that bounces like 33 .semalt .com and 42 .semalt .com I even found traffic coming from hosts on unrelated domains like semalt .somedomain .com etc. not sure if the owners are aware they are running the semalt crawler or not.

    Also the code in the .htaccess file which blocked the referrer, still actually resolved to a 403 error page and still I wanted to keep it off our sites completely.

    So we decided to just redirect the traffic back.

    Here is the code we ended up using:

    Code:
    # Block visits from semalt.com
    RewriteEngine on
    RewriteCond %{HTTP_REFERER} ^http://([^.]+\.)*semalt\.com [NC]
    RewriteRule (.*) http://www.semalt.com [R=301,L]
    
    Just in case you might like to know.


    Thanks to Oz for thinking of sending the traffic back to them... :)


    Cheers All
     
    Last edited: Sep 4, 2014
  8. chris

    chris Administrator

    Joined:
    Mar 7, 2010
    Messages:
    2,253
    Likes Received:
    829
    auDA Member:
    Yes
    Thanks for sharing Peter, what a PITA eh!
     
  9. James

    James Membership: VIP

    Joined:
    Sep 9, 2010
    Messages:
    2,782
    Likes Received:
    460
  10. nt81

    nt81 Membership: Community

    Joined:
    Jul 22, 2013
    Messages:
    443
    Likes Received:
    44
    We've also blocked them for clients at work.

    Quite a few hosting accounts were getting smashed. Nobody gives any fark for search engine traffic other than Google / Bing / Yahoo anyway.

    Seems to have fixed quite a few bandwidth issues lately. Bots be loco!
     
  11. Ash

    Ash Membership: Trader

    Joined:
    Dec 21, 2010
    Messages:
    1,145
    Likes Received:
    63
    Thanks Peter - just used this on 4 of my sites. I wonder though, any risk of them biting back?
     
  12. petermeadit

    petermeadit Membership: VIP

    Joined:
    Jul 13, 2012
    Messages:
    893
    Likes Received:
    153
    auDA Member:
    Yes
    Biting back? Not sure, as James says we have seen them become quite defensive on twitter, and other platforms. They claim what they are doing is legit.

    Of course the main issue we have with them is if they are going to crawl the site that is fine, but at least send traffic as a crawler, instead of a referrer that bounces 100%. Of course if this or any other referrer bounces 100% for all of the pages on my site, over and over again, then we will block it. This goes for any unwanted traffic, it gets blocked in the .htaccess file.
     
  13. petermeadit

    petermeadit Membership: VIP

    Joined:
    Jul 13, 2012
    Messages:
    893
    Likes Received:
    153
    auDA Member:
    Yes
    Finding that this code is not blocking referrer any more. Not sure how they are doing it, but the traffic still hits the site and bounces 100%.
    This really is referrer spam, and if you go to the website and follow the instructions and enter your website to get it removed, be prepared for further problems.

    Of course the major impact of this is the User Signals, which is a major ranking factor. So if you are getting lots of referral traffic that bounces then that may affect your rankings.

    Also other sites have started using referrer spam in the same way.
     
  14. petermeadit

    petermeadit Membership: VIP

    Joined:
    Jul 13, 2012
    Messages:
    893
    Likes Received:
    153
    auDA Member:
    Yes
    Just another update to this one. The code I was using in my .htaccess stopped working and I was still getting hits from semalt that were bouncing 100% for sites with less traffic, this can have a big impact.
    Here is a bit a bit of code to use. I have not seen any traffic in about a week now, so it really looks to be working.

    BTW I found this on a WordPress thread somewhere, so I did not come up with it, but thanks to whoever did.

    Code:
    # Block semalt
    SetEnvIfNoCase Via evil-spam-proxy spammer=yes
    SetEnvIfNoCase Referer evil-spam-domain.com spammer=yes
    SetEnvIfNoCase Referer evil-spam-keyword spammer=yes
    SetEnvIfNoCase Via pinappleproxy spammer=yes
    SetEnvIfNoCase Referer semalt.com spammer=yes
    SetEnvIfNoCase Referer poker spammer=yes
    
    Order allow,deny
    Allow from all
    Deny from env=spammer
    
    # End Block semalt
     
  15. whiterabit

    whiterabit Membership: Community

    Joined:
    Apr 20, 2011
    Messages:
    69
    Likes Received:
    4
    I've been hit by the referrer spam and I have found a possible solution but it means that you have to update your .htaccess file. can anyone confirm if either of the following code snips would work:

    Version 1
    #block semalt
    RewriteCond %{HTTP_REFERER} ^http://([^.]+\.)*semalt\.com [NC]
    RewriteRule (.*) http://www.semalt.com [R=301,L]

    Version 2
    #block referring domains
    RewriteEngine on
    RewriteCond %{HTTP_REFERER} semalt\.com [NC]
    RewriteRule .* – [F]

    Thanks
    MIKE
     
  16. petermeadit

    petermeadit Membership: VIP

    Joined:
    Jul 13, 2012
    Messages:
    893
    Likes Received:
    153
    auDA Member:
    Yes
    Hi whiterabit
    I tried the kind of blocking you are trying with no luck. There is a plugin called Analytics Spam Blocker. It is by Luke Williamson http://lukewilliamson.com.au/ It creates the following output which you could just paste into your .htaccess file if you like. But might be better to install the plugin.

    # Analytics Spam Blocker - Start
    SetEnvIfNoCase Referer semalt.com spambot=yes
    SetEnvIfNoCase Referer darodar.com spambot=yes
    SetEnvIfNoCase Referer buttons-for-website.com spambot=yes
    SetEnvIfNoCase Referer fbdownloader.com spambot=yes
    SetEnvIfNoCase Referer descargar-musicas-gratis.com spambot=yes
    SetEnvIfNoCase Referer baixar-musicas-gratis.comsavetubevideo.com spambot=yes
    SetEnvIfNoCase Referer srecorder.com spambot=yes
    SetEnvIfNoCase Referer kambasoft.com spambot=yes
    SetEnvIfNoCase Referer ilovevitaly.com spambot=yes
    SetEnvIfNoCase Referer ilovevitaly.co spambot=yes
    SetEnvIfNoCase Referer ilovevitaly.ru spambot=yes
    SetEnvIfNoCase Referer blackhatworth.com spambot=yes
    SetEnvIfNoCase Referer priceg.com spambot=yes
    SetEnvIfNoCase Referer backgroundpictures.net spambot=yes
    SetEnvIfNoCase Referer embedle.com spambot=yes
    SetEnvIfNoCase Referer extener.com spambot=yes
    SetEnvIfNoCase Referer extener.org spambot=yes
    SetEnvIfNoCase Referer fbfreegifts.com spambot=yes
    SetEnvIfNoCase Referer feedouble.com spambot=yes
    SetEnvIfNoCase Referer feedouble.net spambot=yes
    SetEnvIfNoCase Referer japfm.com spambot=yes
    SetEnvIfNoCase Referer joinandplay.me spambot=yes
    SetEnvIfNoCase Referer joingames.org spambot=yes
    SetEnvIfNoCase Referer iskalko.ru spambot=yes
    SetEnvIfNoCase Referer musicprojectfoundation.com spambot=yes
    SetEnvIfNoCase Referer myprintscreen.com spambot=yes
    SetEnvIfNoCase Referer slftsdybbg.ru spambot=yes
    SetEnvIfNoCase Referer edakgfvwql.ru spambot=yes
    SetEnvIfNoCase Referer openfrost.com spambot=yes
    SetEnvIfNoCase Referer openfrost.net spambot=yes
    SetEnvIfNoCase Referer openmediasoft.com spambot=yes
    SetEnvIfNoCase Referer serw.clicksor.com spambot=yes
    SetEnvIfNoCase Referer socialseet.ru spambot=yes
    SetEnvIfNoCase Referer sharebutton.net spambot=yes
    SetEnvIfNoCase Referer cityadspix.com spambot=yes
    SetEnvIfNoCase Referer screentoolkit.com spambot=yes
    SetEnvIfNoCase Referer softomix.com spambot=yes
    SetEnvIfNoCase Referer softomix.net spambot=yes
    SetEnvIfNoCase Referer softomix.ru spambot=yes
    SetEnvIfNoCase Referer gobongo.info spambot=yes
    SetEnvIfNoCase Referer myftpupload.com spambot=yes
    SetEnvIfNoCase Referer websocial.me spambot=yes
    SetEnvIfNoCase Referer luxup.ru spambot=yes
    SetEnvIfNoCase Referer ykecwqlixx.ru spambot=yes
    SetEnvIfNoCase Referer soundfrost.org spambot=yes
    SetEnvIfNoCase Referer seoexperimenty.ru spambot=yes
    SetEnvIfNoCase Referer cenokos.ru spambot=yes
    SetEnvIfNoCase Referer star61.de spambot=yes
    SetEnvIfNoCase Referer superiends.org spambot=yes
    SetEnvIfNoCase Referer vapmedia.org spambot=yes
    SetEnvIfNoCase Referer econom.co spambot=yes
    SetEnvIfNoCase Referer vodkoved.ru spambot=yes
    SetEnvIfNoCase Referer adcash.com spambot=yes
    SetEnvIfNoCase Referer videofrost.com spambot=yes
    SetEnvIfNoCase Referer youtubedownload.org spambot=yes
    SetEnvIfNoCase Referer zazagames.org spambot=yes
    SetEnvIfNoCase Referer 7makemoneyonline.com spambot=yes
    SetEnvIfNoCase Referer ranksonic.info spambot=yes
    SetEnvIfNoCase Referer hulfingtonpost.com spambot=yes
    SetEnvIfNoCase Referer viandpet.com spambot=yes
    SetEnvIfNoCase Referer a-hau.mk spambot=yes
    SetEnvIfNoCase Referer cfsrating.sonicwall.com:8080 spambot=yes
    SetEnvIfNoCase Referer yougetsignal.com spambot=yes
    SetEnvIfNoCase Referer cenoval.ru spambot=yes
    SetEnvIfNoCase Referer bestwebsiteawards.com spambot=yes
    SetEnvIfNoCase Referer bestwebsitesawards.com spambot=yes
    SetEnvIfNoCase Referer simple-share-buttons.com spambot=yes
    SetEnvIfNoCase Referer adviceforum.info spambot=yes
    SetEnvIfNoCase Referer o-o-6-o-o.com spambot=yes
    SetEnvIfNoCase Referer o-o-8-o-o.com spambot=yes
    SetEnvIfNoCase Referer humanorightswatch.org spambot=yes
    SetEnvIfNoCase Referer smailik.org spambot=yes
    SetEnvIfNoCase Referer s.click.aliexpress.com spambot=yes
    SetEnvIfNoCase Referer social-buttons.com spambot=yes
    SetEnvIfNoCase Referer 4webmasters.org spambot=yes
    SetEnvIfNoCase Referer best-seo-offer.com spambot=yes
    SetEnvIfNoCase Referer best-seo-solution.com spambot=yes
    SetEnvIfNoCase Referer buttons-for-your-website.com spambot=yes
    SetEnvIfNoCase Referer www.Get-Free-Traffic-Now.com spambot=yes
    SetEnvIfNoCase Referer free-share-buttons.co spambot=yes
    SetEnvIfNoCase Referer theguardlan.com spambot=yes
    SetEnvIfNoCase Referer googlsucks.com spambot=yes
    SetEnvIfNoCase Referer buy-cheap-online.info spambot=yes
    SetEnvIfNoCase Referer forum69.info spambot=yes
    SetEnvIfNoCase Referer meendo-free-traffic.ga spambot=yes
    SetEnvIfNoCase Referer www.kabbalah-red-bracelets.com spambot=yes
    SetEnvIfNoCase Referer pornhub-forum.ga spambot=yes
    SetEnvIfNoCase Referer pornhubforum.tk spambot=yes
    SetEnvIfNoCase Referer anal-acrobats.hol.es spambot=yes
    SetEnvIfNoCase Referer youporn-forum.ga spambot=yes
    SetEnvIfNoCase Referer sexyteens.hol.es spambot=yes
    SetEnvIfNoCase Referer amanda-porn.ga spambot=yes
    SetEnvIfNoCase Referer generalporn.org spambot=yes
    SetEnvIfNoCase Referer depositfiles-porn.ga spambot=yes
    SetEnvIfNoCase Referer rapidgator-porn.ga spambot=yes
    SetEnvIfNoCase Referer torture.ml spambot=yes
    SetEnvIfNoCase Referer domination.ml spambot=yes
    SetEnvIfNoCase Referer webmaster-traffic.com spambot=yes
    SetEnvIfNoCase Referer youporn-forum.uni.me spambot=yes
    SetEnvIfNoCase Referer www.event-tracking.com spambot=yes
    SetEnvIfNoCase Referer free-share-buttons.com spambot=yes
    SetEnvIfNoCase Referer free-social-buttons.com spambot=yes
    SetEnvIfNoCase Referer guardlink.org spambot=yes
    SetEnvIfNoCase Referer redtube-talk.ga spambot=yes
    SetEnvIfNoCase Referer sanjosestartups.com spambot=yes
    SetEnvIfNoCase Referer trafficmonetize.org spambot=yes
    SetEnvIfNoCase Referer sitevaluation.org spambot=yes
    SetEnvIfNoCase Referer 100dollars-seo.com spambot=yes
    SetEnvIfNoCase Referer dailyrank.net spambot=yes
    SetEnvIfNoCase Referer howtostopreferralspam.eu spambot=yes
    Order allow,deny
    Allow from all
    Deny from env=spambot
    # Analytics Spam Blocker - End

    Here is the link to the plugin https://wordpress.org/plugins/analytics-spam-blocker/

    HTH