Get all your news in one place.
100’s of premium titles.
One app.
Start reading
The Guardian - UK
The Guardian - UK
Technology
Jack Schofield

Blocking spiders knocks your site off Google's SafeSearch

A Harvard report on Google's optional SafeSearch (censorship) facility found that many innocuous sites were blocked for no apparent reason. It turns out that if Webmasters use a robots.txt file to stop Google's spiders from crawling parts of the site, Google excludes it from SafeSearch on the very reasonable grounds that if it can't search a site, it can't tell whether it is safe or not. Read more at CNet.

Sign up to read this article
Read news from 100’s of titles, curated specifically for you.
Already a member? Sign in here
Related Stories
Top stories on inkl right now
One subscription that gives you access to news from hundreds of sites
Already a member? Sign in here
Our Picks
‘People Do Indeed Ask’: Katt Williams Drops Claims About What He Saw When People Hit Up Diddy’s Parties
Katt Williams bring his perspective.
Cinemablend
Cinemablend
Florida man picks up cold blooded iguana from under a tree and then makes tacos with its meat
TikTok user @gray.davis has a habit of making a meal out of everything, but no one was prepared for this one. Exactly a week ago, he uploaded a video on his profile in which he picked up an iguana from under a tree and used its meat to make tacos,…
The Mary Sue
The Mary Sue
‘They were disgusted that I was even talking’: Nashville dad takes kids to McDonald’s, ends up realizing they don’t want him there at all
Remember the golden age of McDonald’s? Bright red booths, a dedicated cashier who actually smiled, and a self-serve soda fountain that felt like a sanctuary of free refills. According to one Nashville father, those days are officially buried under a pile of sterile kiosks and gray paint. 
The Mary Sue
The Mary Sue
Miami woman brought a family heirloom to Chicago airport and loses it to the TSA. But why is nobody feeling bad for her?
We’ve all had those mask-off moments at the TSA. The frantic realization that your expensive moisturizer is 4 ounces instead of 3.4, or the internal panic when you forget a stray water bottle in your bag. But one traveler just lived through a tragedy that has the internet screaming at…
The Mary Sue
The Mary Sue
8 Household Items That Are Secretly Making You Tired (The Toxic Home List)
Do you ever wake up feeling like you haven’t slept a wink, despite getting a full eight hours? We often blame our busy schedules, our age, or our stress levels for our chronic fatigue. However, the culprit might hide in plain sight within your four walls. A hidden system of…
Budget and the Bees
Budget and the Bees
"He says, 'Everyone knows what ****er means there. You can't show that in England. They'll edit it out.'" A wrestling legend shares how Ozzy Osbourne was "the voice of reason" when he appeared on WWE RAW
Trust Ozzy Osbourne to give American broadcasters a lesson in British swearing
Louder
Louder
Fourteen days free
Download the app
One app. One membership.
100+ trusted global sources.
Download on the AppStore Get it on Google Play