Question - which search engines have YOU used which have directed you to the Coffee Shop?

Background - why I am asking

We have ten times the number of visitors to the Coffee Shop than we had a year ago. "Users Online" is telling us around 2,000 good requests in the last 15 minutes compared to around 200 good requests on a typical look last year. A very high proportion of the additional traffic is from search engine crawlers and other automata, and to a very great extent it's what I had anticiated as we moved from http to https.  But I need to monitor it and ask / tell / force irrellevant search engines to ignore or go light on our pages - for example, Baiduspider and ClaudeBot are discouraged - told not to index us and if they persist are given a quick forbidden response.

Over the 24 hours to 04:30 this morning, almost 300,000 requests were made of our Coffee Shop worker server to the Coffee Shop URLs. 50,000 of those were from Amazon AI to feed Alexa, 20,000 from Bingbot to feed Bing and 10,000 from Googlebot to feed Google.  Those are the best known; in total a dozen different robots have been indexing us with in excess of 500 page requests each.

Comments welcome on where you might expect to find the Coffee Shop but have not done so.

 
I can't remember how I initially found the Coffee Shop. (via Chrome ((edit))
After becoming a regular lurker, I signed up and bookmarket it.
When I joined Firstbus, I verbally told our manager I was a member, but with a few changes of depot, many changes of managers, guessing that was lost in the mists of time.
Always posted carefully, but honestly, where it involved bus items; trying not to bring First into disrepute.

Now retired, I still try to post about what I hear and see about Firstbus (still in contact with current working drivers).

 

Same here.

Meanwhile, I use DuckDuckGo.

 
Using Brave these days

 

Interesting ...


My highlighting.   No - they are failing to say who they are - no must about that - it's their choice and as a content provider, perhaps it should be our right at the Coffee Shop to chose who indexes us. As written, this description of the Brave Search Crawler suggests they are intentionally muddying the waters to index us (and use our resources and bandwidth in doing to so) in a misleading way.

 
I registered as a member of the Coffee Shop forum on 23 December 2007, at 00:12:25.  I remember doing so, as I had become increasingly frustrated at the constant delays I had experienced getting home from Bristol on the train, missing our child's early swimming lessons, for example.

However, I have no recollection at all as to how I found this particular forum on the internet to vent my frustrations.  You were all unlucky. 

My current search engine of choice is Google.

CfN. 

 
No utter shocks in the early results - but some fascinating data which is helping inform me as to where all the access are coming from, which are effective in helping us, and which might be a waste of bandwidth / data harvesting for no discernable reason.   Much appreciated, keep comments coming.

 

I don't think I used Brave to find the Coffee Shop.  That was many moons ago and probably involved either Firefox or the old MS Internet Explorer

Coffee Shop is bookmarked on the Brave browser window, however.

 
I found the Coffee Shop when using Google to look for details of the Reading Station rebuild.  I was then a lurker and later became a member.

 
Am a confirmed DuckDuckGo user now. Can't remember how/when I found the Coffee shop - perhaps Google was the only one then !

 
I used Google back in 2006 to look for content on when FGW shortened all their fleet of 158s from 3 coaches to 2 and the chaos that ensued on the Cardiff to Portsmouth line as a result and I’ve been here ever since.

 
15 out of 22 of you (who voted) have been directed to the Coffee Shop from Google, 5 of you from DuckDuckGo, 2 from another search engine, 2 from Bing. And one from Yahoo.  4 of you have never been directed to the Coffee Shop by a search engine, and three of you never even use a search engine.

Thank you for these inputs.

The Coffee Shop servers get a huge amount of crawling traffic from search engines and other spiders and they have an impact on performance.   Part of the monitoring I do in the background is to see where our resources are going, and to ask some of these spiders to go easy, and if necessary to take action to force them to back off.

I am reading the inputs in association with general search engine popularity surveys,  and for the UK Google is even more dominant in general than would be suggested by our survey. Amazingly, the load from the "Googlebot" on our servers is very low and  I would encourage it more rather than less if I could.  DuckDuckGo doesn't seem to be notably indexing us directly - and they do say they use other sources too.

With other crawlers, I take a look at their crawler descriptions - why they want the information, what they are doing with it, whether that's a use relevant to the Coffee Shop and how much of an impact they're having on our server performance.  In my view, there is strength in supporting some AI crawlers to help inform them into the future, and in contrast we should really not impact our user base because of aggressive crawling to inform search engine users who are in countries far, far away.   Those crawlers run by companies who sell onwards services such as SEO optimisation or plagiarism reporting but do not make any significant services available without people paying them (for the data they have trawled for free) are typically told to go away if their traffic levels reach a level on our servers that I notice.

The change from http to https last December adds a shell of respectable security to our content, and - as  I would have expected (and indeed hoped) there has been hugely increased indexing activity since. I am somewhat disappointed that the increased activity has not extended to valid new member applications - many reasons and under discussion Behind the Counter, some probably under our control and others not.    The survey helps inform that - only tangentially though; it's rather like asking people on the train why they don't use the train more when really you should be asking people in the street ...

 
A bit of a read on the woes caused by AI web site crawlers, from the Register.

Mark

https://www.theregister.com/2025/08/29/ai_web_crawlers_are_destroying/

 

The full headline is "AI web crawlers are destroying websites in their never-ending hunger for any and all content" ... dramatically put and if I hadn't put certain measures in place, our servers would be swamped - and it's an ongoing "game".  Thank goodness I can jump on and very quickly label visitors that match a certain profile as "naughty_boys" and then do a "deny from naughty_boys" and there are some things visitors can do that get themselves automatically added to that group.

The Coffee Shoo IS on shared servers as described in that article, but shared with "my" other sites so if something else such as https://www.twhc.org.uk/person.html - the Well House Collection with records for 52,000 former Melksham residents - gets hit, I am responsible there too - and indeed my server tool are across all sites on the server.   But these days, 4 requests out of 5 are to the Coffee Shop.