MastodonTech.de

Tomas Norre :verified:Today 14th of Juli, It's 10 Years since I did my first <a href="https://phpc.social/tags/TYPO3" class="mention hashtag" rel="nofollow noopener" target="_blank">#TYPO3</a> <a href="https://phpc.social/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a> contribution, and now I'm the maintainer of it.<a href="https://blog.tomasnorre.dk/blog/typo3-crawler-10years/" rel="nofollow noopener" translate="no" target="_blank">https://blog.tomasnorre.dk/blog/typo3-crawler-10years/</a><a href="https://phpc.social/tags/HappyCrawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#HappyCrawling</a> <a href="https://phpc.social/tags/OpenSource" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenSource</a>

MisterOpenDataEin Diagramm zu den Zugriffen auf das Open-Data-Portal Schleswig-Holstein. Es wird versucht, Bots zu erkennen. Ab etwa Oktober 2023 nimmt die Belastung durch aggressives KI-Training zu. <a href="https://norden.social/tags/opendata" class="mention hashtag" rel="nofollow noopener" target="_blank">#opendata</a> <a href="https://norden.social/tags/schleswigholstein" class="mention hashtag" rel="nofollow noopener" target="_blank">#schleswigholstein</a> <a href="https://norden.social/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a> <a href="https://norden.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a>

@francksOur small team vs millions of bots<a href="https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots" rel="nofollow noopener" translate="no" target="_blank">https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots</a><a href="https://mstdn.fr/tags/fsf" class="mention hashtag" rel="nofollow noopener" target="_blank">#fsf</a> <a href="https://mstdn.fr/tags/freesoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#freesoftware</a> <a href="https://mstdn.fr/tags/ddos" class="mention hashtag" rel="nofollow noopener" target="_blank">#ddos</a> <a href="https://mstdn.fr/tags/javascrip" class="mention hashtag" rel="nofollow noopener" target="_blank">#javascrip</a> <a href="https://mstdn.fr/tags/anubis" class="mention hashtag" rel="nofollow noopener" target="_blank">#anubis</a> <a href="https://mstdn.fr/tags/botnet" class="mention hashtag" rel="nofollow noopener" target="_blank">#botnet</a> <a href="https://mstdn.fr/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mstdn.fr/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mstdn.fr/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://mstdn.fr/tags/proprietarysoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#proprietarysoftware</a> <a href="https://mstdn.fr/tags/malware" class="mention hashtag" rel="nofollow noopener" target="_blank">#malware</a>

Arie van Deursen“The problem is whether you create content to sell ads, sell subscriptions, or just to know that people value what you've created, an AI-driven web doesn't reward content creators the way that the old search-driven web did. And that means the deal that Google made to take content in exchange for sending you traffic just doesn't make sense anymore.”— Matthew Prince, Cloudflare<a href="https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/" rel="nofollow noopener" translate="no" target="_blank">https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/</a><a href="https://mastodon.acm.org/tags/cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudflare</a> <a href="https://mastodon.acm.org/tags/contentcreators" class="mention hashtag" rel="nofollow noopener" target="_blank">#contentcreators</a> <a href="https://mastodon.acm.org/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://mastodon.acm.org/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mastodon.acm.org/tags/search" class="mention hashtag" rel="nofollow noopener" target="_blank">#search</a>

Erik C. ThauvinCloudflare Introduces pay per crawl: enabling content owners to charge AI crawlers for access<a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mastodon.social/tags/cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudflare</a> <a href="https://mastodon.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a><a href="https://blog.cloudflare.com/introducing-pay-per-crawl/" rel="nofollow noopener" translate="no" target="_blank">https://blog.cloudflare.com/introducing-pay-per-crawl/</a>

𝕂𝚞𝚋𝚒𝚔ℙ𝚒𝚡𝚎𝚕»Pay up or stop scraping – Cloudflare program charges bots for each crawl: Cloudflare now beta testing pay-per-crawl feature to stop endless AI scraping. Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites.«This is certainly a good idea, but on the other hand, the competition is trying to eliminate each other. I'm curious… 🍿😎<a href="https://arstechnica.com/tech-policy/2025/07/pay-up-or-stop-scraping-cloudflare-program-charges-bots-for-each-crawl/" rel="nofollow noopener" translate="no" target="_blank">https://arstechnica.com/tech-policy/2025/07/pay-up-or-stop-scraping-cloudflare-program-charges-bots-for-each-crawl/</a><a href="https://chaos.social/tags/cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudflare</a> <a href="https://chaos.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://chaos.social/tags/payorstop" class="mention hashtag" rel="nofollow noopener" target="_blank">#payorstop</a> <a href="https://chaos.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://chaos.social/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a> <a href="https://chaos.social/tags/website" class="mention hashtag" rel="nofollow noopener" target="_blank">#website</a> <a href="https://chaos.social/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#bots</a> <a href="https://chaos.social/tags/aibots" class="mention hashtag" rel="nofollow noopener" target="_blank">#aibots</a>

DJM (freelance for hire)Ah aha ah... sorry... Ah ah ah ahA proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time. <a href="https://llmstxt.org/" rel="nofollow noopener" translate="no" target="_blank">https://llmstxt.org/</a>Remember that you can block their robots from crawling your site/blog: <a href="https://www.didiermary.fr/bloquer-ai-bots-chatgpt-openai/" rel="nofollow noopener" translate="no" target="_blank">https://www.didiermary.fr/bloquer-ai-bots-chatgpt-openai/</a><a href="https://masto.ai/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://masto.ai/tags/IA" class="mention hashtag" rel="nofollow noopener" target="_blank">#IA</a> <a href="https://masto.ai/tags/AGI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AGI</a> <a href="https://masto.ai/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a> <a href="https://masto.ai/tags/Robot" class="mention hashtag" rel="nofollow noopener" target="_blank">#Robot</a> <a href="https://masto.ai/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a>

teufelswerkDer aus Deutschland (Karlsruhe) stammende B2B Databroker "dealfront.com" brüstet sich mit Aussagen wie "Die Daten, die wir in Dealfront bereitstellen, sind DSGVO-konform und entsprechen den höchsten Datenschutz- und Privatsphäre-Standards." Da dieser Anbieter ohne mein Wissen und meine Einwilligung u.a. meine personenbezogen, aber auch unsere Unternehmensdaten an Spammer verkauft, gecrawlte Daten anreichert und aus diversen Quellen KI-gestützt zusammenführt, stehe ich diesem Anbieter mehr als sehr skeptisch 🖕 gegenüber. Ich werde ihm jetzt auf den Zahn fühlen und euch darüber berichten.<a href="https://social.tchncs.de/tags/dealfront" class="mention hashtag" rel="nofollow noopener" target="_blank">#dealfront</a> <a href="https://social.tchncs.de/tags/databroker" class="mention hashtag" rel="nofollow noopener" target="_blank">#databroker</a> <a href="https://social.tchncs.de/tags/b2b" class="mention hashtag" rel="nofollow noopener" target="_blank">#b2b</a> <a href="https://social.tchncs.de/tags/ki" class="mention hashtag" rel="nofollow noopener" target="_blank">#ki</a> <a href="https://social.tchncs.de/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://social.tchncs.de/tags/profiling" class="mention hashtag" rel="nofollow noopener" target="_blank">#profiling</a> <a href="https://social.tchncs.de/tags/dsgvo" class="mention hashtag" rel="nofollow noopener" target="_blank">#dsgvo</a> <a href="https://social.tchncs.de/tags/datenschutz" class="mention hashtag" rel="nofollow noopener" target="_blank">#datenschutz</a> <a href="https://social.tchncs.de/tags/spam" class="mention hashtag" rel="nofollow noopener" target="_blank">#spam</a>

Kevin Karhan :verified:<a href="https://mastodon.pnpde.social/@spielleitung" class="u-url mention" rel="nofollow noopener" target="_blank">@spielleitung</a> also ich mach's mir professionell einfacher und blockiere einfach alle bekannten <a href="https://infosec.space/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a>...<ul><li>Wenn ihr weitere Ranges an <a href="https://infosec.space/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://infosec.space/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a>|n kennt: Immer her damit! </li></ul><a href="https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv" rel="nofollow noopener" translate="no" target="_blank">https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv</a>

Hacker NewsJava Virtual Threads Ate My Memory: A Web Crawler's Tale of Speed vs. Memory<a href="https://dariobalinzo.medium.com/virtual-threads-ate-my-memory-a-web-crawlers-tale-of-speed-vs-memory-a92fc75085f6" rel="nofollow noopener" translate="no" target="_blank">https://dariobalinzo.medium.com/virtual-threads-ate-my-memory-a-web-crawlers-tale-of-speed-vs-memory-a92fc75085f6</a><a href="https://mastodon.social/tags/HackerNews" class="mention hashtag" rel="nofollow noopener" target="_blank">#HackerNews</a> <a href="https://mastodon.social/tags/Java" class="mention hashtag" rel="nofollow noopener" target="_blank">#Java</a> <a href="https://mastodon.social/tags/Virtual" class="mention hashtag" rel="nofollow noopener" target="_blank">#Virtual</a> <a href="https://mastodon.social/tags/Threads" class="mention hashtag" rel="nofollow noopener" target="_blank">#Threads</a> <a href="https://mastodon.social/tags/Web" class="mention hashtag" rel="nofollow noopener" target="_blank">#Web</a> <a href="https://mastodon.social/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a> <a href="https://mastodon.social/tags/Memory" class="mention hashtag" rel="nofollow noopener" target="_blank">#Memory</a> <a href="https://mastodon.social/tags/Management" class="mention hashtag" rel="nofollow noopener" target="_blank">#Management</a> <a href="https://mastodon.social/tags/Speed" class="mention hashtag" rel="nofollow noopener" target="_blank">#Speed</a> <a href="https://mastodon.social/tags/Optimization" class="mention hashtag" rel="nofollow noopener" target="_blank">#Optimization</a>

Camelia :tranarchy_a_nonbinary: 🇵🇸Well, that's it. I've officially started using <a href="https://tech.lgbt/tags/Anubis" class="mention hashtag" rel="nofollow noopener" target="_blank">#Anubis</a> to protect my self-hosted <a href="https://tech.lgbt/tags/Forgejo" class="mention hashtag" rel="nofollow noopener" target="_blank">#Forgejo</a> instance.I didn't want to do it at first, but my nginx and fail2ban configurations weren't efficient enough.Down with LLMs!<a href="https://tech.lgbt/tags/LLM" class="mention hashtag" rel="nofollow noopener" target="_blank">#LLM</a> <a href="https://tech.lgbt/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a>

Tomas Norre :verified:Just released version 12.0.8 of the <a href="https://phpc.social/tags/TYPO3" class="mention hashtag" rel="nofollow noopener" target="_blank">#TYPO3</a> <a href="https://phpc.social/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a>, with a number of fixes.Thanks for all contributions. <a href="https://phpc.social/tags/HappyCrawling" class="mention hashtag" rel="nofollow noopener" target="_blank">#HappyCrawling</a><a href="https://github.com/tomasnorre/crawler/releases/tag/12.0.8" rel="nofollow noopener" translate="no" target="_blank">https://github.com/tomasnorre/crawler/releases/tag/12.0.8</a>

noisioHow do you deal with web crawlers and bots on your personal websites? There is a lot of automated traffic I can detect at my own. Some of yours have this short living captchas, that don't even need to be filled out and disappear in just a second. While starting investigating in this topic, I can find as many info about anti-webcrawlers as anti-anti webcrawlers and get lost very soon.What are solutions you have found for this?<a href="https://chaos.social/tags/website" class="mention hashtag" rel="nofollow noopener" target="_blank">#website</a> <a href="https://chaos.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://chaos.social/tags/bot" class="mention hashtag" rel="nofollow noopener" target="_blank">#bot</a> <a href="https://chaos.social/tags/defense" class="mention hashtag" rel="nofollow noopener" target="_blank">#defense</a>

Leszek<a href="https://mastodon.social/@tdp_org" class="u-url mention" rel="nofollow noopener" target="_blank">@tdp_org</a> unbelievable! I've set up <a href="https://genomic.social/tags/nepenthes" class="mention hashtag" rel="nofollow noopener" target="_blank">#nepenthes</a> tarpit in my personal blog and reached over 1 million requests from a <a href="https://genomic.social/tags/amazon" class="mention hashtag" rel="nofollow noopener" target="_blank">#amazon</a> <a href="https://genomic.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> alone in less than 3 months! Other bots typically gave up after crawling tens of thousands of pages of bulshit. Naturally, my robots.txt informs not to crawl the tarpit ...

Larvitz :fedora: :redhat:Anubis - Weigh the soul of incoming HTTP requests using proof-of-work to stop AI crawlers (<a href="https://anubis.techaro.lol" rel="nofollow noopener" translate="no" target="_blank">https://anubis.techaro.lol</a>)I give that a try. Maybe it can reduce the AI crawler mess a little bit on my servers.<a href="https://burningboard.net/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://burningboard.net/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://burningboard.net/tags/aicrawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#aicrawler</a> <a href="https://burningboard.net/tags/fckai" class="mention hashtag" rel="nofollow noopener" target="_blank">#fckai</a> <a href="https://burningboard.net/tags/anibus" class="mention hashtag" rel="nofollow noopener" target="_blank">#anibus</a>

Orhun Parmaksız 👾Is this the future of terminal gaming? 🤯⚔️ **ratthew** — A 3D dungeon crawler in the terminal.🦀 Written in Rust!🏗️ Built with <a href="https://fosstodon.org/@ratatui_rs" class="u-url mention" rel="nofollow noopener" target="_blank">@ratatui_rs</a> + <a href="https://mastodon.social/@bevy" class="u-url mention" rel="nofollow noopener" target="_blank">@bevy</a> ⭐ GitHub: <a href="https://github.com/cxreiff/ratthew" rel="nofollow noopener" translate="no" target="_blank">https://github.com/cxreiff/ratthew</a> (WIP)<a href="https://fosstodon.org/tags/rustlang" class="mention hashtag" rel="nofollow noopener" target="_blank">#rustlang</a> <a href="https://fosstodon.org/tags/ratatui" class="mention hashtag" rel="nofollow noopener" target="_blank">#ratatui</a> <a href="https://fosstodon.org/tags/tui" class="mention hashtag" rel="nofollow noopener" target="_blank">#tui</a> <a href="https://fosstodon.org/tags/terminal" class="mention hashtag" rel="nofollow noopener" target="_blank">#terminal</a> <a href="https://fosstodon.org/tags/gaming" class="mention hashtag" rel="nofollow noopener" target="_blank">#gaming</a> <a href="https://fosstodon.org/tags/3d" class="mention hashtag" rel="nofollow noopener" target="_blank">#3d</a> <a href="https://fosstodon.org/tags/bevy" class="mention hashtag" rel="nofollow noopener" target="_blank">#bevy</a> <a href="https://fosstodon.org/tags/commandline" class="mention hashtag" rel="nofollow noopener" target="_blank">#commandline</a> <a href="https://fosstodon.org/tags/dungeon" class="mention hashtag" rel="nofollow noopener" target="_blank">#dungeon</a> <a href="https://fosstodon.org/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a>

Yannick PaulsenDer Druck auf öffentliche und gemeinnützige Infrastrukturen steigt. Ob nun Open-Access-Repositorien oder wie hier im Text <a href="https://openbiblio.social/tags/Wikimedia" class="mention hashtag" rel="nofollow noopener" target="_blank">#Wikimedia</a>: <a href="https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/" rel="nofollow noopener" translate="no" target="_blank">https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/</a>Es ist gut, dass die Inhalte in das Training von Künstlicher Intelligenz einbezogen werden, aber bei der unverhältnismäßigen Belastung als Ergebnis von kommerziellen Interessen überlege ich, ob es nicht eigentlich einen Ausgleich braucht.<a href="https://openbiblio.social/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a> <a href="https://openbiblio.social/tags/KI" class="mention hashtag" rel="nofollow noopener" target="_blank">#KI</a> <a href="https://openbiblio.social/tags/OpenAccess" class="mention hashtag" rel="nofollow noopener" target="_blank">#OpenAccess</a>

Michiel ScholtenWhy I am sort of afraid to share my projects now, and link to my own Git server <a href="https://dammit.nl/afraid-to-git.html" rel="nofollow noopener" translate="no" target="_blank">https://dammit.nl/afraid-to-git.html</a> <a href="https://mastodon.social/tags/git" class="mention hashtag" rel="nofollow noopener" target="_blank">#git</a> <a href="https://mastodon.social/tags/github" class="mention hashtag" rel="nofollow noopener" target="_blank">#github</a> <a href="https://mastodon.social/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mastodon.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://mastodon.social/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a>

LinuxNews.deNachdem diverse <a href="https://social.anoxinon.de/tags/ki" class="mention hashtag" rel="nofollow noopener" target="_blank">#ki</a> <a href="https://social.anoxinon.de/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://social.anoxinon.de/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> besonders respektvoll mit den öffentlichen Ressourcen von Open Source Projekten umgehen, habe ich mich dazu entschlossen eben diese auszusperren. Wir hatten in der Vergangenheit crawls, die im <a href="https://social.anoxinon.de/tags/monitoring" class="mention hashtag" rel="nofollow noopener" target="_blank">#monitoring</a> als <a href="https://social.anoxinon.de/tags/ddos" class="mention hashtag" rel="nofollow noopener" target="_blank">#ddos</a> gewertet wurden. Diverse AS erfreuen sich nun einem dauerhaften 429, einige wenige die es für alle kaputt machen…

AlexSad to see <a href="https://mastodon.org.uk/tags/lore" class="mention hashtag" rel="nofollow noopener" target="_blank">#lore</a> had to check I wasn't a bot to get through to the archive. I guess these are the <a href="https://mastodon.org.uk/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> ridden times we live in now.

Frühere Suchanfragen

Suchoptionen

Verwaltet von:

Serverstatistik:

#crawler