MastodonTech.de

Austin Huang ❤️Since people are dunking on <a href="https://mstdn.party/tags/Meta" class="mention hashtag" rel="nofollow noopener" target="_blank">#Meta</a> <a href="https://mstdn.party/tags/scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraping</a> again I'll share one tidbit: when <a href="https://mastodon.neat.computer/@jonah" class="u-url mention" rel="nofollow noopener" target="_blank">@jonah</a> and I was investigating some performance issues, I noticed that Meta-ExternalAgent was scraping /auth/sign_up and one specific invite link with different `accept` parameters (which indicates acceptance of rules), however because Mastodon returns 200 (and shows the rules again) on invalid `accept` parameters the <a href="https://mstdn.party/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> just keeps going...

🌈 vanta rainbow black 🌈that first one is only accessible via http, NOT https<a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a> <a href="https://cyberpunk.lol/tags/fediblock" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediblock</a>

🌈 vanta rainbow black 🌈i found some more fedi search scraper indexer thingies i think<a href="http://fedilogue.xyz" rel="nofollow noopener" translate="no" target="_blank">http://fedilogue.xyz</a><a href="https://fedsearch.io" rel="nofollow noopener" translate="no" target="_blank">https://fedsearch.io</a><a href="https://cyberpunk.lol/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://cyberpunk.lol/tags/indexer" class="mention hashtag" rel="nofollow noopener" target="_blank">#indexer</a> <a href="https://cyberpunk.lol/tags/fediblock" class="mention hashtag" rel="nofollow noopener" target="_blank">#fediblock</a>

Rod2ik 🇪🇺 🇨🇵 🇪🇸 🇺🇦 🇨🇦 🇩🇰 🇬🇱☮🕊️<a href="https://mastodon.social/tags/Darkdump" class="mention hashtag" rel="nofollow noopener" target="_blank">#Darkdump</a> - L'outil <a href="https://mastodon.social/tags/OSINT" class="mention hashtag" rel="nofollow noopener" target="_blank">#OSINT</a> ( <a href="https://mastodon.social/tags/Open" class="mention hashtag" rel="nofollow noopener" target="_blank">#Open</a> <a href="https://mastodon.social/tags/Source" class="mention hashtag" rel="nofollow noopener" target="_blank">#Source</a> <a href="https://mastodon.social/tags/Intelligence" class="mention hashtag" rel="nofollow noopener" target="_blank">#Intelligence</a> ) qui fouille le <a href="https://mastodon.social/tags/dark" class="mention hashtag" rel="nofollow noopener" target="_blank">#dark</a> <a href="https://mastodon.social/tags/web" class="mention hashtag" rel="nofollow noopener" target="_blank">#web</a> pour vousEn gros,vous tapez un mot-clé,et l’outil va <a href="https://mastodon.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> les sites .onion pour en extraire des emails, des métadonnées,des mots-clés,des images,des liens vers les réseaux sociaux<a href="https://korben.info/darkdump-outil-osint-fouille-dark-web.html" rel="nofollow noopener" translate="no" target="_blank">https://korben.info/darkdump-outil-osint-fouille-dark-web.html</a>

MDZG (Markdown Zen Garden)🔍 / <a href="https://mastodon.uno/tags/software" class="mention hashtag" rel="nofollow noopener" target="_blank">#software</a> / <a href="https://mastodon.uno/tags/automation" class="mention hashtag" rel="nofollow noopener" target="_blank">#automation</a> / <a href="https://mastodon.uno/tags/API" class="mention hashtag" rel="nofollow noopener" target="_blank">#API</a> <a href="https://mastodon.uno/tags/ZenRows" class="mention hashtag" rel="nofollow noopener" target="_blank">#ZenRows</a> turns hours of tedious lead hunting into a fast, reliable, and automated data workflow that fills your data pipelines.🐱🔗 <a href="https://laravista.altervista.org/CatLink/links/315" rel="nofollow noopener" translate="no" target="_blank">https://laravista.altervista.org/CatLink/links/315</a><a href="https://mastodon.uno/tags/catlink" class="mention hashtag" rel="nofollow noopener" target="_blank">#catlink</a> <a href="https://mastodon.uno/tags/SoftwareAutomation" class="mention hashtag" rel="nofollow noopener" target="_blank">#SoftwareAutomation</a> <a href="https://mastodon.uno/tags/SoftwareAutomationAPI" class="mention hashtag" rel="nofollow noopener" target="_blank">#SoftwareAutomationAPI</a> <a href="https://mastodon.uno/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mastodon.uno/tags/DataExtraction" class="mention hashtag" rel="nofollow noopener" target="_blank">#DataExtraction</a>

@francksOur small team vs millions of bots<a href="https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots" rel="nofollow noopener" translate="no" target="_blank">https://www.fsf.org/blogs/sysadmin/our-small-team-vs-millions-of-bots</a><a href="https://mstdn.fr/tags/fsf" class="mention hashtag" rel="nofollow noopener" target="_blank">#fsf</a> <a href="https://mstdn.fr/tags/freesoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#freesoftware</a> <a href="https://mstdn.fr/tags/ddos" class="mention hashtag" rel="nofollow noopener" target="_blank">#ddos</a> <a href="https://mstdn.fr/tags/javascrip" class="mention hashtag" rel="nofollow noopener" target="_blank">#javascrip</a> <a href="https://mstdn.fr/tags/anubis" class="mention hashtag" rel="nofollow noopener" target="_blank">#anubis</a> <a href="https://mstdn.fr/tags/botnet" class="mention hashtag" rel="nofollow noopener" target="_blank">#botnet</a> <a href="https://mstdn.fr/tags/llm" class="mention hashtag" rel="nofollow noopener" target="_blank">#llm</a> <a href="https://mstdn.fr/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://mstdn.fr/tags/crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#crawler</a> <a href="https://mstdn.fr/tags/proprietarysoftware" class="mention hashtag" rel="nofollow noopener" target="_blank">#proprietarysoftware</a> <a href="https://mstdn.fr/tags/malware" class="mention hashtag" rel="nofollow noopener" target="_blank">#malware</a>

Pyrzout :vm:Cloudflare bloqueará por defecto los rastreadores de contenido web IA <a href="https://blog.elhacker.net/2025/07/cloudflare-bloqueara-por-defecto-los-rastreadores-bots-trafico-web.html" rel="nofollow noopener" translate="no" target="_blank">https://blog.elhacker.net/2025/07/cloudflare-bloqueara-por-defecto-los-rastreadores-bots-trafico-web.html</a> <a href="https://social.skynetcloud.site/tags/inteligenciaartificial" class="mention hashtag" rel="nofollow noopener" target="_blank">#inteligenciaartificial</a> <a href="https://social.skynetcloud.site/tags/CloudFlare" class="mention hashtag" rel="nofollow noopener" target="_blank">#CloudFlare</a> <a href="https://social.skynetcloud.site/tags/rastreo" class="mention hashtag" rel="nofollow noopener" target="_blank">#rastreo</a> <a href="https://social.skynetcloud.site/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://social.skynetcloud.site/tags/robots" class="mention hashtag" rel="nofollow noopener" target="_blank">#robots</a> <a href="https://social.skynetcloud.site/tags/CDN" class="mention hashtag" rel="nofollow noopener" target="_blank">#CDN</a>

Kuketz-Blog 🛡»Cloudflare Introduces Default Blocking of A.I. Data Scrapers«Nett, wird aber kaum funktionieren. Weil: Fortgeschrittene Scraper nutzen Browser-Emulation und rotierende IPs, um sich als echte Nutzer auszugeben und technische Erkennung zu umgehen. Da es sich nur um eine serverseitige Maßnahme ohne rechtliche Bindung handelt, können solche Akteure die Sperren leicht und folgenlos ignorieren.<a href="https://www.nytimes.com/2025/07/01/technology/cloudflare-ai-data.html" rel="nofollow noopener" translate="no" target="_blank">https://www.nytimes.com/2025/07/01/technology/cloudflare-ai-data.html</a><a href="https://social.tchncs.de/tags/cloudflare" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloudflare</a> <a href="https://social.tchncs.de/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://social.tchncs.de/tags/ki" class="mention hashtag" rel="nofollow noopener" target="_blank">#ki</a> <a href="https://social.tchncs.de/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>/kuk

eicker.TV ▹ TechnewsMastodon untersagt das Training von KIs mit Inhalten der eigenen Plattform.Neue <a href="https://mastodon.social/tags/Nutzungsbedingungen" class="mention hashtag" rel="nofollow noopener" target="_blank">#Nutzungsbedingungen</a> gegen <a href="https://mastodon.social/tags/KITraining" class="mention hashtag" rel="nofollow noopener" target="_blank">#KITraining</a> 🚫 <a href="https://mastodon.social/tags/Mastodon" class="mention hashtag" rel="nofollow noopener" target="_blank">#Mastodon</a> verbietet ab 1. Juli explizit das <a href="https://mastodon.social/tags/Scrapen" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scrapen</a> und die Nutzung von User-Daten zum <a href="https://mastodon.social/tags/Training" class="mention hashtag" rel="nofollow noopener" target="_blank">#Training</a> von <a href="https://mastodon.social/tags/KIModellen" class="mention hashtag" rel="nofollow noopener" target="_blank">#KIModellen</a> auf seinem <a href="https://mastodon.social/tags/Hauptserver" class="mention hashtag" rel="nofollow noopener" target="_blank">#Hauptserver</a>.Klarer <a href="https://mastodon.social/tags/Schutz" class="mention hashtag" rel="nofollow noopener" target="_blank">#Schutz</a> der <a href="https://mastodon.social/tags/Community" class="mention hashtag" rel="nofollow noopener" target="_blank">#Community</a> 🤝 Die neuen Regeln untersagen automatisierte Tools wie <a href="https://mastodon.social/tags/Bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#Bots</a> und <a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a>, um Daten abzugreifen – mit Ausnahme von normalen Suchmaschinen und Browsern. (1/2)

Kevin Karhan :verified:<a href="https://mastodon.pnpde.social/@spielleitung" class="u-url mention" rel="nofollow noopener" target="_blank">@spielleitung</a> also ich mach's mir professionell einfacher und blockiere einfach alle bekannten <a href="https://infosec.space/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a>...<ul><li>Wenn ihr weitere Ranges an <a href="https://infosec.space/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://infosec.space/tags/Crawler" class="mention hashtag" rel="nofollow noopener" target="_blank">#Crawler</a>|n kennt: Immer her damit! </li></ul><a href="https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv" rel="nofollow noopener" translate="no" target="_blank">https://github.com/greyhat-academy/lists.d/blob/main/scrapers.ipv4.block.list.tsv</a>

Torf und SchneeThe most disgusting feature of this relatively new <a href="https://c.im/tags/AI" class="mention hashtag" rel="nofollow noopener" target="_blank">#AI</a> <a href="https://c.im/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> |s plague is that they are about to defile everything we like in the *good* internet.Images with relevant <a href="https://c.im/tags/AltText" class="mention hashtag" rel="nofollow noopener" target="_blank">#AltText</a>? Perfect training materials for text-to-image generative models.Static webpages? No <a href="https://c.im/tags/Anubis" class="mention hashtag" rel="nofollow noopener" target="_blank">#Anubis</a> - no problem to scrape.<a href="https://c.im/tags/Anubis" class="mention hashtag" rel="nofollow noopener" target="_blank">#Anubis</a> uses proof-of-work ( <a href="https://c.im/tags/PoW" class="mention hashtag" rel="nofollow noopener" target="_blank">#PoW</a> ), which implies either <a href="https://c.im/tags/JavaScript" class="mention hashtag" rel="nofollow noopener" target="_blank">#JavaScript</a> or manual instructions. No, it is a good solution... Best of the worst (as if there were any good ones...)Last days I learned that (1) <a href="https://c.im/tags/Tor" class="mention hashtag" rel="nofollow noopener" target="_blank">#Tor</a> has a <a href="https://c.im/tags/PoW" class="mention hashtag" rel="nofollow noopener" target="_blank">#PoW</a> mechanism (2) Anubis seems to somehow whitelist <a href="https://c.im/tags/lynx" class="mention hashtag" rel="nofollow noopener" target="_blank">#lynx</a> browser, allowing no-JS Lynx users in (a big favour for <a href="https://c.im/tags/accessibility" class="mention hashtag" rel="nofollow noopener" target="_blank">#accessibility</a> and <a href="https://c.im/tags/smolweb" class="mention hashtag" rel="nofollow noopener" target="_blank">#smolweb</a> ). Good (let's hope all these will persist).

KiwixMWoffliner, the <a href="https://wikis.world/@mediawiki" class="u-url mention" rel="nofollow noopener" target="_blank">@mediawiki</a> <a href="https://mastodon.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> has been released in version 1.15!1.15 brings a significant amount of improvements: * Support of wide used (outside Wikimedia) "ActionParse" API * Use latest libzim (we were stuck with an older version) which fixes many suggestion problems with non-latin alphabets * Move to Node.js 24 + many install fixes * Better & sophisticated remote error handlingFull changelog at <a href="https://github.com/openzim/mwoffliner/releases/tag/v1.15.0" rel="nofollow noopener" translate="no" target="_blank">https://github.com/openzim/mwoffliner/releases/tag/v1.15.0</a>Available as container image and Npmjs package!

@reiver ⊼ (Charles) :batman:3/For more on scraping (as in web-scraping) see here: <a href="https://mastodon.social/@reiver/114353728684249608" rel="nofollow noopener" translate="no" target="_blank">https://mastodon.social/@reiver/114353728684249608</a>CC: <a href="https://mastodon.social/@404mediaco" class="u-url mention" rel="nofollow noopener" target="_blank">@404mediaco</a> <a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:2/Scraping (as in Web Scraping) is the act of extracting data from HTML web-pages where the data is NOT machine-legible.If the data, even in an HTML web-page, is in a machine-legible format, then it is NOT scraping....And, getting data in JSON (key-value pairs) is definitely NOT scraping — as JSON's purpose is to communicate data in a machine-legible manner.CC: <a href="https://mastodon.social/@404mediaco" class="u-url mention" rel="nofollow noopener" target="_blank">@404mediaco</a> <a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebScraping</a>

@reiver ⊼ (Charles) :batman:1/If these researchers used a typical HTTP-based API that returns JSON, then —What these researchers did is NOT scraping.CC: <a href="https://mastodon.social/@404mediaco" class="u-url mention" rel="nofollow noopener" target="_blank">@404mediaco</a>RE: <a href="https://www.404media.co/researchers-scrape-2-billion-discord-messages-and-publish-them-online/" rel="nofollow noopener" translate="no" target="_blank">https://www.404media.co/researchers-scrape-2-billion-discord-messages-and-publish-them-online/</a><a href="https://mastodon.social/tags/Scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraper</a> <a href="https://mastodon.social/tags/Scraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#Scraping</a> <a href="https://mastodon.social/tags/WebScraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebScraper</a> <a href="https://mastodon.social/tags/WebScraping" class="mention hashtag" rel="nofollow noopener" target="_blank">#WebScraping</a>

Patrick Wu :neocat_flag_bi:And it's much betterfuck you OpenAI and Amazon<a href="https://o0o.social/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://o0o.social/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a> <a href="https://o0o.social/tags/cloud" class="mention hashtag" rel="nofollow noopener" target="_blank">#cloud</a> <a href="https://o0o.social/tags/openai" class="mention hashtag" rel="nofollow noopener" target="_blank">#openai</a> <a href="https://o0o.social/tags/amazon" class="mention hashtag" rel="nofollow noopener" target="_blank">#amazon</a>

Paolo AmorosoUpdate: I reported the bot. Thanks.A Mastodon bot account at mastodon.cloud scans the fediverse, scrapes selected web pages shared there, rewrites them with AI, posts them to its own site, and shares on Mastodon as tech news the rewritten AI slop. The bot scraped a post of mine (including the attached image) within minutes of my federated blog publishing it.Is it worth flagging the bot and reporting it to its instance? Are the mods likely to take action?<a href="https://fosstodon.org/tags/mastodon" class="mention hashtag" rel="nofollow noopener" target="_blank">#mastodon</a> <a href="https://fosstodon.org/tags/moderation" class="mention hashtag" rel="nofollow noopener" target="_blank">#moderation</a> <a href="https://fosstodon.org/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://fosstodon.org/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>

Polychrome :clockworkheart:<a class="u-url mention" href="https://metalhead.club/@thomas" rel="nofollow noopener" target="_blank">@thomas</a> I didn't discover this. It's been making the rounds on tech news sites for awhile now. <a class="hashtag" href="https://poly.cybre.city/tag/ai" rel="nofollow noopener" target="_blank">#ai</a> <a class="hashtag" href="https://poly.cybre.city/tag/bots" rel="nofollow noopener" target="_blank">#bots</a> <a class="hashtag" href="https://poly.cybre.city/tag/scraper" rel="nofollow noopener" target="_blank">#scraper</a>

Tom :damnified:Anubis: self hostable scraper defense software | Anubis<a href="https://anubis.techaro.lol/" rel="nofollow noopener" translate="no" target="_blank">https://anubis.techaro.lol/</a><a href="https://metalhead.club/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://metalhead.club/tags/bots" class="mention hashtag" rel="nofollow noopener" target="_blank">#bots</a> <a href="https://metalhead.club/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>

Paolo AmorosoWikimedia is buckling under the weight of AI scrapers."Our content is free, our infrastructure is not".<a href="https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects" rel="nofollow noopener" translate="no" target="_blank">https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects</a><a href="https://fosstodon.org/tags/wikipedia" class="mention hashtag" rel="nofollow noopener" target="_blank">#wikipedia</a> <a href="https://fosstodon.org/tags/ai" class="mention hashtag" rel="nofollow noopener" target="_blank">#ai</a> <a href="https://fosstodon.org/tags/scraper" class="mention hashtag" rel="nofollow noopener" target="_blank">#scraper</a>

Frühere Suchanfragen

Suchoptionen

Verwaltet von:

Serverstatistik:

#scraper