The basic software is open source, and the list is free if you're running the tool and contributing detections back. They do have some curated lists that you have to pay for.
It's quite a bit less expensive than most other commercial products of this kind that I've looked at.
We implemented CrowdSec for bot/scraping mitigation. The architecture is sound, but it ended up having an unacceptable false positive rate for us. This may be an issue with any kind of IP reputation approach. After a couple of months of work getting it ready to go I had to turn it off after a couple of days.
We had the main community blocklist and several of their pricey paid blocklists enabled in a PoC capacity. We had a lot of legitimate users end up blocked. In some cases these may have been VPN exit nodes, or users on CG-NAT, or devices on a shared network with some other compromised / bot device. I didn't get 100% of the details, just that we were inundated with support requests from real users that ended up blocked.
Given the number of residential proxies I see scraping the couple of sites I have responsibility for I don't find IP address-based blocking useful anymore. That ship has sailed.
Behavioral and client fingerprint analysis (ugh-- having to run Javascript just to view a static site) is the only way (at least until we get user "age" and identity attestation rammed down our throats).
I run an attractive site to AI scrapers and yes the IP ship has indeed sailed in favor of residential proxies. They are non-stop and could have ingested all of our content many times over every day. JA4 is useless too because they have the ability to spoof this or use a very common fingerprint like Chrome that you cannot block without blowback. These bots will only use a given IP a couple of dozen times and then use another node. They use 10s of thousands of IPs. You can tell because they will come in waves from a residential or mobile network
What kind of fingerprinting are you thinking of? JA4? I haven't found a way to do that inexpensively at our scale, but we may have to go that route - looking at CloudFront bot mitigation.
For behavioral, we have Anubis honeypot functionality turned on, but it doesn't seem to be effective for 99% of scrapers. Anubis is also running behind TLS termination, so I don't think it can do full JA4. It does have the less robust JA4H apparently, but I'm not sure how effective that will be.
Edit: Oh yeah, forgot to mention - it's almost 100% residential proxies. Primarily China Telecom and China Unicom. Unfortunately those providers are HUGE and also host a ton of legitimate users all over Asia.
> What kind of fingerprinting are you thinking of?
I'm minding a proprietary application where this is being discussed. The vendor is talking about running Javascript in the client to detect headless browsers, bots, etc. I'm guessing they're going to license something off-the-shelf. The technical specifics aren't being discussed. They're at the "wow the stakeholders" phase.
I'd like to do something ourselves but, at the size of the org I'm talking about (with virtually no in-house dev resources), COTS rules the day.
Interesting, did you implement only IP reputation (via blocklist) or did you deploy the WAF as well? Regarding bot scrapping, you would probably want to try the new bot detection feature recently released
I have written my own honeypots to reduce the false positive rate. I simply have things like a VM with RDP and SSH open to the internet and any IP that tries to login gets banned at the firewall for x days. It works really well.
My local coffee shop only takes long government currency.
My bank already allows me to keep my bank account money in stocks and sells units whenever I spend. (We have no capital gains tax here, so this is less insane than it sounds.)
I just finished reading a scifi book called Venemous Lumpsucker which had this type of system as a minor plot point. An interesting twist was that there were essentially smart-contract based non-disclosure agreements that could effectively disable attestation for photos and videos on a specific device that had consented to the NDA.
As someone who operates a large non-profit public data driven website, I have some VERY strong feelings about scrapers. We looked into various commercial solutions (Datadome, HUMAN) and based on our traffic estimates from logs we'd be looking at at least 250k/yr for bot mitigation. Anubis is offering a temporary reprieve, but after reading the recent kernel.org article [0] it's increasingly clear that this is a temporary bandaid.
The cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons.
Looked into it - the bot operators also own, or are associated with, the sites that show the ads. The site showing the ads gets some percentage from Google for each click through conversion, eg app install. So app owner pays $10, google keeps $4, bot / website operator keep $6 or whatever percentages.
This is the bread and butter for commercial VPN services. The VPN knows who you are (maybe, unless you pay crypto/cash), and the rights-holders know your VPN address (from joining/logging a BitTorrent swarm, for instance), but the better providers keep no logs, and I'm guessing there's too many individuals to target effectively.
I've received a nasty letter from the ISP over BitTorrent (someone's phone joined my wifi, torrenting without my knowledge), but using a VPN seems sufficient. I think the Hetzener suggestion is a VPS for running the software stack. You could alternatively self host on hardware in your home.
It's pretty cheap to run a fight re token cost depending on model. I think Opus is $0.15 for a brawl. It comes down to your budget for agent leisure activities. You do get a cool rendering as a gift for your fight.
There's a lot of room for improvement. Thinking about adding clawtokens to join a fight. Winner takes the pot.
Don't know what sport AlpacaJones is talking about but I can think of Gaelic Football and Hurling. Two sports that are very popular in Ireland (whose population is smaller than many major cities) but barely known anywhere else in the world.
The total number of fans is way too small to justify a typical video game budget, but if you made a really good game based on either sport, it would probably sell well in Ireland.
Sure, but this "invention" isn't new. It seems to make the internet rounds every few months. Here's one from 11 years ago that looks almost identical: https://www.youtube.com/watch?v=uPVQMZ4ikvM - I wonder how their "preliminary patent application" went...
It's quite a bit less expensive than most other commercial products of this kind that I've looked at.
reply