still have the ability to disable the crawler via robots.txt flags
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
The name Suckerberg I mean Zuckerberg is so strange. That's just raw german:
Zucker = Sugar
Berg = Mountain (or a big hill, but typically it refers to a mountain)
So, a "mountain of sugar" (lots of sugar).
Suckerberg kind of fits better, since "sucker" is not german (berg still is of course; I think scandinavian countries also have berg, though that could be burg, aka castle). Plus, Suckerberg describes the evil behaviour of Facebook better. The USA really has a problem with its billionaires.
Blocking by UA is pointless now, meta-externalagent rotates and a chunk comes off non-Meta ASNs. We matched on ASN plus TLS JA4 and that actually stuck.
Never rely on robots.txt. Configure it, but do not rely on it. A bot will do what a bot can do. Their bots are trivial to spot so close the connection, bonus if done silently closing the socket and inhibiting first RST from web daemon and second RST from the OS.
Zucker = Sugar Berg = Mountain (or a big hill, but typically it refers to a mountain)
So, a "mountain of sugar" (lots of sugar).
Suckerberg kind of fits better, since "sucker" is not german (berg still is of course; I think scandinavian countries also have berg, though that could be burg, aka castle). Plus, Suckerberg describes the evil behaviour of Facebook better. The USA really has a problem with its billionaires.
News at never.