User-agent: wget Disallow: /p/ Disallow: /wiki/ Disallow: /cgi-bin/ Disallow: /mailman/ Disallow: /pipermail/ User-agent: WDG_SiteValidator User-agent: * Disallow: /pollux Disallow: /p/ Disallow: /wiki/ Disallow: /qc/ Disallow: /cgi-bin/ Disallow: /~ Disallow: /% Disallow: /mailman/ Disallow: /pipermail/ Disallow: /gallery/heidi_ja_miika Disallow: /gallery/heidi_ja_miika/ Disallow: /gallery/mummi80v Disallow: /gallery/mummi80v/ Disallow: /gallery/paula_rippiaiset Disallow: /gallery/paula_rippiaiset/ # You hog 3x the bandwidth of all other bots together, go away! User-agent: msnbot Disallow: / # You crawl sites way too fast, go away! User-agent: Slurp Disallow: /