{"id":26794,"date":"2022-04-26T11:37:00","date_gmt":"2022-04-26T09:37:00","guid":{"rendered":"https:\/\/perseus.de\/web-crawler\/"},"modified":"2026-03-30T11:17:10","modified_gmt":"2026-03-30T09:17:10","slug":"web-crawler","status":"publish","type":"post","link":"https:\/\/perseus.de\/en\/web-crawler\/","title":{"rendered":"Web crawler"},"content":{"rendered":"\t\t
\n\t\t\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Web crawler<\/h1>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
\n\t\t\t\t\t
\n\t\t
\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Webcrawler translates as “crawler through the Internet”. Web crawlers are computer programs that independently search the Internet and analyze web pages. <\/p>

<\/h4>

What does web crawler mean in detail?<\/h4>

Web crawlers are bots<\/a> \u2013 computer programs that automatically and independently perform certain tasks. The figurative term “web crawler” is derived from the way these bots work. They “crawl” from link to link through the Internet. This is how they get from website to website. Search engines use web crawlers to analyze websites and add them to their directory. Each search engine uses its own web crawlers. Your business website is visited by Googlebot, Bingbot, and other search engine crawlers. <\/p>

Web crawlers can be used to automatically collect and analyze data from websites. Therefore, they are also used, for example, for price comparison websites to find the lowest prices for certain products.
But web crawlers are also used for shady or illegal purposes. For example, to automatically collect email addresses, to which spam<\/a> is then sent. <\/p>

<\/h4>

Where do I encounter the topic of “web crawlers” in my everyday work?<\/h4>

You encounter it indirectly every time you use a search engine. The search results are the results of the work of the web crawlers. Unusual e-mail address information on websites such as “info(at)unternehmen(dot)com” is also justified by web crawlers. This format is designed to make the email address unreadable to shady web crawlers. However, e-mail addresses provided in this way are no longer barrier-free and can no longer be used, for example, by people with severely impaired vision. Furthermore, many shady web crawlers are now programmed to recognize such alternative spellings. <\/p>

<\/h4>

What can I do to improve my security?<\/h4>

Protect email addresses accessible on your company website from shady web crawlers. There are several ways to do this. Two examples: <\/p>

  • Replace email addresses with contact forms. Contact forms cannot be used by bots, but can be made accessible to humans. <\/li>
  • Replace email addresses with a forwarding via HTTP redirect. In this case, the e-mail address becomes accessible via a detour that is not comprehensible to bots. Contact your IT department to identify and implement your website best practices. <\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\n\t\t\t\t\t\t<\/span>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

    Web crawler Webcrawler translates as “crawler through the Internet”. Web crawlers are computer programs that independently search the Internet and analyze web pages. What does web crawler mean in detail? Web crawlers are bots \u2013 computer programs that automatically and independently perform certain tasks. The figurative term “web crawler” is derived from the way these […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[57],"tags":[],"class_list":["post-26794","post","type-post","status-publish","format-standard","hentry","category-cyber-glossary"],"acf":[],"yoast_head":"\nWeb crawler - Perseus Technologies<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/perseus.de\/en\/web-crawler\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Web crawler - Perseus Technologies\" \/>\n<meta property=\"og:description\" content=\"Web crawler Webcrawler translates as “crawler through the Internet”. Web crawlers are computer programs that independently search the Internet and analyze web pages. What does web crawler mean in detail? Web crawlers are bots \u2013 computer programs that automatically and independently perform certain tasks. The figurative term “web crawler” is derived from the way these […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/perseus.de\/en\/web-crawler\/\" \/>\n<meta property=\"og:site_name\" content=\"Perseus Technologies\" \/>\n<meta property=\"article:published_time\" content=\"2022-04-26T09:37:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-03-30T09:17:10+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/perseus.de\/wp-content\/uploads\/2025\/05\/Headerbild-Glossary-Page.png\" \/>\n\t<meta property=\"og:image:width\" content=\"800\" \/>\n\t<meta property=\"og:image:height\" content=\"800\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Anastasia Pamoukis\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Anastasia Pamoukis\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/\"},\"author\":{\"name\":\"Anastasia Pamoukis\",\"@id\":\"https:\\\/\\\/perseus.de\\\/#\\\/schema\\\/person\\\/6c87a2feea6439d0ead16c8f0f07e40b\"},\"headline\":\"Web crawler\",\"datePublished\":\"2022-04-26T09:37:00+00:00\",\"dateModified\":\"2026-03-30T09:17:10+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/\"},\"wordCount\":362,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/perseus.de\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/Headerbild-Glossary-Page-300x300.png\",\"articleSection\":[\"Cyber Glossary\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/\",\"url\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/\",\"name\":\"Web crawler - Perseus Technologies\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/perseus.de\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/Headerbild-Glossary-Page-300x300.png\",\"datePublished\":\"2022-04-26T09:37:00+00:00\",\"dateModified\":\"2026-03-30T09:17:10+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#primaryimage\",\"url\":\"https:\\\/\\\/perseus.de\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/Headerbild-Glossary-Page-300x300.png\",\"contentUrl\":\"https:\\\/\\\/perseus.de\\\/wp-content\\\/uploads\\\/2025\\\/05\\\/Headerbild-Glossary-Page-300x300.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/perseus.de\\\/en\\\/web-crawler\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/perseus.de\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Web crawler\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/perseus.de\\\/#website\",\"url\":\"https:\\\/\\\/perseus.de\\\/\",\"name\":\"perseus-web.de\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/perseus.de\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/perseus.de\\\/#organization\",\"name\":\"perseus-web.de\",\"url\":\"https:\\\/\\\/perseus.de\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/perseus.de\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/perseus.de\\\/wp-content\\\/uploads\\\/2025\\\/03\\\/PerseusTechnologie_Color-2.png\",\"contentUrl\":\"https:\\\/\\\/perseus.de\\\/wp-content\\\/uploads\\\/2025\\\/03\\\/PerseusTechnologie_Color-2.png\",\"width\":536,\"height\":172,\"caption\":\"perseus-web.de\"},\"image\":{\"@id\":\"https:\\\/\\\/perseus.de\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/perseus.de\\\/#\\\/schema\\\/person\\\/6c87a2feea6439d0ead16c8f0f07e40b\",\"name\":\"Anastasia Pamoukis\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/88febef1895859fc3a543c774ae6bc44cde0f02199f7f792af67eaf196406342?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/88febef1895859fc3a543c774ae6bc44cde0f02199f7f792af67eaf196406342?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/88febef1895859fc3a543c774ae6bc44cde0f02199f7f792af67eaf196406342?s=96&d=mm&r=g\",\"caption\":\"Anastasia Pamoukis\"},\"sameAs\":[\"https:\\\/\\\/perseus.de\"],\"url\":\"https:\\\/\\\/perseus.de\\\/en\\\/author\\\/anastasia-pamoukis\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Web crawler - Perseus Technologies","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/perseus.de\/en\/web-crawler\/","og_locale":"en_US","og_type":"article","og_title":"Web crawler - Perseus Technologies","og_description":"Web crawler Webcrawler translates as “crawler through the Internet”. Web crawlers are computer programs that independently search the Internet and analyze web pages. What does web crawler mean in detail? Web crawlers are bots \u2013 computer programs that automatically and independently perform certain tasks. The figurative term “web crawler” is derived from the way these […]","og_url":"https:\/\/perseus.de\/en\/web-crawler\/","og_site_name":"Perseus Technologies","article_published_time":"2022-04-26T09:37:00+00:00","article_modified_time":"2026-03-30T09:17:10+00:00","og_image":[{"width":800,"height":800,"url":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/05\/Headerbild-Glossary-Page.png","type":"image\/png"}],"author":"Anastasia Pamoukis","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Anastasia Pamoukis","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/perseus.de\/en\/web-crawler\/#article","isPartOf":{"@id":"https:\/\/perseus.de\/en\/web-crawler\/"},"author":{"name":"Anastasia Pamoukis","@id":"https:\/\/perseus.de\/#\/schema\/person\/6c87a2feea6439d0ead16c8f0f07e40b"},"headline":"Web crawler","datePublished":"2022-04-26T09:37:00+00:00","dateModified":"2026-03-30T09:17:10+00:00","mainEntityOfPage":{"@id":"https:\/\/perseus.de\/en\/web-crawler\/"},"wordCount":362,"commentCount":0,"publisher":{"@id":"https:\/\/perseus.de\/#organization"},"image":{"@id":"https:\/\/perseus.de\/en\/web-crawler\/#primaryimage"},"thumbnailUrl":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/05\/Headerbild-Glossary-Page-300x300.png","articleSection":["Cyber Glossary"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/perseus.de\/en\/web-crawler\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/perseus.de\/en\/web-crawler\/","url":"https:\/\/perseus.de\/en\/web-crawler\/","name":"Web crawler - Perseus Technologies","isPartOf":{"@id":"https:\/\/perseus.de\/#website"},"primaryImageOfPage":{"@id":"https:\/\/perseus.de\/en\/web-crawler\/#primaryimage"},"image":{"@id":"https:\/\/perseus.de\/en\/web-crawler\/#primaryimage"},"thumbnailUrl":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/05\/Headerbild-Glossary-Page-300x300.png","datePublished":"2022-04-26T09:37:00+00:00","dateModified":"2026-03-30T09:17:10+00:00","breadcrumb":{"@id":"https:\/\/perseus.de\/en\/web-crawler\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/perseus.de\/en\/web-crawler\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/perseus.de\/en\/web-crawler\/#primaryimage","url":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/05\/Headerbild-Glossary-Page-300x300.png","contentUrl":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/05\/Headerbild-Glossary-Page-300x300.png"},{"@type":"BreadcrumbList","@id":"https:\/\/perseus.de\/en\/web-crawler\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/perseus.de\/"},{"@type":"ListItem","position":2,"name":"Web crawler"}]},{"@type":"WebSite","@id":"https:\/\/perseus.de\/#website","url":"https:\/\/perseus.de\/","name":"perseus-web.de","description":"","publisher":{"@id":"https:\/\/perseus.de\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/perseus.de\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/perseus.de\/#organization","name":"perseus-web.de","url":"https:\/\/perseus.de\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/perseus.de\/#\/schema\/logo\/image\/","url":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/03\/PerseusTechnologie_Color-2.png","contentUrl":"https:\/\/perseus.de\/wp-content\/uploads\/2025\/03\/PerseusTechnologie_Color-2.png","width":536,"height":172,"caption":"perseus-web.de"},"image":{"@id":"https:\/\/perseus.de\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/perseus.de\/#\/schema\/person\/6c87a2feea6439d0ead16c8f0f07e40b","name":"Anastasia Pamoukis","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/88febef1895859fc3a543c774ae6bc44cde0f02199f7f792af67eaf196406342?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/88febef1895859fc3a543c774ae6bc44cde0f02199f7f792af67eaf196406342?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/88febef1895859fc3a543c774ae6bc44cde0f02199f7f792af67eaf196406342?s=96&d=mm&r=g","caption":"Anastasia Pamoukis"},"sameAs":["https:\/\/perseus.de"],"url":"https:\/\/perseus.de\/en\/author\/anastasia-pamoukis\/"}]}},"_links":{"self":[{"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/posts\/26794","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/comments?post=26794"}],"version-history":[{"count":1,"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/posts\/26794\/revisions"}],"predecessor-version":[{"id":26795,"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/posts\/26794\/revisions\/26795"}],"wp:attachment":[{"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/media?parent=26794"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/categories?post=26794"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/perseus.de\/en\/wp-json\/wp\/v2\/tags?post=26794"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}