Newly Added Website Not Crawled, Identified, or Analyzed

Newly Added Website Not Crawled, Identified, or Analyzed

If you've recently added a website to our system and are encountering issues with crawling, identification, or analysis, this guide will help you understand the potential reasons and the supported features of our system.

System Support

Nytro Systems is designed to efficiently work with a wide range of websites, ensuring a seamless and comprehensive analysis.

Supported Website (CMS) Types

  • Standard HTML-based Websites: These are the most common types of websites, where the content is served directly from the server in HTML format.
  • Server Side Rendering (SSR): Websites that dynamically generate content on the server side before sending it to the client in HTML format.
  • Standard CMS: e.g Wordpress

Unsupported Features

There are certain types of websites and technologies that Nytro Systems does not currently support:

  • CSR (Client-Side Rendering) Websites: These websites render content in the browser using JavaScript. Unfortunately, our system does not offer support for crawling and analyzing these types of websites at present

Common Issues Leading to Non-Crawling

In some cases, a website may not be identified or crawled due to specific issues or configurations. Below are common scenarios where this might occur:

  1. Robots.txt Blocking: If your website's robots.txt file disallows crawling, our system will not be able to crawl it.

  2. Crawling Blocked by Firewalls or Services: Websites protected by firewalls or services like Cloudflare may block our crawling attempts.

  3. Empty or Linkless Home Page: If the home page of your website has no content or links, our system might not be able to initiate the crawling process.

  4. Home Page Errors:

    • Not Found (404 Error)
    • Server Down or Not Responding
    • Access Denied (403 Error)
    • Other HTTP error codes
  5. Expired Domain: If the domain of the website has expired, crawling is not possible.

  6. Single, Non-Canonical Page: Websites with only one page that is not marked as canonical can lead to crawling issues.

  7. Unexpected Failures in Our System:

    • Server Downtime
    • Logical Errors in the Crawling Process
  8. Website Offline: If your website is not connected to the internet or the server is down, crawling cannot be initiated.

Resolution Steps

If you encounter any of these issues, we recommend checking your website’s configuration and settings to ensure compatibility with our crawling and analysis system.

If no such issue is identified, please contact our support team for assistance.

We are continuously working to improve our system and expand the range of supported technologies.


    • Related Articles

    • Adding the NytroSEO Optimization snippet to your website

      The NytroSEO JS (Script) Optimization Snippet allows for the automatic SEO optimization of your webpage code. You will need to copy and paste the JS into the <HEAD> of your website. JS Snippet Setup Guide Add your website Firstly, you will need to ...
    • Adding Keyword search terms to a Website

      Adding Keywords to a Website This guide provides a detailed walkthrough of adding keyword search terms to the Nytro system for website optimization, rank monitoring, and pre-evaluation of meta-tags effectiveness. You can add Keyword search terms to ...
    • What are the steps and process for optimizing a website with NytroSEO?

      How does the process work? Utilizing advanced AI based technology, NytroSEO automatically and continually optimizes websites code by adding the relevant meta-tags & meta data to each page, based on its content and the optimal Keyword search terms. ...
    • SETTINGS: Website Automation setup

      Website Settings: Configuration Guide for Nytro SEO Automated Software This guide outlines the process of configuring your website’s main settings using the Nytro SEO Automated Software. Selecting Your Optimization Mode Objective: Choose the most ...
    • Adding a Website to your account

      Add Website - NytroSEO Video Tutorial You may add multiple websites to your NytroSEO Account. In this video tutorial, we will show you how to add a new website to your account. Just follow the displayed instructions. Step By Step How to Add your ...