Crawler SDR frequently asked questions

Crawler SDR is a desktop SEO audit tool. It crawls websites and detects internal and external URLs, redirects, HTTP status codes, indexability, titles, meta descriptions, H1/H2, canonicals, hreflang, images, content, JavaScript, structured data, and actionable report data.

Crawler SDR covers a complete technical SEO audit workflow: crawling, JavaScript rendering, exports, warning/error detection, and comparisons with external sources. The product focuses on useful, clear, exportable results.

You can analyze corporate sites, blogs, ecommerce sites, JavaScript-heavy pages, multilingual websites, projects with sitemaps, robots.txt rules, and sites that need indexability, architecture, and performance checks.

Yes. It is designed to generate technical audits, prioritize issues, and deliver HTML/XLSX reports with KPIs and filterable tables. During beta, validate the results before using them as final client deliverables.

No. The main crawler runs as a Windows desktop application. The website is used for downloads, beta license management, support, and documentation.

During beta, you can generate a temporary free PRO license from your account area. It lets you test advanced features while the first public version is being validated.

Log in, open your dashboard, and use the free PRO beta license button. The license appears in your Licenses section with its status and expiry date.

The beta license is a temporary license with a limited expiry, usually one month. Duration may change while the product is in beta.

Yes. The tool uses a hashed device identifier to bind the license to the machine without storing a readable hardware identifier.

Even when the installer is signed, Windows and Chrome can show warnings if an application is new and does not yet have enough reputation. This usually improves with real downloads, consistent signing, clean detections, and publisher reputation.

The goal is to distribute digitally signed installers. Signing helps identify the publisher, but it does not automatically remove all early SmartScreen or reputation warnings.

The tool checks whether new versions are available. When an update is detected, it should inform the user and ask for confirmation before updating.

Crawler SDR includes a disk-space control: if less than 10 GB is available on the disk used for temporary crawl files, the crawl should stop and should not resume until enough space is freed.

Yes. The tool can respect robots.txt and mark blocked URLs or resources so you can decide whether the block is expected or affecting important pages.

Yes. The PRO version includes adaptive JavaScript rendering, dynamic URL extraction, and dedicated workers for pages where content or links depend on browser rendering.

Yes. Advanced settings let you adjust scope, depth, delay, resource type, JavaScript URL extraction, and parameter filters to control how the crawl runs.

The crawler can detect HTML, CSS, JS, images, and other resources. Some comparisons, such as GSC, GA4, PageSpeed, and CrUX, apply only to HTML URLs because comparing images or CSS against organic performance data is not useful.

Address is the originally discovered URL. Final URL is the URL reached after redirects. For titles, meta, H1, H2, and content data, Final URL should be prioritized because the original URL may not have its own content if it redirects.

Yes. It checks signals such as noindex, canonical to another URL, robots rules, redirects, blocking, and non-200 statuses to classify indexability and indexability status.

Yes. It can identify missing, multiple, canonicalized, canonical-to-redirect, or canonical-to-non-indexable cases when the crawl has enough data.

Yes. It includes a hreflang tab to review annotations, references, languages/regions, non-200 or non-indexable URLs, and common multilingual issues.

Yes. The crawler collects structured data and can show warnings or errors when it detects incomplete or invalid markup based on available crawl data.

Yes. It reports internal links, external links, inlinks, outlinks, crawl depth, discovery sources, HTTP status codes, and link-related errors.

The HTML report includes an executive summary, KPIs, warning/error types, crawl tabs, filterable tables, and additional sections for GSC, GA4, PageSpeed, CrUX, and server logs when those analyses have been run.

Yes. Main KPIs can filter the lower table to show affected URLs, such as pages with duplicate H1s, non-indexable URLs, or missing meta descriptions.

Yes. Warning audit and error audit sections can export XLSX files per issue so each problem can be reviewed in a cleaner, prioritized table.

Each warning should show only relevant fields. For example, a duplicate title issue shows Final URL, title, occurrence count, and duplicate URLs; generic crawl columns are not needed if they do not add context.

Yes. The tool can save and load crawl projects so you can continue an analysis, review results, or generate reports later.

It lets you upload an Excel file with a URL column and compare those URLs against the crawl. It helps find URLs that exist only in the file, only in the crawl, or in both.

Yes, but only if those comparisons or enrichments have been executed before downloading it. If there is no GSC, GA4, PageSpeed, CrUX, or log data, the report should not invent or mention them as real results.

It compares crawled HTML URLs with pages returned by Search Console. This helps identify crawl-only URLs, GSC-only URLs, coverage, clicks, impressions, CTR, and average position.

It compares crawled HTML URLs with pages that have sessions, users, views, or engagement in GA4. This helps detect traffic pages that are not internally linked or crawled pages with no usage signal.

Its main purpose is to create Crawl vs GSC and Crawl vs GA4 comparisons. It does not replace the technical crawl; it enriches it with organic and behavioral data when the user authorizes access.

The app requests read-only permissions for Search Console and Google Analytics when you connect Google. The purpose is to list properties and query the data needed for comparison reports.

Not exactly. Search Console and GA4 use OAuth. PageSpeed Insights and CrUX use a Google Cloud API key configured by the user, like many desktop SEO tools do, so the tool provider does not consume the user quota.

These APIs consume quota from the Google Cloud project linked to the API key. That is why the tool limits volume and lets you select the HTML URLs you want to analyze.

Only selected HTML URLs. You can search, check URLs, and run the query for important pages to avoid spending quota on irrelevant URLs.

It lets you upload server logs and cross real bot hits with crawl URLs. This helps detect which URLs receive bot activity, which do not, and where crawl budget noise or indexing opportunities may exist.

Yes. External connections, advanced comparisons, and enrichment with third-party sources are part of the PRO workflow and may have URL limits to control performance and API quotas.

Connections are stored locally in the user installation. API keys and OAuth tokens should not be uploaded to crawl files or shareable reports.

The tool uses authorized data to create comparisons inside your installation. The privacy policy explains how data is handled on the website and for user-authorized connections.

Yes. In Connections you can disconnect Google to remove the active connection and stop using those properties in new comparisons.

You should use it responsibly and respect laws, site terms, robots.txt when applicable, and reasonable limits. Responsibility for crawler usage belongs to the person who runs it.

Yes. The website has a consent banner to manage required, analytics, and advertising cookies. You can review or change preferences from the cookie policy.

They may be blocked by robots.txt, headers, security rules, authentication, or site-level protections. The crawler should show the status so you can check whether the block is expected.

It may mean it was not fetched, was blocked, timed out, or was recorded only as an external link without being fully crawled.

Because GSC and GA4 are mainly page-based. Including images, CSS, or JS in those comparisons would create noise and false gaps.

Review JavaScript rendering, workers, delay, depth, robots rules, heavy resources, limits, and disk space. On complex sites, JavaScript rendering is often the slowest part.

Readability only makes sense when there is enough text. On thin pages, redirects, resources, or non-textual content, the result may be empty or not relevant.

Use the support area in your account or the contact page. Include the URL, tool version, crawl type, and a screenshot or export if the issue is reproducible.