Skip to main content

​
Web pages

Realm can sync any publicly accessible website and add its content to your knowledge base.

​
Setting up web scraping

1

Navigate to Data sources

Go to the Data sources page in Realm.
2

Find Web Scraping

Open “Web Scraping”, then click Add Site.
3

Enter URL

Insert the URL you wish to sync (e.g., https://example.com/docs/).Realm only syncs this page and pages whose URL path is under it, so use the root of the section you want. For example, https://example.com/docs/ syncs the pages under /docs/, while https://example.com/docs/intro.html syncs only that single page.
4

Configure Patterns (Optional)

If you want to sync only some of the pages under the start URL, add include or exclude patterns.
5

Save

Click Save to start the sync.

​
Sync scope

The start URL defines the scope of the sync. Realm follows links from the start URL, but only syncs pages on the same domain whose URL path begins with the path of the start URL. Pages outside that scope are skipped, even if the start page links to them. The one exception is direct PDF links: a link ending in .pdf found on a synced page is synced even when the file lives outside the path of the start URL. PDF links that do not end in .pdf (for example, download endpoints with query parameters) follow the normal scope rules. To sync a whole site, use the site root (e.g., https://example.com/) as the start URL. To sync one section, use that section’s root (e.g., https://example.com/docs/).

​
URL patterns

Patterns narrow the sync further within the scope of the start URL. They cannot extend the sync beyond pages under the start URL. Patterns use glob syntax and are relative to the start URL. The start page itself is always synced, even when it does not match the patterns. The examples below assume a start URL of https://example.com/.

​
Include patterns

When include patterns are set, only pages matching at least one of them are synced:

​
Exclude patterns

Pages matching an exclude pattern are skipped: Patterns match the full URL, including any query string. Use /**/*.pdf* rather than /**/*.pdf so links such as /files/manual.pdf?download=1 are also matched.

​
Use cases

Product Documentation

Sync public product docs from your website

Help Centers

Import external help center content

Competitor Research

Track competitor public documentation

Partner Content

Sync partner documentation and resources

​
Best practices

Only scrape websites that allow crawling. Check the site’s robots.txt file.
Be specific with include/exclude patterns to avoid syncing irrelevant content.
Periodically review scraped content to ensure it remains relevant and accurate.
Begin with a specific section of a website before expanding to more pages.

​
Limitations

  • Only works with publicly accessible websites
  • Only pages under the path of the start URL are synced (see Sync scope)
  • Pages are discovered by following links and sitemaps, so pages that nothing links to may not be found, and very large sites may not be fully covered due to crawl depth and size limits
  • Dynamic content (JavaScript-rendered) may not be fully captured
  • Rate limiting may apply to prevent overloading target sites