Invisible Web Sources in An Agent

The Invisible Web

The invisible web refers to those parts of the internet that cannot be seen by most search engines.  Examples of this are

• websites that require a log in and a password to access the content
• search result pages from search engines hosted on company pages or institutional sites

For some sites you can very easily use the Smart Tracker in an agent to monitor what is mentioned behind search forms and also behind authentification pages.  You can also use the keyword function of Smart Tracker to monitor a range of search engines across the internet.  Click on the links of each one to find out more.

It is also possible to set up invisible web sources for your Digimind platform.  You may want to do this if you find that your source is not compatible with the search form options in the advanced configuration of an agent and / or if the site in question contains a lot of pages.

Set up a Search Engine Category

The first step is to set up a search engine category.  Contact our technical team who will explain the steps and inform you of the cost involved.

Once you have set up the search engines you can start to use them as sources.

Add invisible Web Sources to an Agent

Inside the advanced configuration of an agent, in the Invisible Web 2 section, click on add an invisible web source:

On the next page, you will be able to name your source (this is how it will appear in your agent) and specify the keywords you would like to look for in that search engine.

Then check the box to include your search engine category and then click on OK.

You will now see your source inside your agent:

Good to know

1. This kind of source does not use the query that has been added to the agent.  The flow of information is based on the keywords you have specified while setting up the source.
2. Digimind tracks the invisible web once a week.