How to use Selenium to crawl websites

OpenSearchServer 1.5 introduces a powerful scripting feature. It is now possible to drive a web browser (Firefox, PhantomJS) using a written script.

The new scripting feature can:

  • Open a web browser,
  • Execute Javascript,
  • Extract data using an XPATH query, a CSS selector, or the ID of the web element,
  • Insert documents in an index.

A set of REST APIs was created to manage the scripts. Each script can be executed/stored/updated/deleted by calling its name.

This new feature is particularly useful when crawling websites protected by a login/password form, websites making heavy use of Javascript to display content -- or when the values to be indexed can only be defined by precise CSS or XPATH selectors.

JSON script

Each command is a JSON structure with a command value and an array of parameters.

{
  "command": "WEBDRIVER_OPEN",
  "parameters": [ "FIREFOX"]
}

A script is an array of JSON command structures. Here is an example:

[
 { "command": "WEBDRIVER_OPEN", "parameters": [ "PHANTOMJS"] },
 { "command": "WEBDRIVER_SET_TIMEOUTS", "parameters": [ 60, 60 ] },
 { "command": "WEBDRIVER_RESIZE", "parameters": [ 1024, 768 ] },
 { "command": "WEBDRIVER_GET", "parameters": [ "{url}" ] },
 { "command": "SLEEP", "parameters": [ 1 ] },
 { "command": "INDEX_DOCUMENT_NEW", "parameters": [ "english" ] },
 { "command": "INDEX_DOCUMENT_ADD_VALUE", "parameters": [ "url", "{url}" ] },
 { "command": "INDEX_DOCUMENT_ADD_VALUE", "parameters": [ "title", "Test" ] },
 { "command": "CSS_SELECTOR_INDEX_FIELD", "parameters": [ "content", "div ul li a" ] },
 { "command": "INDEX_DOCUMENT_UPDATE" }
]

This script executes the following actions:

  • open a PHANTOMJS window,
  • set a timeout duration of one minute,
  • the width and the height of the window are set to 1024×768,
  • a WEB page is loaded, based on the {url} parameter,
  • the script waits for one second,
  • an index document is created,
  • the fields url and title are set,
  • a CSS selector locates a text element within the web page. These elements are stored in the content field.
  • the document is inserted within the index.

RESTful API

A set of RESTful API is available.

Saving the script

To store the script, use the following call:

  • URL: http://localhost:9090/services/rest/index/script/script/{script_name}
  • HTTP Method: PUT
  • HTTP Header: Content-Type: application/json
  • Payload: The JSON script

Running the script

To run the script use the following call:

  • URL: http://localhost:9090/services/rest/index/script/script/{script_name}/run
  • HTTP Method: POST
  • HTTP Header: Content-Type: application/json
  • Payload: The JSON structure with the variable
  • The JSON structure for the variables:

{"url": "http://www.open-search-server.com", "name": "John Doe" }

Subscript

A script can call a subscript for each web element found by a selector.

[
 { "command": "WEBDRIVER_OPEN", "parameters": [ "PHANTOMJS" ] },
 { "command": "WEBDRIVER_SET_TIMEOUTS", "parameters": [ 60, 60 ] },
 { "command": "WEBDRIVER_RESIZE", "parameters": [ 1024, 768 ] },
 { "command": "WEBDRIVER_GET", "parameters": [ "http://www.dmoz.org/" ] },
 { "command": "SLEEP", "parameters": [ 3 ] },
 { "command": "CSS_SELECTOR_SUBSCRIPT", "parameters": [ "dmoz_sub", "div#catalogs span a" ] },
 { "command": "WEBDRIVER_CLOSE" }
]

This script will extract all the root categories of the homepage of Dmoz.org. Then, for each category found, the subscript dmoz_sub will be called.

Full list of commands

  • WEBDRIVER_OPEN: open a web browser
    • parameter 1: name of driver. Possible values: PHANTOMJS, FIREFOX.
  • WEBDRIVER_CLOSE: close the browser
  • WEBDRIVER_NEW_WINDOW: open a new window and keep the session running
  • WEBDRIVER_CLOSE_WINDOW: close the current window
  • WEBDRIVER_SET_TIMEOUTS: define a timeout delay after which script execution will be stopped.
    • parameter 1: delay, in seconds
  • WEBDRIVER_RESIZE: resize window
    • parameter 1: width
    • parameter 2: height
  • WEBDRIVER_GET: access an URL
    • parameter 1: URL. Variables can be used within this parameter.
  • SLEEP : pause script execution
    • parameter 1: time, in seconds
  • CSS_SELECTOR_SUBSCRIPT, XPATH_SELECTOR_SUBSCRIPT: select an element and run a subscript. The selected element must be an <a or an <img. The href or src attribute will be passed as the {url} variable of the subscript.
    • parameter 1: name of the script
    • parameter 2: selector, CSS or XPATH depending on the command.
  • WEBDRIVER_JAVASCRIPT: execute some Javascript.
    • parameter 1: Javascript commands to execute. Variables can be used within this parameter.
  • SCRIPT: call another script
    • parameter 1: name of the script
  • CSS_SELECTOR_CLICK_AND_SCRIPT, XPATH_SELECTOR_CLICK_AND_SCRIPT: select an element, click on it, wait some time and run a script
    • parameter 1: selector, CSS or XPATH depending on the command.
    • parameter 2: name of the script
    • parameter 3: time to wait between the click and the script execution
  • INDEX_DOCUMENT_NEW: create a new document to be indexed
    • parameter 1: name of the new document
  • INDEX_DOCUMENT_ADD_VALUE: add a value in the current new document
    • parameter 1: field
    • parameter 2: value. Variables can be used in this parameter.
  • INDEX_DOCUMENT_UPDATE: commit the current new document to the index
  • VAR_NEW_REGEX : create a new variable by using a regexp on a variable. The new variable can then be used in the script and subscripts.
    • parameter 1: variable on which to apply the regexp
    • parameter 2: regexp. The capture group will give the new variable its value.
    • parameter 3: new variable name
  • CSS_SELECTOR_DOWNLOAD, XPATH_SELECTOR_DOWNLOAD : select an element and download the URL within the href attribute. Selected element must be an <a.
    • parameter 1: directory into which the file will be downloaded. It will be fully (recursively) created if it does not exist. Variables can be used within this parameter.
    • parameter 2: selector, CSS or XPATH depending on the command.
  • WEBDRIVER_DOWNLOAD: download the current file indicated by the {url} variable.
    • parameter 1: directory into which the file will be downloaded. It will be fully (recursively) created if it does not exist. Variables can be used within this parameter.
  • PARSER_MERGE: merge every PDF document from a directory to a new PDF document.
    • parameter 1: name of the PDF parser. Value can only be "PDF parser" at this stage.
    • parameter 2: directory that contains the PDF files to merge.
    • parameter 3: full path (including filename) to the PDF to be created
  • SEARCH_TEMPLATE_JSON: execute a query on the index and run an action depending on the result.
    • parameter 1: name of the query template to use for the search
    • parameter 2: keywords to use for the search query
    • parameter 3: JSON path to match a specific part of the result
    • parameter 4: action to take. Values can be:
      • EXIT_IF_NOT_FOUND: exit current script if the JSON path does not match
      • IF_FOUND: if the JSON path matches:
        • parameter 5: must be NEXT_COMMAND
        • parameter 6: next command to run. Will often be WEBDRIVER_CLOSE_WINDOW

View/edit on GitHub


comments powered by Disqus