Starting expecco via Command Line

Aus ExpeccoWiki

Wechseln zu: Navigation, Suche

expecco can be executed automatically via the crontab (Unix) or scheduled tasks (Windows) mechanism of the operating system. Thus, you can have tests to execute unattended and without GUI "overnight".

Notice, that this is more of a "poor man's test-automation" - not the kind of automation and control that an expeccoNET automation environment would give you.


Inhaltsverzeichnis

Command Line

The simplest command line to execute a test suite is:

expecco --execute <testsuite filename> ...
<testsuite filename> is the name as saved from within the expecco GUI application. This should be a saved ".ets" testSuite file. If more than one test suite filename is given, each is processed in sequence. All top-level test plans as found in the test suite(s) are executed (see below for more options).
By default, the overall execution outcome is returned by the command's return value (exit code), which is
0 - all PASSED
non-zero - any ERROR, FAILURES or INCONCLUSIVE items.

The return value is useful if expecco is started via a makefile, or from jenkins for example.

Command Line Options

Command line options are handled at two places: the runtime system (Smalltalk/X VM) and the expecco application itself. When expecco is started, the VM is first to look for command line arguments and processes any of the VM options, until a non-VM option is encountered. It passes the remaining options to expecco's main entry. Thus, any option which is given after the first non-VM option will not be handled by the VM, but by expecco. Therefore, no VM options should be placed after the first non-VM option. VM options are marked as "VM option" in the list below.

Also, some options are specific to the operating system, on which expecco is executed. These are marked as "Windows", "Linux" etc.

Recommended Command Line Options

A typical command line is:

expecco --noBanner --noTray --verdicts --execute <testsuite filename>

or, if you want to use a special parameter-file:

expecco --noBanner --noTray --verdicts --parameter <param-file> --execute <testsuite filename>

All options are described below.

Additional Options

Information/Debugging/Logging

  • --help
    Prints an up-to-date list of possible options.
  • --version
    Prints the expecco release identification and exits.
  • --verbose (VM option)
    Additional startup, execution and result info is sent in human readable format to the stderr output. That may be too much of printout; try "--verdicts" instead.
  • --noBanner (VM option)
    Suppress the splash startup banner.
  • --console (VM option, Windows only)
    Open a debug console window for Stdout and Stderr (1.5.3).
  • --debug
    Turns on startup debugging. By default, if an error occurs during startup, expecco terminates itself. With this option enabled, a command line debugger is opened instead. This is probably only useful for plugin developers.

Test Selection / Test Execution

  • --testSuiteName <namePattern>
    Only execute testSuites with a matching name (1.5.3).
  • --testSuiteTag <tagPattern>
    Only execute testSuites with a matching tag (1.5.3).
  • --testSuiteID <uuid>
    Only execute that particular testSuite (repeat to execute multiple) (1.5.3).
  • --parameter <fileName>
    Load parameter set from filename. A parameter set contains values for a test suite's top environment. This is in XML format and can be generated with the "save-parameter-set" menu function, or provided by a QM system (such as expeccoNET, HP Quality Center or similar), or be generated by another controlling program.
  • --scriptFile <fileName>
    read and execute a script (see scripting below) from a file.

Reporting

  • --verdicts
    Send execution and result info in human readable format to the stderr output. (see example below.)
  • --logFile <fileName>
    Used to change the name of the generated log file, where stderr is written to.
    This is only useful if running in server mode.
  • --exitCodeOnError <n>
    Changes the exit code in case any test ends with a test-error status. The default is <1>. If you run expecco out of make, jenkins or other tools which check for the exit status, you may want to change this to <0> (zero), in order for the calling program to not interpret this status as if expecco itself failed to execute.
  • --exitCodeOnFail <n>
    Changes the exit code in case any test ends with a test-failure status. The default is <1>. If you run expecco out of make, jenkins or other tools which check for the exit status, you may want to change this to <0> (zero), in order for the calling program to not interpret this status as if expecco itself failed to execute.
  • --csvReportFile <csvFilename>
    Writes a summary report in CSV format. Each row consists of:
    • TESTSUITE - the filename
    • TESTPLAN - the test plan
    • TESTCASE - the test case
    • VERDICT - the test result; one of SUCCESS, FAILED, ERROR or INCONCLUSIVE
    • INFO - additional info in case of failure
Additional pseudo verdicts such as STARTTIME, ENDTIME etc. are written. A typical CVS output file can be found in the examples.
  • --xmlReportFile <xmlFilename>
    Writes a summary report in XML format. A typical XML output file can be found in the examples.
  • --textReportFile <textFilename>
    Writes a summary report in a human readable textual format. A typical text output file can be found in the examples.
  • --pdfReportFile <pdfFilename>
    Writes a summary report in pdf format.
  • --elfReportFile <elfFilename>
    Writes a summary report in expecco logfile format
  • --htmlReportFile <htmlFilename>
    Writes a summary report in html format.
  • --junitReportFile <htmlFilename>
    Writes a summary report in a junit compatible XML format. Useful, when expecco is to be started by jenkins or similar driver programs.

Operation

Startup
  • --noTray
    Disable the tray control icon (server and execute modes only).
  • --noWindow
    Do not open the main window (useful when scripting).
  • --noPlugins
    Disable plugin loading. This makes startup slightly faster. You can still force individual plugins to be loaded using the "--loadPlugin" option.
  • --loadPlugin <pluginName>
    Force loading a specific plugin. This is only useful if a "noPlugins" option was given before. pluginName is the name of the folder in the "plugin" folder (under the expecco installation folder).
  • --noUpdateCheck
    If configured by the user's settings, expecco automatically checks for updates and patches when started (by checking the exept web-server for the presence of new patch files). This option disables that check, even if configured in the settings. Notice, that this is usually controlled by the user settings; however, in situations where settings are shared among multiple machines (network drive) and a test host has no internet access, it may be useful to disable this from the command line.
Services
  • --server
    Starts expecco in server mode. In this mode, HTTP and SOAP requests can be used to remote-control expecco's execution; especially, to execute tests, and upload results. The most obvious use for this is for expecco to act as an execution host for an expeccoNET server.
    However, as standard protocols are used, this may also be used to interface the execution engine to any other QM management or automation system (HP Quality center).
  • --port <portNr>
    Used to change the HTTP and SOAP port numbers, when running in server mode.
    The default is 9090 (server mode only).
  • --info
    Provide an additional info service which allows for monitoring and control of an expecco server via a web browser (server mode only).
  • --scripting <portNr>
    Enable scripting (remote control) on portNr (see more below).
  • --allowHost <hostName>
    Allow hostName to connect to the scripting port. Repeat for multiple hosts (see more below) (1.5.3).


(1.5.3) these options are only supported by rel 1.5.3 and higher (see Release Notes expecco )

Output Examples

The following output files were generated by executing the test suite from the file:

"C:\Programme\exept\expecco\testsuites\projects\Example.ets"

which contains a single test plan named:

"Example Testplan"

which contains three test cases named:

"Test1"
"Test2"
"Test3-Should fail"

of which the first two are supposed to PASS and the last one generates an ERROR.

All timestamp information is written in ISO8601 format.

CSV File Example

By default, entries are separated by ";"-characters. Additional double quotes (") are used to delimit strings with embedded ";" or spaces. Any double quote within such a string is doubled.

TESTSUITE;TESTPLAN;TESTCASE;VERDICT;INFO
;;;StartTime;2008-10-31T08:57:13.689
"C:\Programme\exept\expecco\testsuites\projects\Example.ets";"Example Testplan";Test1;PASSED;  
"C:\Programme\exept\expecco\testsuites\projects\Example.ets";"Example Testplan";Test2;PASSED;  
"C:\Programme\exept\expecco\testsuites\projects\Example.ets";"Example Testplan";"Test3 - Should fail";ERROR;   
;;;EndTime;2008-10-31T08:57:18.746

XML File Example

<SummaryReport>
  <StartTime>2008-10-31T08:56:49.774</StartTime> 
  <TestSuite>
    <File>C:\Programme\exept\expecco\testsuites\projects\Example.ets</File> 
    <TestPlan>
      <Name>Example Testplan</Name> 
      <TestCase>
        <Name>Test 1</Name> 
        <Verdict>PASSED</Verdict> 
      </TestCase>
      <TestCase>
        <Name>Test 2</Name> 
        <Verdict>PASSED</Verdict> 
      </TestCase>
      <TestCase>
        <Name>Test 3 - Should fail</Name> 
        <Verdict>ERROR</Verdict> 
      </TestCase>
    </TestPlan>
  </TestSuite>
  <EndTime>2008-10-31T08:56:55.142</EndTime> 
</SummaryReport>

Textual File Example

Test Execution Summary
  Start: 2008-10-31 08:56:12.751
  TestSuite: "C:\Programme\exept\expecco\testsuites\projects\Example.ets
    TestPlan: Example Testplan
      Test 1 ....................... PASSED
      Test 2 ....................... PASSED
      Test 3 - Should fail ......... ERROR
  Finish: 2008-10-31 08:56:18.009


Verdicts Example

the "--verdicts" option generates a simple trace on the standard error (to be redirected into a file):

C:\selftest\commandLineArgs> expecco --noBanner --noTray --verdicts 
--parameter paramesForTest.xpar --execute parameterTest.ets
Expecco [info]: start testSuite "parameterTest.ets"... Expecco [info]: using parameters "paramesForTest.xpar"... Expecco [info]: start testPlan "Test Parameters From CommandLine" Expecco [info]: start testCase "Validate Parameters" Expecco [info]: done testCase "Validate Parameters"; STATUS=PASSED Expecco [info]: done testPlan "Test Parameters From CommandLine"; STATUS=PASSED Expecco [info]: done testSuite "parameterTest.ets"; STATUS=PASSED

Scripting Example

Expecco can be scripted either remote controlled via a telnet like command-line interface, or by reading a script from a file. This is useful for self-testing expecco, to use another expecco to control its GUI or to look into its internal data structures, or to execute complex test scenarios and meta tasks. It can also be used to execute and automate more complex scenarios without using expeccoNET.

The scripting interface expects expressions in JavaScript syntax (but with a Smalltalk object model), similar to the language used for elementary block definition.

When used with a remote script conenction, additional connection- and execution related commands are introduced by lines starting with the "~" (tilde) escape character (similar to telnet). For example, "~." (tilde-period) is used to close the connection. Press "~?" (tilde-question) for help.

The remote scripting service is enabled when a "--scripting" command line argument is given. By default, the service only accepts connections from the local host. Using the "--allowHost" option, more hosts can be gained access to the scripting feature - but be aware, that this opens a potential security hole, as all of the internal functions can be accessed via the scripting system.

The following is a trace of a typical scripting session:

c:\programs\expecco> expecco --scripting 8008
c:\programs\expecco>     ... expecco now running with GUI ...

c:\programs\expecco> telnet localhost 8008
Welcome to ST/X (Type ~? for help) > print(expecco); an Expecco::Browser > expecco.menuNewProject(); ...expecco performs the new-function from its menu... > expecco.loadProjectFromFile("C:\Users\cg\testsuites\webedition\projects\BearerTest.ets"); ...expecco loads the testSuite... > expecco.close();
Lost Connection. c:\programs\expecco>

Scripting API Overview

A few objects are predefined by name; among them are "expecco", which refers to the browser itself (i.e. the UI) and "environment", which refers to a collection of variable bindings.
New variables can defined by entering a "var <name>;" statement. For example:

    var ws;

    ws = new ExpeccoWorkspaceApplication;
    ws.open();
    ws.close();

expecco Object

The variable named "expecco" refers to the opened GUI-object, which represents the expecco application itself.

Menu Functions

These execute the underlying menu functions (as if clicked by the user).

  • menuNewProject()
    creates a new testSuite
  • menuOpen()
    opens a dialog requesting a testSuite
  • menuExit()
    closes the expecco application
  • loadProjectFromFile(fileNameString)
    load a testSuite
  • saveProjectToFile(fileNameString)
    save the testSuite
  • selectTestPlanWithName(testPlaneNameString)
    select a testplan item by name
  • selectTestPlanWithFunctionalId(uuid)
    select a testplan item by uuid
  • testSuite()
    retrieves the testSuite object

testSuite Object

  • authorName()
    get the author name string
  • authorName(aString)
    set the author name string
  • versionString()
    get the version string
  • versionString(aString)
    set the version string
  • projectsWorkingDirectory()
    the directory, where attachments etc. are found
  • ensureEnvironment()
    an instantiated environment (concrete values)

environment Object

[ ... to be continued ... ]



Back to Online Documentation

Meine Werkzeuge