Exporting data out of LAVA¶

LAVA supports two methods of extracting data: the REST API and XML-RPC. Results are made available while the job is running via the results API. Direct links from the test log UI are not populated until after the job completes, due to performance issues.

In addition to these methods of pulling data out of LAVA, there are also two methods for pushing information about its activity: notifications and publishing events.

REST API¶

REST API is available with /api URL. The API is based on django-rest-framework. All of the docs from django-rest-framework apply. LAVA uses token based authentication. To obtain a token POST request has to be made to /api/<version>/token/ endpoint. The request has to contain username and password fields. These are the same as used for web UI authentication. Call returns either existing AuthToken or creates new one. The tokens used for REST API and XML-RPC API are the same objects. Example:

$ curl -d '{"username":"john.doe", "password":"FooBar"}' -H "Content-Type: application/json" -X POST "https://master.lavasoftware.org/api/v0.1/token/"

Reply will contain a token that identifies the user when using the REST API:

{"token":"ezwpm1wytdwwnbbu90e6eo02bligzw21b0ibyc1ikbc19zkin6639f3wodce5u9oc3lndoqn0asfewrw0bclfii4mgtweokrxa0mztohj46n2rdi0qinezsbobfauqf0"}

v0.2¶

Currently there are 9 endpoints available:

/api/v0.2/aliases/
/api/v0.2/jobs/
/api/v0.2/devicetypes/
/api/v0.2/devices/
/api/v0.2/permissions/devicetypes
/api/v0.2/permissions/devices
/api/v0.2/system/
/api/v0.2/tags/
/api/v0.2/workers/

In addition TestJob object (/api/v0.2/jobs/id/) contains following routes:

logs (/api/v0.2/jobs/<job_id>/logs)

metadata (/api/v0.2/jobs/<job_id>/metadata)

Following nested routes are available. The allow for filtering in suites and tests:

suites (/api/v0.2/jobs/<job_id>/suites)
tests (/api/v0.2/jobs/<job_id>/tests)

The results are also available in JUnit, TAP13, CSV or YAML formats at:

junit (/api/v0.2/jobs/<job_id>/junit/)
tap13 (/api/v0.2/jobs/<job_id>/tap13/)
CSV (/api/v0.2/jobs/<job_id>/csv)
yaml (/api/v0.2/jobs/<job_id>/yaml)

Extra actions¶

Additional actions are available on the following endpoints with GET request:

device dictionary (/api/v0.2/devices/<device_hostname>/dictionary/)
device type health check (/api/v0.2/devicetypes/<device_type_name>/health_check/)
device type template (/api/v0.2/devicetypes/<device_type_name>/template/)
worker configuration (/api/v0.2/workers/<worker>/config/)
worker environment (/api/v0.2/workers/<worker>/env/)

Additional actions are available on the following endpoints with POST request:

device dictionary (/api/v0.2/devices/<device_hostname>/dictionary/)
device type health check (/api/v0.2/devicetypes/<device_type_name>/health_check/)
device type template (/api/v0.2/devicetypes/<device_type_name>/template/)
worker configuration (/api/v0.2/workers/<worker>/config/)
worker environment (/api/v0.2/workers/<worker>/env/)
job resubmit (/api/v0.2/jobs/<job_id>/resubmit)
job validate (/api/v0.2/jobs/validate)
- required parameters:
  
  definition (string)
- optional parameters:
  
  strict (bool)

Objects in all endpoints can be filtered and sorted as described in django-rest-framework docs: http://www.django-rest-framework.org/api-guide/filtering/ Searching is currently disabled.

Examples specific to LAVA objects:

Filtering¶

https://validation.linaro.org/restapi/v0.2/jobs/?health_check=true

Filtering fields

TestJob object¶

submitter (can be traversed to User object)
viewing_groups (can be traversed to Group object)
description (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
health_check (bool field)
- exact
requested_device_type (can be traversed to DeviceType object)
tags (can be traversed to Tag object)
actual_device (can be traversed to Device object)
submit_time (datetime field)
- exact
- lt
- gt
start_time (datetime field)
- exact
- lt
- gt
end_time (datetime field)
- exact
- lt
- gt
state (Submitted, Scheduling, Scheduled, Running, Canceling, Finished)
- exact
health (Unknown, Complete, Incomplete, Canceled)
- exact
priority (integer field)
- exact
- in
- lt
- gt
- lte
- gte
definition (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
original_definition (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
multinode_definition (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
failure_tags (can be traversed to JobFailureTag object)
failure_comment (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
- isnull

DeviceType object¶

name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
architecture (can traverse to Architecture object)
processor (can traverse to ProcessorFamily object)
cpu_model (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
aliases (can traverse to Alias object)
bits (can traverse to BitWidth object)
cores (can traverse to Core object)
core_count (integer field)
- exact
- in
description (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
health_frequency (integer field)
- exact
- in
disable_health_check (bool field)
- exact
- in
health_denominator (hours, jobs)
- exact
display (bool field)
- exact
- in

Architecture object¶

name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

ProcessorFamily object¶

name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

Alias object¶

name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

Core object¶

name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

BitWidth object¶

width (integer field)
- exact
- in

Device object¶

hostname (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
device_type (can traverse to DeviceType object)
device_version (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
physical_owner (can traverse to User object)
physical_group (can traverse to Group object)
description (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
tags (can traverse to Tag object)
state (Idle, Reserved, Running)
- exact
health (GOOD, UNKNOWN, LOOPING, BAD, MAINTENANCE, RETIRED)
- exact
worker_host (can traverse to Worker object)

Worker object¶

hostname (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
description (text field)
- exact
- in
- contains
- icontains
- startswith
- endswith
last_ping (datetime field)
- exact
- lt
- gt
state (Online, Offline)
- exact
health (Active, Maintenance, Retired)
- exact

Tag object¶

description (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

JobFailureTag object¶

description (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

User object¶

group (can traverse to Group object)
username (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith
email (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

Group object¶

name (char field)
- exact
- in
- contains
- icontains
- startswith
- endswith

Nested filtering¶

It is possible to filter objects using their relations. This is achieved by using double underscore notation from django. Example:

https://validation.linaro.org/restapi/v0.1/jobs/?requested_device_type__cores__name=kirin

In the example above requested_device_type comes from TestJob object. It’s a related field of DeviceType. DeviceType contains cores field which is of type Core. Core object contains name field. So the example above queries database for all TestJob objects which requested device type that use cores with name kirin.

Sorting¶

https://validation.linaro.org/restapi/v0.1/jobs/?ordering=start_time

Sorting fields for TestJob object:

id
start_time
end_time
submit_time

Sorting fields for Device object:

hostname
device_type
device_version
physical_owner
physical_group
description
tags
state
health
worker_host

Sorting fields for Worker object:

hostname
description
last_ping
state
health

Sorting fields for Alias object:

name

Sorting fields for Tag object:

name
description

Creating and modifying objects¶

DjangoRestFramework allows to create and modify objects using POST and PUT http requests respectively. These operations require successful authentication and sufficient permissions. LAVA utilizes Token authentication using http request header. Example:

$ curl -H "Authorization: Token xxxxxx"

Same token is used for REST API and XML-RPC API authentication.

Submitting a test job¶

Sending a POST request on a /jobs/ endpoint will result in attempted job submission in LAVA. There is only one argument which needs to be passed in the request and that is definition. Example:

$ curl -H "Authorization: Token xxxxxx" -d '{"definition": "your-testjob-definition-here"}' -H "Content-Type: application/json" -X POST "https://your.lava.hostname/api/v0.2/jobs/"

Other database object can be created in a similar way. Different objects require different fields.

Deprecated semi-REST API for retrieving results¶

LAVA makes the test results available directly from the instance, without needing to go through lavacli. The results for any test job which the user can view can be downloaded in CSV or YAML format.

For example, the results for test job number 123 are available in CSV format using: https://validation.linaro.org/results/123/csv. The same results for job number 123 are available in YAML format using: https://validation.linaro.org/results/123/yaml

If you know the test definition name, you can also download the results for that specific test definition only in a similar way: https://validation.linaro.org/results/123/0_singlenode-advanced/csv for the data in CSV format and https://validation.linaro.org/results/123/0_singlenode-advanced/yaml for the YAML format.

Test definition names use a prefix to show the sequence within the test job, for each namespace. The list of test definitions within a test job can be retrieved using the Summary: https://validation.linaro.org/results/123/yaml_summary

Some test jobs can be restricted to particular users or groups of users. The results of these test jobs will be restricted in the same way. To download these results, you will need to specify your username and one of your Authentication Tokens when using the REST API. Remember to quote the URL if using it on the command line, to avoid the & character being mis-interpreted by your shell, for example:

$ curl 'https://validation.linaro.org/results/123/0_singlenode-advanced/yaml?user=user.name&token=yourtokentextgoeshereononeverylongline'

Use the Username as specified in your Profile - this may differ from the username you use if logging in via LDAP.

Caution

Take care of your tokens - avoid using personal tokens in scripts and test definitions, or any other files that end up in public git repositories. Wherever supported, use https:// when using a token to avoid it being sent in plain-text.

Chunking test suite results¶

When jobs have a large number of test results in a particular test suite, it is advisable to use test case chunking, to provide pagination for downloading test cases via the REST API.

Two special query string arguments are supported for allowing users to chunk the test cases when downloading test suite results.

Limit - determines how many results to read in this chunk
offset - the number of results already received

Limit and offset are supported only for test suite exports (both csv and yaml). Example:

$ curl 'https://validation.linaro.org/results/123/0_singlenode-advanced/yaml?limit=100&offset=200'

$ curl 'https://validation.linaro.org/results/123/0_singlenode-advanced/yaml?user=user.name&token=yourtokentextgoeshereononeverylongline&limit=100&offset=200'

To retrieve the full count of testcases in a single test suite (useful for pagination), you can use the testcase-count REST API method, like so:

$ curl 'https://validation.linaro.org/results/123/0_singlenode-advanced/+testcase-count?user=user.name&token=yourtokentextgoeshereononeverylongline'

Note

Test cases will be ordered by ID regardless of whether pagination is used or not. This applies to downloading test cases for a particular test suite only.

XML-RPC¶

Lots of methods are available to query various information in LAVA.

Warning

When using XML-RPC to communicate with a remote server, check whether https:// can be used to protect the token. http:// connections to a remote XML-RPC server will transmit the token in plain-text. Not all servers have https:// configured. If a token becomes compromised, log in to that LAVA instance and delete the token before creating a new one.

The general structure of an XML-RPC call can be shown in this python snippet:

# Python3
import json
import xmlrpc.client

config = json.dumps({ ... })
server=xmlrpc.client.ServerProxy("http://username:API-Key@localhost:8001/RPC2/")
jobid=server.scheduler.submit_job(config)

XML-RPC can also be used to query data anonymously:

# Python3
import xmlrpc.client

server = xmlrpc.client.ServerProxy("http://sylvester.codehelp/RPC2")
print server.system.listMethods()

Individual XML-RPC commands are documented on the API Help page.

User specified notifications¶

Users can request notifications about submitted test jobs by adding a notify block to their test job submission.

The basic setup of the notifications in job definitions will have criteria, verbosity, recipients and compare blocks.

Criteria tell the system when the notifications should be sent
Verbosity tells the system how much detail should be included in the notification
Recipients tells the system where to send the notification, and how
Compare is an optional block that allows the user to request comparisons between results in this test and results from previous test

Here is an example notification setup. For more detailed information see User notifications in LAVA.

Example test job notification¶

notify:
  criteria:
    status: incomplete
  verbosity: quiet
  recipients:
  - to:
     user: neil.williams
     method: irc

Event notifications¶

Event notifications are handled by the lava-publisher service on the master. By default, event notifications are disabled.

Note

lava-publisher is distinct from the publishing API. Publishing events covers status changes for devices and test jobs. The publishing API covers copying files from test jobs to external sites.

http://ivoire.dinauz.org/linaro/bus/ is the home of ReactOWeb. It shows an example of the status change information which can be made available using lava-publisher. Events include:

metadata on the instance which was the source of the event; and
description of a status change on that instance.

Event notifications are disabled by default and must be configured before being enabled.

Example metadata¶

Date and time
Topic (for example org.linaro.validation.staging.device)
Message UUID
Username

The topic field is configurable by lab administrators. Its intended use is to allow receivers of events to filter incoming events.

Event notification types¶

Device event notifications are emitted automatically when a device changes state (e.g. Idle to Running) or health (e.g. Bad to Unknown). Some events are related to testjobs, some are due to admin action.
Testjob event notifications are emitted automatically when a testjob changes state (e.g. Submitted to Running).
System event notifications are emitted automatically when workers change state.
Test Shell event notifications are emitted only when requested within a Lava Test Shell by a test writer and contain a customized message.

Example device notification¶

{
   "device": "staging-qemu05",
   "device_type": "qemu",
   "health_status": "Pass",
   "job": 156223,
   "pipeline": true,
   "status": "Idle"
}

Example testjob notification¶

{
    'health_check': False,
    'description': 'QEMU pipeline, first job',
    'state': 'Scheduled',
    'visibility': 'Publicly visible',
    'priority': 50,
    'submitter': 'default',
    'job': 'http://calvin.codehelp/scheduler/1995',
    'health': 'Unknown',
    'device_type': 'qemu',
    'submit_time': '2018-05-17T11:49:56.336847+00:00',
    'device': 'qemu01'
}

Example log event notification¶

2018-05-17T12:12:15.238331 .codehelp.calvin.worker lavaserver - [worker01] state=Online health=Active

Example test event notification¶

Test writers can cause event notifications to be emitted under the control of a Lava Test Shell. This example uses an inline test definition.

    - repository:
        metadata:
          format: Lava-Test Test Definition 1.0
          name: apache-server
          description: "test events"
          os:
          - debian
          scope:
          - functional
        run:
          steps:
          - lava-test-event demonstration
      from: inline
      name: test-event
      path: inline/test-event.yaml

2018-05-17T11:51:22.542416 org.linaro.validation.event lavaserver - {"message": "demonstration", "job": "1995"}

Write your own event notification client¶

It is quite straightforward to communicate with lava-publisher to listen for events. This example code shows how to connect and subscribe to lava-publisher job events. It includes a simple main function to run on the command line if you wish:

python zmq_client.py -j 357 --hostname tcp://127.0.0.1:5500 -t 1200

zmq_client.py script:

import argparse
import signal
import ssl
import sys
import xmlrpc.client
from urllib.parse import urlsplit

import yaml
import zmq

FINISHED_JOB_STATUS = ["Complete", "Incomplete", "Canceled"]


class JobEndTimeoutError(Exception):
    """Raise when the specified job does not finish in certain timeframe."""


class Timeout:
    """Timeout error class with ALARM signal. Accepts time in seconds."""

    class TimeoutError(Exception):
        pass

    def __init__(self, sec=0):
        self.sec = sec

    def __enter__(self):
        signal.signal(signal.SIGALRM, self.timeout_raise)
        if not self.sec:
            self.sec = 0
        signal.alarm(self.sec)

    def __exit__(self, *args):
        signal.alarm(0)

    def timeout_raise(self, *args):
        raise Timeout.TimeoutError()


class JobListener:
    def __init__(self, url):
        self.context = zmq.Context.instance()
        self.sock = self.context.socket(zmq.SUB)

        self.sock.setsockopt(zmq.SUBSCRIBE, b"")
        self.sock.connect(url)

    def wait_for_job_end(self, job_id, timeout=None):
        try:
            with Timeout(timeout):
                while True:
                    msg = self.sock.recv_multipart()
                    try:
                        (topic, uuid, dt, username, data) = msg[:]
                    except IndexError:
                        # Dropping invalid message
                        continue

                    data = yaml.safe_load(data)
                    if "job" in data:
                        if data["job"] == job_id:
                            if data["health"] in FINISHED_JOB_STATUS:
                                return data

        except Timeout.TimeoutError:
            raise JobEndTimeoutError(
                "JobListener timed out after %s seconds." % timeout
            )


def lookup_publisher(hostname, https):
    """
    Lookup the publisher details using XML-RPC
    on the specified hostname.
    """
    xmlrpc_url = "http://%s/RPC2" % (hostname)
    if https:
        xmlrpc_url = "https://%s/RPC2" % (hostname)
    server = xmlrpc.client.ServerProxy(xmlrpc_url)
    try:
        socket = server.scheduler.get_publisher_event_socket()
    except ssl.SSLError as exc:
        sys.stderr.write("ERROR %s\n" % exc)
        return None
    port = urlsplit(socket).port
    listener_url = "tcp://%s:%s" % (hostname, port)
    print("Using %s" % listener_url)
    return listener_url


def main():
    """
    Parse the command line
    For simplicity, this script does not handle usernames
    and tokens so needs a job ID. For support submitting
    a test job as well as watching the events, use lavacli.
    """
    parser = argparse.ArgumentParser()
    parser.add_argument("-j", "--job-id", type=int, help="Job ID to wait for")

    parser.add_argument(
        "--https", action="store_true", help="Use https:// for this hostname"
    )

    parser.add_argument("-t", "--timeout", type=int, help="Timeout in seconds")
    parser.add_argument("--hostname", required=True, help="hostname of the instance")

    options = parser.parse_args()

    try:
        publisher = lookup_publisher(options.hostname, options.https)
    except xmlrpc.client.ProtocolError as exc:
        sys.stderr.write("ERROR %s\n" % exc)
        return 1

    if not publisher:
        return 1

    if options.job_id:
        listener = JobListener(publisher)
        print(listener.wait_for_job_end(options.job_id, options.timeout))
    print("\n")
    return 0


if __name__ == "__main__":
    main()

Download or view the complete example: examples/source/zmq_client.py

If you are interested in using event notifications for a custom frontend, you may want also to look at the code for the ReactOWeb example website: https://github.com/ivoire/ReactOWeb

Submit a job and wait on notifications¶

A common request from LAVA users is the ability to submit a test job, wait for the job to start and then monitor it directly as it runs. Recent versions of lavacli support this directly.