The information on these pages may be out of date, or may refer to
resources that have moved or have been made read-only.
For more information please refer to the
Apache Attic
Note
This document is not complete. The final version will be finalized as the framework development is completed and associated feedback is recieved.
To be part of the project and successfully contribute to it this is important to know about test and release process, automation for testing, available test suites and scenarios. So the main target of this document is to provide a developer with all required information. The information is divided into the next sections:
A joint decision about a release schedule and acceptance criteria is made on the Harmony mailing list. Currently only milestone builds are released - the project has no official releases. Usually the milestone builds are released on 2-3 month schedule and named using the next scheme: Apache Harmony 5.0M<milestone number>. Before a release date there is 1 week feature freeze period and 1 week code freeze period. During feature freeze new functionality, big changes and code redesign are forbidden; only bugs fixes and code tidy-up are allowed. During code freeze period only fixes for critical/blocker issues are committed.
The Harmony team does its best to deliver well tested implementation. As the project’s code-base consists of big number of code-lines the testing became not a trivial task. So the clear testing process was established, automation for testing and several test suites were developed.
Currently the testing process consists of two testing cycles: integrity testing and snapshot testing. Both cycles are based on build-test framework
Integrity testing
cycle is aimed to verify that code base is buildable and runs selected
number of suites to catch regressions early. The cycle works on debug
build. The integrity testing cycle is triggered by a developer commit
(i.e. when a code is updated in the source code repository).
Results of testing are
published
almost immediately
Snapshot testing cycle is aimed to provide tested binaries on regular basis. Based on integrity testing results (i.e. the build is not broken and most of tests are passing) a revision for the next snapshot is chosen. When the snapshot is build it is uploaded to the web site and that in turn triggers the testing cycle. For each build snapshot there is a page with tests results summary. The page is updated dynamically as soon as results are ready.
The
build-test framework
is targeted to assist a developer with running different suites and
automation of testing process. The framework is a set of ANT scripts
that provides necessary support to simplify installation, setup of
Harmony build, test suites and configuring
CruiseControl.
You need to obtain and install the same tools, as for building the Apache Harmony components.
Get the infrastructure by checking it out from the project Subversion® repository:
svn co http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/infra
With Java, Ant and SVN installed, change into the infra
directory and type:
ant -Dtest.suites="<list of suites>" install
This command fetches external dependencies (including
CruiseControl), checks out Harmony build and test suites
(if necessary), sets up testing environment. After install
is complete it may request to specify unset required properties
for selected suites. The required-parameters.properties
file contains a list of all required properties.
If a property has empty or unresolved value
then a property has to be set in framework.local.properties
file and install target has to be re-run.
The next step is to setup suites. (Note: the framework saves the list
of installed suites so the parameter -Dtest.suites=...
is not required any more) Just simply type:
ant setup
This command launches setup target for each installed suite.
The suites' run can be launched in two ways: by the framework or by
CruiseControl tool. If a single run is needed type:
ant run
The command launches run target for each installed suite.
In case when a continuous build and test process is required type:
ant run-cc
The given command launches CruiseControl with the full test set.
To check status, point your browser to:
http://localhost:8080/
Please use the next steps for how to organize:
The Harmony testing base can be divided into 2 main parts: suites and scenarios.
A test suite is a collection of quite simple/short test cases that is used to verify that the implementation follows specified behavior. The project has several own test suites as well uses 3rd party (external) test suites.
An application testing scenario (or simply testing scenario) is a set of actions/operations that models typical application's usage/functioning.
Note
Most of suites/scenarios below can be run with and without build-test framework. Please consult a suite's documentation for details.