Apache Harmony is retired at the Apache Software Foundation since Nov 16, 2011.

The information on these pages may be out of date, or may refer to resources that have moved or have been made read-only.
For more information please refer to the Apache Attic

Introduction

Note

This document is not complete. The final version will be finalized as the framework development is completed and associated feedback is recieved.

To be part of the project and successfully contribute to it this is important to know about test and release process, automation for testing, available test suites and scenarios. So the main target of this document is to provide a developer with all required information. The information is divided into the next sections:

Back to top

Harmony Test and Release Process

A joint decision about a release schedule and acceptance criteria is made on the Harmony mailing list. Currently only milestone builds are released - the project has no official releases. Usually the milestone builds are released on 2-3 month schedule and named using the next scheme: Apache Harmony 5.0M<milestone number>. Before a release date there is 1 week feature freeze period and 1 week code freeze period. During feature freeze new functionality, big changes and code redesign are forbidden; only bugs fixes and code tidy-up are allowed. During code freeze period only fixes for critical/blocker issues are committed.

The Harmony team does its best to deliver well tested implementation. As the project’s code-base consists of big number of code-lines the testing became not a trivial task. So the clear testing process was established, automation for testing and several test suites were developed.

Currently the testing process consists of two testing cycles: integrity testing and snapshot testing. Both cycles are based on build-test framework

Integrity testing cycle is aimed to verify that code base is buildable and runs selected number of suites to catch regressions early. The cycle works on debug build. The integrity testing cycle is triggered by a developer commit (i.e. when a code is updated in the source code repository). Results of testing are published almost immediately

Snapshot testing cycle is aimed to provide tested binaries on regular basis. Based on integrity testing results (i.e. the build is not broken and most of tests are passing) a revision for the next snapshot is chosen. When the snapshot is build it is uploaded to the web site and that in turn triggers the testing cycle. For each build snapshot there is a page with tests results summary. The page is updated dynamically as soon as results are ready.

Back to top

Framework for Testing and Automation

The build-test framework is targeted to assist a developer with running different suites and automation of testing process. The framework is a set of ANT scripts that provides necessary support to simplify installation, setup of Harmony build, test suites and configuring CruiseControl.

Prerequisites

You need to obtain and install the same tools, as for building the Apache Harmony components.

Setup

Get the infrastructure by checking it out from the project Subversion® repository:

svn co http://svn.apache.org/repos/asf/harmony/enhanced/buildtest/trunk/infra

With Java, Ant and SVN installed, change into the infra directory and type:

ant -Dtest.suites="<list of suites>" install

This command fetches external dependencies (including CruiseControl), checks out Harmony build and test suites (if necessary), sets up testing environment. After install is complete it may request to specify unset required properties for selected suites. The required-parameters.properties file contains a list of all required properties. If a property has empty or unresolved value then a property has to be set in framework.local.properties file and install target has to be re-run.

The next step is to setup suites. (Note: the framework saves the list of installed suites so the parameter -Dtest.suites=... is not required any more) Just simply type:

ant setup

This command launches setup target for each installed suite.

Run

The suites' run can be launched in two ways: by the framework or by CruiseControl tool. If a single run is needed type:

ant run

The command launches run target for each installed suite.

In case when a continuous build and test process is required type:

ant run-cc

The given command launches CruiseControl with the full test set. To check status, point your browser to:

http://localhost:8080/

How To

Please use the next steps for how to organize:

Back to top

Test Suites and Scenarios

The Harmony testing base can be divided into 2 main parts: suites and scenarios.

A test suite is a collection of quite simple/short test cases that is used to verify that the implementation follows specified behavior. The project has several own test suites as well uses 3rd party (external) test suites.

An application testing scenario (or simply testing scenario) is a set of actions/operations that models typical application's usage/functioning.

Note

Most of suites/scenarios below can be run with and without build-test framework. Please consult a suite's documentation for details.

Back to top