Nowadays many companies define, model and use business processes (BP) for several tasks. BP management has become an important research area and researchers have focused their attention on the development of mechanisms for searching BP models on repositories. Despite the positive results of the current mechanisms, there is no defined collaborative methodology to create a closed repository evaluation for these search mechanisms. This kind of repository contains some closed BP predefined lists representing queries and ideal answers to these queries with the most relevant BPs based on a set of evaluation metrics. These evaluation metrics can measure the consensus degree in the results, therefore confirming the methodology feasibility to create an open access, scalable and expandable closed BP repository with new BP models that can be reusable in future research.

At the bottom of this page you can find the links to download our closed test collection as well as our repository of Busines Processes.

A business processes (BP) model captures the set of procedures or interrelated activities that collectively develop a common business goal within the context of an organizational structure.

Nowadays there is a need for development of mechanisms to search and discover reusable components for defining new BP adjustable to current requirements of the organization.

Build a closed test repository of BPs while taking into account the opinion of an expert group from a collaborative perspective.

The proposed collaborative evaluation methodology contains three stages where each item of the resulting list is evaluated using a Likert scale containing the following concepts: very relevant, relevant, quite relevant, not very relevant, and irrelevant.

Individual Evaluation: At this stage, each evaluator or judge executes each query Q and the system returns a list of results. Then the evaluator issues a judgment of similarity.
Searching for consensus on discordant evaluations: At this stage, each evaluator reviews the results obtained in the previous stage, and compares them with the judgments obtained by other judges. Accordingly, evaluators can analyze how concordant or discordant their judgment are against each item taking into account the judgment of other evaluators.
Results Refinement: At this stage, the results are filtered according to values of relevance level (nr) that range from 50% to 60% (these parameters can be adjusted). As in the previous step, evaluators can re-analyze their judgments and modify them based on the judgements of the other evaluators.
Methodology Objectives: A fundamental task for building a BP test repository is the definition of an intuitive evaluation process where the evaluators collaboratively agree their judgments in order to clarify the similarity criteria about the results retrieved from an automatic BP search system.

The repository includes 100 BPs modeled with BPMN (Business Process Modeling Notation). In order to evaluate the proposed methodology we generated a set of relevant set of BPs. To generate this set, we based our evaluation on a collaborative strategy, where 59 expert reviewers (evaluators) from different institutions executed
comparisons between pairs of BPs (a query BP and each BP of the repository).

The average concordance (AVG) values between the evaluators were obtained: 0.284 for stage 1, 0.256 for stage 2 and 0.250 for stage 3.

In addition, it has a 9.7% of concessive improvement in (MCF) between stage 1 and stage 2, and stage 3 for each query. Finally the repository gets 11.8% of general concessive improvement (MCG) .

CONCLUSIONS: The application methodology proposed demonstrated to be useful for generating stable evaluations of BP repositories in order to make them more maintainable and reusable.

* More information about this methodology can be found in our paper published at the 16th International Conference on Enterprise Information Systems (ICEIS 2014).

Please cite as follows: Ordóñez, H, Corrales, J C, Cobos, C, Krug Wives, L, Thom, L H. Collaborative Evaluation to Build Closed Repositories on Business Process Models. In International Conference on Enterprise Information Systems (ICEIS). Volume 3 pg 311 - 318. 2014.

The repository is the central BP storage unit, similar to a database that shares information about the engineered artifacts produced or used by an enterprise. It is responsible for storing and representing all the attributes of information present in the BP (roles, description of activities, timers, messages, and service calls). Our repository is composed of 100 BPs obtained from real processes of diverse enterprises of the telecommunications and georeferencing domains.

The repository was built using diverse definition languages such as BPMO (Business Process Modeling Ontology) and BPMN (Business Process Model Notation).

We made freely available the description files of our repository including the relevance judgments in the next links.

Download the closed test collection of Business Processes in BPMN.

The BPs modeled with BPMO were semantically enriched with concepts of the SeTOM (semantic enhanced Telecom Operations Map) and SSID (Semantic Shared Information/Data Model) ontologies from the telecommunications domain. Then semantic concepts from these ontologies where manually added by experts to each element of the BPs.

Download the repository of semantically enriched Bussiness Processes in BPMO.