Computerized standardsetting using 3DC

The Data Driven Direct Consensus procedure (3DC) is a new and very flexible method for standard setting. As the name implies, it borrows considerably from the Direct Consensus procedure but it adds psychometrical rigour and guidance for the panellists by providing extra information about the relative difficulty of the clusters in relation to each other and to the test score.

“Efficient and innovative standard-setting”

Efficient and innovative standardsetting

The 3DC method allows for a variety of dichotomously or polytomously scored item types and any prediction method can be used to construct the form.

In a typical workflow, a psychometrican constructs an ability scale based on assessment data.  Items are then grouped in clusters, preferably in a logical manner based on their content. The ability scale and the clusters are used to build a form using random draws from a general partial credit model (an R script to construct the form data is provided). The form data is saved and later used by the 3DC application during a standard setting session.

How does it work?
For this session, a small network of laptops is set up, often using the panelists own laptops. The only requirement of participants’ laptops is that a recent version of Google Chrome is installed. Each panelist can provide his or her indicators by clicking on the form. Black circles are used for round 1. After a central discussion panelists can - if so desired - correct their indicators in round 2 which will then be shown in red.

During a session the group leader can review all selected performance indicators of all the panelists. Some basic statistics and useful graphs are also provided in real time. The application also allows participants to browse through and view the items in each cluster. Items in QTI or HTML format are natively supported - including multimedia - and there are workarounds to view items in other formats. 

Keuning, J. Straat, J.H. & R.C.W. Feskens (in press). The Data-Driven Direct Consensus (3DC) Procedure: A New Approach to Standard Setting. In “Standard setting: International state of research and practices in the Nordic countries”, Springer “Educational Measurement” series (eds. Veldkamp & von Davier).