Abstract

Validation of conceptual framework with user-centred action research

Methodology or methods/research instruments or sources used:

Literature review, online survey instrument, quantitative data analysis, qualitative findings

Conclusions or expected outcomes or findings:

Good match of remote and face to face user feedback which correlates well with initial hypothesis

References (including own publications):

• Admiraal, W., de Laat, M., Lockhorst, D., Rubens, W. (2004) "ICT and learning in small- and medium sized enterprises (SMEs)".
• Simons, P.R.J. (2004) "Metaphors of learning at work and the role of ICT".
• Pye, J. (2006). Tool evaluation report. Marchmont Observatory, University of Exeter.

ABSTRACT:

The Leonardo eAssessment project has been funded since 2004 in order to prove a central hypothesis. The project plan asserts that employees in small and medium sized businesses across Europe learn primarily through informal means, for which the application of technology based skills diagnostics - coupled with appropriate learning strategies - provides a flexible learning environment that can be tailored to suit individual business contexts. The eAssessment project has set out a structured research programme to substantiate this position, which in stages has consisted of:

• Initial literature review
• Conceptual design study
• Early tool development
• Expert consultation
• Implementation and piloting with transnational partners
• Final tool development.

The literature review at the outset of the project confirmed preferences for informal learning and standards of uptake of information and communications technologies by European small businesses. The subsequent conceptual design study consulted directly with companies themselves to identify types of online diagnostic software and design features that they would find useful, coupled with a market review of available packages. Once the preliminary specification was determined an open source solution was adapted and tested in the United Kingdom with expert groups.

Trialing of the diagnostic tool was conducted both remotely by email and face to face at a workshop in the United Kingdom, and feedback received from both sets of experts. Those consulted represented a range of interested groups, some of whom worked extensively with small businesses sourcing training and development provision. Others responding were themselves experts in elearning and already familiar with comparable software against which to assess the utility of the tool.

Both sets of experts tested the same version of the tool, although separate feedback was requested by each group which reflected their different modes of consultation.

Experts were invited to comment remotely via a structured questionnaire, whilst those accessing the tool at the conference also benefited through step-by-step 'real time' interaction and guidance via the project's developer partner.

Experts at the conference felt that the methodology integrated in the eAssessment tool could be embedded into other applications beyond those associated specifically with measuring competence or training needs in the workplace. The point was made however that staff expectations for follow up action after assessment must be recognised and appropriately taken forward. Beyond assessment of individual employees the tool was considered an appropriate platform for organisational analysis, perhaps as a means of gathering feedback from its workforce. Generally it was validated as complementary to other existing methods of assessment that learning brokers or training and development professionals could offer client firms.

Consultation comments given without benefit of a live demonstration again indicated that the tool is a useful implementation with further scope for development of content. Improvements in the presentation of results were suggested that would make the tool more usable, particularly in the provision of clear learning advice strategies in relevant language that would be meaningful to the user group. Feedback from conference delegates agreed well with individual experts' comments from the consultation and have been included in subsequent tool developments for the piloting phase, currently underway with transnational partners.

The conference presentation will compare responses received from the range of stakeholders consulted during the early stages of tool development with reactions from European small businesses themselves who are at present piloting the refined tool, and for which results will be available by mid 2007.

Full Papers:

[Word version - full paper] eValidating eAssessment for Workplace Skills

Preview Info - Jo Pye ECER 2007.doc - 70.00 Kb

Jo Pye, Marchmont Observatory University of Exeter, School of Education and Lifelong Learning