ARNS highlight report – ARTP spirometry survey
(Last updated: )
The ARTP spirometry survey was released between 20th October 2023 and 13th November 2023. There were 267 respondents to the survey; mostly nurses, with over 50% reporting they were on the ARTP register and most on the “performing and interpreting” and “adults” register.
30% respondents were not planning on registering with ARTP. The main reasons were completing alternative courses/training, retiring or finding the process too difficult. Lack of time, spirometry not being part of their job role and cost were additional reasons.
The overall pass rate for ARTP assessment (OSCE, portfolio and MCQ) was 53% (first attempt) and 79% (after second attempt).
The vast majority reported the OSCE assessment was as at least satisfactory. 90% passed on the first attempt; and 96% passed by their second attempt.
The majority of respondents reported the portfolio assessment as at least satisfactory. But, almost a third of respondents reported their overall experience as poor / very poor. 50% passed on their first attempt; whereas 79% passed by their second attempt.
The majority reported that the MCQ assessment was at least satisfactory. 25% felt that questions were not relevant to their practice; with the most common concerns from Primary care colleagues. 76% of candidates passed on their first attempt; 94% passed by their second attempt.
Of the respondents who had contacted ARTP in the last year, most reported at least a satisfactory experience. A fifth reported a poor or very poor experience. When asked to comment further, the most common answers were a lack of support, delays in or lack of adequate feedback, inconsistent marking and the cost and/or stress of completing the course. It should be noted that these comments were based on only a few responses to this question on the survey so should be interpreted with caution.
Regarding the renewal of ARTP registration, most respondents found the process straightforward. Some respondents would have preferred to have more notice for when they were due for renewal. Though very few respondents were in the 5% who have had to submit further evidence of their practice, almost all had a good or very good experience.
Suggested action points for ARTP:
1. Aim to improve the uptake of candidates completing the ARTP course in the UK.
2. Review of pass rates for different parts of the ARTP assessment as follows;
- OSCE: Due to a very high pass rates; a review of ARTP data on pass rates should be reviewed on its accuracy and marking criteria if appropriate.
- Portfolio: Due to lower pass rates and reports of poorer experience, an analysis of the assessment and marking criteria, as well as candidate feedback should be obtained and reviewed to improve and experience.
- MCQ: A review of the content and wording should be reviewed to ensure that questions are relevant to practice and easy to understand for candidates.
3. Use of second marking and/or external validation for ARTP assessment for more consistent marking and feedback.
4. Improve accessibility to support candidates who have queries on ARTP processes and assessment, ensuring timely and constructive feedback.
5. Repeat ARTP Spirometry questionnaire via ARNS in 12 months.
Response from ARTP
We would like to thank ARNS for conducting the survey, and everyone who took time to provide feedback. Overall the feedback is positive regarding experience of undertaking the spirometry certification which is reassuring. However there are clearly areas which we can improve on. Since the survey was conducted, ARTP has already made some changes which we hope have made a positive impact on candidates’ experience of the certification.
1. Whilst undergoing the certification process is not mandatory, ARTP encourages those healthcare professionals who routinely perform spirometry to complete the certification to ensure quality assured diagnostic spirometry is delivered, and to provide continuing professional development and confidence to these professionals. We will continue to work with all healthcare organisations to improve the process which will hopefully increase enrolment.
2.
– The MCQ exam pass rate has been reduced from 80% to 70% in line with other professional qualifications. Candidates receive their results immediately upon completion. Anecdotally we are getting less calls regarding MCQ questions now which hopefully indicates people are happy with this element of the certification
– The OSCE’s overall appear to be a positive experience and we are glad that candidates are satisfied with this element. We have recently begun recording the OSCE’s to improve transparency if there are any questions regarding results, and in line with other qualifications.
– We are aware that the portfolio generates the most concerns and questions and are looking at ways we can improve this. We plan to audit the portfolios to understand which are the most common areas that candidates are failing in. We can then review whether these elements need removing, changing, or we need to increase communication to ensure candidates know what is expected of them. There is a new version of the portfolio which is going live shortly which should reduce discrepancies as there are clear objective criteria for assessors.
It is important to emphasise that ARTP do not provide training, and are only the assessors of spirometry. We do not endorse training providers. If candidates feel there are elements of the certification they are struggling with relating to understanding spirometry they should discuss this with their training provider. All training providers are encouraged to provide mentorship to candidates throughout their enrolment.
3. We are in discussions regarding developing an external validation process to ensure transparency, and will partner with ARNS in this process. We are aware of issues with inconsistencies between assessors, and we have updated our process when taking on new assessors. New assessors are internally verified and reviewed before being allowed to independently mark portfolios. We have regular meetings with assessors to allow discussion and feedback on queries when marking.
4. There is a dedicated support platform called “Freshdesk” which all candidates have access to, with a dedicated clinical assessor who responds to all queries. We have developed a Spirometry newsletter, which will go out 4 times a year, in which we aim to provide clarity on common questions regarding spirometry. This should improve communication and support to candidates going through the certification, but also to all healthcare professionals who perform spirometry. We encourage people to contact ARTP spirometry with suggestions about what they would like to see included in the newsletter. We are also discussing using social media to provide more information. We will look at the timeframe in which we are responding to feedback, and will provide this information to all so that people know when to expect a response. We can then review this data to see how often we are meeting the target and put steps in place to improve this.
5. We will continue to collaborate with ARNS and other healthcare partners to improve the spirometry certification process, and look forward to hearing feedback from members in 12 months’ time.