Impact and Challenge model of CPD credits

Impact and Challenge model of CPD credits Pilot report June 2009 Christopher Price RCGP CPD fellow Introduction A pilot of a system of CPD credit rating based on the impact and challenge model (described in appendix 1) ran between September 2008 and May 2009. 10 geographical locations spread across all four home nations contributed to the pilot. 232 doctors completed an appraisal using the impact and challenge model to rate their CPD activity. These doctors and 90 appraisers completed an online questionnaire. The results of a questionnaire are detailed in this report. Contents Executive summary Demographic data The appraisees view The appraisers view Report on pilot objectives Conclusions Appendix 1 -the impact and challenge model of CPD credits Appendix 2 - credits project plan Acknowledgements 1 Executive summary , The pilot examined a system of accreditation of CPD activity based on the following two statements:- o A credit is a unit of professional development which is a product of the impact of a developmental activity and to a lesser extent the challenge involved in its completion. o Credits are self assessed and verified at appraisal. , 232 doctors and 90 appraisers completed the pilot , All four home nations contributed to the pilot , Overall the system has been received positively but with a significant minority expressing the counter view , Many doctors and appraisers found the system acceptable with many expressions of concern over the complexity and subjectivity of the system , The absence of time spent as a factor was perceived to be positive , Little difference was seen in the results from GP educators and those not working in GP education , A small group of freelance GPs contributed to the pilot. The acceptance of the by this group was relatively lower than GP partners , Refinements will be needed before the system is rolled out Demographic data 2 The pilot was completed by 232 doctors and 90 appraisers. A variety of regional leads, working both within the educational establishment and as appraisal leads, were recruited from 10 geographical areas throughout the UK. 232 doctors contributed to the pilot of which 201 discussed credits at appraisal and 31 discussed credits with a tutor. The overwhelming majority of doctors describe themselves as a partner, 25 said they were salaried, 32 were freelance, 4 were retainers and 4 worked exclusively in out of hours. There was an even split between doctors describing themselves as full or part time. 54% had a role in medical education outside the appraisal process. Region Number of appraisers Number of doctors appraised East Midlands 8 16 Kent Surrey and Sussex 30 58 London 18 27 Mersey 1 25 North Western 0 6 Northern 19 32 Northern Ireland 4 9 Scotland 2 25 Wales 5 31 Wessex 3 3 The appraisees view 3 Context It is imperative that an accreditation system for CPD is acceptable to the profession. Furthermore it needs to measure “the right thing”. It should be easily understood, fair, be applicable across different working environments and recognise different learning styles. The initial concept of the impact and challenge model was to remove “time spent”, rewarding the impact of an activity and, to a lesser extent, the challenge involved in making that impact. Results 232 doctors contributed to the pilot of which 201 discussed credits at appraisal and 31 discussed credits with a tutor. The overwhelming majority of doctors describe themselves as a partner, 25 said they were salaried, 32 were freelance, 4 were retainers and 4 worked exclusively in out of hours. There was an even split between doctors describing themselves as full or part time. 54% had a role in medical education outside the appraisal process. Slightly more than 50% felt the definition was easy or very easy to understand, less than a quarter rated it quite hard or very hard. 84% felt the definition described the system in which they participated. 82% found the definition acceptable. 94% self assessed their credits. 90% had the majority of their credits and verified by the appraiser or tutor, with 86.6% declaring that the process of self-assessment is acceptable to them. 87.5% found the verification of credits at appraisal acceptable. In order to assess the ease of use and the time spent the following questions were asked (percentages are related to the total number of respondents which is the whole number in the right-hand box) Did you receive training in the impact and challenge model of CPD credits? Yes, face to face: 40.5% 94 Yes, paper/electronic 37.9% 88 based: No: 21.6% 50 If no It was offered but I 2.0% 1 declined: It was offered but the 6.0% 3 date was unsuitable: It was not offered: 92.0% 46 Did the accompanying paperwork (electronic documentation) help you assess the impact of 4 your CPD? Yes: 86.2% 200 No: 6.0% 14 Did not receive 7.8% 18 paperwork: Did the accompanying paperwork (electronic documentation) help you assess the challenge involved in obtaining your CPD? Yes: 84.9% 197 No: 6.9% 16 Did not receive 8.2% 19 paperwork: When assessing the number of credits for an individual activity did you find assessing:- Impact Always easy: 2.2% 5 Mostly easy: 37.1% 86 Neither easy nor hard: 30.2% 70 Mostly hard: 25.0% 58 Always hard: 5.6% 13 Challenge Always easy: 1.7% 4 Mostly easy: 39.2% 91 Neither easy nor hard: 31.9% 74 Mostly hard: 23.3% 54 Always hard: 3.9% 9 Number of credits per activity Always easy: 0.9% 2 Mostly easy: 24.6% 57 Neither easy nor hard: 30.6% 71 Mostly hard: 37.1% 86 Always hard: 6.9% 16 How much time did you spend in preparing your credit claim 20 minutes or less: 9.1% 21 5 20-30 minutes: 13.8% 32 30-40 minutes: 19.8% 46 40-50 minutes: 7.3% 17 50-60 minutes: 15.9% 37 over an hour: 34.1% 79 How much time did discussing the credits discussion add to your appraisal discussion 10 minutes or less: 16.8% 39 10-20 minutes: 34.1% 79 20-30 minutes: 28.9% 67 30 minutes plus: 20.3% 47 70.7% of the participants felt that their traditional CPD activity fitted easily into the impact and challenge model. The remaining 29.3% felt that it did not. When asked about future CPD planning in relation to the impact and challenge model 76.3% felt that it would easily fit. 87.9% of respondents rated impact as a legitimate factor in assessing CPD credits, 88.8% rated challenge as a legitimate factor. The respondents were asked to give overall ratings on Likhert scales the results are reproduced below:- Overall please rank your feeling about the system where 1 is low and 10 is high Ease of use 6 1: 4.7% 11 2: 3.4% 8 3: 9.5% 22 4: 9.9% 23 5: 10.8% 25 6: 9.1% 21 7: 19.0% 44 8: 22.4% 52 9: 8.6% 20 10: 2.6% 6 Ranking Statistics Median rank: 7 Mean rank: 6.0 Acceptability 1: 4.7% 11 2: 4.3% 10 3: 4.3% 10 4: 7.3% 17 5: 11.6% 27 6: 10.3% 24 7: 19.8% 46 8: 22.4% 52 9: 8.2% 19 10: 6.9% 16 Median rank: 7 Mean rank: 6.4 Logical system 1: 5.2% 12 7 2: 5.2% 12 3: 4.7% 11 4: 3.4% 8 5: 13.8% 32 6: 9.1% 21 7: 21.1% 49 8: 20.3% 47 9: 11.6% 27 10: 5.6% 13 Median rank: 7 Mean rank: 6.4 Measures the "right" thing 1: 5.6% 13 2: 5.6% 13 3: 5.2% 12 4: 5.2% 12 5: 14.7% 34 6: 9.1% 21 7: 15.5% 36 8: 19.4% 45 9: 14.2% 33 10: 5.6% 13 Median rank: 7 Mean rank: 6.3 Overall rating 1: 4.3% 10 8 2: 4.7% 11 3: 4.7% 11 4: 5.6% 13 5: 13.8% 32 6: 7.8% 18 7: 24.1% 56 8: 22.8% 53 9: 9.1% 21 10: 3.0% 7 Median rank: 7 Mean rank: 6.3 The Bristol online survey (BOS) system allows anonymous data manipulation giving the overall results against any single question. It is therefore possible to analyse any differences that may be evident from different demographic groups. An important distinction, made explicit at the inception of the pilot, was around the doctor’s working environment. Pilot participants not involved in medical education were more likely to be freelance or salaried and were far more likely to be part-time than the equivalent group involved in medical education. They were less likely to have been offered training in the impact and challenge model, far greater numbers did not receive the paperwork associated with the pilot. When asked did the model fit easily with their CPD activity? 65.9% of those involved in education replied yes. Of those not involved in education 76.5% replied yes. Apart from this one difference the remainder of the results were strikingly similar. The overall rating was virtually mirrored between the educators and the non-educators. When working arrangements were supplied as a filter it was clear that freelance GPs had misgivings about the system. 86.2% of partners found the definition acceptable as opposed to 68.75% of freelance GPs. 73% of partners felt the system applied easily to their CPD activities in the freelance group this was only 62%. These misgivings were reflected in the overall rating of the system with the freelance GPs giving lower rankings than partners. Despite the low numbers of freelance GPs participating in the pilot (32) these findings must be given due cognizance. There were no significant differences between those who work full-time and those who work part-time. The overall ratings tended to be very slightly lower for those working part-time. The quantitative results reproduced above are supported by a wealth of free hand qualitative data. The themes that emerge contain entries that are highly supportive of the system and a smaller but strident minority of critical comments. The full unedited list of comments may be requested from Chris@pustule.Demon.co.uk (227 pages long). Examples are reproduced below:- 9 A thoughtfully designed model that should work well once we get used to the 'currency'. Acknowledgement of the effort required for some activities Allows you to use a wide variety of activities, including self-directed learning, for credits and the content can be much more tailored to individual requirements. I like the large element of personal responsibility and accountability After reading all the instructions and information that came with it the actual process of evaluating the activities was relatively easy Being able to score my own activities A wide range of activities can be included, more than I would realise Easy to use. Helps encourage reflective learning. Educationally a better system than "hours clocked" and does get you think about outcomes of learning. Honest attempt to measure the immeasurable!! I found it really made me look at my activities over the year and to help plan what to do in the forthcoming year - for example this year for me was very busy with a lot of activities and this means that I may focus in the forthcoming year on fewer activities which are more challenging and have a higher impact I immediately enjoyed the different way of reflecting on each experience - which made the process more worthwhile It didn't take too long to do, it encouraged reflection, and although I easily achieved more than the required 50 credits, I realised that I could have easily scored even more highly by building on activities and extending them. EG - an SEA relating to ACEI use in women of childbearing years scored 1-2 credits. Had I done an audit/search for ACEI's in all women registered with the practice of childbearing potential that would have had a greater impact and would have been so simple to do! Thus, the system promotes reflection. It has helped me focus on why I do CPD It's a new way to think about CPD rather than going through the motions Makes reflection easier, concentrates the mind and encourages one to keep continual update of appraisal activities Surprisingly it worked and did feel 'right'(reading the bumf I was not convinced it would but it did) When my partner initially brought this project to me I was sceptical. However, after reading the guides I felt it was a system that could work as it tries to move away from a pure time concept. 10 addresses not only the learning done but the wider implication of this to patients/practice etc It is far better than bums on seats hours and recognises the outcomes of learning It requires thought as to the purpose of the educational exercise - who is it going to help and to what extent- and also measures the effort expended in achievement and so invites questioning of whether the benefits merit the amount of effort required. By slightly weighting patient benefit it does not disallow credits for putting lots of effort into studying subjects of personal interest but possibly dubious relevance but encourages a practical outcome focussed approach to learning and revision. It rewards doctors for thinking deeply about education and taking things to the next level It seems long overdue for GPs to be given credit for educational activity that they share more widely in the interests of patient care, and result in improvements in health care systems. It takes account of the reality of learning, that it is often stimulated by day to day events It tries to quantify the value of work rather than just time it took It's better than just "hours on courses" Seems to make as much sense as anything else; I especially like that it gets away from the 'time served' model of simply turning up to random events to collect points. This definition has assisted my concept of education no end and has helped immensely with my educational needs It is still difficult to work out how many credits you have and not to consider time It is too complicated It promotes reflection on what I have learned, what I have achieved as a result, and the learning journey. more focused than merely attending courses and claiming I think it will be quite variable how people rate their own scores > return to the old PGEA is almost better as that had different categories to ensure a broad educational agenda need to have some benchmark my appraiser is objective and together the process was easily done In order to meet the minimum requirement of 50 credits I had to make sure I did 'more than enough' in order to account for variation with the appraiser. This extra work is time consuming 11 and adds pressure on my normal work. The appraiser cannot understand your personal constraints fully unless they have a very good relationship with you. The themes that emerge from the qualitative data can be placed in the broad headings:- , not counting hours being a positive thing , the lack of hours introduces subjectivity into the process , very simple and easy to understand , very complicated , good support documentation easy to follow , support documentation too complicated and full of jargon , adds to the appraisal process , detracts from the appraisal process , measures the right thing , very difficult to achieve impact in day-to-day practice especially as a freelance GP The appraisers view 12 Context All appraisers that contributed to the pilot were volunteers (as were all the doctors). As part of the results it appears that significant extra time was required for some appraisers in order to complete the credit discussion. The geographical distribution of the pilot allowed the credit system to be trialled in multiple different appraisal systems. Results 90 appraisers completed the survey; just over 50% completed only one credit assessment. 20% completed two and 15% completed three, one appraiser completed eight, two completed six and three completed four and five. 55% of appraisers also work in medical education in a different role. One third participated in the pilot as an appraisee in addition to as an appraiser. 46.7% found the definition easy to understand with 25.5% finding it quite hard or hard. 82.2% felt the definition described the system in which they participated. 77.8% of appraisers found the definition of a CPD credit acceptable and 78.9% felt that self assessment was acceptable. 67.8% of appraisers found the verification at appraisal appropriate, the perceived effect that this verification had on appraisal is shown below:- Did the verification affect the appraisal process? Introduced an extra dimension (positive) in all appraisals: 31.3% 26 in most: 13.3% 11 in some: 18.1% 15 in a minority: 8.4% 7 in none: 21.7% 18 Additional 7.2% 6 comments: Stimulated a discussion focussed on the individual's development in all appraisals: 25.9% 22 in most: 17.6% 15 in some: 21.2% 18 in a minority: 10.6% 9 in none: 18.8% 16 Additional 5.9% 5 comments: Interfered with the process 13 in all appraisals: 22.1% 19 in most: 5.8% 5 in some: 14.0% 12 in a minority: 9.3% 8 in none: 37.2% 32 Additional 11.6% 10 comments: 85% of appraisers felt that the electronic documentation and help them verify the impact and challenge of the doctors CPD activity. When verifying the number of credits for an individual activity did you find assessing:- Impact Always easy: 0.0% 0 Mostly easy: 42.2% 38 Neither easy nor hard: 23.3% 21 Mostly hard: 28.9% 26 Always hard: 5.6% 5 Challenge Always easy: 1.1% 1 Mostly easy: 32.2% 29 Neither easy nor hard: 25.6% 23 Mostly hard: 36.7% 33 Always hard: 4.4% 4 Number of credits per activity Always easy: 0.0% 0 Mostly easy: 24.4% 22 Neither easy nor hard: 25.6% 23 Mostly hard: 42.2% 38 Always hard: 7.8% 7 When asked “Did the impact and challenge model apply easily to the type of CPD activity you normally discuss as an appraiser?” 73.3% responded yes, 67.8% felt that easily applied to the construction of the PDP. 14 When examining the application of the model, the appraisers were asked three questions. The first two related to the overall number of credits for the year:- Overall the total number of credits that doctors estimated were in harmony with my estimation Agree strongly: 5.6% 5 Agree: 52.2% 47 Neither agree nor 25.6% 23 disagree: Disagree: 13.3% 12 Disagree strongly: 3.3% 3 I saw some instances where my estimation of the credit total varied widely from the doctors overall total Yes: 38.9% 35 No: 61.1% 55 And the third to individual activities:- I saw some instances where my estimation of the credit varied widely from the doctor's estimation on individual items Yes: 50.0% 45 No: 50.0% 45 The ease-of-use of the verification process was examined using the following questions:- When assessing a single item for the impact were you able to classify it into the five point low to high impact scale? Yes always: 10.0% 9 Mostly: 56.7% 51 Half of the time: 22.2% 20 Rarely: 6.7% 6 No never: 4.4% 4 Were there any particular activities that you found hard to classify in terms of impact into the five point low to high impact scale? Yes: 55.6% 50 15 No: 44.4% 40 When thinking of ease of classification of impact -- please rank below Before the discussion Very easy: 3.3% 3 Easy: 16.7% 15 Neither easy nor hard: 44.4% 40 Hard: 28.9% 26 Very hard: 6.7% 6 At the discussion Very easy: 1.1% 1 Easy: 30.0% 27 Neither easy nor hard: 47.8% 43 Hard: 14.4% 13 Very hard: 6.7% 6 After the discussion Very easy: 2.2% 2 Easy: 32.2% 29 Neither easy nor hard: 43.3% 39 Hard: 16.7% 15 Very hard: 5.6% 5 When assessing a single item for the challenge were you able to classify it into the five point low to high challenge scale? Yes always: 6.7% 6 Mostly: 65.6% 59 Half of the time: 18.9% 17 Rarely: 6.7% 6 No never: 2.2% 2 Were there any particular activities that you found hard to classify in terms of challenge into the five point low to high challenge scale? Yes: 46.7% 42 16 No: 53.3% 48 When thinking of ease of classification of challenge -- please rank below Before the discussion Very easy: 3.3% 3 Easy: 13.3% 12 Neither easy nor hard: 46.7% 42 Hard: 30.0% 27 Very hard: 6.7% 6 At the discussion Very easy: 0.0% 0 Easy: 26.7% 24 Neither easy nor hard: 54.4% 49 Hard: 12.2% 11 Very hard: 6.7% 6 After the discussion Very easy: 1.1% 1 Easy: 28.9% 26 Neither easy nor hard: 51.1% 46 Hard: 12.2% 11 Very hard: 6.7% 6 The appraisers were asked to judge the coverage of subject matter by the doctors appraised:- Did doctors claim credits for subject matter that was appropriately diverse? Yes always: 22.2% 20 Mostly: 55.6% 50 Half of the time: 14.4% 13 Rarely: 3.3% 3 Never: 4.4% 4 The range of different activities presented:- Did doctors claim credits for a variety of activities? 17 Yes always: 36.7% 33 Mostly: 43.3% 39 Half of the time: 10.0% 9 Rarely: 5.6% 5 Never: 4.4% 4 The appraisers were asked for an overall rating of the system:- Overall please rank your feeling about the system where 1 is low and 10 is high Ease of use 1: 6.7% 6 2: 6.7% 6 3: 8.9% 8 4: 7.8% 7 5: 11.1% 10 6: 13.3% 12 7: 12.2% 11 8: 27.8% 25 9: 5.6% 5 10: 0.0% 0 Ranking Statistics Median rank: 6 Mean rank: 5.7 Acceptability 1: 7.8% 7 2: 6.7% 6 18 3: 11.1% 10 4: 8.9% 8 5: 15.6% 14 6: 14.4% 13 7: 8.9% 8 8: 21.1% 19 9: 5.6% 5 10: 0.0% 0 Median rank: 5.5 Mean rank: 5.4 Logical system 1: 7.8% 7 2: 1.1% 1 3: 7.8% 7 4: 8.9% 8 5: 22.2% 20 6: 12.2% 11 7: 13.3% 12 8: 14.4% 13 9: 11.1% 10 10: 1.1% 1 Median rank: 6 Mean rank: 5.7 Measures the "right" thing 1: 8.9% 8 2: 4.4% 4 19 3: 10.0% 9 4: 7.8% 7 5: 8.9% 8 6: 20.0% 18 7: 10.0% 9 8: 18.9% 17 9: 8.9% 8 10: 2.2% 2 Median rank: 6 Mean rank: 5.7 Overall rating 1: 8.9% 8 2: 5.6% 5 3: 5.6% 5 4: 5.6% 5 5: 18.9% 17 6: 16.7% 15 7: 13.3% 12 8: 20.0% 18 9: 5.6% 5 10: 0.0% 0 Median rank: 6 Mean rank: 5.6 The appraisers in addition were asked to rate their degree of comfort with the system with reference to the number of times the system was used:- How comfortable did you feel in using the system:- The first time Very comfortable: 3.3% 3 20 Comfortable: 27.8% 25 Neither comfortable 22.2% 20 nor uncomfortable: Uncomfortable: 34.4% 31 Very uncomfortable: 10.0% 9 Not applicable: 2.2% 2 After 2-3 appraisals Very comfortable: 4.4% 4 Comfortable: 16.7% 15 Neither comfortable 17.8% 16 nor uncomfortable: Uncomfortable: 5.6% 5 Very uncomfortable: 2.2% 2 Not applicable: 53.3% 48 For the fourth and subsequent times Very comfortable: 4.4% 4 Comfortable: 6.7% 6 Neither comfortable 2.2% 2 nor uncomfortable: Uncomfortable: 3.3% 3 Very uncomfortable: 1.1% 1 Not applicable: 82.2% 74 Finally they were asked to estimate the additional amount of time and effort expended:- How much extra time did the credits discussion add to the appraisal interview 21 Less than 10 minutes: 4.4% 4 10-20 minutes: 28.9% 26 20-30 minutes: 34.4% 31 30-40 minutes: 13.3% 12 40-50 minutes: 5.6% 5 50-60 minutes: 6.7% 6 over 60 minutes: 6.7% 6 Please think about the extra effort you have put in to each appraisal on this pilot - has it been:- Significant extra 27.8% 25 effort: A moderate amount: 54.4% 49 Some extra: 16.7% 15 Minimal extra effort: 1.1% 1 The reporting system also collected qualitative data. The full unedited report may be requested from Chris@pustule.Demon.co.uk (113 pages). Examples are reproduced below:- A move away from collecting certificates of attendance and totting up the hours to a more reflective way of how a learning experience impacts on the individual and others is a welcome move. Allows reflection on impact of learning event As a principle, it's great for the breadth which is general practice. The current scoring allows a huge amount of subjectivity from both parties. Maybe that's a good thing at its inception but perhaps unfair potentially in the longer term - hawks and doves Can't think of a better way and is superior to purely counting time I think that it is a useful device to move away from strictly hours based education. Impact on practice is a good measure of educational worth, though not the whole picture. I think this system is better than just counting up hours of educational activity Initial difficulty with the definition lessened over time and with use It moves away from the PGEA approach and gives value to learning (including less formal learning) which has an effect on day to day work It works but is vague and very subjective 22 Quite happy to use credits instead of number of hours The combination of challenge and impact is a useful one The idea of a 'personalised' credit related to the learning as it applies to the appraisee seems entirely appropriate for a system that aims to encourage personal development Again I think that the concept of credit is difficult to grasp. Why not divide the learning credits into hours of learning -which should be easy to do and give people an idea of how many hours they have done, that is what GPs are used to I remain unconvinced of the issue of challenge, I feel impact should probably be the sole criterion I understand the aspect of impact and challenge but the BASE unit is not clear i.e. what is 1 unit? e.g. 1hr or 1 day of activity , 1 lecture, 1 series of lectures. Like appraisal itself it will take a while to become accepted and bed in. Good reflective tool. Too complicated, and in practice has shown much room for disagreement between appraisee and appraiser It will penalise the doctor who is conscientious and produces large amounts of evidence or work through the year as it will add considerably to their time in keeping such a record and deciding the value of each piece of work they enclose. It will only be workable for people with minimal evidence in their folders I found it a very interesting part of the discussion - although there were no contentious ones in my cohort - it actually helped me , as the appraiser to realise the true significance of one doctor's learning that was involved in many aspects of primary care that I have had no experience of. It should be verified separately and preferably before the appraisal process. It needs separate funding and should not just be bolted as it will diminish the appraisal. A good method of discussing and planning PDP A general feeling of 'oh god, yet another new thing to master' 'yet another bloody hoop' 23 The themes that emerged from the qualitative data supplied by the appraisers were broadly:- , an appropriate and useful addition to the appraisal process , concerns over making judgements , time-consuming , not counting hours being a positive thing , concerns over altering the appraisal dynamic , encourages reflection , some concern over “ordinary” GPs struggling with the system Report on pilot objectives The pilot objectives were made explicit in the pilot project plan (appendix 1) and were to answer the following questions:- o Is this definition of a credit acceptable? o Is the system easy to understand and use? o Are GPs able to produce evidence easily? o Are the examples of credits self-accredited justifiable? o Are appraisers easily able to verify an individual’s credits in terms of challenge or impact? o What if an appraiser disagrees with the doctor? o Are appraisers comfortable with this system? o Are GPs comfortable with this system? o Are we seeing diversity of subject? o Are we seeing diversity of method? o Is this an appropriate system for all GPs (sessional, OOH, overseas)? o Are there further training issues for GPs or appraisers? o What are the local resource issues of the system? Is this definition of a credit acceptable? 82% of appraisees and 77.8% of appraisers found the definition acceptable. Is the system easy to understand and use? 24 Slightly more than half of appraisees and slightly less than half of appraisers found the definition easy to understand. The accompanying paperwork was rated by about 85% of all doctors in the survey as being helpful. Over a third of those appraised and nearly half of the appraisers found assessing credits hard. The overall ranking for ease of use was 6 and 5.7 on a scale of 1 to 10 by the appraised doctors and the appraisers respectively. Are GPs able to produce evidence easily? Only 70.7% of the appraisees in the pilot felt that the system fitted in with their traditional CPD activities. 76.3% felt that it could fit in with their future activity. 73.3% of appraisers felt the traditional CPD activities fit easily into the system and 67.8% felt that the system could be used in constructing a PDP. Are the examples of credits self-accredited justifiable? 90% of the doctors felt that the appraiser had more or less agreed with them regarding their credit estimation. This high ranking was reflected in the appraisers rating around this subject. Are appraisers easily able to verify an individual’s credits in terms of challenge or impact? 66% of appraisers felt that they were able to classify impact on the five point scale, there were however 55% that had difficulties with one or more item. Around a quarter felt that even after discussion classifying impact was hard. The classification of challenge produced similar numbers. A half of all appraisers had a situation where their credit estimation varied widely from the doctor's estimation on an individual item. And over a third had seen their overall credit estimation vary from that of the doctor’s. On this last point it is clear from the freehand comments that in most instances the doctor underestimates the number of credits that the appraiser feels an activity would attract. What if an appraiser disagrees with the doctor? The evidence mainly from free text entries is that doctors tend to underestimate the credits. The following quotes are from the 10.3% of doctors who felt their appraiser did not verify their credits 50:50. I tended to underestimate the credit. Particularly the time needed [as lead] to produce a good SEA, drawing on educational skills from course organise role. Actually very difficult to do.. easy for the small fry.. but the whales are more difficult... how can you quantify a very high challenge task with an indefinable impact? He felt that I had not claimed enough credits and pointed out large amounts of work which I had not considered educationally exceptional which he felt were worthy of inclusion in the assessment 25 process— i.e. multiple parts of the appraisal documents that I had filled in- i.e. significant events audits etc not only the courses I was counting My appraiser reduced every credit that we looked at because her perception of an educational challenge was more related to intellectual challenge rather than the factors I listed in Q9.We didn’t have time to look at the last bit of my list-SEAS PUNS and DENS and reading. My appraiser made me think I had over estimated the value of all the work I had done. If I could be assured that my estimates were acceptable I would be happy with the system. Invariably there will be disagreements as appraisers are trained to look for certain things & follow guidelines. As explained above it erodes further at my self respect, amongst other things. I don't trust my peers to be reliably and consistently objective enough. These comments were in the minority, it seems that in the main negotiation and agreement took place rather than confrontation. Are appraisers comfortable with this system? Nearly half of the appraisers were uncomfortable using the system the first time. This high proportion quickly dropped with second and subsequent appraisals. The numbers that performed more than two appraisals in the system are so low it may be that this data is unreliable. It is evident that there is a degree of disquiet amongst the appraisers using this system. Are GPs comfortable with this system? The appraised doctors gave a mean score of 6.4 on a 10 point scale when asked about acceptability of the system. Are we seeing diversity of subject? 77.8% of the appraisers felt that the diversity of subject was present mostly or always. Only 7.7% felt diversity was they rarely or never. Are we seeing diversity of method? 80% of appraisers felt there was diversity of method in all or most that they appraised. 10% felt it was rarely or never present. 26 Is this an appropriate system for all GPs (sessional, OOH, overseas)? This question has remained unanswered from the results of the pilot. The group of freelance GPs has yielded some differing results from GPs in partnerships; the numbers involved though are small. It is clear that there are concerns over the ability to show impact amongst freelance GPs. The overall rating for the system is lower than that by partners. Are there further training issues for GPs or appraisers? A number of themes emerged from the pilot, firstly that there were great differences in the information was supplied to both the appraisers and the doctors. Some received face-to-face training others received the paperwork associated with the pilot and a small group received neither. When asked directly if further training was required, again there were mixed responses. Many felt that using the system was all the training was required, others felt that they were great training needs particularly for those that had not been involved in the pilot. What are the local resource issues of the system? The regional leads in the pilot areas all completed the feedback questionnaire. Full results of this are available on request (10 pages). Themes that emerged from the feedback from the regional leads echoed the results seen with reference to acceptability of the system. There were however significant resource implications highlighted, training for GPs, appraisers and administrative support were all mentioned as requirements for rollout. Different approaches were taken in different regions; one region had great success with a PCT wide GP meeting. It was commented that this meeting attracted the highest attendance of all previous meetings. Several comments mentioned using the opportunity to get the revalidation agenda to the forefront. Many of the regional leads mention resistance from both GPs and appraisers to the process and yet other areas found that appraisers were enthusiastic, one area recruited all eligible GPs and the majority completed. Much of the feedback mentions the need for support material both physical and online. Most regions recognised that there was a need for systematic appraiser training. One region expended significant administrative time due to the large number of PCTs within that region. The resource implication of any credit-based system is likely to be large. Whereas the regions within the pilot were volunteers cognizance needs to be taken of the impact this may have locally when full roll-out occurs. Conclusions 27 The pilot has delivered on the numbers of participants and has stayed within timeframe and budget. The stated objectives of the pilot had been met however there are mixed messages about the system. Clear themes have emerged which must be utilised in refining a CPD credits system. There is overall general acceptance of the system but with a significant minority opposed to it. In general using impact and challenge as factors in determining credit received high ranking and positive freehand feedback. The absence of hours was welcomed by most, there were however a number of comments that hours should be reintroduced to counter the subjectivity seen in this system. Two main negative themes emerged, the complexity and the subjectivity. Many felt that the documentation and the system was too complex, the subjectivity created disquiet amongst both doctors and appraises when they were discussing the credit value of an individual development. Refinements will need to be made to the system prior to roll out. The system will need to be simplified, subjectivity will need to be addressed and benchmarking will need to continue. Appendix 1 -the impact and challenge model of CPD credits 28 CPD credits pilot Impact and Challenge model August 2008 Christopher Price RCGP CPD Fellow 29 1. What is a credit? The Academy of Royal Colleges has a consensus view that every doctor should demonstrate a minimum of 50 CPD credits in a year and 250 CPD credits in a five year cycle to support a positive revalidation decision. This has been embraced as RCGP policy as part of our managed CPD strategy. The RCGP is to pilot a credit system between autumn 2008 and late spring 2009 based on the following statements: “A credit is a unit of professional development which is a product of the impact of a developmental activity and to a lesser extent the challenge involved in its completion.” “Credits are self assessed and verified at appraisal.” Impact in this context may include , Impact on patients (e.g. a change in practice, initiating a new drug – this has obvious overlaps with personal development) , Impact on the individual (personal development) , Impact on service (e.g. becoming a training practice, teaching others, implementing a clinic system) Challenge in this context may be , Context related (e.g. more challenging to become a new training practice than a trainer in an established training practice) , Related to circumstances (e.g. a sessional GP undertaking audit is often faced with problems around the data and follow up) , Related to personal ability (e.g. personal disability, prior skills, prior experience etc.) , Related to effort expended (e.g. attending an ophthalmology clinic for a whole day 40 miles away to gain experience) 30 2. The credits system and Revalidation This approach to Continuing Professional Development is also endorsed in the Report of the Chief Medical Officer for England's Working Group, Medical revalidation - principles and next 1 stepsas follows “6.3 It will be desirable to increase the linkage between CPD and appraisal. Appraisal focuses on meeting agreed educational objectives. Monitored systems which define College or Faculty approved educational activities may assist the meeting of those objectives. Presently most College or Faculty schemes are based on acquiring credits. The advantage of this system is that the time devoted to CPD can be measured and recorded. The disadvantage is that it is insensitive to the quality and relevance of the various CPD activities. The more that credits can encompass the value of the learning and not simply the time spent engaged in CPD, the more it will be valued by doctors and the better a measure it will be of their CPD activities. 6.4 Effective CPD schemes are flexible and largely based on self-evaluation. This lets doctors develop what they do in the context of their individual professional practice while providing evidence for external scrutiny. There is no single correct way of doing CPD. The methods chosen will depend on spheres of practice, learning styles and personal preference. 6.5 The principles underpinning CPD schemes therefore need to be as simple as possible while providing a good foundation on which to build an appropriate portfolio unique to the individual doctor.” The important points within these three sections are: , Increasing the linkage between CPD and appraisal , Credits being based on the value of the learning , Flexibility of CPD schemes , Developing doctors in the context of their professional practice , Evidence for external scrutiny , Multiple methods appropriate , Building a portfolio The proposed Impact and Challenge model addresses all of these points in a simple to administer scheme where the evidence is presented at appraisal. The working group report firmly intertwines relicensing and recertification as a single process and sets appraisal at the heart of it; it seems logical therefore to include the credits system into the appraisal process. 31 3. Impact and challenge in the appraisal year The development for an individual in an appraisal year should involve the following 4 steps: Appraisal Discussion Recording of development and Construction of PDP reflection on the impact Developmental activity either derived from PDP or from other sources Impact and challenge fit into this year in the following model: Discussion of developmental impact Setting challenge and impact goals Appraisal Discussion Recording of development and Construction of PDP reflection on the impact Impact Developmental activity either derived from PDP or from other sources Challenge 32 4. Relative value of Impact and Challenge In the RCGP scheme, the impact of an activity will have greater weight than the challenge. This relative weighting will: , Encourage activities that produce positive change, both for individuals and the system, for the benefit of patients. , Encourage reflection; an individual claiming higher impact will need to reflect on how the learning process has affected the final outcome. , Reward quality of CPD. The system could not entirely rely on the impact of activity as there needs to be recognition of the differing situations in which doctors practice as GPs and the diversity of the generalist. Some GPs have highly developed knowledge in specialist areas and others have remote or difficult working environments. The recognition of challenge in producing an impact will to some extent redress this balance and allows an individual’s working pattern to be recognised by the appraiser in confirming the self apportioned credits. The relative weighting of impact and challenge are illustrated in table 1: Table 1 1-2 Credits 2-4 Credits 3-6 Credits 4-8 Credits 5-10 + Credits Example 1 1-3 Credits 2-5 3-7 5-10 6-12 + Credits Example 2 2-4 Credits 3-6 4-8 6-12 8-15 + Credits Example 3 Example 4 3-5 Credits 4-7 5-11 7-15 + 10-20 + Credits 4-6 Credits 5-10 6-14 + 10-20 + 33 Examples for Interpreting the Credits Table Example 1 I have been involved in examining prescribing in two areas on behalf of the practice. We share the workload on this issue and I have reflected my role in my form 3 documentation. I looked at the pros and cons of two drug switches (on purely cost grounds) the short documents I produced are available in my form 3. In both cases the savings a switch would have produced are more than outweighed by the impact on the patients and the practice. I reflect in my form three that perhaps this activity has “run its day” – Claim 1 credit Example 2 New indication for antibiotic prescribing in splenectomised patients – we have 4 – I have examined the records to ensure we are following the guidelines – we are – this is of some importance to this small group of patients (written up in form 3 evidence) – claim 2 credits Example 3 Gave a presentation to 25 GPs on minor surgery techniques – well received and in particular a number of them indicated that they would change their practice following the lecture – this has an impact on me (in that I am keener to provide more sessions) and hopefully on the service (see form 3) – claim 4 credits Example 4 Used the Essential General Practice on line module on Chronic fatigue syndrome/myalgic encephalomyelitis (or encephalopathy) – this was focused on the NICE guidance – firstly this has helped me in understanding at least four of my personal patients problems much better. I have had a very productive consultation with one of them and realized (along with the patient) that fluoxetine is not helping and we have agreed withdrawal. I used the suggested exercise to “check out what I did in my own practice” and found that the four cases I could identify all had significant periods of misdiagnosis (usually depression) and probably inappropriate treatment before the penny dropped. I have further evidence in my form 3 to show the significance of the change at a personal level and the reduction in prescribing of anti depressants in this group – claim 8 credits Example 5 Have been approved as a trainer and our practice has become a training practice for the first time – see form 3 highlighting the changes we need to make within the practice and the process of convincing two of my partners of the value of this – we all seem to be excited with the prospect of our first registrar this August – claim 50 credits More extensive examples are given in appendix 1 34 5. Defining impact and challenge Table 1 arbitrarily divides impact and challenge into five categories. Guidance is given below. The examples are neither exhaustive nor prescriptive, credits are self assessed. Table 2: Impact Low , Mainly confirming current practice impact , Little change necessary within the practice , No examination of current practice (e.g. data collection) , Knowledge gained is minimal or of low value , Mainly for personal benefit , Anything that does not reach a higher level Minor , Confirming current practice although new knowledge acquired which aids understanding impact , Some change in practice required (but not necessarily followed through systematically) , May involve others (e.g. discussion on new NICE guidance at practice meeting) but probably falls short of changing practice protocols , Initial data collection for audit discussed but change not yet evaluated , Minor audit (few patients, minimal change and low level gain) Moderate , Demonstrating current practice against accepted best practice (e.g. completed audit cycle) impact , Change in practice in response to new information (e.g. essential general practice – followed through to examining own practice) , Would usually involve others (e.g. practice protocol, presenting audit data or implementing change) , Teaching session that demonstrates a change in the learners through evaluation , Working with organisations to influence change in others (e.g. PCO guideline development) , Becoming a trainer in a well established training practice Significant , Major change in practice involving an important condition. This should be in response to a change in the accepted evidence (e.g. the use of atenolol in treating uncomplicated hypertension – re designing the impact practice protocol and reviewing patients taking atenolol considering a switch) , Influencing others to change in response to new evidence either through (evaluated) teaching or through guideline and protocol development on a regional basis , Introducing a new service for patients (e.g. starting a monitoring system for DMARDS / Warfarin, starting a minor surgery clinic from scratch) , Introducing a new service to your team (e.g. a new palliative care team, an “intermediate care” team) , Becoming a trainer to fill the gap left by the retirement of the only other trainer in the practice High , Anything the individual feels is of higher impact than the lower levels impact , Major change in the practice (e.g. becoming a new training practice, becoming a research practice within a recognised research network etc.) , Major contribution or lead on projects that change or confirm professional practice. This would be at a regional or national level , Personal development to implement a new service in practice (e.g. using a recognised scheme to gain a skill and then set up a service – RCGP certificate in substance misuse – new clinic in practice – possibly recognised as a GPwSI) 35 Table 3: Challenge Low , Easily available challenge , Passive learning (e.g. lecture with little or no interaction) , No self testing (e.g. on line module without knowledge test) , Occurs without planning (e.g. at practice based meeting with no planning or prior effort by the individual) , Probably not part of PDP , General untargeted reading (e.g. reading BMJ every week) , Anything that does not reach a higher level Minor , Some planning involved – either as a result of the PDP or in response to an identified need (e.g. a patient encounter) challenge , Learning involves the individual (e.g. if it’s a meeting it would be mainly workshop style or targeted reading – topic covered by more than one article or by reading of nice guidance etc.) , Learning has not been applied to practice/patients/self yet , May be a degree of self testing but no standard needs to be reached Moderate , Planned learning - either as a result of the PDP or in response to an identified need (e.g. a patient encounter) challenge , Learning is focussed on the individual (either self directed, practice based or interactive facilitated style) , There is a method of self testing to which standards apply (e.g. on-line MCQ with pass mark, data collection of performance or reflection on change present) , The learning although part of a planned needs driven activity involves a degree of difficulty in the organisational sense (e.g. attending ophthalmology outpatients for a day to fulfil a learning plan) Significant , Planned learning involving an organised literature search – multiple sources identified challenge , PDP based mainly – may involve learning then audit of the topic , Systematic learning focussed on a topic and/or disease entity using a number of different learning methods (e.g. attend meeting on diabetes, complete an on-line module with a ranked MCQ and either changed protocol within practice or performed audit) , PDP based unusual topic requiring unusual effort to fulfil need (e.g. doctor is a mountain rescue worker and there is an annual national meeting 400 miles away which is a requirement to maintain registration) , Activity made unusually challenging due to individual’s working circumstance (e.g. audit is sometimes difficult for sessional GPs without a regular practice commitment) High , Anything the individual feels is of higher challenge than the lower levels challenge , PDP based or needs based activity systematically exploring the subject, almost certainly involving multiple learning methods with either an external method of assessment (exam, award, publication, change in status becoming a GPwSI or trainer etc.) , PDP based or needs based activity – the individual has identified a system change. Systematic implementation of evidence based practice. (e.g. taken over asthma clinic, re written protocol along new NICE guidelines, 8 criterion audit performed) , Academic award (e.g. diploma/certificate) 36 6. Roles The role of the doctor The individual GP being appraised in this pilot should examine the CPD activity they would normally bring to the appraisal discussion and decide on the resultant impact of that activity using tables 2 & 3 above as guidance. The challenge involved in that activity should also be estimated. Using the two descriptors for impact and challenge the doctor should estimate the credit claim using table 1. For the purposes of the pilot we are not asking doctors to change their CPD activity but simply to record an estimate of the number of credits under the definitions above. Some doctors will have significantly more than 50 credits and one of the outcomes of the project will be to examine the current range of “credit” activity. Some doctors will have less than 50 credits and for the purposes of the pilot this shortfall is an equally valid measure. Individuals with less than 50 credits should remember that this is a new and different way of examining development and that their activity planned and/or executed for the year was not based on this recording system. All doctors involved in the pilot will be asked to fill in a short on-line survey and to allow their records of credits (anonymised) to be examined and amalgamated with the other doctors from their region. The role of the appraiser The appraiser should verify that the credits claimed by the doctor being appraised are a reasonable estimate of the impact and challenge. The appraiser will have examined the form 3 evidence for appraisal and will have the opportunity to discuss in further depth any aspects of the credits claimed that need clarification. There are three scenarios that should lead to further discussion , The appraiser feels that overall the doctor has claimed too many credits , The appraiser feels that overall the doctor has claimed too few credits , The appraiser feels that the numbers of credits claimed for individual items bear little relation to the impact and challenge mode In any of the three scenarios the appraiser should ask the doctor being appraised to indicate on table 2 the level of impact the doctor feels the activity produced and by cross referencing the challenge level should plot the position on table 1. The number of credits claimed should normally fall within the range stated but there may be exceptional circumstances that need to be taken into account. If at the end of the appraisal discussion consensus is not reached this fact should be recorded on the credits table (appendix 2). However, it is for the doctor being appraised to assess the number of credits and for the appraiser to verify those that they can (which could be equal to, higher or lower than the claimed credits) 37 All appraisers taking part in the pilot will be required to complete a short on-line survey which will ask them to record their experiences of the system – these responses will be anonymised and utilised to inform the project report. The appraiser’s role in helping the doctor develop a PDP for the subsequent year is vital and 2. In the context of the credits is covered to a great extent in Developing guidance of the PDP system the appraiser may take on a number of roles helping the doctor ensure that: , The PDP is a reflection of true developmental need in addition to wants , The PDP contains items of sufficient impact to allow credits to be accumulated , The PDP allows the doctor to examine a wide range of subjects over a five year cycle and does not become a narrow “interests based” plan , The PDP allows the use of different learning and implementation tools that are appropriate to maximise the impact of the doctor’s development The role of the tutor The role of the tutor and appraiser in the pilot may vary from site to site and indeed there may be a number of models running in parallel within a project site. The possible roles of the GP tutor are listed below – it will be vital to understand the role adopted by each site to ensure that the correct data is collected. , The tutor in a wholly supporting role. The regional lead may wish to utilise the GP tutor network to publicise the project and help in the training of the doctors involved in the project. In this model the appraisal discussion remains the focus of verification of credits however the tutor will have a role in helping doctors convert their CPD activity into the impact and challenge model of credits. , The tutor in a mixed role. In this situation the tutor may have the role described above and in addition will interview some of the doctors who wish to take part in the pilot but whose appraisal discussion is scheduled to take place outside the timeframe of the credits pilot. , The tutor in the verifying role. In this situation the tutor is conducting credit verification outside the local appraisal system as the regional lead has not been able to negotiate the inclusion of this process in the appraisal discussion. The value of including the local tutor network is recognised within the project however to avoid duplication of effort the appraisal model is preferred if there is a choice. The involvement of the tutor as described in the first bullet point above is the role envisaged in the final scheme with credits being discussed and verified at appraisal and the tutor facilitating the production of the evidence by the individual doctor. The role of the tutor in the two lower scenarios will obviously include the verification of the doctor’s credit claim and will also require the completion of a short on-line survey. 38 7. Conclusion The impact and challenge model described is to be the subject of a pilot and will be examined in terms of its acceptability, feasibility and practicability. This paper describes the reasons behind the system and its advantages in terms of measuring CPD activity in positive outcome based units or credits. Examples are given and the system will be refined as appropriate drawing on the experiences of the participants in the pilots. References 1. Department of Health, Medical revalidation - principles and next steps, London, July 2008 Guidance/DH_086430 2. Rughani A, Field N, Holmes S, Howard J, Lane P. Developing guidance of the PDP. London: RCGP Professional Development Board; 2007 39 Appendix 1 Examples of credit estimation These examples are neither exhaustive nor prescriptive, as with the tables above these should be used as guides enabling GPs to self-assess the credit score. Audit Audit now occurs in every General Practice in the UK as part of the QOF target system. It is inappropriate to claim credits for the process of data collection or QOF achievements. The process of improving or maintaining QOF points is of course a quality exercise in itself that has impact and would be eligible. Audit outside QOF may also qualify for credits, it must be remembered however that it’s the impact that leads to the credit not the process of audit. Distance Learning (including on-line) Many on-line and distance learning packs have a number of “hours” attached. Hours do not equate to credits. The credit value is related to the impact of the exercise – for example a pack given “4 hours approval” may contain little of value and merely confirms current practice, no changes are necessary within the individual’s practice and there is no self examination of current practice – this would be low impact and low to minor challenge. Another pack has “1 hour approval” but within it there is a major piece of new information that changes the individual’s practice – this would be minor to moderate impact and potentially the credits claimed would at least equal the first scenario. Meetings As with distance or on-line learning the impact of knowledge gained at meetings should be measured. Practice based colleagues are often the source of nuggets of information that change what the individual does. More formal meetings are excellent resources for up to date information and professional interaction although sometimes the impact on practice may vary from doctor to doctor in the same meeting. Practical Skills Practical skills are as important as knowledge in some areas of an individual’s practice. The demonstration of acquisition or mastery of a new skill can be used in the credit system. It is not just new skills that may be used in the system. For instance teaching others practical skills has some impact and certainly examining your own results either through audit or other markers is a legitimate exercise (% diagnosis of skin lesions subsequently proved correct on histology, or % complete excision of BCC). 40 Practice Developments Building a new surgery or buying an expensive piece of equipment often involves doctors in a managerial role, there may be the opportunity to gain some credits but only for development – for example managing the transition from old practice premises to new would present a challenge and patients presumably would benefit, however the credit claim would be related to the learning involved in management. Statements like “I discovered new ways to motivate the team” or “this process far from causing conflict has engendered a team spirit” would demonstrate change. Developing a new service (e.g. Insulin initiation) would certainly have impact and if this were measured by data collection or audit the impact could be shown to be significant. Taking on a new role, with new responsibilities could involve development (e.g. leading on the staff appraisal system, becoming the finance partner), the doctor should reflect on the changes to estimate credits. Puns and Dens (patient case reports and clinical experiences) Patients are a rich source of learning opportunities; most will be familiar with Richard Eve’s model of PUNS and DENS (see BMJ leaning module). PUNS and DENS relies on the doctor having a need. Patient reports and experiences can be used as a narrative to demonstrate good practice, highlight a good experience or use a bad experience to examine the need for change. A quote from 1905 (Cabot RC, Locke EA Boston Med Surg J. 1905;153:461-5.) is as true today as it was then :- "Learning medicine is not fundamentally different from learninganything else. If one had 100 hours in which to learn to ridea horse or to speak in public, one might profitably spend perhapsan hour (in divided doses) in being told how to do it, fourhours in watching a teacher do it, and the remaining 95 hoursin practice, at first with close supervision, later under generaloversight." Recording what happens in a consultation (or case study) would be in the “general oversight” category and demonstrating that an individual was; using best practice, dealing with problems appropriately, responding to emergencies, dealing with difficult patients, up to date with palliative care; using the BTS/SIGN guidance etc. The unusual presentation, the rare condition, the referral on instinct that turns out to be significant, the wrong word that changed the consultation, the last extra of the day with rectal bleeding or similar scenarios provoke thought, reflection and action all of which may have impact on future behaviour. This learning by experience or from an anecdote from others often goes unrecognised; the impact associated with this day to day learning can be converted into credits. Reading Reading can be arbitrarily divided into structured and unstructured, both have merit. An example of structured reading would be researching a condition – for example the use of the latest 41 hypoglycaemic medication – this is likely to be of relatively high impact as compared to the challenge (as it is targeted). Unstructured would be reading every issue of a journal, the individual is likely to gain some impactful information but the effort (or challenge) expended is relatively higher than in the first example. In both cases the reflection on the impact of this activity is the important thing when assessing the credits. Significant Events Significant events as a learning tool have gained widespread acceptance. Adverse events or near misses can be used to address system or personal issues however positive significant events can be used to demonstrate impact and learning. An early diagnosis, dealing with an emergency, a medication review leading to significant improvements in a patient’s well being are all positive examples that can be shared with the team as learning points and can attract credits. Structured Learning (including certificates/diplomas etc) Structured learning within the auspices of a higher education institution can also be used for credits. The planning and structured nature of the course is likely to rank highly on the challenge scale. The learning toward a goal (and possibly an assessment) may add to that. Impact will probably also be toward the upper end of the scale as presumably this will have been planned for a service or personal reason. Surveys (patient, 360’) The impact of surveys will vary from individual to individual. There may be few learning points that can be gleaned from the exercise, or the feedback may include factors beyond the control of the individual. There may be instances however where changes are required and those changes when made have an impact on the way a GP works. The reflection on the results and subsequent changes are the areas to examine when judging the impact. 42 Appendix 2 Standard credit claim form The standard credit claim form should not contain the evidence for an activity, this should be presented elsewhere (in the appraisal folder or to the tutor). The examples below are sufficient for the purposes of the pilot. 4 Attendance and Audit in N Practice use of outcome – use supporting BB and of evidence for spironolactone spironolactone appraisal low – patients and beta reviewed and blockers in appropriate heart failure changes made audit 2 Participation 2 SEA Y Last year at and reflection documents appraisal on SEA system involving self in discussed SEA in practice appraisal system in documentation practice and and evidence have taken lead that practice in redesign and has SEA system implementation of new system 43 Appendix 3 FAQs I use the table to estimate the credits allocated to an activity and I find that the number of credits suggested is well below my estimation. What should I do? In this situation the first thing you should consider is the impact of your activity, if you feel the impact is worthy of higher credit then rate the activity to the level you feel you can justify. In this situation you should list your reasons in your form 3. If you feel the challenge was higher than the definitions given then you may feel that the justification in form 3 will cover this difference. You should however remember that impact is weighted more highly than challenge. What if my credit total is less than 50? This is a pilot and it has no bearing on you. Your development over the pilot year was not focussed to aim at a target of 50 credits. If this system is introduced you will have sufficient notice to tailor your development if that is required. What if my credit total is more than 50? Many doctors will be able to demonstrate more than 50 credits of development – just as many doctors bring excellent work to appraisal Can I fail in this system? No – this is a pilot; however the system may become adopted as part of the evidence required for revalidation What if the appraiser disagrees with my estimation of credits? This is one of the things that the pilot is set up to measure – it is unlikely there will be universal agreement and although this will have no punitive effect it is vital that such disagreements are recorded in the surveys How can I contribute to the development of this system? By participating you already will have. The surveys are vitally important in shaping the future and completion of the appropriate survey in a frank and open manner will be invaluable 44 Appendix 4 Scenarios 1. The doctor and the appraiser have different views on the number of credits:- Dr Smith has claimed 85 credits and when his appraiser examines the record the overall total seems to them to be more like 50. The appraiser feels that this may lead to conflict within the discussion and seeks advice from the regional lead. He is advised to go ahead with the discussion and to examine four of the individual items in some detail. Dr Smith’s appraisal discussion seems to go well; the appraiser initiates the credit discussion by highlighting the four individual items and asking Dr Smith about the general process and how he arrived at the numbers of credit per item. Using the quick reference guide it is clear to both the appraiser and Dr Smith that two of the four items rank lower on impact than initially thought. Dr Smith had given a higher credit rating because of the challenge involved. The appraiser verifies 65 credits and Dr Smith is happy. The appraiser completes an incident report survey and Dr Smith discusses the issues on his survey. 2. An individual item is given a seemingly high credit rating:- Dr Jones has claimed 63 credits overall and her appraiser is able to verify most of them. There is one item, an audit of ace inhibitor prescribing, for which 12 credits have been self assessed. The appraiser highlights the item and agrees that the audit cycle has been completed and that substantial numbers were involved. Dr Jones then describes the circumstances – during this audit she had identified one patient who had been prescribed an ace inhibitor and whose renal function had deteriorated. Dr Jones was involved in the patient’s care and had felt that the renal function was deteriorating due to diabetes. Once medication was rationalised (stopped ACE and NSAID) the renal function and the patient’s condition improved markedly. This single case had a profound effect on Dr Jones who had changed her practice and was meticulous about ACE prescribing and monitoring of renal function, and was far more aware of drug interactions. This audit had changed her practice. The appraiser agreed that the audit, significant event, change in practice and learning experience was a significant development and verified the credits. 3. There is no agreement Dr Thomas and his appraiser cannot agree on the credit rating of his CPD. His activities don’t seem to fit into the descriptors on the easy reference forms and for some activities he has added a few credits because “it was hard work”. The appraiser remembered that “Credits are self assessed and verified at appraisal” and verified about 2/3rds of the credits. Dr Thomas was not happy with this, the appraiser pointed out that this is only a pilot scheme and that Dr Thomas now had the opportunity to shape the outcome by completion of the survey, furthermore the appraiser would also feedback that there was some 45 difficulty in this rating. Dr Thomas was assured that as this is a pilot there are no consequences of not getting 50 credits and that more work would be done on the system after the pilot. 46 Appendix 2 - credits project plan RCGP CPD Credits Project plan Christopher Price RCGP CPD Fellow 47 Introduction The Academy of Royal Colleges has a consensus view that every doctor should demonstrate a minimum of 50 CPD credits and 250 CPD credits in a five year cycle to support a positive revalidation decision. The RCGP is to pilot a credit system between Autumn 2008 and late Spring 2009 based on the following statements:- A credit is a unit of professional development which is a product of the impact of a developmental activity and to a lesser extent the challenge involved in its completion. Credits are self assessed and verified at appraisal. There are at least seven geographical areas that have expressed an interest in participating in the pilot and these represent England, Wales, Northern Ireland and Scotland as well as interest being expressed on behalf of overseas members. Project purpose The project has been commissioned to test the proposed Impact and Challenge model and to answer the following questions:- o Is this definition of a credit acceptable? o Is the system easy to understand and use? o Are GPs able to produce evidence easily? o Are the examples of credits self-accredited justifiable? o Are appraisers easily able to verify an individual’s credits in terms of challenge or impact? o What if an appraiser disagrees with the doctor? o Are appraisers comfortable with this system? o Are GPs comfortable with this system? o Are we seeing diversity of subject? o Are we seeing diversity of method? o Is this an appropriate system for all GPs (sessional, OOH, overseas)? o Are there further training issues for GPs or appraisers? o What are the local resource issues of the system? 48 Project Personnel The RCGP CPD fellow (CP) will lead on all national aspects of the project, he will be responsible for the design of the project, production of project literature and troubleshooting during the run-time of the project. The evaluation of data collected and the production of a report for the RCGP will be his responsibility and he will report directly to the PDB through the chair. Regional Leads. A single regional lead (RL) will be identified for each geographical project area. The RL will be responsible for the local conduct of the pilot including recruitment. RLs will meet together prior to project launch and will co-ordinate training (in conjunction with CP) and dissemination of the project information. The RL will also collate local information on content of credit claims and will amalgamate and annonymise the information and feed it back once complete to CP. Appraisers will need to be recruited locally by the RL. They will be supplied with training material and will be asked to verify credits self-assessed. They will be required to complete an on-line survey at the completion of the project. Doctors due to undergo appraisal during the project-run time will need to be recruited and supplied with training material through the RL in conjunction with CP. They will need to share their “credit claim” with the RL and agree to the anonymised data being used for evaluation purposes. The doctor will be required to complete an on-line questionnaire on completion of appraisal. Tutors. It is recognised that some of the regions in which the project will run do not have the co-operation of, or indeed access to, appraisers to contribute to the pilot. Furthermore there may be a number of doctors that wish to contribute to the pilot whose appraisal discussion is scheduled outside the project timeframe. It may be possible in these situations for a meeting to occur between the doctor and a local tutor specifically to participate in the credits pilot – different documentation is being developed for this eventuality. Administrative support This term refers to the support that will be required centrally for the project to run successfully. This is likely to be a “moveable feast” – the internal requirements at the RCGP are dealt with by an internal document. CP will provide an email support system for the RLs however any local administration support will be a local issue. 49 Project design The project is intended to run in geographical areas within the UK. It will run for six months and report within eight months of launch. The credit based system will be used as part of the normal appraisal process in the pilot areas and the appraisers and doctors involved will complete on-line questionnaires which will in part inform the project outcomes. To ensure sufficient numbers for analysis each region will be asked to recruit a minimum of 20 doctors with no maximum. This should ensure a sample size of 150+ which would provide sufficient data to allow analysis and reporting. The first flow chart below illustrates the information flow within the roll out phase of the project, the second illustrates the information gathering during the project proper. 50 CP Central planning meeting RL BOS (on-line) survey Regional recruitment RL If required CP will Appraiser be involved Regional training GP Appraiser GP Appraisal discussion Standard credit claim form 51 CP Central planning meeting RL BOS (on-line) survey Regional recruitment RL If required CP will Appraiser be involved Regional training GP Appraiser GP Appraisal discussion To RL for collation and anonymisation Standard credit claim form 52 The regional lead is vital in ensuring the project runs well on the ground, while the appraisers and GPs need to be well informed regarding the purpose and components of the scheme. Project timeline To April / May 2009 June/July 2009 July- September September – 2008 November 2008 Recruitment of regions, recruitment of appraisers and GPs. Pilot launch Pilot Runs Analysis Report How will the project answer the questions posed? , Is this definition of a credit acceptable? This will be a subject of the on-line (BOS) survey, there will be three distinct surveys one for the RL one for the appraisers and on for the GPs. The acceptability will be a standard question in all 3 surveys , Is the system easy to understand and use? BOS survey , Are GPs able to produce evidence easily? BOS survey coupled with the analysis of the standard credit claim form 53 , Are the examples of credits self-accredited justifiable? BOS survey coupled with the analysis of the standard credit claim form , Are appraisers easily able to verify an individual’s credits in terms of challenge or impact? BOS survey coupled with the analysis of the standard credit claim form , What if an appraiser disagrees with the doctor? BOS survey coupled with the analysis of the standard credit claim form , Are appraisers comfortable with this system? BOS survey , Are GPs comfortable with this system? BOS survey , Are we seeing diversity of subject? BOS survey coupled with the analysis of the standard credit claim form although this may be beyond the scope of the pilot as it is only one appraisal cycle , Are we seeing diversity of method? BOS survey coupled with the analysis of the standard credit claim form , Is this an appropriate system for all GPs (sessional, OOH, overseas)? BOS survey coupled with the analysis of the standard credit claim form. Will rely on recruitment methods to ensure that all GPs are represented , Are there further training issues for GPs or appraisers? BOS survey coupled with the analysis of the standard credit claim form – there will also need to be analysis of the overall outcome of the project (the answer to this one is obviously yes the outcome should probably reflect the onward training needs in terms of content) , What are the local resource issues of the system? The Regional Leads will be asked to quantify time and other resource issues in running the pilot 54 The role of the GP tutor (if utilised) The role of the tutor and appraiser in the pilot may vary from site to site and indeed there may be a number of models running in parallel within a project site. The possible roles of the GP tutor are listed below – it will be vital to understand the role adopted by each site to ensure that the correct data is collected. , The tutor in a wholly supporting role. The regional lead may wish to utilise the GP tutor network to publicise the project and help in the training of the doctors involved in the project. In this model the appraisal discussion remains the focus of verification of credits however the tutor will have a role in helping doctors convert their CPD activity into the impact and challenge model of credits. , The tutor in a mixed role. In this situation the tutor may have the role described above and in addition will interview some of the doctors who wish to take part in the pilot but whose appraisal discussion is scheduled to take place outside the timeframe of the credits pilot. , The tutor in the verifying role. In this situation the tutor is conducting credit verification outside the local appraisal system as the regional lead has not been able to negotiate the inclusion of this process in the appraisal discussion. The value of including the local tutor network is recognised within the project however to avoid duplication of effort the appraisal model is preferred if there is a choice. The involvement of the tutor as described in the first bullet point above is the role envisaged in the final scheme with credits being discussed and verified at appraisal and the tutor facilitating the production of the evidence by the individual doctor. In the cases of the second two bullets above the following flow charts explain the process. 55 CP Central planning meeting RL BOS (on-line) survey Regional recruitment RL If required CP will Tutor be involved Regional training GP GP Standard credit claim form Appraisal discussion Outside project scope 56 CP Central planning meeting RL BOS (on-line) survey Regional recruitment RL If required CP will Tutor be involved Regional training GP GP To RL for collation and anonymisation Standard credit claim form 57 Detailed Project Tasks May – September 2008 Project design phase Production of initial project documentation Purchase of BOS licence – late August/early September Production of BOS surveys as illustrated above Production of training materials Production of comprehensive project documentation for regional leads September 2008 Project design phase Meeting of regional leads (presumably London ? at RCGP) September-October 2008 Project implementation phase Recruitment and training in regions – BOS surveys will need to be functional by the middle of October – data gathering phase begins November 2008 Project implementation phase Stock take on regional uptake January 2009 Project implementation phase Meeting of project exec group February 2009 Project implementation phase Stock take on regional uptake April 2009 Project implementation phase Meeting of project exec group to discuss end date (? Need for extension) May 2009 Project implementation phase New doctors accepted to end of month and then recruitment 58 closed June 2009 Project data gathering phase This phase which has been ongoing since October 2008 will close June 2009 Write up and reporting phase Project report to be produced by the end of the month for project exec meeting July 2009 Write up and reporting phase Final document produced for PDB and Council Conclusion The description of the project contained in this document lacks much in the detail. The purpose of this document is to define the purpose, explain the design, to superficially define the timelines of the pilot. To explain how the questions posed to the pilot will be addressed and to inform discussion at the initial meeting of the regional leads. 59 Acknowledgements Pilot steering group Christopher Price, RCGP CPD fellow - pilot lead Nigel Sparrow, RCGP Chair of PDB Caroline Turnbull, RCGP Head of Education & Professional Development Sam McNabb, RCGP Project Manager Pilot Regional Leads East Midlands: Val Evans Kent, Sussex, Surrey: Professor Abdol Tavabie London: Dr Julia Whiteman Mersey: Dr Tahir Awan Northern: Dr Di Jelley Northern Ireland: Dr Claire Loughrey Northwest: Dr Rebecca Baron Scotland: Dr Alan Melville Wales: Dr Lynne Rees Wessex: Dr Steve Scott I would also like to thank all the doctors and appraises that took part in the pilot as well as the unseen administrators in the regions. 60