JOURNAL TRANSCRIPT
DIABETES TECHNOLOGY & THERAPEUTICS Volume 8, Number 5, 2006 © Mary Ann Liebert, Inc.
Diabetes Information Technology & WebWatch Retrospective Pilot Feedback Survey of 200 Users of the AIDA Version 4 Educational Diabetes Program. 2—Qualitative Feedback Data ELDON D. LEHMANN, Ph.D., F.R.C.R.,1,2 SUKHDEV S. CHATU, M.R.C.P. (UK),3 and S. SABINA H. HASHMY, M.B. B.S.4
ABSTRACT This column continues the report of a detailed, questionnaire-based, post-release feedback survey of 200 users of the AIDA educational diabetes simulator. AIDA is a free software program that permits the interactive simulation of insulin and glucose profiles for educational, demonstration, self-learning, and research purposes. Since its Internet launch in 1996 well over 700,000 visits have been logged to the AIDA Websites—including www.2aida.org—and over 200,000 program copies have been downloaded without charge. The main goals of the present study were: (1) to establish what people have thought about the AIDA software, (2) to assess the utility of the program, and (3) to ascertain how much people have actually used it. In Part 1 of this column (Diabetes Technol Ther 2006;8:419–432) an analysis was undertaken of the first 200 feedback forms that were returned by AIDA users. The questionnairebased survey methodology was found to be robust and reliable. The study established the feasibility of using a simple feedback form to survey a substantial number of diabetes software users. In addition, it yielded interesting data in terms of who are the main users of the AIDA program, and also provided technical (computer) information that aided the release of a freeware upgrade to the software. The present Diabetes Information Technology & WebWatch column overviews qualitative feedback comments left by users as part of the questionnaire-based survey. In general, respondents seemed to find the software of educational value. The implications of these findings will be discussed.
1Department of Medical Imaging, University of Toronto, Toronto, Ontario, Canada. 2Department of Imaging (MRU), Imperial College of Science, Technology and Medicine (NHLI), Royal Brompton Hospital; 3St. Bartholomew’s and the Royal London Hospitals Medical School; and 4Guy’s, King’s & St. Thomas’ Hospital Medical School, London, United Kingdom. The AIDA software referred to in this report is an independent, non-commercial development that is being made available free-of-charge via the Internet—at a dot org (.org) not-for-profit Website—as a non-commercial contribution to continuing diabetes education. Dr. Lehmann is a co-developer of the AIDA diabetes simulator, and Webmaster of the www.2aida.org Website.
602
INFORMATION TECHNOLOGY & WEBWATCH
INTRODUCTION
A
IDA IS A FREE PROGRAM that permits the interactive simulation of insulin and glucose profiles for demonstration, teaching, self-learning, and research purposes. It has been made freely available via the Internet as a non-commercial contribution to continuing diabetes education. In the 10 years since its Web launch well over 700,000 visits have been logged at the AIDA Websites—including www.2aida.org and www. 2aida.net—and more than 200,000 copies of the program have been downloaded free-of-charge. Further copies have been made available, in the past, on diskette by the system developers and from the British Diabetic Association (now called “Diabetes UK”), London, U.K.1 The main aims of the current n 200 survey study were threefold: (1) to establish what people have thought about the AIDA software, (2) to assess the utility of the program, and (3) to ascertain how much people have actually used it. Three ancillary objectives for the study were: (a) to identify any problems that users may have experienced with the software since its launch, with the intention being for any critical problems to be addressed in future upgrades; (b) to identify features that users felt were important to have incorporated in later releases of the program; and (c) to establish the feasibility of undertaking such a post-release diabetes software survey primarily via electronic mail. The qualitative section of the evaluation form2—available on the Web at www.2aida.org/ questionnaire—also asked users to submit feedback comments. This is part of an ongoing process that aims to assess how people are making use of AIDA and what users would like to see in future versions of the software, as well as take into account any problems that they may have encountered. Qualitative data in the form of selected user free text feedback comments received from survey respondents are documented below.
FREE TEXT COMMENTS Comments received from people with diabetes in response to the question “Do you think AIDA has some utility?” included:
603
• “I see it as being very useful for the inquisitive patient who wants to understand the effects of different insulins. It could be demonstrated on a large screen in the doctor’s office while they wait for an appointment.” • “As a newly diagnosed type 1 [diabetic patient] AIDA helped me to achieve [a] better understanding of diabetes. I think it could be used with other patients as well.” • “It helps to confirm my intuitions about how I change my carbohydrate intake and insulin intake to changing conditions.” • “I see it first as a way of showing people (health care professionals and diabetics) what typical BG [blood glucose] profiles look like, and [how] things such as eating more, injecting less, etc. can affect the pattern. . . . Second I see it being used by diabetics when they start to want to find out how to move beyond the rules that are issued, as a way to see what size of experiments is a good level to start at. . . . It also gives an indication of the time when problems may happen—so that the BG can be checked by meter, driving and use of dangerous machinery avoided etc. Third I see it as a way for experienced diabetics to think about areas for improvement and how they might be made.” • “It gives a better understanding of how the levels of insulin and BG fluctuate and [one] can thus anticipate what kind of effect any change of diet or insulin might have. It also provides some clue as to what might have gone wrong when highs or lows occur and when might be a good time to have occasional tests to see what might be happening.” • “I have been able to use AIDA to test different insulin and food regimens so that my blood sugar measurements remain more stable—i.e. so that I can avoid hypoglycemic reactions.” • “Very good educational tool for people who want to manage their diabetes. This is the new generation of help with diabetes management. Home blood glucose monitoring is essential. But, AIDA permits us to simulate various scenarios to gain better control based on blood glucose levels and carbohydrate intake.” • “As a simulation of what might happen if you do not follow your diabetes prescriptions.”
604
• “A very good training tool and a good way of getting a computer literate diabetic to understand his management.” • “Evaluation of control between doctor’s appointments. Better understanding of why insulins have different effects, and can help to give a better understanding of how to change dosages to get different results.” • “To illustrate the relationships that exists in a type 1 diabetic. To perform a ‘what if?’ with the data which may take some of the guess work out of how we deal with this disease.” • “Some people need practical demonstrations of food/insulin/exercise interaction. They have not been instructed in adjusting their own insulin. They need some degree of practice in adjustment prior to adjusting their own, as a confidence builder and education. I liked it.” • “As a diabetic patient it is a very useful educational tool. Can be of assistance in helping to assess possible changes in routine.” • “I see it as being useful to new type 1’s who need to learn how to adjust their insulin regimen to correct a problem and see that their blood sugar can go through significant changes in between their blood tests.” • “As a way of teaching new diabetics what is happening in their bodies regarding insulin and food.” • “The obvious place is for those who can be shown what they have had explained to them. The old story of recalling what you’ve been told is about 30%, what you’ve been shown about 40%, and when the two are done together is 70%—holds true. It can also show the way in which insulin and diet interact. It may even help the professionals in the field understand the matter better.” • “I could see it being used in a classroom scenario as instructions for diabetic patients, care givers and medical service providers and also at home by interested parties such as myself.” • “I think this could be a valuable tool for diabetics, if the program is simpler to set-up than my experience would indicate. I can see this being used by diabetes educators on a one to one basis with their patients. American dieticians could also utilise this program. Of course, every diabetic patient who uses
INFORMATION TECHNOLOGY & WEBWATCH
insulin can benefit from the added knowledge.” • “I think dieticians and people on a new type of insulin to their normal type will find it of great interest.” • “Can be helpful in projecting the impact of changes in insulin injection levels. I now realise that there are many variables that affect insulin assimilation, but the simulations can be helpful.” • “What I did was to put a couple of my information into it and see that I needed help so I’m in the process of getting a pump. It has helped me to get a pump.” Feedback from diabetes educators and healthcare professionals in response to the same question—“Do you think AIDA has some utility?”—included: • “By educators and people like me who have to explain what happens with your blood glucose levels in case one plays with insulin or food.” • “Very interesting program which I used with the endocrine fellows at Mount Sinai [Hospital] and selected patients for teaching.” • “For medical students to make them aware of the temporal nature of insulin therapy and nutritional constraints.” • “I am thinking about to start using this programme next year when students have to use computer programs or other audiovisual material to evaluate the metabolic processes in IDDM [insulin-dependent diabetes mellitus] and NIDDM [non–insulin-dependent diabetes mellitus].” • “In our area which is rural there are no diabetes specialists. Diabetes is managed by family practice or internists which is fine except most in this area do not manage blood glucose levels effectively. If they could plug in their treatments they would see rapidly the possible effects of the pharmaceuticals (granted real people are not as predictable as computers . . . but AIDA would help). I would like to have my patients use AIDA and see the projected effect of subtle changes in their dosages prior to making the change.” • “I teach a graduate level interdisciplinary course in diabetes and the simulations will
INFORMATION TECHNOLOGY & WEBWATCH
•
•
•
•
•
•
•
•
•
•
provide an interesting additional practice. I teach continuing nursing education courses and just yesterday I gave the group the address of AIDA. In the undergraduate courses I think that the simulations will help students to learn the integration of the effects of components of [the] management of diabetes.” “As continuing education tool for diabetes programme staff. Have shown AIDA to other members of our diabetes education staff.” “I think that it can and should be used in endocrine physiology lessons for undergraduate students.” “We precept/mentor pharmacy students from the university. This will allow students to simulate what dosage adjustments cause what blood glucose readings, etc.” “For medical student/physician intern education regarding interrelationship of therapies as well [as] nurse/dietician/pharmacist education. For many people with diabetes, would be an interesting way for them to learn how changes in therapeutics affects their diabetes control.” “I think it could be a valuable service to diabetes educators and would like to be able to put patients into these situations and simulate outcomes. I have not been fully interactive but I really like the concept and the ability to look at treatment options that way.” “I’ve used it to teach nurse practitioner students and to help me plan some presentations on insulin adjustment.” “As an educational tool for healthcare providers involved in educating other staff as well as patients.” “As an interactive tool to demonstrate factors contributing to blood glucose regulation in an endocrine physiology lesson.” “I have two diabetes education and support groups that I run. Sometimes it’s hard for them to see their trends. With this program it would show them the problems more easily and make it easier for them to understand what’s happening.” “Training health care providers . . . especially medical students, interns and others wishing to advance their diabetes management skills.”
605
• “Comparing AIDA output/results with actual results in a MD approved treatment program. Compare actual changes in blood sugar with projected changes. Provide insight for expected results prior to implementing a change in insulin dose, carbohydrate change, time shifts, etc.” • “I would use it in my waiting room and make it mandatory that diabetics use as well as provide the service for them to ‘play’.” Feedback from other users (e.g., researchers and relatives of patients) in response to the same question—“Do you think AIDA has some utility?”—included: • “Tool for people coming to grips with using insulin to see how changes they make will affect their control, giving them more confidence to adjust their own medication/diet appropriately and to become more empowered. Also a good tool for doctors to learn about insulin regimes.” • “It can help to explain the basic principles of diabetic control to patients and helpers.” • “Demonstrating the effect of changing levels of carbohydrate intake and insulin in both quantity and timing. Greater utility would arise if the effect of exercise could also be demonstrated.” • “As a parent of a fairly recently diagnosed child the simulations help me understand the general dynamics of insulin types and the likely effects on blood glucose control.” • “To get much better understanding of the glycaemic process, i.e. that it is dynamic process where not only the absolute values of insulin/food intake but also gradients of blood glucose changes have importance. Better understanding means better balancing of the insulin regimen.” • “Help the user visualise the time variants of glucose and insulin following intake of carbohydrate and insulin—from this develop a feeling of the effects of varying the times and amounts of intakes.” • “For explanation of some control processes, that are difficult to explain only qualitatively (in words). The diagrams in books are only static and difficult to individualise and for interpretation. AIDA illustrates the mecha-
606
INFORMATION TECHNOLOGY & WEBWATCH
nism of combination of different types of insulin in time.” Comments received in response to the question “Please outline any problems/difficulties/bugs that you have encountered” are documented below: • “Well when I entered my insulin regime it would not allow me to put my NPH in any higher than 40 units. I take 42 units twice a day.” • “I had some problems with installation which were probably mostly related to my lack of DOS skills; since installation no problems.” • “It seems that advice is given only for adjusting insulin intakes. Does it suggest changes in food also? I am still a novice both on the computer and on your programme, in the near future I hope to understand both better.” • “A bit slow at processing changes in therapy and the impact. I’d like to have it on disk so I could load it on my laptop to use with patients. I really enjoyed the programme and look forward to working with it again!! I’d like to be able to enter data about a patient I have and have the programme allow me to make simulated adjustments and see the impact.” • “No problems encountered. I found the scenarios to be a beneficial learning tool for myself and other healthcare providers. Just a few comments: . . . I could not discern whether comparison of new graph to “previous” was the immediate preceding graph or the graph from the original scenario. Thanks for making this resource available over the Internet at no cost.” • “The carbohydrate intake of my son exceeds 80 g per meal. Including snacks he takes 6 meals a day. This cannot be simulated.”
OTHER COMMENTS Other free text comments in response to other questions—in none of the above categories—received from survey respondents, included:
• “Excellent tool for patients to play with and get a feel for how their insulin works.” • “My colleagues that give endocrine physiology lectures use it. I think that they will still use it on a regular basis.” • “The program provides very useful information. Being able to interactively change the insulin and meal regimes and observe the probable effect on glucose levels is very enlightening. It is interesting to enter one’s current regime and see how the glucose profile compares to the actual readings. Finding possible explanations for the difference can give some interesting insights into one’s own condition. I think this programme can be very beneficial for both parents and doctors for learning about managing diabetes.” • “The nurse/educator I was working with was very interested in AIDA and immediately demanded the URL [Web address].” • “Actually I found it very useful because it indicated just how the two insulins combined and indicated just where the peaks and dips were likely to be. In my case I had just switched over to Ultra and Humalog, from NPH and Humalog, and wanted to get some idea how the Ultra doses overlapped each other. This is very difficult to do unless you have a clear idea of the slope and length of the curves. Of course, since you do not have Lispro insulin incorporated I knew that substituting regular (Humulin R) would not indicate the steepness of the Humalog’s effect. However it did indicate two things I had not been sure about: a high peak in mid-afternoon and a very flat line during the night. I checked these out by testing and found it to be the case. I also didn’t put too much credence in the actual BG [blood glucose] values given but found the shape of the curve very interesting.” • “It seems to me that you have a good powerful tool here.” • “I really enjoyed playing with this tool. I think it would be a useful element for training family docs who are now being forced to take care of complicated diabetics.” • “Would like a version that lets you put in your weight in pounds versus kg [kilograms]. I know I can easily convert pounds to kg by multiplying by 2.2 but that is a has-
INFORMATION TECHNOLOGY & WEBWATCH
sle. Don’t worry about it. It is just a personal thing that worries me (I did not use the program for a month because I did not know what a pound was in kg!). Overall I give the programme an ‘A’.” • “Although this is an outstanding programme for educational purposes (undergraduate). It would become much more powerful if it included a physiology section, that students could use to study blood glucose control. . . . ” Question: AIDA worth wider distribution? • “Definitely—this software tool is not perfect (because of the way people react to insulin), but there is nothing else that even comes close.” Question: Think AIDA may be unsafe? • “May be some people may not read the notices stating that AIDA is a learning tool, not pure fact. It is for learning purposes only, and should not be used as a diagnosis.” • “I think anyone interested enough in the program to be able to run it would be able to realise that it was not a prescription to be followed literally and was a theoretical model of what would happen under controlled or ideal conditions.” • “Not per se but it frightened the living day lights out of me when I ran a guesstimate of my diet and no insulin!” • “Not with all the clear warnings/caveats, diabetics should know these caveats very well to a point of irritation (since they often know more than their doctor about their condition).” • “For some even a pencil is dangerous.” • “No, AIDA includes ample warnings.” Question: Program caveats ok/clear? • “Most people who are intelligent enough to use AIDA will understand the limitations it has.” • “Too many warning[s] makes use annoying.” • “You will still have some that will try to use AIDA as a cure. I don’t think that can be stopped. When a patient gets a bottle of in-
607
sulin you have no control if they choose to adjust the dosage incorrectly. Misuse will happen . . . but more good can come of AIDA’s usage.” Question: Understood AIDA not intended for individual simulation? • “Yes, of course it was! But do you really think an IDDM patient curious enough to run the program would refrain from seeing how it modeled his own situation?” A subset of the spell-checked free text comments, received from survey respondents and overviewed in this column, can be found on the Web at: www.2aida.org/survey-text DISCUSSION The current pilot study has confirmed the feasibility of using a standardized questionnaire-based survey as a primary medium to obtain feedback from users of diabetes software. It has also provided useful and interesting information regarding what healthcarer and patient users have thought about the software. What has been particularly informative has been the wide variety of ways in which different people see the program being used. There seem to have been generally high levels of satisfaction among respondents. Clearly those who have chosen to respond to such a survey may be a self-selected group—but nevertheless the free text comments do provide an interesting insight into how people have been applying and viewing the program. The study also raises interesting issues about how to present and deal with so much largely descriptive qualitative feedback data. One hurdle with detailed questionnaires is to get users to complete the feedback forms. However, once they do, then another issue becomes how to handle all the data, and analyze and summarize them all in a meaningful way. In Part 3 of this column (manuscript in preparation) the results and implications of this pilot study will be discussed in detail. One concept that will be conveyed is the drug paradigm for medical software.
608
INFORMATION TECHNOLOGY & WEBWATCH
Even after a drug has been tested and evaluated, and released—pharmaceutical firms still have systems in place to monitor usage and any complications. The same should be the case for medical informatics tools and software—with post-release monitoring of programs taking place routinely. The exact method by which this is done still requires further work and elucidation, but we would propose that monitoring feedback from software users should form an important and integral part of the post-release monitoring process for software programs, just as for pharmaceuticals.
ACKNOWLEDGMENTS The authors would like to thank the 200 AIDA users who took the time and trouble to provide their feedback, and respond to this survey.
FURTHER TOPICS If you would like to suggest further topics or Websites for future Diabetes Information Technology & WebWatch columns, please e-mail in-
formation—with a brief description of the site/suggestion—to Dr. E.D. Lehmann:
[email protected] (please write Diabetes WebWatch in the subject line). You can also fax information to: (503) 218-0828, quoting Diabetes Information Technology & WebWatch.
REFERENCES 1. Lehmann ED, Deutsch T, Broad D: AIDA: An Educational Simulator for Insulin Dosage and Dietary Adjustment in Diabetes. London: British Diabetic Association, 1997. 2. Lehmann ED, Chatu SS, Hashmy SSH: Retrospective pilot feedback survey of 200 users of the AIDA version 4 educational diabetes program. 1—Quantitative survey data. Diabetes Technol Ther 2006;8:419–432.
Address reprint requests to: Dr. Eldon D. Lehmann c/o www.2aida.org Diabetes Simulator Development Team P.O. Box 38265 London, NW3 7XZ, UK E-mail:
[email protected] Web: www.2aida.org/lehmann