Guideline development process

From Cancer Guidelines Wiki

Contents

Background

In 2014, Cancer Council Australia and Melanoma Institute Australia partnered as guideline developers and initiated the project to revise the Clinical Practice Guidelines for the Management of Melanoma in Australia and New Zealand . Due to the advancements in treatment options, the 2008 guidelines are no longer up to date. The evidence base and management of melanoma has significantly changed since 2008, particularly for the treatment of stage III and stage IV disease emerging over the past few years. Importantly, targeted and systemic therapy drugs are now registered for use within Australia and there are significant publications demonstrating the improvement for life expectancy in melanoma patients due to the improved treatment options.

Cancer Council Australia and Melanoma Institute Australia contributed in kind resources consisting of project staff, facilities, systems and travel budget to revise the 2008 guidelines. In 2015, Skin Cancer College Australasia joined the project and provided funding to enable employment of an additional full-time Project Officer in the Systematic Review team.

Back to top

Project governance, guidelines scope and guidelines development group

Cancer Council Australia and Melanoma Institute Australia appointed a small Management Committee that were members of the 2008 working party, to oversee the guidelines revision project (see working party members and contributors). The Management Committee is responsible for the overall management and strategic leadership of the guidelines review process. This includes the establishment of the wider multidisciplinary guidelines working party and question-specific sub-committee members in consultation with the lead authors and the evaluation of declarations of interest and, if necessary, implementing management strategies for conflict/s of interest.

During a face-to-face meeting in November 2014, the Management Committee assessed the clinical questions addressed the 2008 guidelines and determined the priority clinical questions to be included in this revision. Twenty-three questions were identified to be of greatest importance, covering issues related to diagnosis, staging and management of cutaneous melanoma (see list of clinical questions).

The Management Committee proposed lead authors for each included clinical question. The nominated individuals were invited to join the (see multidisciplinary working party). In addition, the Management Committee identified and nominated two consumer representatives and two GP representatives to join the multidisciplinary working party.

In consultation with the question lead author, sub-committees consisting of members with relevant expertise and experience were established for each question (see multidisciplinary working party).

Declarations of interest were collected from all nominated members and evaluated (see COI register). All members were advised to update their declarations of interest over the course of the project and received reminders to review their declarations prior to every formal working party meeting.

Back to top

Guidelines development approach

The Management Committee agreed to use Cancer Council Australia’s Cancer Guidelines Wiki Platform and approach to develop the guidelines. The Wiki Platform is web-based and supports all processes of guidelines development, such as the literature search, critical appraisal, data extraction, evidence assessment and summary processes, as well as content and recommendation development, online consultation, review and web publication. It is in line with the NHMRC guidelines requirements, designated standards of quality, process and grading system for recommendations.[1][2] An infrastructure is set in place to process literature updates and continuously update content as new evidence emerges and is reviewed.

The Development of Clinical Practice Guidelines using Cancer Council Australia’s Cancer Guidelines Wiki Handbook[3] illustrates the steps in the development of Cancer Council Australia’s web-based clinical practice guidelines. It provides information to assist working party members and staff members to develop concise clinical questions in PICO format, construct sound search strategies, systematically search the literature, critically appraise, summarise the evidence and formulate guidelines recommendations.

The Management Committee was approached by the German guidelines development group, which developed the guidelines “Malignant Melanoma S3-Guideline Diagnosis, Therapy and Follow-up of Melanoma”[4] in 2012 and adapted some sections from the 2008 Australian guidelines. The systematic review team assessed the German guidelines using the AGREE II assessment tool[5] and found the guidelines to be high quality. As many exhaustive systematic reviews were undertaken to answer critical clinical questions in the melanoma diagnosis and management guidelines, it was decided to adapt the German systematic reviews and update the literature searches, where possible, rather than undertaking new systematic reviews for the same clinical questions (see also 3b. If a relevant clinical practice guidelines was found and assessed as suitable for adaption). The data extractions and quality appraisals of any new studies will be shared with the German group.

Rather than waiting until systematic reviews and content for all included clinical questions have been finalised, the Management Committee agreed to publish finalised question content and the associated recommendations in stages. The group decided that it is important to publish content and results as soon as it is finalised by the working party to ensure that the medical community receives up-to-date information without any publication delay. Prior to publication, feedback would be sought from guidelines stakeholders about the clinical questions content (See also Public consultation).

The first set of completed draft contents is now being released for public consultation (refer to set of questions).

  • What are the clinical features of melanoma and how do atypical melanomas present?
  • What type of biopsy should be performed for a suspicious pigmented skin lesion?
  • When is a sentinel node biopsy indicated?
  • What are the recommended safety margins for radical excision of primary melanoma?

Subsequent clinical questions and associated recommendations will be published in 2016 and 2017.

The detailed steps in preparing the question content, conducting the literature searches, appraising the literature and formulating and grading recommendations, are outlined below.

Back to top

Steps in preparing clinical practice guidelines

For every clinical question the following steps were completed:

1. Develop a structured clinical question in PICO format

2. Search for existing relevant guidelines and systematic reviews answering the clinical question

3. Perform systematic review process, depending on if a relevant clinical practice guideline is identified or not

3a If no relevant clinical practice guideline was found


Developing the systematic review protocol and systematic literature search strategy for each PICO question

Conducting the systematic literature search according to protocol

Screening of literature results against pre-defined inclusion and exclusion criteria

Critical appraisal and data extraction of each included article

Create body evidence table of all included literature


3b If a relevant clinical practice guideline was found and assessed as suitable for adaption

Undertake systematic literature search update for the question of the existing clinical practice guideline

Screening of literature update results against pre-defined inclusion and exclusion criteria

Critical appraisal and data extraction of each new included article

Update body evidence table of evidence review of existing guideline with new literature update results


4. Summarise the relevant data

5. Assess the body of evidence and formulate recommendations

6. Write the content narrative

Back to top

Step 1. Develop a structured clinical question

All included questions were reviewed on the basis of their purpose, scope and clinical importance to the target audience and were structured according to the PICO (populations, interventions, comparisons, outcomes) framework. The lead authors provided the systematic review team with feedback to refine the PICO questions and inclusion and exclusion criteria for the systematic review.

Back to top

Step 2. Search for existing relevant guidelines and systematic reviews

For each PICO question, the National Guideline Clearinghouse, the Guidelines Resource Centre and the scoping search for the PICO question were scanned for relevant clinical practice guidelines that could potentially be suitable for adaption.

Full systematic reviews were then performed as outlined in the sections below (Developing a systematic search strategy; Conducting the systematic literature search according to protocol; Screening of literature results against pre-defined inclusion and exclusion criteria; Critical appraisal and data extraction of each included article).

If an existing relevant guideline was identified, the guideline was assessed with the AGREEII assessment tool[5] to ensure the guideline is of high quality. The ADAPTE process was then followed.[6]

Back to top

Step 3. Perform systematic review process

Step 3a. If no relevant clinical practice guideline was found

Developing a systematic search strategy

For each PICO question, systematic literature search strategies were developed by the technical team. Searches were limited or widened as necessary according to the PICO structure using keywords or MESH and subject terms. Systematic search strategies were derived from these terms for each included electronic databases. The included standard databases searched were Pubmed, Embase, Trip database, Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects and Health Technology Assessment for all questions. The psychosocial questions also included CINAHL and PsycINFO databases to retrieve relevant literature.

Back to top

Conducting the systematic literature search according to protocol

Clinical practice guidelines should be based on systematic identification and synthesis of the best available scientific evidence.[1] For each clinical question that required a systematic literature review, literature searches were conducted systematically from 2007 onwards. The following electronic databases were part of the systematic literature search strategy:

  • PubMed – bibliographic references and abstracts to articles in a range of languages on topics such as clinical medical information and biomedicine, and including the allied health fields, biological and physical sciences.
  • EMBASE – major pharmacological and biomedical database indexing drug information from 4550 journals published in 70 countries.
  • Trip Database – A medical database with focus on Evidence based medicine and clinical practice guidelines with content available from Cochrane and Bandolier.
  • Database of Abstracts of Reviews of Effects and Health Technology Assessment – Contains details of systematic reviews that evaluate the effects of healthcare interventions and the delivery and organisation of health services.
  • The Cochrane Database of Systematic Reviews.
  • Cinahl – Bibliographic references and abstracts to journal articles, book chapters, pamphlets, audiovisual materials, software, dissertations, critical paths, and research instruments on topics including nursing and allied health, biomedicine, consumer health, health sciences librarianship, behavioral sciences, management, and education
  • Psychinfo – Bibliographic references and abstracts to journal articles, book chapters, dissertations and technical reports on psychology; social, clinical, cognitive and neuropsychology; psychiatry, sociology, anthropology and education, with source material from a wide range of languages.

Additional relevant papers from reference lists and, where appropriate, clinical trial registries, were also identified for retrieval as part of the snowballing process.

The full detailed systematic literature search strategy for every clinical question is fully documented in the appendix of the clinical question.

Back to top

Screening of literature results against pre-defined inclusion and exclusion criteria

Part of the systematic review process is to screen all retrieved literature results against the pre-defined inclusion and exclusion criteria in two stages.

a) First screen – During the first screening round, the titles and abstracts of all retrieved literature were screened by one reviewer. All irrelevant, incorrect and duplicates were removed.

b) Second screen – A second screen was undertaken based on the full article. Two reviewers assessed each article for inclusion against the pre-defined inclusion and exclusion criteria for each question. In the case of a disagreement between the reviewers, a third independent reviewer assessed the article against the inclusion and exclusion criteria. Articles that met the inclusion criteria were forwarded for quality assessment and data extraction.

Back to top

Critical appraisal and data extraction of each included article

Two assessors independently assessed the risk of bias of each of the included studies using a study design specific assessment tool and where necessary pre-specified criteria. For all quality assessment tools, see link to pdf.

Any disagreements were adjudicated by a third reviewer.

For all included articles, the relevant data was extracted and summarised in study characteristics and evidence tables. Each data extraction was checked by a second assessor. These tables are available in the appendix of each question.

Back to top

Step 3b. If a relevant clinical practice guidelines was found and assessed as suitable for adaption

Undertake systematic literature search update for the question of the existing clinical practice guideline If an existing clinical practice guideline of high quality was found that directly addresses the clinical question to be reviewed, an update search of the original systematic literature search was performed covering the time period between the literature cut-off of the original review until now across all relevant databases (see also Conducting the systematic literature search according to protocol).

Screening of literature update results against pre-defined inclusion and exclusion criteria

All retrieved literature results from the update search were screened against the pre-defined inclusion and exclusion criteria in two stages.

a) First screen – During the first screening round, the titles and abstracts of all retrieved literature were screened by 1 reviewer. All irrelevant, incorrect and duplicates were removed.

b) Second screen – A second screen was undertaken based on the full article. Two reviewers assessed each article for inclusion against the pre-defined inclusion and exclusion criteria for each question. In the case of a disagreement between the reviewers, a third independent reviewer assessed the article against the inclusion and exclusion criteria. Articles that met the inclusion criteria were forwarded for quality assessment and data extraction.

Back to top

Critical appraisal and data extraction of each included article

Two assessors independently assessed the risk of bias of each of the included studies using a study design specific assessment tool and where necessary pre-specified criteria. For all quality assessment tools, see link to pdf.

Any disagreements were adjudicated by a third reviewer.


Back to top

Step 4. Summarise the relevant data

The study results, level of the evidence, risk of bias due to study design and the relevance of the evidence for each included study were summarised in a body of evidence table.

When a systematic review from an existing guidelines was updated to answer and develop recommendations for a clinical question, the new evidence was added to the existing body of evidence table. Where required, the levels of evidence were translated to the NHMRC levels of evidence. The NHMRC levels of evidence are outlined below:

Table 1. Designations of levels of evidence according to type of research question (NHMRC, 2009)

Level Intervention Diagnosis Prognosis Aetiology Screening
I A systematic review of level II studies A systematic review of level II studies A systematic review of level II studies A systematic review of level II studies A systematic review of level II studies
II A randomised controlled trial A study of test accuracy with: an independent, blinded comparison with a valid reference standard, among consecutive patients with a defined clinical presentation A prospective cohort study A prospective cohort study A randomised controlled trial
III-1 A pseudo-randomised controlled trial (i.e. alternate allocation or some other method) A study of test accuracy with: an independent, blinded comparison with a valid reference standard, among non-consecutive patients with a defined clinical presentation All or none All or none A pseudo-randomised controlled trial (i.e. alternate allocation or some other method)
III-2 A comparative study with concurrent controls:

Non-randomised, experimental trial

Cohort study

Case-control study

Interrupted time series with a control group

A comparison with reference standard that does not meet the criteria required for Level II and III-1 evidence Analysis of prognostic factors amongst untreated control patients in a randomised controlled trial A retrospective cohort study A comparative study with concurrent controls:

Non-randomised, experimental trial

Cohort study

Case-control study

III-3 A comparative study without concurrent controls:

Historical control study

Two or more single arm study

Interrupted time series without a parallel control group

Diagnostic case-control study A retrospective cohort study A case-control study A comparative study without concurrent controls:

Historical control study

Two or more single arm study

IV Case series with either post-test or pre-test/post-test outcomes Study of diagnostic yield (no reference standard) Case series, or cohort study of patients at different stages of disease A cross-sectional study Case series

Source: National Health and Medical Research Council. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Canberra: NHMRC; 2009. (https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf)

Back to top

Step 5. Assess the body of evidence and formulate recommendations

The body of evidence table for each clinical question was forwarded to the lead author for assessment. The lead author in collaboration with the systematic reviewer (who conducted the systematic reviews and extracted the data and performed risk of bias assessment) assessed the body of evidence and completed the evidence assessment matrix in regard to the volume of the evidence, its consistency, clinical impact, generalisability and applicability and developed evidence statements for each recommendation.

The process is described in NHMRC additional levels of evidence and grades for recommendations for developers of guidelines (2009).[7]

Following grading of the body of evidence and development of evidence statements, authors were asked to formulate evidence-based recommendations based on the results of the systematic review summarised in the body of evidence table. The method of grading recommendations is shown in Table 2.

Table 2. Grading of recommendations

Component of Recommendation Recommendation Grade
A

Excellent

B

Good

C

Satisfactory

D

Poor

Volume of evidence 1** one or more level I studies with a low risk of bias or several level II studies with a low risk of bias one or two level II studies with a low risk of bias or a systematic review/several level III studies with a low risk of bias one or two level III studies with a low risk of bias, or level I or II studies with a moderate risk of bias level IV studies, or level I to III studies/systematic reviews with a high risk of bias
Consistency 2** all studies consistent most studies consistent and inconsistency may be explained some inconsistency reflecting genuine uncertainty around clinical question evidence is inconsistent
Clinical impact very large substantial moderate slight or restricted
Generalisability population/s studied in body of evidence are the same as the target population for the guideline population/s studied in the body of evidence are similar to the target population for the guideline population/s studied in body of evidence differ to target population for guideline but it is clinically sensible to apply this evidence to target population3 population/s studied in body of evidence different to target population and hard to judge whether it is sensible to generalise to target population
Applicability directly applicable to Australian healthcare context applicable to Australian healthcare context with few caveats probably applicable to Australian healthcare context with some caveats not applicable to Australian healthcare context

1 Level of evidence determined from level of evidence criteria

2 If there is only one study, rank this component as ‘not applicable’

3 For example results in adults that are clinically sensible to apply children OR psychosocial outcomes for one cancer that may be applicable to patients with another cancer.

**For a recommendation to be graded A or B, the volume and consistency of evidence must also be graded either A or B!

Source: National Health and Medical Research Council. NHMRC additional levels of evidence and grades for recommendations for developers of guidelines. Canberra: NHMRC; 2009. (https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf)


The overall recommendations grade are shown in Table 3.

Table 3. Overall recommendation grades

Grade of recommendation
Description
A
Body of evidence can be trusted to guide practice
B
Body of evidence can be trusted to guide practice in most situations
C
Body of evidence provides some support for recommendation(s) but care should be taken in its application
D
Body of evidence is weak and recommendation must be applied with caution

Source: National Health and Medical Research Council. NHMRC levels of evidence and grades for recommendations for developers of guidelines. Canberra: NHMRC; 2009. (https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf)

The NHMRC approved recommendation types and definitions are shown in Table 4.

Table 4. NHMRC approved recommendation types and definitions

Type of recommendation
Definition
Evidence-based recommendation
A recommendation formulated after a systematic review of the evidence, indicating supporting references
Consensus-based recommendation
A recommendation formulated in the absence of quality evidence, after a systematic review of the evidence was conducted and failed to identify admissible evidence on the clinical question
Practice point
A recommendation on a subject that is outside the scope of the search strategy for the systematic review, based on expert opinion and formulated by a consensus process


Source: National Health and Medical Research Council. Procedures and requirements for meeting the NHMRC standard for clinical practice guidelines. Melbourne: National Health and Medical Research Council, 2011


In addition to developing evidence-based recommendations as a result of the systematic review for a clinical question, expert authors could also draft consensus-based recommendations in the absence of evidence after having performed a systematic review or practice points, when a matter was outside the scope of the search strategy for the systematic review.

Back to top

Step 6. Write the content narrative

For each question, the assigned lead authors were asked to draft their guidelines chapter using the following format:

  • Background to the clinical question, including its clinical importance and historical evidence, where relevant
  • Review of the evidence, including the number, quality and findings of studies identified by the systematic review
  • Evidence summary in tabular form including evidence statements, levels of evidence of included studies, and reference citations
  • Evidence-based recommendation(s) and corresponding grade(s), consensus-based recommendations and practice points
  • Discussion, including unresolved issues, relevant studies currently underway, and future research priorities
  • References.

The content draft was then reviewed by all sub-committee members. The draft documents underwent several iterations until agreement between the members of the sub-committee on these drafts was reached.

Back to top

Review of the draft chapters

Each set of draft content was circulated to the Working Party. The whole group was asked to review the content and submit feedback. Members were asked to submit further suggestions on consensus-based recommendation and practice points.

A face-to-face meeting with all working party members was scheduled to review and finalise the draft content for public consultation. Prior to this meeting, the latest iteration drafts were circulated. All panelists were asked to review the content, individual recommendations and practice points in detail, identify and note any controversies and points to be discussed at the meeting. During the meeting, each recommendation and practice point was tabled as an agenda point. Each was reviewed and approved by consensus, which was reached by voting. The Chairperson nominated a particular recommendation/practice point to be reviewed and the panelists had the opportunity to discuss any issues and suggest revisions to recommendations and practice points. Each recommendation and practice point was approved once the eligible panelists reached consensus.

Back to top

Public consultation

This guideline is being developed in a staged process.

  • The first set of draft clinical questions (Features of Melanoma, Biopsy, Sentinel Node Biopsy, Excision Margins) were made available on the wiki for public consultation from 14 May to 14 June 2016.
  • The second set of draft clinical questions (Diagnostic aids for melanoma (Dermoscopy) and Confocal microscopy) were made available on the wiki for public consultation from 23 January to 17 February 2017.

During each public consutlation period, submissions were invited from the general public and professional societies and groups and other relevant stakeholders. Relevant professional societies and groups, consumer groups and other relevant stakeholders were contacted.

All feedback on the draft received during the consultation periods were compiled and sent to the relevant lead author (and subcommittee, when required) to review the draft content, assessing and considering the submitted comments. Any additional submitted paper during public consultation was assessed by the methodologist team against the review protocol.

Wider Working Party review of the public consultation comments and suggested amendments was facilitated by email or teleconference. Subsequent changes to the draft were agreed by consensus, based on consideration of the evidence and, in the absence of evidence, expert opinion. The same consensus process that was followed during the face-to-face working party meeting prior to public consultation was followed again. All changes resulting from the public consultation submission reviews will be documented and made accessible by request once the guidelines are published.

Back to top

Dissemination and implementation

A multi-strategy approach will be followed for the dissemination and implementation of the guidelines, as this has shown to positively influence guidelines uptake.[8][9]

Once all clinical questions that are part of the guidelines revision are completed, the guidelines will be distributed directly to relevant professional and other interested groups and through meetings, national and international conferences, and other professional development and continuing medical education (CME) events. Local expert leaders will be identified and approached to facilitate dissemination and act as champions for the guidelines.

A significant effort will be made to have the guidelines introduced to senior undergraduate medical students and to encourage the relevant learned colleges to support the guidelines and to foster their integration into hospital and community practice through resident and registrar education activities.

The guidelines will be made available as online guidelines via the Cancer Council Australia Cancer Guidelines Wiki. The online guidelines version increases availability as well as accessibility, and usage will be tracked and analysed with a web analytics solution. The Cancer Guidelines Wiki is a responsive website that is optimised for mobile and desktop access.

Interlinking and listing the guidelines on national and international guideline portal is also an important part of the digital dissemination strategy. Important Australian health websites, such as EviQ and healthdirect Australia will be approached to link to the online guidelines. The guidelines will also be listed on national and international guideline portals such as Australia’s Clinical Practice Guidelines Portal, Guidelines International Network guidelines library and National Guidelines Clearinghouse.

The Cancer Guidelines Wiki is based on semantic web technology, so the guidelines are available in a machine-readable format, which offers the possibility to easily integrate the guidelines content with systems and web applications used in the Australian healthcare context. Use of the guidelines as part of core curriculum in specialty exams will be encouraged.

It is recognised that a planned approach is necessary to overcome specific barriers to implementation in particular settings and to identify appropriate incentives to encourage uptake of guidelines recommendations. Implementation of the guidelines will require a combination of effective strategies and may include further CME initiatives and interactive learning, the development and promotion of computer-assisted decision aids and electronic decision-support systems, and the creation of audit and other clinical tools.

Back to top

Future updates

The Development of Clinical Practice Guidelines Using Cancer Council Australia’s Cancer Guidelines Wiki: Handbook for section authors and the guideline working party outlines Cancer Council Australia’s guidelines updating processes. The incoming literature updates will continue to be monitored for each systematic review question. The Working Party will notify the Technical Team if any clinical question requires revision because new high level evidence has been published. External stakeholders are encouraged to use the comment feature and notify us of any new evidence for a specific topic.

References

  1. 1.0 1.1 National Health and Medical Research Council. Procedures and requirements for meeting the NHMRC standard for clinical practice guidelines. Melbourne; 2011.
  2. National Health and Medical Research Council. NHMRC levels of evidence and grades for recommendations for guideline developers. Canberra: National Health and Medical Research Council; 2009 Available from: https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf.
  3. Clinical Guidelines Network Cancer Council Australia. Development of Clinical Practice Guidelines using Cancer Council Australia’s Cancer Guidelines Wiki. Handbook for section authors and the guideline working party. CCA Sydney; 2014 Available from: http://wiki.cancer.org.au/australiawiki/images/9/9b/CCA_Clinical_Practice_Guideline_Development_Handbook.pdf.
  4. Pflugfelder A, Kochs C, Blum A, Capellaro M, Czeschik C, Dettenborn T, et al. Malignant melanoma S3-guideline "diagnosis, therapy and follow-up of melanoma". J Dtsch Dermatol Ges 2013 Aug;11 Suppl 6:1-116, 1-126. doi: 10.1111/ddg.12113_suppl.
  5. 5.0 5.1 Brouwers M, Kho ME, Browman GP, Burgers JS, Cluzeau F, Feder G, et al. AGREE II: Advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010;doi:10.1503/cmaj.090449 Abstract available at http://www.agreetrust.org/agree-ii/.
  6. ADAPTE Collaboration, Fervers B, Burgers JS, Voellinger R, Brouwers M, Browman GP, et al. Guideline adaptation: an approach to enhance efficiency in guideline development and improve utilisation. BMJ Qual Saf 2011 Mar;20(3):228-36 Abstract available at http://www.ncbi.nlm.nih.gov/pubmed/21209134.
  7. National Health and Medical Research Council. NHMRC levels of evidence and grades for recommendations for guideline developers. Canberra: National Health and Medical Research Council; 2009 Available from: https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf.
  8. National Institute of Clinical Studies. Do guidelines make a difference to health outcomes?; 2006 Available from: https://www.nhmrc.gov.au/_files_nhmrc/file/nics/material_resources/Do%20guidelines%20make%20a%20difference%20to%20health%20care%20outcomes.pdf.
  9. Francke AL, Smit MC, de Veer AJE, Mistiaen P. Factors influencing the implementation of clinical guidelines for health care professionals: A systematic meta-review. BMC Med Inform Decis Mak 2008;8, (38).

Back to top