Special Acupuncture Research Considerations

There are several aspects that make reading acupuncture research different from reading other clinical research, and key terminology that is important to understand before diving in. This information will likely be of most interest to other acupuncturists or acupuncture students as well as anyone who does have some general research knowledge but is beginning to read more of the literature associated with acupuncture and East Asian Medicine.

What is a Systematic Review? What is a Cochrane Review?

A Systematic Review (SR) is a compilation of the data from all the research trials (which meet certain predefined criteria) studying a particular topic, for example, acupuncture for treating lower back pain or pelvic pain during pregnancy. The studies are always Randomized Controlled Trials (RCT’s), seen as the “gold standard” in a clinical study. The authors look at all the existing studies, examine their similarities and differences, and perform a statistical analysis of the results. This allows more certain conclusions to be drawn since there’s a larger pool of data. The larger the pool of data, generally the more certainty we have that the results are not due to chance. Different groups of authors might all write their own SR on a particular topic, and as new studies are published, new SR’s will be published.

This is not to say that a Systematic Review’s conclusions are set in stone: some of these papers have problems with the way they were written or analyzed the data, or they might still be based on a relatively small amount of information. However they are an important part of performing research and hold more weight than any single research study.

A Cochrane Review is a systematic review on a topic in health care which is written according to certain criteria set forth by Cochrane. It’s generally seen as the highest quality type of Systematic Review, and only looks at RCT’s. There is a Cochrane Review available for almost any health care topic, and they are usually updated every few years as new research becomes available. The authors of a Cochrane Review follow a specific procedure in choosing their subject matter, the evidence included and the way it’s analyzed and reported. The SR is reviewed by the Cochrane Editorial Board for quality and accuracy. The authors are independent and Cochrane does not accept commercial funding or any funding that might pose a conflict of interest.

One aspect of Cochrane Reviews that I appreciate is that they make free plain-language summaries available for each topic. So even if you aren’t a researcher, you can search for a summary and understand what conclusions have been made and why.

Cochrane reviews do not address the quality of the intervention- meaning, how applicable it was to real-life practice. For more information on this issue, see the next section. Also, it’s important to remember that Cochrane reviews only include Randomized Controlled Trials, not other types of studies. These other studies may still have value and lend an important contribution to the body of evidence, so keep in mind that outside of the Cochrane review there may be other important information to consider.

How Do We Know The Acupuncture (Points, Method, Etc.) Used In A Study Was Clinically Relevant? How Do We Know A Study Was of Good Quality?

Imagine this scenario: A study is performed to find out how effective antibiotics are for ear infections. However the antibiotic chosen for use in the study is one that we know is totally inappropriate for treating ear infections. Therefore the researchers conclude that because the infection wasn’t cleared, antibiotics aren’t an effective treatment for ear infections. Of course we can understand this is quite bogus, but if you aren’t familiar with what antibiotics are generally used for ear infections, how would you know that the study should be ignored completely?

Unfortunately, we do at times come across these types of issues in acupuncture research. Although quality standards have been developed by researchers in order to improve reporting of acupuncture trials (for example, did the authors report on the possible types of bias? Did they outline in detail what was done so it may be repeated in the future?), until recently there wasn’t a way to assess the clinical relevance of the acupuncture or other East Asian Medicine technique utilized. Thankfully a quality rating scale called NICMAN has been developed to address this issue.

The NICMAN scale was initially developed in 2011 and has been gradually refined. If the paper you are reading does not assess the quality of the acupuncture intervention using NICMAN, it’s a good idea as an acupuncturist to apply the scale to the study yourself! If you’re not an acupuncturist, ask one what they think of the points used and style of acupuncture. The other way to make sure a research paper has followed accepted and recommended reporting standards is to check on whether they have followed STRICTA or at a minimum CONSORT guidelines. STRICTA is a set of guidelines specifically for acupuncture research, and is an official extension of CONSORT, developed in collaboration with the CONSORT group. CONSORT is a set of general standards for reporting of clinical trials.

For more detailed information and links on these three different sets of guidelines, see below:

STRICTA– STandards for Reporting Interventions in Clinical Trials of Acupuncture. This is an official extension of CONSORT, developed in collaboration with the CONSORT group. First published in 2001. Click the link for the official checklist.

CONSORT– CONsolidated Standards Of Reporting Trials. Industry-wide accepted general standards for reporting of randomized controlled trials.

NICMAN– National Institute for Complementary Medicine Acupuncture Network. This is a network of acupuncture researchers in Australia who developed the standards, initially developed 2011. It is intended to be used in conjunction with STRICTA, and evaluates the quality of the acupuncture used in an acupuncture randomized controlled trial or systematic review.

Important Writings on The Various Guidelines:

The impact of STRICTA and CONSORT on reporting of randomized control trials of acupuncture: A systematic methodological evaluation- by Simen Svenkerud and Hugh MacPherson

This paper, published in 2017, looked at 207 acupuncture trials between 1994-2015. The researchers found an improvement in reporting of STRICTA and CONSORT items over this time period. “Journals that endorse STRICTA have a better record in terms of reporting quality. Some evidence suggests that the publication of STRICTA has had a positive impact on reporting quality.”

Revised STandards for Reporting Interventions in Clinical Trials of Acupuncture: Extending the CONSORT Statement: Hugh McPherson et al.

This paper, published in 2010, describes the process of revising the STRICTA guidelines (first developed in 2001). It describes the collaborative effort of the CONSORT group, STRICTA group as well as the Chinese Cochrane Center and the Chinese Center for Evidence-based Medicine. Each item in the revised STRICTA guidelines is described in detail.

Reliability of the NICMAN Scale An Instrument To Assess the Quality of Acupuncture Administered in Clinical Trials: Caroline Smith et al.

This paper describes why NICMAN was needed in addition to STRICTA: “There are now clear standards for publications to report acupuncture parameters used in clinical studies, and the time is right to address the related but separate issue of evaluating the quality of acupuncture performed during these studies. Ensuring adequate reporting of a study may contribute to improving its reliability but does not address the quality of intervention applied (i.e., validity, standards, or adequacy).”

What is the Difference Between Effectiveness vs Efficacy?

“Efficacy and effectiveness trials have different goals, use different tools, and generate different messages.” – Porzsolt et al 2015

Since learning the very large difference between Efficacy and Effectiveness, I’ve noticed that the terms are misused absolutely all the time, especially on blogs, news articles and even by experts (who simply aren’t also research experts). They seem to be used interchangeably, and they should not be. They are very different.

Efficacy refers to “a biologic effect (which) can be observed under ideal study conditions.”- Porzsolt et al 2015. One example of efficacy in acupuncture research would be a study of the change in blood levels of neurotransmitters when the LI 4 point is stimulated in a very specific way.
Effectiveness “means an effect is detected not under ideal but under real-world conditions.” – Porzsolt et al 2015. One example of effectiveness in acupuncture research would be the change in the level of pain reported after a series of individualized acupuncture treatments are given.

Porzsolt et al. state an important point: “The efficacy of an intervention demonstrated under ideal study conditions (explanatory trial) will not necessarily predict the effectiveness of the same intervention described under real-world conditions (pragmatic trial).” This means that just because that LI 4 point (see example above) produced a change in blood levels of neurotransmitters, doesn’t necessarily mean it will make a meaningful difference in the life of the person. This also brings up a discussion on the difference between statistical significance and clinical significance.

A treatment is said to create statistically significant change when the effect is large enough (and there are enough data points) to show that it likely wasn’t due to chance alone. This is undoubtedly an important measure of the true effect of a treatment, but it isn’t everything! The result of a treatment can be statistically significant, but it may or may not also be clinically significant. Something is clinically significant if it makes a difference in the actual outcome, such as a decreased rate of cesarean births or decrease in the rate of epidurals given during labor.

The key thing to remember is that most (though not all) acupuncture trials measure effectiveness, and should be discussed in such terms. If you hear someone say “The efficacy of acupuncture treatments for lower back pain was studied”, it may be worth looking more closely to determine if they are really talking about effectiveness, and if so, whether this a reliable source of information?

What is A Placebo in Acupuncture Research?

The topic of placebo or sham acupuncture controls is a very complicated one in acupuncture research, and not one where there is universal agreement. However I’ll outline some of the key findings and things to keep in mind whenever you see a study that compares acupuncture to a placebo, sham or other type of minimal acupuncture as a control.

Researchers have been attempting to come up with an acceptable placebo control for acupuncture treatments for many years. These placebos have run the gamut from tapping a point with no needle insertion to inserting a needle in a non-acupuncture point, to a special device that gives the appearance that an acupuncture needle is inserted into the body, when really it has not penetrated the body at all.

A placebo should be physiologically inert; such as when comparing a new medication to the placebo of a sugar pill or empty capsule. A placebo should not create any of the physiological effects of the treatment. There isn’t yet clarity on what exactly the physiological effects specific to acupuncture are, so this of course makes it difficult to create a sham acupuncture treatment which doesn’t have any of these effects.

After much analysis, researchers have found that acupuncture placebos/sham acupuncture treatments are not physiologically inert. How do we know this? According to Langevin et al., “Sham acupuncture treatments have consistently been shown to have greater therapeutic effects than conventional placebos (2).” In addition, when comparing studies where acupuncture treatment is compared to no treatment at all or usual care, the effect size is consistently larger than studies where acupuncture treatment is compared to a “sham” acupuncture placebo. These “sham” acupuncture treatments are thought to have some degree of physiological effect in combination with some degree of placebo effect. The amount of each is impossible to know at present time, because the exact mechanisms of acupuncture are still being determined.

At the current time, researchers (including at the NIH) are prioritizing acupuncture mechanisms research as well as studies that compare acupuncture to usual care or to another modality (for example, acupuncture compared to physical therapy for lower back pain) (1).

A Few Other Important Research Topics with Links

Article explaining the value of pragmatic studies and qualitative analysis in acupuncture research: “Why Acupuncture Is More Than Just Needling, and the Implications for Research”- Hugh MacPherson
An article discussing the often overlooked issue of acupuncture dosage in studies- what is an appropriate and effective course of treatment and how can research begin to examine this more closely? Bauer, Matthew et al. “Is acupuncture dose dependent? Ramifications of acupuncture treatment dose within clinical practice and trials.”

Resource Links

Society for Acupuncture Research (SAR)

AACMA (Australian Acupuncture and Chinese Medicine Association)

British Acupuncture Council Factsheets

Evidence Based Acupuncture

Acupuncture Now Foundation

The Acupuncture Evidence Project

Debra Betts’ Perinatal Acupuncture Research Summaries

MAMPS Resources Guide: Links to Individual Studies and Cochrane Reviews on Perinatal Acupuncture

Other Useful Articles:

Is this study legit? 5 questions to ask when reading news stories of medical research

Sources

McDonald J, Janz S. The Acupuncture Evidence Project: A Comparative Literature Review (Revised edition). Brisbane: Australian Acupuncture and Chinese Medicine Association Ltd; 2017. http://www.acupuncture.org.au.
Helene M. Langevin, Peter M. Wayne, Hugh MacPherson, Rosa Schnyer, Ryan M. Milley, Vitaly Napadow, Lixing Lao, Jongbae Park, Richard E. Harris, Misha Cohen, Karen J. Sherman, Aviad Haramati, Richard Hammerschlag, "Paradoxes in Acupuncture Research: Strategies for Moving Forward", Evidence-Based Complementary and Alternative Medicine, vol. 2011, Article ID 180805, 11 pages, 2011. https://doi.org/10.1155/2011/180805
https://www.acunow.org/putting-acupuncture-research-into-perspective.html

Special Acupuncture Research Considerations

Resource Links

Sources

Health Topics

Lower back pain or pelvic pain

Nausea/Vomiting and Hyperemesis Gravidarum

Prebirth and Labor Induction

Pain Management in Labor

Safety of Acupuncture During Pregnancy

Anxiety and Depression

Moxa for Breech Presentation

Threatened Miscarriage

Pre-Eclampsia

Insomnia