Evidence-Based Medicine

.title[
# Evidence-Based Medicine - Part 1
]
.author[
### Austin Meyer, MD, PhD, MPH, MS, FAAP, etc.
]
.date[
### August 25, 2023
]

---

# Agenda

**Basic Statistics - times**
 
1. Some Terms - 3 minutes
 
2. Accuracy versus Precision - 5 minutes
 
3. Statistical inference - 5 minutes
 
5. Brief hypothesis testing (will get more with epi stuff) - 10 minutes
 
6. Probability - 27 minutes
---

# Basic Statistical Terms

---

# There are some important terms

- **Generalizability**
  - How applicable is a finding from a particular sample to the population
--

- **P-value**
  - Probability of finding a value this extreme by random chance (given the assumption of a true null)
--

- **Confidence Interval**
  - Interval over which population (true?) value is expected to be found with a specified probability (e.g. 95%)
--

- **Efficacy**
  - Performance of treatment under ideal circumstances
--

- **Effectiveness**
  - Performance of treatment under real world circumstances

---

# Regarding generalizability

**The boards wants you to know these terms**

- **Internal Validity**
  - Is the extent to which the study shows that the indendepent variable is causing the depdent variable
  - _For instance, if a drug trial finds that a new medication reduces symptoms of a disease, internal validity would address whether this reduction is truly due to the medication and not some other factor like participants' expectations, the placebo effect, or other treatments they might be receiving._

--
- **External Validity**
  - Is the extent to which the study can be generalized to pts not in the study population
  - Typically an intuitive judgment call unless external data exists for analysis
  
---

# Accuracy and Precision

---

# Reminder of Accuracy v. Precision

---

# Accuracy v. Precision in words

1. Accuracy is how close an estimate is to the true value

2. Precision is how much variation exists in the estimate

---

# Statistical Inference

---

# Statistical distributions have invariant properties

---

# Real distributions can have one or multiple peaks

---

# Skew describes the direction of the tail

---

# Question
 
.pull-left[
Which of the following corresponds to the measures of central tendency on the graph from **left to right**?

1. mean, median, mode
2. mode, mean, median
3. median, mode, mean
4. mode, median, mean
5. mean, mode, median
]

---

# Answer
 
.pull-left[
Which of the following corresponds to the measures of central tendency on the graph from **left to right**?

1. mean, median, mode
2. mode, mean, median
3. median, mode, mean
4. **mode, median, mean**
5. mean, mode, median
]

---

# Notes about statistical inference

- The goal of these measures is to produce a single number to represent the whole population

- Obviously we generally can't get data on the whole population; so we sample

- When we sample and estimate a value, we try to report external validity with a confidence interval

- The confidence interval at some probability means the estimated probability of finding the true population value in that interval

- If the distribution looks weird, you could generate a CI by bootstrapping... describe

---

# Hypothesis Testing

---

# Measures of association

### Data type goes with hypothesis test
--

- **Counts of 2 or more groups:** Chi-square test, Fisher exact (only 2x2 contingency)
 
- **Quantitative variable + grouping variable(s):** T-test (data approximates a t-distribution), U-test (distribution not well defined), ANOVA
 
- **Quantitative variable + ranking variable:** Spearman correlation
 
- **Quantitative and 2 variables:** Pearson correlation

---

# Was on my boards for both Peds and IM

.pull-right[.left[**Which of the following is the best method to assess the association between OCP use and blood clots?** A. Two sample T-test B. Analysis of variance C. Pearson correlation D. Chi-square test E. Spearman correlation ]]

<table style="margin-left:6%;">
<tr><th align="left"></th><th align="right">Clot</th><th align="right">No Clot</th><th align="right">Total</th></tr>
<tr><td align="left">OCP Use</td><td align="right">500</td><td align="right">400</td><td align="right">900</td></tr>
<tr><td align="left">No OCP Use</td><td align="right">80</td><td align="right">20</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">580</td><td align="right">420</td><td align="right">1000</td></tr>
</table>

---

# Was on my boards for both Peds and IM

---

# Why?

The data are counts.

---

# The null hypothesis `\(H_0\)` is the default

- Assume:
  - There are two or more groups being compared, or 
  - One group is being compared to zero, or
  - One group is being compared to expectation.
  - Slope of a line is zero

--

- For boards, probably safe to assume null is always rejected with `\(p < 0.05\)`.
- For ratios (e.g. Relative Risk, Odds Ratio), a 95% CI __not__ overlapping 1 is significant.
- For two sample tests, it is less straightforward how the CI relates to the p-value so don't worry about it.

--

- Once `\(H_0\)` is rejected, we accept the alternative hypothesis `\(H_A\)`.

---

# Hypothesis testing: 4 possible outcomes

.pull-right[
- Correct - Reject a false `\(H_0\)`
  - Probability of success is called "power"
  - Power depends on sample size
  - bigger sample = bigger power

- Correct - Fail to reject a true `\(H_0\)`
  - Probability determined by `\(\alpha\)` as `\(1−\alpha\)`

- Type 1 - Incorrect rejection of a true `\(H_0\)`
  - False Positive

- Type 2 - Failure to reject a false `\(H_0\)`
  - False Negative]

---

# Common Clinical/Epi Metrics

**Calculating NNT was on my boards**

1. Odds Ratio (OR)
2. Risk Ratio (RR)
3. Risk Difference (RD)
4. Number Needed to Treat (NNT)

---

# Here is some data that I randomly sampled

---

# Here is the same data

<table>
<tr><th align="left"></th><th align="right">Disease</th><th align="right">No Disease</th><th align="right">Total</th></tr>
<tr><td align="left">Exposed</td><td align="right">30</td><td align="right">70</td><td align="right">100</td></tr>
<tr><td align="left">Not Exposed</td><td align="right">20</td><td align="right">80</td><td align="right">100</td></tr>
<tr><td align="left">Total</td><td align="right">50</td><td align="right">150</td><td align="right">200</td></tr>
</table>

---

**How do I calculate odds?**

**How do I calculate risk?**

---

# Odds Ratio (OR)

.pull-right[
.left[
`\(OR = \frac{D_E * ND_{NE}}{D_{NE} * ND_E} = \frac{30 * 80}{20 * 70} =\)` **1.71**
 
`\(CI = e ^ {ln(OR) \pm 1.96 * sqrt(\frac{1}{a} + \frac{1}{b} + \frac{1}{c} + \frac{1}{d})}=\)` (**0.89**, **3.29**)
]
]

---

# Risk Ratio (RR)
**Only use on random samples**

.pull-right[
.left[
`\(R_E = \frac{D_E}{D_E + ND_E} = \frac{30}{30 + 70} = 0.3\)` 
`\(R_{NE} = \frac{D_{NE}}{D_{NE} + ND_{NE}} = \frac{20}{20 + 80} = 0.2\)` 
`\(RR = \frac{R_E}{R_{NE}} = \frac{0.3}{0.2} =\)` **1.5**
 
`\(CI = e ^ {ln(RR) \pm 1.96 * sqrt(\frac{1}{a} + \frac{1}{c} \pm \frac{1}{a+b} - \frac{1}{c+d})}=\)` (**0.92**, **2.64**)
]
]

---

# Risk Difference (RD)

---

# Number Needed to Harm or Treat (NNH or NNT)

.pull-right[
.left[
`\(R_E = \frac{D_E}{D_E + ND_E} = \frac{30}{30 + 70} = 0.3\)` 
`\(R_{NE} = \frac{D_{NE}}{D_{NE} + ND_{NE}} = \frac{20}{20 + 80} = 0.2\)` 
`\(RD = R_E - R_{NE} = 0.3 - 0.2 = 0.1\)` 
`\(NNH = \frac{1}{0.1} =\)` **10**
]
]

---

# Brief probability

---

**The goal here is not that you should memorize how to calculate these. It is that you should be familiar with them, know they they are relatively accessible even for non-statisticians, and be able to look up if needed.**

---

# What are sensitivity and specificity?

---

# Tests are cutoffs on a continuous variable
 
![](index_files/figure-html/unnamed-chunk-16-1.png)

---

# Sensitivity:

Efficiency of finding **TRUE POSITIVES TESTS** in All Real Positive Cases
 
![](index_files/figure-html/unnamed-chunk-17-1.png)

---

# Specificity:

Efficiency of finding **TRUE NEGATIVES TESTS** in All Real Negative Cases

![](index_files/figure-html/unnamed-chunk-18-1.png)

---

# PPV and NPV

.pull-left[
- __Positive Predictive Value__
- Chance that person has the disease after a positive test result
  - `\(PPV = \frac{TP}{TP + FP}\)`
]

.pull-right[
- __Negative Predictive Value__
- Chance that person does not have disease after a negative test result
  - `\(NPV = \frac{TN}{TN + FN}\)`
]

- PPV and NPV depend on how prevalent the disease is in the population

---

# Sensitivity and Specificity

--
- __These are the defining characteristics of any clinical finding (e.g. history, physical, test, image)__.

--
- They do not depend on anything... they are intrinsic to the exam/test

--
- If someone quotes the positive or negative predictive value of a test, they are wrong.

--
- Therefore, if you do not know the sensitivity or specificity of a test, you are missing information
- Without sensitivity and specificity, you cannot make a ROC curve

--
- Be skeptical of a paper, poster, or company presentation that does not include a ROC curve
  
.pull-left[
- __Sensitivity__
  - `\(Sensitivity = \frac{TruePositives}{AllRealPositives}\)`
  - `\(AllRealPositives = TP + FN\)`
]

.pull-right[
- __Specificity__
  - `\(Specificity = \frac{TrueNegatives}{AllRealNegatives}\)`
  - `\(AllRealNegatives = TN + FP\)`
]

---

# PPV and NPV vary with probability

Unlike Sens and Spec, PPV and NPV vary with pre-test probability

![](index_files/figure-html/unnamed-chunk-19-1.png)

---

# Predictive values versus test metrics

![](index_files/figure-html/unnamed-chunk-20-1.png)

---

# Predictive values versus test metrics

![](index_files/figure-html/unnamed-chunk-21-1.png)

---

# Predictive values versus test metrics
PPV and NPV are post-test probabilities that are chained to the prevalence of disease in the population where the sensitivity and specificity were calculated

---

# Has anyone seen one of these?

**These nomograms are silly and ridiculous, and shouldn't exist**

---

# Calculating post-test probability #1

**If you have a contingency table or sensitivity/specificity these are easy to calculate**

`\(S_e = \frac{a}{a+c}\)` 
`\(S_p = \frac{d}{b+d}\)`

--

**You can use disease prevalence as a naive pre-test probability**

**Think flu prevalence in the winter versus summer for a presenting toddler**
---

# Calculating post-test probability #2

**You can make likelihood ratios from sensitivity and specificity**

`\(LR_+ = \frac{S_e}{(1-S_p)} = \frac{\frac{a}{a+c}}{\frac{b}{b+d}}\)` 
`\(LR_- = \frac{(1-S_e)}{S_p} = \frac{\frac{c}{a+c}}{\frac{d}{b+d}}\)`

---

# Calculating post-test probability #3

- Once you know if test was positive or negative you pick the LR you need 
  - `\(LR_+\)` for positive test 
  - `\(LR_-\)` for negative test

1. **Convert pretest probability to pretest odds**
  - `\(odds_{pretest} = \frac{prob_{pretest}}{1-prob_{pretest}}\)`
  
2. **Multiply odds by relevant likelihood ratio to get post test odds**
  - `\(odds_{posttest} = odds_{pretest} * LR\)`
  
3. **Convert posttest odds back to probability**
  - `\(prob_{posttest} = \frac{odds_{posttest}}{odds_{posttest}+1}\)`

---

# Calculating post-test probability #4

**Here is a full example**

--
- Prevalence of HIV is ~1 in 300 in the US. So 0.333%

--
- Sensitivity of Gen-4 HIV test outside of window is 99.7% 
- Specificity of Gen-4 HIV test outside of window is 99.8%

--
- Let's say someone tests positive - what is the probability they are really infected?

1. Calculate `\(LR_+\)` for positive test: `\(LR_+ = \frac{0.997}{1-0.998} = 498.5\)`

2. Convert pretest probability (prevalence) to pretest odds
  - `\(odd_{pretest} = \frac{prob_{pretest}}{1-prob_{pretest}} = \frac{0.00333}{1-0.00333} = 0.00334\)`
  - note pretty much the same as probability, which is always true for low prevalence/probability diseases
  
--

3. Multiply pretest odds by LR to get posttest odds
  - `\(odds_{posttest} = odds_{pretest} * LR_+ = 0.00334 * 498.5 = 1.66499\)`
  
--

4. Convert posttest odds to posttest probability
  - `\(prob_{posttest} = \frac{odds_{posttest}}{odds_{posttest}+1} = \frac{1.66499}{1.66499+1} = 0.6247\)`

**So probability of true infection out of window with positive 4th gen test is 63%**

---

# The End