7 3/13 Lab VI | Experiments
7.1 Preparation
library(knitr)
library(haven)
library(tidyverse)
library(car)
library(memisc)
load("~/GOVT702/Data/EgyptProtests.RData")Williamson and Malik (2020) used a survey experiment to examine the public response to how Egyptians responded to different framings regarding a protest in which protesters were killed by police.
All respondents were given the following statement: “Last year, police raided an apartment in 6 October City. During the raid, they killed 9 members of the Muslim Brotherhood.”
Respondents were assigned with equal probability to a control group that got no more information or one of three treatment groups that received additional information.
Respondents in the first treatment group were provided a paragraph that justified the killings from the security forces perspective.
Respondents in the second treatment group were provided a paragraph that criticized the killings from a human rights perspective.
Respondents in the third treatment group were provided both the security forces and human rights perspectives.
Two outcomes were measured:
if respondents thought the police tactics were justified, and
if the police should the police be held accountable for killing the nine men. So that both variables are coded in same direction, this variable is coded as 1 if the respondent gave a pro-police answer (saying the police did not need to be held accountable).
Data description
| Variable | Description |
|---|---|
| just_binary_main | 1 if the respondent said the police tactics were justified; 0 otherwise. |
| not_acc_binary_main | 1 if the respondent said the police should definitely or probably not be held accountable; 0 otherwise. We call this variable no-accountability |
| treat_b1 | 1 if respondent was given the pro-police paragraph |
| treat_b2 | 1 if respondent was given the human rights paragraph |
| treat_b3 | 1 if respondent was given both the pro-police and human rights paragraphs |
| sisi_vote | 1 if the respondent voted for Sisi |
| not_voteboycott | 1 if the respondent did not vote for some other reason than boycotting the election |
| opposition_vote_boycott | 1 if the respondent either voted for the opposition candidate or boycotted the election |
| education | from 1 (no formal education) to 8 (graduate degree). Values from 6 to 8 went to college. |
| income | 1 = Less than 500 pounds per month, 2 = 500 to 1000 pounds per month, 3 = 1000 to 4000 pounds per month, 4 = 4000 to 10000 pounds per month, 5 = More than 10000 pounds per month |
| male | 1 for men, 0 otherwise |
| age_cat | age categories: 1 is less than 35, 2 is 35 to 50 and 3 is over 50. |
Published paper (paywalled)
7.2 Why is an experiment useful in this context?
Simply asking respondents if they agree or disagree with the police does not answer if and how different messages work. If respondents were simply asked if they agree with the either the pro-police or human rights messages it may be difficult to separate out whether the message was persuasive or whether the respondent was predisposed to approve of the argument.
7.3 Is there evidence of balance in the treatment? Why is this relevant? Check for balance of each treatment with respect to age category, gender, education, income, voting for Sisi and not voting for non-boycott reasons. Use two different procedures to conduct your balance tests. You do not need to use TOST!
## Removing NAs
dta <- dta[is.na(dta$just_binary_main)==0 & is.na(dta$not_acc_binary_main)==0,]
dep_vars <- dta %>%
dplyr::select(age_cat, male, education, sisi_vote, not_voteboycott, income)
b_test <- lapply(dep_vars, function(bal) {
lm(bal ~ dta$treat_b1 + dta$treat_b2 + dta$treat_b3)})
## Extracting Slopes and P Values
slopes <- round(sapply(b_test, function(x) x$coefficients), 2)
p <- round(sapply(b_test, function(x) {
summary(x)$coefficients[,4]}), 2)
## Showing Results
balance_results = data.frame(b1= slopes[2,], b1.p = p[2,],
b2= slopes[3,], b2.p = p[3,],
b3= slopes[4,], b3.p = p[4,])
balance_results## b1 b1.p b2 b2.p b3 b3.p
## age_cat 0.05 0.60 0.01 0.89 -0.01 0.89
## male 0.01 0.77 0.00 0.90 0.05 0.18
## education 0.45 0.01 0.27 0.12 0.04 0.82
## sisi_vote 0.02 0.74 0.05 0.36 0.09 0.12
## not_voteboycott -0.02 0.71 -0.02 0.76 -0.05 0.37
## income 0.18 0.10 0.06 0.55 0.24 0.03We can do the same with a loop.
## DV Creation
dvs <- c("dta$age_cat", "dta$male", "dta$education", "dta$income",
"dta$sisi_vote", "dta$not_voteboycott")
## Loop for Treatments
for(i in 1:length(dvs)){
model <- paste("model",i, sep="")
m <- lm(as.formula(paste(dvs[i],"~treat_b1 + treat_b2 + treat_b3")),
data=dta)
assign(model,m)}
mtable(model1, model2, model3, model4, model5, model6)There is some evidence of imbalance.
7.4 What is the effect of the treatments on whether respondents viewed the police tactics as justified? Run one model with only the treatment variables and another that includes controls for age category, gender, education, income, voting for Sisi and not voting for non-boycott reasons.
## Without Controls
dta %>%
lm(just_binary_main~treat_b1 + treat_b2 + treat_b3, data=.) %>%
summary()##
## Call:
## lm(formula = just_binary_main ~ treat_b1 + treat_b2 + treat_b3,
## data = .)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.4589 -0.3453 -0.3154 0.5411 0.6846
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.31544 0.03923 8.042 5.02e-15 ***
## treat_b1 0.14347 0.05576 2.573 0.0103 *
## treat_b2 0.01342 0.05547 0.242 0.8089
## treat_b3 0.02989 0.05646 0.529 0.5968
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4788 on 579 degrees of freedom
## Multiple R-squared: 0.01409, Adjusted R-squared: 0.008977
## F-statistic: 2.757 on 3 and 579 DF, p-value: 0.04167## With Controls
dta %>%
lm(just_binary_main~treat_b1 + treat_b2 + treat_b3 + factor(age_cat) +
male + education + income + sisi_vote + not_voteboycott, data=.) %>%
summary()##
## Call:
## lm(formula = just_binary_main ~ treat_b1 + treat_b2 + treat_b3 +
## factor(age_cat) + male + education + income + sisi_vote +
## not_voteboycott, data = .)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.7901 -0.1992 -0.1011 0.3328 0.9448
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.0771076 0.1015603 0.759 0.4481
## treat_b1 0.1189739 0.0499768 2.381 0.0176 *
## treat_b2 -0.0249841 0.0496725 -0.503 0.6152
## treat_b3 -0.0169674 0.0506942 -0.335 0.7380
## factor(age_cat)2 -0.0362495 0.0425449 -0.852 0.3946
## factor(age_cat)3 0.0005895 0.0471999 0.012 0.9900
## male 0.0494751 0.0534350 0.926 0.3549
## education -0.0007622 0.0125688 -0.061 0.9517
## income -0.0010071 0.0209086 -0.048 0.9616
## sisi_voteTRUE 0.5494024 0.0414417 13.257 <2e-16 ***
## not_voteboycottTRUE 0.0806941 0.0453807 1.778 0.0760 .
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.4036 on 516 degrees of freedom
## (56 observations deleted due to missingness)
## Multiple R-squared: 0.2926, Adjusted R-squared: 0.2789
## F-statistic: 21.34 on 10 and 516 DF, p-value: < 2.2e-167.5 Test whether the effects of treatment 1 (police-favorable message) are the same as treatment 3 (both sides) for each dependent variable. Feel free to use the linearHypothesis function in the car package.
reg.1 <- lm(just_binary_main~treat_b1 + treat_b2 + treat_b3 +
factor(age_cat) + male + education + income +
sisi_vote + not_voteboycott, data=dta)
linearHypothesis(reg.1, "treat_b1=treat_b3")## Linear hypothesis test
##
## Hypothesis:
## treat_b1 - treat_b3 = 0
##
## Model 1: restricted model
## Model 2: just_binary_main ~ treat_b1 + treat_b2 + treat_b3 + factor(age_cat) +
## male + education + income + sisi_vote + not_voteboycott
##
## Res.Df RSS Df Sum of Sq F Pr(>F)
## 1 517 85.236
## 2 516 84.062 1 1.1742 7.2075 0.007494 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 17.6 In some political contexts, messages have a different effect on people with less engagement in politics. Are there heterogeneous treatment effects by those who did not vote for reasons other than to boycott the election and those who did not? Feel free to explore multiple other ways there could be heterogeneous effects (e.g., perhaps there are different effects on Sisi supporters), but you only need to report on possible differential effects of the treatments on the “not-accountable” dependent variable.
dta %>%
lm(not_acc_binary_main ~ not_voteboycott*treat_b1 + not_voteboycott*treat_b2 +
not_voteboycott*treat_b3 +factor(age_cat) + male +
education + income + sisi_vote, data = .) %>%
summary()##
## Call:
## lm(formula = not_acc_binary_main ~ not_voteboycott * treat_b1 +
## not_voteboycott * treat_b2 + not_voteboycott * treat_b3 +
## factor(age_cat) + male + education + income + sisi_vote,
## data = .)
##
## Residuals:
## Min 1Q Median 3Q Max
## -0.58236 -0.22916 -0.05162 0.03384 0.99812
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 0.022268 0.095838 0.232 0.81636
## not_voteboycottTRUE 0.250014 0.075859 3.296 0.00105 **
## treat_b1 0.105574 0.053546 1.972 0.04919 *
## treat_b2 0.038517 0.053542 0.719 0.47223
## treat_b3 0.144074 0.054048 2.666 0.00793 **
## factor(age_cat)2 0.009428 0.039361 0.240 0.81078
## factor(age_cat)3 0.015627 0.043724 0.357 0.72094
## male -0.017692 0.049679 -0.356 0.72189
## education -0.007508 0.011635 -0.645 0.51902
## income 0.006283 0.019388 0.324 0.74602
## sisi_voteTRUE 0.434241 0.038357 11.321 < 2e-16 ***
## not_voteboycottTRUE:treat_b1 -0.091467 0.105260 -0.869 0.38527
## not_voteboycottTRUE:treat_b2 -0.212318 0.104539 -2.031 0.04277 *
## not_voteboycottTRUE:treat_b3 -0.337778 0.108394 -3.116 0.00193 **
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.3733 on 513 degrees of freedom
## (56 observations deleted due to missingness)
## Multiple R-squared: 0.2461, Adjusted R-squared: 0.2269
## F-statistic: 12.88 on 13 and 513 DF, p-value: < 2.2e-16