Structural after Measurement vs. vanilla SEM estimation - a Case Study on Adversarial Collaboration for Simulation Studies
The Generalizability Challenge & Adversarial Collaboration
An Adversarial Simulation Framework
Background (SAM vs. SEM)
Research Questions
Time for questions
Results
Collaboration
Evaluation and Future Directions
Discussion
general & verbal
researcher’s degrees of freedom
→ ambiguity
→ persistent disagreement
specific & empirical
In Simulation Studies
general & verbal
researchers degrees of freedom
→ ambiguity
→ persistant disagreement
specific & empirical
Identify general & verbal Disagreement → joint Research Question
Agree on:
- Operationalizations
- Test Design
- Interpretation
→ Reduce ambiguity
→ Increase generalizability
unify
| Steps of a Simulation Study | Content |
|---|---|
| 1. Research Question | Verbal description of Research Goals |
| 2. Population Model | Type, size, complexity |
| 3. Data Generation | E.g. resampling vs. parametric draw |
| 4. Experimental Design | Specifiy conditions (e.g., sample size) |
| 5. Method Selection | Type, implementation, number |
| 6. Estimands | Population level parameter values |
| 7. Performance Metrics | E.g. Bias, Coverage etc. |
| 8. Software | Applies to steps 2-7 |
| 9. Analysis | Decision criteria, graphical display etc. |
Siepe et al. (2023), Paxton et al. (2001), Morris et al. (2019)
| Round 1 | Round 2 | Round 1 | |
|---|---|---|---|
| Steps | Collaborator 1 | Joint Study | Collaborator 2 |
| 1. Research Question | Agreed upon prior to Round 1 | ||
| 2. Population Model | → | ← | |
| 3. Data Generation | → | ← | |
| 4. Experimental Design | → | ← | |
| 5. Method Selection | → | ← | |
| 6. Estimands | → | ← | |
| 7. Performance Metrics | → | ← | |
| 8. Software | → | ← | |
| 9. Analysis | → | ← |
Problems:
Two-phase process:
→
Rosseel & Loh (2022)
| Rosseel & Loh (2022), Dhaene & Rosseel (2023) | Robitzsch (2022) |
|---|---|
| - SAM outperformed SEM in terms of convergence, bias & RMSE in small samples x low item reliability, especially under misspecifications |
- SAM did not generally outperform traditional SEM in challenging conditions. - SAM appears better: general negative small sample bias of SAM cancels out positive bias from positive misspecifications. |
→ Basis for a case study on adversarial collaboration
| Rosseel & Loh (2022), Dhaene & Rosseel (2023) | Robitzsch (2022) |
|---|---|
| - SAM outperformed SEM in terms of convergence, bias & RMSE in small samples x low item reliability, especially under misspecifications |
- SAM did not generally outperform traditional SEM in challenging conditions. - SAM appears better: general negative small sample bias of SAM cancels out positive bias from positive misspecifications. |
| Replication by Kriegmair | Replication by Kosanke |
→ Basis for a case study on adversarial collaboration
Is Adversarial Collaboration a viable approach to address ambiguity and increase generalizability in simulation studies?
Substantive Questions wihtin the Case Study (agreed upon prior to individual studies)
Replication Results
Adversarial Collaboration
| Kriegmair | Kosanke |
|---|---|
| based on Rosseel & Loh (2022) and Dhaene & Rosseel (2023) | based on Robitzsch (2022) |
1.1 no misspecifications
1.2 cross loadings
1.3 correlated residuals
1.4 structural misspecification
| Kriegmair | Kosanke | |
|---|---|---|
| Verbal Dispute | low convergence rate and high rate of improper solutions only for SEM in challenging conditinos | no convergence issues |
| Method selection | un-bounded ML SEM | bounded ML SEM |
| Analysis | condition-wise rates | global rate |
direct:lavInspect(fit, "converged") |
indirect:quietly(safely(simulation_study_)) |
| Kriegmair | Kosanke | |
|---|---|---|
| Verbal Dispute | “SEM performes worse than SAM in low reliability x low sample size x misspecification” | “SAM generally did not outperform traditional SEM in small to moderate samples.” “[under] unmodelled negative cross-loadings and residual correlations, SAM tended to perform worse than traditional SEM” |

Sample Size
|
|||||||
|---|---|---|---|---|---|---|---|
| Method | 50 | 100 | 250 | 500 | 1000 | 2500 | 100000 |
| SEM ML | -0.205 | -0.175 | -0.166 | -0.168 | -0.161 | -0.166 | -0.164 |
| SEM ULS | -0.139 | -0.145 | -0.159 | -0.167 | -0.163 | -0.170 | -0.169 |
| LSAM ML | -0.498 | -0.385 | -0.272 | -0.225 | -0.196 | -0.189 | -0.180 |
| LSAM ULS | -0.497 | -0.385 | -0.272 | -0.225 | -0.196 | -0.189 | -0.180 |
| GSAM ML | -0.497 | -0.385 | -0.272 | -0.225 | -0.196 | -0.189 | -0.180 |
| GSAM ULS | -0.496 | -0.385 | -0.272 | -0.225 | -0.196 | -0.189 | -0.180 |
Relative bias of \(\hat{\phi}\) in conditions with two negative unmodelled residual correlations in a 2-factor-CFA
| Kriegmair | Kosanke | |
|---|---|---|
| Verbal Dispute | “SEM performs worse than SAM in low reliability x low sample size x misspecification” | “SAM generally did not outperform traditional SEM in small to moderate samples.” “[under] unmodeled negative cross-loadings and residual correlations, SAM tended to perform worse than traditional SEM” |
| 4. Experimental Design | N:100-6400 Reliability via Θ Positive cross loadings & correlated residuals |
N:50-100000 Reliability via \(\lambda\) Positive & negative cross loadings and correlated residuals (only in CFA) |
| Kriegmair | Kosanke | |
|---|---|---|
| Verbal Dispute | “SEM performs worse than SAM in low reliability x low sample size x misspecification” | “SAM generally did not outperform traditional SEM in small to moderate samples.” “[under] unmodeled negative cross-loadings and residual correlations, SAM tended to perform worse than traditional SEM” |
| 4. Experimental Design | N:100, 400, 6400 Reliability via \(\Theta\) Positive cross loadings & correlated residuals |
N:50, 100, 250, 500, 1000, 2500, 100000 Reliability via \(\Lambda\) Positive & negative cross loadings & correlated residuals (only in CFA) |
| 5. Population Model | 5-factor-SEM | 2-factor CFA & 5-factor SEM |
| 6. Analysis | Aggregated relative values and parameter-wise | Aggregated relative bias |
| Round 1 | Round 2 | Round 1 | |
|---|---|---|---|
| Steps | Kriegmair | Joint Study | Kosanke |
| Population Model | 5-factor-SEM | 5-factor-SEM |
5-factor-SEM 2-factor-CFA |
| Data Generation | parametric & normally distributed | parametric & normally distributed | parametric & normally distributed |
| Experimental Design |
Misspecifications (+) N: 100 - 6400 Reliability via Θ |
Misspecifications (+/-) N: 50, 100, 250, 400 Reliability via Θ |
Misspecifications (+/-) N: 50 - 100,000 Reliability via λ |
| Method Selection | SEM-ML, gSAM, lSAM (ULS & ML) | bounded SEM-ML, gSAM, lSAM (ULS & ML) | bounded SEM (ULS & ML), gSAM, lSAM (ULS & ML) |
| Round 1 | Round 2 | Round 1 | |
|---|---|---|---|
| Estimands | β: Fixed at 0.1 and varied | β: Fixed at 0.1 |
β: Fixed at 0.1 φ: Fixed and varied |
| Performance Metrics |
Absolute bias in absolute values Signed Relative bias 95% CI coverage Convergence & Improper Solutions |
Absolute bias in absolute and signed values RMSE 95% CI coverage Convergence & Improper Solutions |
Absolute bias in absolute values Signed Relative bias RMSE 95% CI coverage |
| Software | lavaan::simulateData() | lavaan::simulateData() | lavaan::simulateData() |
| Analysis |
Aggregated across parameters, heat maps |
Aggregated, parameter-wise, decision criteria heat maps |
Aggregated across parameters, decision criteria |
Updating Verbal Claims
SAM has no convergence rate advantage over bounded ML SEM
SAM outperforms SEM despite of negative bias under positive and negative cross loadings in small samples and small to moderate reliability
Diverging operationalizations:
- Method
- Design
- Reliability etc.
→ Reduced ambiguity
→ Increased generalizability?
unified
still limited to specific (somewhat less) arbitrary choices for simulation
→ empirically ground simulations by sampling models (and data) from the literature (Taxonomy.jl)
→ Establish the practice of (at least low-code) adversarial simulation by all co-authors of a study?