You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
My (quite inexpert) understanding is that CFI and RMSEA are the goodness-of-fit measures par excellence when it comes to CFA. It seems that Chi-squared is also useful.
@desilinguist, I'll have to double-check some of the formulas below, but I think RMSEA is pretty easy and do-able. (You can find a couplereferences here.)
Here are some of the steps, as far as I understand:
Implement the Chi-squared test statistics. I believe this is just chi2 = self.n_obs * res.fun, where fun comes directly from the minimize results object (see here).
Calculate the degrees of freedom. I believe this is (k * (k - 1) / 2), or dof = self.n_obs * (self.n_obs - 1) / 2, but we may also have to subtract out the number of model parameters at the end.
Calculate the p-value. Once we have the test statistics and the degrees of freedom, this should be as simple as 1 - scipy.stats.chi2.cdf(chi2, dof).
Calculate Root Mean Square Error of Approximation (RMSEA). There are a few different formulas I've seen, but I think they all reduce to rmsea = 0 if chi2 < dof else np.sqrt((chi2 / (dof * self.n_obs) - 1) / (self.n_obs - 1)) or something like that. I'll have to double check this , too. (We can look at psych to see how they implement.)
Getting the Comparative Fit Index (CFI) is a little more involved, since you have to calculate the Chi-squared test statistics for the baseline/null model, where all the variables are independent/uncorrelated. I'll read up on that a bit, but it does seem like a useful thing to add.
As an FYI, I believe that semopy has all of these implemented, but it's been a while since I've looked at that package. Not sure whether/how much we can borrow from that package. Looks like it's under the MIT license. (See here.)
Reacted by Nitin Madnani, Jimmy Bruno, nielsle, Kerby Shedden and eli knaap
This is amazingly helpful, @jbiggsets! I'll try and take a stab at RMSEA soon. It's funny you mention semopy since that's exactly what @jimmybru is using right now. I told him he should look into replacing that with factor analyzer but he needs these metrics to do that.
Reacted by jbiggsets, Jimmy Bruno, nielsle and ZhaoXianglin
Btw: this should probably be another issue, but just in case the answer is obvious, I’ll ask here: I don’t see a way in factor_analyer to specify covariances in the model like you can in semopy. Is that true? If so, is the default that all factors are orthogonal, or is everything allowed to covary with everything?
Activity
jbiggsets commentedon Apr 2, 2022
@desilinguist, I'll have to double-check some of the formulas below, but I think RMSEA is pretty easy and do-able. (You can find a couple references here.)
Here are some of the steps, as far as I understand:
Implement the Chi-squared test statistics. I believe this is just
chi2 = self.n_obs * res.fun, wherefuncomes directly from theminimizeresults object (see here).Calculate the degrees of freedom. I believe this is (k * (k - 1) / 2), or
dof = self.n_obs * (self.n_obs - 1) / 2, but we may also have to subtract out the number of model parameters at the end.Calculate the p-value. Once we have the test statistics and the degrees of freedom, this should be as simple as
1 - scipy.stats.chi2.cdf(chi2, dof).Calculate Root Mean Square Error of Approximation (RMSEA). There are a few different formulas I've seen, but I think they all reduce to
rmsea = 0 if chi2 < dof else np.sqrt((chi2 / (dof * self.n_obs) - 1) / (self.n_obs - 1))or something like that. I'll have to double check this , too. (We can look atpsychto see how they implement.)Getting the Comparative Fit Index (CFI) is a little more involved, since you have to calculate the Chi-squared test statistics for the baseline/null model, where all the variables are independent/uncorrelated. I'll read up on that a bit, but it does seem like a useful thing to add.
As an FYI, I believe that
semopyhas all of these implemented, but it's been a while since I've looked at that package. Not sure whether/how much we can borrow from that package. Looks like it's under the MIT license. (See here.)desilinguist commentedon Apr 2, 2022
This is amazingly helpful, @jbiggsets! I'll try and take a stab at RMSEA soon. It's funny you mention semopy since that's exactly what @jimmybru is using right now. I told him he should look into replacing that with factor analyzer but he needs these metrics to do that.
jimmybru commentedon Apr 2, 2022
You guys are Beautiful!
jimmybru commentedon Apr 2, 2022
Btw: this should probably be another issue, but just in case the answer is obvious, I’ll ask here: I don’t see a way in factor_analyer to specify covariances in the model like you can in semopy. Is that true? If so, is the default that all factors are orthogonal, or is everything allowed to covary with everything?