In a post a few weeks ago, I cited the analysis by R. Davis on some research conducted by Professor Madeline Zavodny of Agnes Scott College. Among other things, she had found that for every 100 immigrant workers with graduate degrees in STEM, 262 jobs (not necessarily in STEM) are created.
Needless to say, the advocates of expansion of foreign tech worker programs like H-1B have been making lots of hay out of Zavodny’s findings. This of course is no coincidence, since those same advocates commissioned Zavodny to do the research.
R. Davis, a Silicon Valley engineer who wishes not to use his full name, has done a very impressive job in trying to verify and revisit Zavodny’s analysis. This has led to a very informative Science Careers article, In the last week or so, I’ve been running some of Davis’ code myself to explore Zavodny’s data, resulting to the update you are now reading.
I ask the reader’s patience, as the issues are as subtle as they are important to the H-1B debate. The details matter. But in outline form, my points will be:
- R. Davis has been able to replicate Zavodny’s findings.
- Zavodny’s findings do not make economic sense.
- Zavodny’s results have serious methodological and statistical issues.
Let’s get the easy issue out of the way first, the replication of Zavodny’s findings. She had generously given R. Davis her data and code (another researcher doing similar work, again funded by the lobbyists, refused to do so), and Davis found that the data had a lot of missing values. Zavodny had coded them as 0s, which seemed wrong, but it turns out that the software she used automatically excluded such values, as she was applying a log transform. Once that was cleared up, Davis was able to replicate all of Zavodny’s numbers.
Davis’ site has a wealth of interesting findings, which I urge readers to peruse, but here is what I regard as the central issue: Are Zavodny’s numbers of any relevance? As far as I can tell, the answer is solidly No.
I always tell my students and consulting clients that one must always check statistical results with what is known qualitatively about the given issue. Do the numbers make sense?
Common sense tells us that for the foreign grad students in STEM to have some sort of special job-creating powers, one of two conditions would have to hold: Either (a) there is a STEM labor shortage or (b) the foreign students are more talented than their American peers. Both (a) and (b) have been shown clearly to be false, using multiple data sources.
Concerning (a), even the H-1B advocates agree that wages in STEM, including in the computer field, have been flat at best. In a previous blog post, I noted that NACE, the National Association of Colleges and Employers, projects that starting salaries for new computer science graduates (Bachelor’s degree level) will be DOWN 9% this year. And NACE’s projected CS Master’s salary for 2015, $71,140, is down from $73,400 in 2013 (I have been unable to get the 2014 figure), which in turn was down 8.7% from 2012. There really is no room left for legitimate debate on the shortage issue.
As to (b), research by two NBER affiliates, as well as my own work, shows that the foreign STEM students tend to be, relative to their American peers, (i) graduates of less-selective institutions, (ii) less likely to file patent applications, (iii) less likely to work in R&D (crucial to job creation!), and so on.
So, if the foreign workers are not remedying labor shortages and aren’t smarter than the Americans, how can they be creating extra jobs? Zavodny’s results counter common sense, and though common sense is sometimes wrong, Zavodny bears the burden of proof to demonstrate that it is wrong.
On the contrary, she herself concedes that the issue of causation is key. Indeed, she writes in her paper,
But one of the fundamental challenges when using cross-state comparisons to show a relationship between immigrants and jobs is that immigrants tend to be more mobile and go where the jobs are. As a result, evidence of high immigrant shares in states with strong economic growth and high employment could be the result of greater job opportunities (as immigrants move to jobs), rather than the cause. Cross-state comparisons would then show an artificially high impact of immigrants on the native employment rate. The study avoids “overcounting” the effects of immigrant workers drawn by a recent economic boom by using an estimation technique (known as “two-stage least squares (2SLS) regression estimation”…
Methodology such as 2SLS is controversial, and highly sensitive to assumptions. Yet almost none of Zavodny’s results, including those employing 2SLS, is statistically significant at the standard 0.05 level. Significance testing itself has its problems, but Zavodny’s work would be highly questioned by journal reviewers if she had submitted it to academia, rather than writing a paid advocacy report as she did. She also uses clustered standard errors (which are used in the significance testing), another controversial technique.
Indeed, state-by-state comparisons themselves have a long history of controversy. As I wrote in my original post,
Worse, region-by-region analyses are notorious for being unreliable and misleading. For example, there have been numerous studies on capital punishment, both pro and con, based on comparing states that do and do not have capital punishment., in terms of murder rates and so on. They can’t all be correct.
There have been similar issues with state-by-state studies of the impact of minimum-wage laws.
To me, the most important flaw in Zavodny’s analysis, also related to the causation issue, is that it doesn’t ask the obvious question: What job creation rate is associated with hiring Americans? If one is going to use Zavodny’s data to advocate for expanding the H-1B program (as she does), then why calculate only the foreign job creation rate? Why not calculate the American job creation rate too, and then compare them? Some of you may recall that this was a major criticism that I made regarding one of Bill Kerr’s papers.
This is crucial. What if those jobs filled by the foreign workers had been filled by Americans?
I do need to make one correction to my original post. I had mentioned how much I liked R. Davis’ graph titled, “Foreign STEM Workers, 2000-2007.” Actually, I still like it, and in fact intend to use something like that as an illustration in the book I’m writing on regression. But I realize now that my comments about the graph were misleading. I had written that it is a good illustration of Simpson’s Paradox, which in fact it is — IF one’s regression analysis does not have terms for the states. Zavodny’s analysis does have those terms.
Nevertheless, my comments still hold. Zavodny’s coefficients for the states are much larger than for the share of foreign workers, just as the graph shows. Whenever there is such a large disparity in sizes of coefficients, one must be very careful, as the small ones are likely very sensitive to the inevitable violations of the assumptions of the model. Of course, the lack of statistical significance of most of Zavodny’s results makes this concern even more acute.
Again, since there is no STEM labor shortage and since the average quality of the foreign workers is lower than that of the Americans, the burden of proof is on Zavodny to make a strong case for her claims. She has not done so.
Finally, I must commend Science Careers blogger Beryl Benderley for not only covering this topic, but also especially for pointing to the fact that Zavodny’s research was sponsored by an advocacy group. The press almost never mentions this.