Recently I reported on a new working paper by Kirk Doran (Notre Dame), Alexander Gelber (UCB) and Adam Isen (U.S. Treasury), that appears to counter work by my UCD colleague Giovanni Peri and his coauthors, as well as by Madeleine Zavodny, a former Fed researcher now at Agnes Scott College. The Peri/Zavodny line of research finds that the hiring of H-1Bs creates lots of new jobs, with the much-cited 2.62 figure being Zavodny’s.
To review, the new work by Doran, Gelber and Isen (DGI) finds that
- “Winning additional H-1B visas has an insignificant effect on patenting within eight years…”
- “H-1Bs substantially crowd out employment of other workers.”
- “We find some evidence that additional H-1Bs lead to lower average employee wages while raising firm profits….and rules out the scenario in which H-1Bs replace natives one-for-one.”
It would be hard to imagine a paper as diametrically opposed to Peri/Zavodny as this one. And certainly quite timely, as the Peri and Zavodny work has been intensely shopped around to Congress and the press by the industry lobbyists seeking expansion of H-1B.
Yesterday Isen gave a talk in a workshop hosted by Giovanni on our UCD campus, with Isen’s two coauthors present as well. Giovanni presented his work following Isen’s talk. Each side had an hour’s time allotted to them, including questions and comments by the rest of the attendees, which enabled a good thorough exploration of both papers. I had not been aware of the workshop until a couple of days before the event, but fortunately heard of it through the grapevine, and did attend both talks. There were about two dozen people in attendance.
Given the stark challenge that DGI is to Giovanni’s work, he deserves a lot of credit for inviting the authors to speak, and he was a very gracious host. Eventually, though, things did get a little heated, though remaining friendly enough that there were smiles all around by those enjoying watching the clash. I particularly enjoyed the following exchange (as verbatim as my memory allows):
Giovanni, to DGI: Your paper consists of nothing by 0s [i.e. findings that the effect size is 0, e.g. 0 gain in employment]! That can’t be true!
DGI: We did find some nonzero results! We found a triple-star effect [i.e. very highly significant in statistical language] of reduced payroll! [An indication that the H-1Bs may be hired as cheap labor.]
Giovanni: But only in some of the cases!
Interestingly, one of Giovanni’s criticisms of DGI was their data source. DGI looked at H-1Bs hired by lottery late in the season. If I understood Giovanni’s point correctly (he speaks very rapidly, with a heavy Italian accent), it was that each “winning” firm got only about 2 H-1B workers in the process, thus making it hard to judge their impact on the firm. (Some of you may recognize that Josh Stern posted a similar comment to my blog post.) DGI, on the other hand, believe that collectively, across the totality of 3,000+ firms, there should have been an impact, if indeed H-1B has the salutatory effect that Giovanni claims. I may be biased, but I would say that given the 8-year time window DGI used to measure results, their defense seems reasonable.
There were various comments from others in attendance. For instance, Giovanni’s PhD student and coauthor, Kevin Shih, suggested that the reason the DGI analysis didn’t find a positive effect of H-1B on patenting may be because the innovative firms tend to file for H-1Bs earlier in the season. But DGI had found that the later filers actually were more prone to patenting.
I posed a question to Giovanni on a point I’ve brought up here in the blog before: If one accepts the research showing that (a) there is no STEM labor shortage, including in CS, and (b) the quality of the H-1Bs is on average somewhat lower than that of their American peers, how can the H-1Bs have a positive effect on employment numbers, relative to what the hiring of Americans would produce? What magic potion do the H-1Bs possess? He replied that he thinks that it’s good to have as many STEM people in the nation as possible, and that, say, an overqualified Indian PhD H-1B doing ordinary work is good. (He didn’t mention whether it’s good for that Indian PhD H-1B to get the job in lieu of an overqualified American PhD.)
In my earlier blog post on DGI, I had suggested that they might do separate analyses for the ordinary 65,000-visa H-1B category, and the ADE category, which allots 20,000 visas for foreign students earning advanced degrees at U.S. universities. Since that latter type of H-1B is a favorite of the industry, supposedly producing so much innovation, it would be useful for DGI to run separate analyses for ADE.
Actually, they responded to this suggestion. But the results were, borrowing from Giovanni’s phrasing, “More 0s.” In other words, the ADE workers did not increase patenting or employment, etc.
By the Giovanni mentioned, good naturedly but I’m sure correctly, that he gets a lot of hate mail. Someone then asked, “Do you respond?”, to which Giovanni replied, again in a lighthearted tone but probably correctly, “I put them all in a big box.” Some of you may recall that Vivek Wadhwa also complains of getting hate mail. For the record, I should add that sometimes I do too, from H-1Bs. Goes with the territory, I guess.
Bottom line, the two talks were both enlightening, and the exchange quite enjoyable.
Upon Closer inspection 
Interesting that they didn’t make a point of inviting you to the workshop…
LikeLike
I complained to Giovanni about that, and he explained that the workshop was only for economists, a bit odd in view of the diverse makeup of the sponsoring group. 🙂 But he welcomed me to the event, and added me to the mailing list for future events, so all is cool.
LikeLike
It’s lies, damned lies, and statistics, Norm.
Of course the ironic thing is Giovani pointing out methodological errors in the DGI paper, one can make no greater errors in methodology than Giovanni does!
Are these things easy to argue, even in academia? Apparently not. Allow a pocket lecture:
The entire field of psychology was trapped in behaviorism for something like seventy years. The entire spectrum of science was trapped in positivism for, well, a century or ten. The symptom is talk of “dispositions” absent a causal model, or correlations ditto. So why do these pathologies persist, even among purportedly educated people? Because there is a certain value to such empirical observations of correlations or dispositions. You may not know the causality, but *something* is very likely there. It can be handled respectably, or direspectably. Asserting causality without proving it is the error. Giovanni makes that error whole-hog.
So why not demand a model? Because models can be wrong, too. And models, well, they’re harder to come up with, and more difficult to talk about. And what about models that follow common sense? Well, they seem “too easy”, who’s going to make an academic reputation documenting the obvious? Good science is supposed to be about showing things that are hidden! So, you often make a better reputation, and maybe a better living, spouting nonsense and lies. Maybe knowingly, maybe not.
Good science is hard, and Sturgeon’s Law applies.
LikeLike
Harry Truman once said he hoped for a one-armed economist, incapable of saying “On the other hand…” 🙂 Accordingly, one does not attain the prominence Giovanni has on the Hill and in the press by being “two-armed.” In this instance, I think your Disraeli/Twain quote about lies and statistics is less apt than Eric Hoffer’s phrase, “the true believer.”
In principle, academia places strong value on the two-armed. But in the case of immigration, especially tech immigration, the picture changes radically. The expansionist side has succeeded in painting the restrictionists as xenophobes if not outright racists, and in portraying American students as being too dumb/lazy to do tech. And remember — I keep harping on this — even many critics of H-1B subscribe to the notion of the Intels being Good and the Infosyses being bad. Thus there is a “consensus” that tech immigration is beneficial, even necessary to our economy and national well-being.
When Giovanni submits his work to academic journals, this mindset is at work among the reviewers. I pointed out during the workshop that both the DGI and Peri papers use highly complex models that could be criticized at a thousand different steps. What reviewer is going to go to so much trouble, especially when “everyone knows” that tech immigration is good?
This is compounded by the fact that very few potential reviewers have enough knowledge of H-1B to spot the errors. In submitting a paper to a journal, one can ask that certain people be blackballed from being selected as reviewers. Once Giovanni told me he suspected that a negative reviewer on an early paper of his as Professor X, who has opposing views. And I believe he was correct. But once certain potential reviewers who know the field are excluded, one is left with some who do not.
LikeLike
It’s a generational thing. It used to be respectable to be two-armed. Modern political discourse has pretty much lost this, each side is one-sided and there is no communication. I would point to just one or two individuals who have fomented this but the MSM has aided and abetted it, they were supposed to know better. Maybe I should blame it on Yoda, “Do or not do, there is no ‘try'”. Or maybe Nike’s “Just do it!”.
A little off-topic, but the software industry is *heavily* invested in this as well. There is no challenge to writing software, you just do it. Hire a million code-monkeys and let them get to it. Throw some bananas in now and then, they’ll be fine. Sometimes they make this explicit, sometimes it’s slightly subtle:
http://www.zerohedge.com/news/2015-05-03/alibaba-job-opening-code-monkey-motivator-who-looks-porn-star
https://www.linkedin.com/pulse/insiders-perspective-why-techhire-matters-shawn-drost
#TechHire goes hand-in-hand with the H-1B flood, both are about debasing the field and do more harm than good.
LikeLike
> I pointed out during the workshop that both the DGI and Peri papers use highly complex models that could be criticized at a thousand different steps. What reviewer is going to go to so much trouble, especially when “everyone knows” that tech immigration is good?
I have often wondered if some models are made more complex partially because it makes them much harder to replicate and critique. At the very least, I’m sure that it’s not lost on researchers that such models tend to receive much less criticism than simpler models which are easier to replicate. They are made even more difficult to replicate by some researchers who decline to release their calculations. Peri did this himself when I requested any of the data or program files with which to replicate the results in his working paper titled “Foreign STEM Workers and Native Wages and Employment in U.S. Cities” which is referenced on the White House website. He sent me an email in which he CC’d his coauthors (Kevin Shih and Chad Sparber) and stated “My coauthors convinced me that the right avenue of action is that, as we spent several months organizing and cleaning the data, they should be available through the journal that publishes the paper.” I received this back in December and I am yet to see any indication that the paper is going to be published. I suspect that I was just getting the brush off. In any event, it seems to me that, as soon as a paper is used to argue for and influence public policy, the means necessary to replicate it should be released.
In any event, was there any mention during the talk about the release of the data and calculations needed to replicate such studies?
LikeLike
Generally people make complicated models to impress their peers. That it also makes it hard to criticize and replicate is a bonus.
Journal review can take many months. Or, maybe the journal already rejected “Foreign STEM Workers and Native Wages and Employment in U.S. Cities” and the authors have submitted to another journal. I’m sure that the authors were not happy by the timing of the DGI paper, coming during the review process of “Foreign STEM Workers…,” and contradicting it.
LikeLike
Yes, long formulas with Greek letters are very impressive! So is the practice of giving results to several decimal places (like 2.62) to give the illusion of precision.
In any case, I didn’t know how this process usually works so I assumed that I was being brushed off. If a paper like “Foreign STEM Workers and Native Wages and Employment in U.S. Cities” is accepted by a journal, will they commonly release the programs or whatever is necessary to replicate the study? Is this often a requirement of the journal? Anyhow, I may check with Peri to see if he has any news on the journal.
LikeLike
The major granting agencies, such as the NSF, now require that data and code for the project be made public. I’m not aware of whether any journals have such a policy.
It may be the case that, as a public university, research data must be accessible to the public. You may wish to look around on ucop.edu, or even talk to suitable people in the state legislature. And don’t forget, the State of California has its own FOIA request mechanism.
The issue of open research is quite big these days, with the current buzzword being reproducible research.
LikeLike
> It may be the case that, as a public university, research data must be accessible to the public. You may wish to look around on ucop.edu, or even talk to suitable people in the state legislature. And don’t forget, the State of California has its own FOIA request mechanism.
Thanks, I’ll do that. Anyhow, it’s good to hear that the push for reproducible research is gaining steam. Now, if we can just get the media to start distinguishing between reproducible research and working papers which have not been replicated or subjected to any serious scrutiny. It would also help if we could point out and possibly embarrass those politicians and other public figures who mindlessly parrot the findings of such working papers. I’ve tried to do this with those who quote the 2.62 and 1.83 numbers at http://econdataus.com/claim262.htm .
By the way, I just noticed that Beryl Benderly posted a blog post about the Doran/Gelber/Isen paper at http://sciencecareers.sciencemag.org/career_magazine/previous_issues/articles/2015_05_05/caredit.a1500116 . It’s good to see this getting more publicity.
LikeLike
(I’m speaking in generality here, not about any specific research.) The journal review process does subject submitted papers to serious scrutiny. Unfortunately, that is often not enough. As I explained here in response to a reader comment the other day, in many cases the subject matter is just so arcane that the journal’s referees, in spite of being excellent number crunchers, don’t know what numbers to crunch. They thus miss major issues in the paper. In addition, there are unconscious preconceptions, such as the “everyone knows we have a STEM labor shortage” view.
LikeLike
How about webcasts of these events? The issue is not only on the coasts. Those of us located in Middle America are impacted by the H-1B crisis as well?
LikeLike
Please change ? to period. My old eyes don’t focus in the AM.
LikeLike
I am sad to hear that Giovanni received hate mail. That should be reserved for corrupt politicians, if anybody.
It’s my opinion that he has a deeply held opinion that H1Bs are a good thing, so his research is biased as a result.
One could say the same about you, but the difference is that you use sound models that double and triple check validation. My son’s Informatics class used one of your studies to demonstrate how to examine whether any data sets are biased. That speaks volumes to me.
Thank you for bringing this study to our attention!
LikeLike
Clearly, after working on research in this area for more than 20 years, I have come to certain strong opinions. This is only natural, but I do work very hard to avoid bias in my analyses. I am always questioning my own assumptions, to see whether circumstances have changed over the years (and whether I was right even in the past).
I hope that this is clear from my writings. For instance, in my first post about DGI a couple of weeks ago, I noted several weaknesses in their paper, which caused them to overlook certain aspects which could potentially be more favorable to the H-1B program. I’ve criticized some research by Hal Salzman, even though he is on the same side as I am regarding H-1B. In my reseearch papers you will find lots of references to Giovanni and Zavodny; I state why I disagree with them, but the point is that I do give them credit for having done work in the field. This is what an academic is supposed to do, build on previous research work. By contrast, you will find NO citations of my work in the Peri and Zavodny papers, nor do they cite work by Hal, Ron Hira etc. or any other academic who is critical of H-1B.
The gravest bias problem, in my view, is that virtually all of the pro-H-1B researchers accept money from the industry and its allies. They claim that the funding has no effect on their research, and they may really believe that, but it should be obvious that the effect does exist. It doesn’t mean that they fake their data, but it does affect what aspects of H-1B they pursue, how they pursue it and so on. “One does not bite the hand that feeds one.”
LikeLike
Odd time stamps on these replies, is there a time zone setting for your blog you could reset?
LikeLike
Will check.
LikeLike
Thank you very much for bringing attention to these important issues
on your blog. We greatly appreciate your helpful suggestions and are
currently revising the paper to take account of them, prior to
submitting the paper to an academic journal for the first time.
In light of the comments on our paper at the UC Davis migration
conference that are reported here, we wanted to send you a few
thoughts about the issues. First, we use randomization, the gold
standard of scientific evidence, and are the only paper on the effects
of H-1Bs on outcomes of the receiving economy that does so. We also
use accurate government data on firm outcomes. Second, we address a
question of great interest to many observers: the effects on a firm of
marginally increasing the number of H-1Bs given to that firm. Third,
our estimates are precise enough to rule out economically relevant and
meaningful alternative hypotheses.
In the context we study, our estimates speak directly to the effects
on firms of marginally increasing or decreasing the number of H-1Bs
they are allowed (holding constant other factors). This is exactly
what our paper estimates, and the effects of marginally changing the
visa cap is a question of great interest to firms, policy-makers, and
other observers. Our paper does not directly speak to the effects of
other changes in the H-1B program, such as the effects of eliminating
the H-1B program, tripling the size of the cap, or the effects of
visas that are not subject to a cap (such as visas given to non-profit
educational institutions). The issue raised by one seminar participant
at the conference of whether other H-1Bs have different effects is
also an interesting question, but that is not what our paper seeks to
address.
As you noted, we find no significant effect on patenting. Finding
zeroes in some contexts can be very meaningful. One hypothesis about
the effect of H-1B visas on firms is that they increase the firm’s
patenting and employment. Another hypothesis is that they do not have
these effects on firms. Finding a zero effect here is useful because
it tells you that those measures are not increasing in response to
granting a firm an extra H-1B visa, contrary to the first hypothesis
and the claims made by some. Crucially, our results allow us to rule
out that winning an extra H-1B leads to more than a modest increase in
patenting or employment at the firm level. For example, in firms with
10 or fewer employees, we rule out that winning an extra H-1B leads
patents over the following eight years rise by more than 1.8 percent,
on a mean base of 0.027 patents over this period. In these firms, we
also rule out that winning an extra H-1B leads the firm’s total
employment to rise by more than 0.11 employees, thus indicating
substantial crowdout of other employment. Thus, we are able to rule
out economically relevant and meaningful alternative hypotheses.
It’s worth noting that we believe that there are insightful ways of
studying H-1Bs that have been used in past literature that do not
involve randomization. We view this work as addressing other
interesting questions that are complementary to our work. For example,
some of this work addresses the effect of H-1Bs on aggregate outcomes.
As we explain in detail in the paper, our work speaks to the effect of
H-1Bs on firms’ outcomes, but we do not believe that our work speaks
directly to the effect of H-1Bs on aggregate outcomes such as the
overall level of employment or measures of innovation in the economy
or in specific geographic areas, or the effect of aggregate increases
in H-1Bs on firm or aggregate outcomes.
Again, thank you for discussing these important issues on your blog.
LikeLike
Thanks for the comments, Alex.
As I remarked during the conference, I believe it’s important to bring in other information, both qualitative and quantitative, in analyzing the H-1B issue. I noted, for instance, that wages in the computer field have been flat, counterindicating a labor shortage. Add to that the work of others (e.g. myself, Bound, Hunt etc.) finding that the H-1Bs hired as students from U.S. universities — which the industry claims are the best H-1Bs — are on average weaker than their American peers on various measures, and it seems pretty clear that H-1Bs do not have a net job-creating effect or a net patent-filing effect. A related point is that even Giovanni Peri has done research indicating that the number of foreign students in a STEM field negatively affects Americans’ decisions as to whether to do graduate work in STEM; note too the 1989 NSF white paper that forecast that the influx of foreign students would hold down STEM wages, thus driving Americans out of STEM study at the PhD level.
I believe that relating your statistical findings to other considerations is especially important for the issue of whether the H-1Bs are hired as cheap labor. Note the vital point that here “cheap” can mean either (a) H-1Bs are paid less than comparable Americans (same education, experience, skill sets etc.) or (b) younger, thus cheaper, H-1Bs are hired in lieu of older, thus more expensive, U.S. citizens and permanent residents. Your finding that hiring H-1Bs reduces a firm’s payroll would be consistent with either (a) or (b). My anecdotal observations, including of public statements made by the industry, is that both factors are at work, but that (b) is a very powerful factor. At the same time, the immobility of the H-1Bs, at least those who are being sponsored for green cards, combined with basic economic theory that immobile workers are paid less ceteris paribus, compels (a).
I do suggest that you do some field work, interviewing employers. The 2001 NRC study, and to some extent the 2003 GAO report, found that employers admitted to paying their H-1Bs less than Americans, and giving them smaller raises and so on. Some years ago, a researcher whom I knew was going to be a “fly on the wall” at Cisco, studying how they hire H-1Bs. Unfortunately, the nonprofit she had been working for suddenly disbanded, and she was not able to do her study, but I can tell you that this kind of information would be a major eye opener.
Thanks again for a stimulating paper and presentation! It’s impressive that the three of you, located in three disparate geographical regions, have collaborated so well.
LikeLike