This brief comment (originally written in May 21, 2013) concerns with the reported first-stage estimates of the (fuzzy) regression kink design used by Lundqvist, Dahlberg and Mörk. The comment is posted on my webpage: http://people.su.se/~pepet. Below I have included the text without the tables and footnotes.

Table 1 reproduces the results from Table 2 in their paper and it shows the first stage estimates (it is noteworthy that four of the estimates are not reported in their table). They write: "It is clear from the table that all estimates are highly statistically significant, irrespective of order of polynomial and bandwidth. The magnitude of the estimates is around 3, although that differs somewhat across the different specifications". However, this conclusion is completely erroneous as can be seen from Table 2 which displays the first-stage estimates for the second and third order polynomial specifications for bandwidths h=10 and h=5 as well as for smaller bandwidths. Table 2 reveals that in almost all specifications with smaller bandwidths and with more flexible specifications the estimates are negative and large and sometimes significantly different from zero. Thus, given that the first stage estimates are very sensitive to the choice of bandwidth and polynomial specification, the conclusion must be that there is no regression kink at the threshold that can be used for a credible identification. The most likely reason for the identification failure of the regression kink design at the threshold is that there are other "kink-points in other components of the cost equalization that—by coincidence—could be close to the kink-point at two percent out-migration" as discussed by and Lundqvist et al. (2013).

References:
Dahlberg, M. E. Mörk, J. Rattsö, H. Ågren (2008), "Using a discontinuous grant rule to identify the effect of grants on local taxes and spending," Journal of Public Economics, 92, 2320-2335.

Lundqvist, H., M. Dahlberg and E. Mörk (2013), "Stimulating Local Public Employment: Do General Grants Work?", American Economic Journal: Economic Policy, forthcoming

It is true that the first-stage estimate starts to vary a lot when using high-order polynomials with the two narrowest bandwidths or with bandwidths smaller than 5, something we also acknowledge in footnote 25. We interpret this as being the result of small sample properties of the regression kink (RKD) estimator. Few observations close to the cut off can be problematic also in regression discontinuity designs (RDD). But because identification in the RKD comes from a discrete change in the slope around the cut off (as opposed to in the intercept), intuitively, the RKD is likely to be even more sensitive to small bandwidths when there is limited data. This intuition is supported by Monte Carlo simulations in Ando (2013), who finds that the introduction of a higher-order polynomial or a smaller bandwidth often yields imprecise and biased RKD estimates in small samples.
Together with the evidence in Figure 4, which shows the graphical version of the first stage, we therefore feel rather confident that the first-stage estimations in the paper indeed isolate exogenous variation in grants stemming from the kinked assignment rule. However, it can be noted that the RKD is a relatively new estimator that have not yet been extensively applied, and there is certainly a need for more studies analyzing the small sample properties of this estimator.

First, l would like to acknowledge that I was in contact with the editor, Alan Auerbach, (May 20, 2012) regarding the submission of my comment to this journal. He responded: "I believe that this comment would be appropriate to post on the journal's web site instead. Let me stress that I am not suggesting that your comment is not useful or important. Rather, I believe that, given its nature, it would be best for it to appear on the web site with the paper, which will make it immediately available to any reader of the paper's electronic version, which as you know is a growing share of the overall readership." Hence, this is the reason for posting my comment on the Journals Web site.

Regarding the authors' response to my comment, it is important to stress that footnote 25 was added only after they received my comment in May 2012. At this time, the paper was already accepted for publication and posted on the Web site as forthcoming. Such type behavior raises concerns about misreporting. Similarly, it also seems quite suspicious that 4 of the total of 12 entries of Table 2 in their paper are left out intentionally by the authors although these results could easily been reported. Most importantly, the omitted results do not lend support the authors' conclusion about the stability of the estimated first-stage effect made on page 181 (On this point, see my original comment). Furthermore, the authors have also failed to follow by now firmly established RD (kink or not) procedure of reporting results from an optimal bandwidth choice as well as from half the optimal bandwidth and twice the optimal bandwidth (e.g., see Imbens and Lemieux 2008). The important point here is that the choice of bandwidth should not be selected by a researcher so as to avoid concerns about data mining. Had they reported these standard sets of estimates, it had been very clear that their first-stage results are highly non-robustness since these estimates are both positive and negative. For example, the optimal bandwidth is 4.4 according to the Imbens and Kalyanaraman (2012) procedure (it is even smaller, 2.9, according to the Calonico et al. (2013) bandwidth selection procedure). In this case the estimated first-stage effect is 1.72 with a standard error of 1.23. The estimated effect for half the optimal bandwidth is -2.83 with a standard error of 2.08. It is also noteworthy that the statistically significant results for the smallest bandwidth of 5 in their Table 2 is no longer significant at conventional levels (5% or 10%) if the bandwidth is only marginally decreased to 4.8. Again, reinforcing the extreme fragility of their results to the choice of bandwidth.

Turning to their argument that the regression kink (RKD) estimator is biased for small bandwidth. This statement is simply wrong since an OLS estimator (i.e., local linear regression with a rectangular kernel) is always unbiased independent of sample size. However, they are right in pointing out that often the sample size is often not very big in an RD design. In such a case, if the analysis is based on a large bandwidth (or if the degree of the polynomial is so small) then it isn't really regression discontinuity that is identifying things. To conclude the results of their study do not survive the conventional set of robustness checks of an RD design (kink or not) and their analysis is not based on a local discontinuity since their results does not hold for the optimal or smaller bandwidths.

When I was preparing my lecture notes on RD kink design for my PhD students, I discovered that this paper misspecifies the functional form of the forcing variable, f(x-c), where x is the forcing variable and c is the cut-off., in their kink design The standard procedure is to interact the function f(x-c) with the indicator variable for treatment D where D=1 if x>c and D=0 if x<c. They have omitted the interactions terms for the second and third order polynomial functions, i.e., the terms (D(x-c)(x-c) and D(x-c)(x-c)(x-c) . This implies that they are simultaneously using information on both sides of the cut-off to estimate f(x-c), which does not sit well with the idea of a RD (kink or not) design, namely local identification at the cut-off without any parametric assumptions. Imposing this type of functional form restrictions is therefore not correct and may therefore yield biased estimate unless the restrictions are true. In order to test whether that is the case, I have re-estimated all their specifications in their Table 2 together with the interaction terms. Although, the two estimates for second-order specification are broadly similar to their estimates (even somewhat larger), the two estimates from the third-order specification are completely different as can be seen by comparing the results from Columns 1 and 2 in my Table with the corresponding results from their Table 2. My estimates are 0.84 for the full sample and -1.44 for the sample with a bandwidth of 15. These estimates should be compared with 3.35 and 4.08 in Table 2, respectively. This large decrease in their estimates clearly illustrates that their results are extremely sensitive to the degree of the polynomial specification. As a result, this new finding together with my previous comment on the extreme sensitivity to choice of bandwidth, the stark conclusion must be that there is no policy kink that can be exploited for identification (it is also noteworthy that only 5 out of 27 estimates in my table are positive and around 3 while 17 are negative, namely those specifications that are more credible since they rely on smaller bandwidths and more flexible functional forms of f(.).