Next: 6 FEAR OF CRIME Up: OUTDOOR LIGHTING AND CRIME, PART 1: LITTLE Previous: 4 SPECIFIC PROBLEMS WITH THE EVIDENCE Contents
A meta-analysis involves pooling separate experimental determinations of some quantity to give a weighted average likely to be more accurate than any of the individual contributing values. The accuracy of the result is likely to be improved if experiments of poor quality are given low weighting or discarded altogether. Important parts of the process are to collect all available relevant studies and to assess them against a rigid set of quality criteria. Provided that some key facts have not been overlooked or misinterpreted, the combined review and meta-analysis process can be expected to give results that are more reliable and accurate than results from typical single studies, and far better than generally indicated by experience, anecdotes or common beliefs.
Farrington and Welsh (2002a,b) [34,35] is a review and meta-analysis of results of UK and US experiments on increased lighting for crime prevention. The search for accounts of street lighting and crime experiments was commendably thorough. The text shows evidence of favouring the view that lighting has a beneficial effect. For example, it has a heading ``How may improved street lighting reduce crime?'' Of the seven paragraphs in this section, six state or imply that lighting is beneficial and only one discusses some possible exceptions. There is also an uncritical reproduction of statements from a book version of Pease (1999) [90].
Eight papers from the USA and five from the UK were selected for the meta-analysis on the basis that they had sufficiently good experimental designs. This was necessarily a compromise between having plenty of results, some of which are of poor quality, or fewer results of better quality.
To provide a common basis for comparison, an odds ratio was calculated for
each experimental result, given by
The odds ratios for the 13 studies ranged from 3.82 to 0.75. The overall weighted odds ratio was 1.25, with a 95% confidence interval of 1.18 to 1.32. This means that a well designed and conducted `standard' lighting intervention could most likely result in a 20% reduction of total crime (1 - 1/1.25 = 0.2) in the relit area as a beneficial outcome. Needless to say, the authors would be justified in feeling pleased at being first to achieve such an important and conclusive result after decades of claim and counterclaim about the overall effect of lighting on the incidence of crime. But there is still some checking to do.
Despite the Tien et al. (1977) [107] warning about inadequate photometry, the situation has not improved markedly. Farrington and Welsh (2002a,b) searched their 13 selected papers for measures of the lighting changes and were able to state the after/before intensity18 ratio in only seven cases. For the US studies, this ratio was given as 7 in Milwaukee, 4 in Atlanta, 3 in Fort Worth and 2 in Portland. They missed one in Quinet and Nunn (1998) [95], where the treatment was to double the number of street lights, presumably doubling the intensity and the mean illuminance. In the UK, it was 5 in Stoke-on-Trent, approximately 2 in Bristol and 2 in Dudley. The arithmetic mean ratio for eight cases is 3.375. Thus, the Farrington and Welsh meta-analysis tells us a lighting increase of about 3.375 times will tend to produce an odds ratio of 1.25, ie a reduction of 20 % in crime in the experimental area relative to the control area.
What is conspicuously missing is the range of illuminance values over which this relationship was derived. For predictive purposes, this range would also be the minimum for which the odds ratio could be expected to apply. The papers available to the writer have quite limited information about the actual illuminance in each experimental area before or after the 13 lighting interventions. The illuminance range therefore has to be estimated. It appears reasonable to search the literature for other clues about before or after values, about values that have been measured in other lighting and crime experiments, and about the values used in lighting practice.
Fisher (1997, Table 5) [37]
gave minimum before values of 0.1 to 1.3 lux and
minimum after values of 2.5 to 4 lux for six UK lighting and crime
studies, and values in his tables of minimum, average and maximum
recommended values from the British and Australian Standards for road and
public lighting range from 
0.07 lux in service to 350 lux (Fisher 1997,
Tables 2, 6).
Nair et al. (1997) [72]
mentioned measured extremes of 1 and 32
lux in their experiment, although the most representative before value was
more than 1 lux.
Vermeulen (1992) [114]
gave a desirable operating range for
ccd video cameras as 8.2 to 32 lux. Data points on graphs in
Boyce and Rea (1990) [13]
range from about 0.1 to 80 lux, and in
Boyce et al. (2000) [14],
about 0.1 to 180 lux.
Philips (2002) [91]
recommended 300 to 500 lux for shop
interior lighting.
Pollard (1994) [92]
gave a maximum of 900 lux for building
floodlighting. The writer has measured peak values of over 1400 lux at a
footpath in Melbourne at night, which is well into the range of natural
daylight. As mentioned in Section 7.2 below, values of over 450 lux
resulted from relighting of railway stations as one of several supposed
crime-prevention measures
(Carr and Spring 1993 [18]).
For the present
discussion only, an upper bound to values that have been used in
relighting experiments, including relit indoor retail areas such as that
in
Poyner and Webb (1997) [94],
is chosen to be 1000 lux as conservative and
convenient.
The lower limit might readily be taken as 0.l lux. However, this happens to be the lowest indication available on typical good-quality hand-held light meters, a reason to expect that lower before values have been, or could have been, present in actual experiments or real-world areas selected for lighting or additional lighting. There is another reason also. Illuminance from the first quarter moon at night is less than 0.1 lux, but common experience is that even this is enough to be seen as markedly brighter and perceived as safer than natural moonless night outdoor illuminances, which can be several factors of ten dimmer than 0.1 lux. Therefore the lower limit for lighting and crime experiments in the real world can be taken to be at least as low as 0.01 lux.
Thus in round figures, every point of the range 0.01 lux to 1000 lux has been, could have been or is still likely to be within the range encompassed by lighting and crime experiments. The experiments included in the meta-analysis appear to have covered most of this range, and the result of the meta-analysis should therefore be representative of lighting changes over this range, at least. There may well be some non-linearity in dose-response over this range, but the unstated assumption of Farrington and Welsh is that the odds ratio for a given lighting change anywhere in the range is constant as a first approximation. From any starting point within this range, a lighting increase of 3.375 times should produce a crime-reduction odds ratio of 1.25 according to the meta-analysis. Even if the `real' value varies with absolute value of illuminance, with the value given being typical or some sort of average, it would not overturn the present argument.
If a lighting intervention takes place from the low end of the range, there is no reason to suppose that the new lighting could not then be usefully increased a second time from 0.03375 lux to 0.114 lux as an intervention for a further crime reduction. After all, the starting points for the 13 studies appear to have been spread over much of the range in question, and at least in the papers to hand there is no mention of how many prior `improvements' have contributed to the present lighting levels and what the effects on crime might have been on each occasion. This process could continue until 1000 lux would be exceeded, at least. In this case, nine serial interventions would bring the after illuminance to 568 lux, and the net odds ratio would be the ninth power of 1.25. This works out at 7.45. From the definition of odds ratio, this means crime in the control area would increase to be 7.45 times that in the experimental area. So by adding lots of light to a really dim area to bring it up to light levels typical of retail sales areas, crime in the treated area relative to the control area should reduce to about 13 % of its original incidence. In practical terms, this would be a dramatic reduction.
Whether such changes are obtained in a single large step or as a succession of two or more smaller steps should not matter - either the derived odds ratio applies uniformly across the lighting range or the total effect of successive applications of a non-constant odds ratio should be the same. Otherwise crime level under a particular lighting level at a given place would have to depend on the history of lighting changes, seemingly a rather implausible proposition. If large lighting increases did indeed produce such large reductions in crime it would have been obvious long ago, but nothing of the sort appears to have been reported, even anecdotally. The conclusion is that the magnitude of the odds ratio derived by Farrington and Welsh is improbably large. The true value, assuming a constant effect over the likely lighting range, must be smaller than 1.25.
Not only should the most likely value for the odds ratio have a credible magnitude but its 95% confidence interval limits should also meet this condition. The present lower value, 1.18, raised to the ninth power, is 4.435. This represents a crime reduction to just 22.5 %, again beyond any likelihood of practical realisation. When raised to the ninth power, the present upper limit of 1.32 leads to a value of 12.17. Crime in the treated area would thus drop to 8.22 %, even further from any reasonable expectation. The meta-analysis result must be erroneous , not mathematically or statistically, but as a guide to the real effect of lighting changes on crime.
If it is accepted that the upper limit for the 95% confidence interval has to be within reason, the overall crime reduction would need to be no more than 50 %, say, and even this might be thought rather optimistic. The total odds ratio for nine typical treatments would thus be 2.0. The ninth root of this is 1.080. To a first approximation, the whole 95% confidence interval for the overall result shown on the Farrington and Welsh forest diagram would need to be shifted leftwards on the logarithmic scale so that its rightmost limit was at the position for an odds ratio of 1.08. When this is done, the lower limit for the 95% confidence interval is actually to the left of the odds ratio = 1.0 axis. This revised result has the overall odds ratio for the 13 studies at about 1.05, which is not significantly different from the `no effect' odds ratio value of 1.0.
Assuming that the meta-analysis processes and algorithms are correct, some or all of the experimental odds ratios must be too large. If they are all inflated by the same proportion as found here, because of some biasing effect such as funding bias or targeting high crime areas, the individual results would all be about 20 % too large. If a correction of this size is applied to all of the 13 studies, in 11 cases the corrected value still lies within the 95% confidence interval. This is not a reason to apply such a correction, merely a demonstration that a systematic bias smaller than the 95% confidence interval can change the overall result from lighting preventing crime to lighting having no effect on crime. It would even be possible for the true result to be a small counterproductive effect of lighting on crime, falling within the 95% confidence interval of the `corrected' odds ratio. It is difficult to accept the meta-analysis result as showing anything definite at all.
Looking again at the data from the studies selected for the meta-analysis, some are confined to measures of crime at night, while others are for crime by day and night. Thus the meta-analysis indiscriminately mixes direct and indirect effects of lighting at night with indirect effects by day. It is possible that some of the indirect effects could have time constants of several years, which is longer than the sampling period generally employed in lighting and crime experiments to date. No account is taken of sampling periods in the meta-analysis, which adds to the uncertainty of what the result is supposed to mean. But there does not appear to be any reasonable way in which the erroneously large result could have arisen simply because of the mixture of direct and indirect effects in the individual odds ratios and in the overall result.
There is no obvious way in which any supposed non-linear dose-response relationship could retrieve the situation for increased lighting as a crime-reducing intervention. This casts suspicion on the review process as not rejecting unacceptably faulty studies. This topic is taken up in Section 5.5 below.
The title of Farrington and Welsh (2002a,b) and the text in places restricts the scope of the experiments reviewed to street lighting but lighting interventions inside a market hall and inside a car park building are included in the analysis. The reasons for their inclusion are not objectionable; instead, the problem lies with the restriction to street lighting. `Public lighting' might have been a better choice. The market and car park studies are of particular interest for other reasons and are discussed in Sections 5.3 and 5.4.
After the relighting treatment in Dudley, the illuminance was a minimum of 2.5 lux and a maximum of 6 lux. The treatment was a lighting increase of more than a factor of 2, but rounded to 2 in Farrington and Welsh (2002a,b). Taking it as 2, the initial luminance comes out as between 1.25 lux and 3 lux. When relighting is eventually extended to all of the surrounding areas, it is quite possible that someone will come along in due course and see a need for relighting what was the experimental area. Given the popularity of lighting for the pride of place and crime prevention mindset, this would appear quite likely. There appears to be no reason why this process should stop there, so it will continue until the streets are lit to near daylight levels or at least to levels currently found in downtown areas of big cities and the entrances to large suburban shopping malls. From a representative starting value of 2 lux, say, eight treatments in succession would bring the representative illuminance to 512 lux.
The Dudley study odds ratio assigned by Farrington and Welsh, 1.44, means that each of the eight treatments would reduce crime in the experimental area, relative to the control, to 1/1.44 of its initial value, ie 0.694. Eight such treatments in succession would therefore reduce crime to the eighth power of 0.694, ie 0.054. If crime really could be reduced by 95% by intense floodlighting of residential streets it would have become standard practice years ago, at least in suburbs populated by the well-off. The experimental result for the Dudley study is therefore improbably large, presumably from confounding by the asymmetric change in police presence or from some unknown factor. Crime reductions like that in the Dudley study or larger were claimed for relighting increments in presentations to conferences of the (UK) Institution of Lighting Engineers on at least two occasions (1989 and 1994), apparently without sufficient comment to alert the researcher to the problem before publication of at least five more papers with similar errors.
In the conclusions to the Painter and Farrington (1997, p 225) paper [82], the authors state:
``Another key issue is the `dose-response' curve relating street lighting and crime; it may be that improved street lighting decreased crime in Dudley because the improvement was so dramatic.''
The total response range of the human eye, from absolute threshold to the brightest tolerable light, is over 11 log units or 1011 (100 billion) in luminance. The lighting increase in Dudley was a factor of a little over 2, equivalent to about 0.3 log unit, quite a small part of the total range. This highlights the contrast between the extensive behavioural data gathering and statistical analysis in the study and the paucity of attention to photometric aspects of the treatment and effects of this on visual performance.
It is not only the result for crime effect that is too large in the Dudley study. The social effects claimed would compound to massive changes with eight successive treatments.19This is further reason to suspect that something is fundamentally wrong with the study.
Similar problems arise with the Stoke-on-Trent results. The odds ratio for crime is 1.72 and the lighting increment was a factor of five (0.7 log unit). From a supposed 1-lux starting point, four successive treatments would reach 625 lux. Crime in the experimental area would be expected to fall to 0.114, ie 11.4 %, after four treatments. The crime result for the initial treatment is again overlarge, and so must be the beneficial social effects reported.
The foregoing discussion would be valid for a uniform lighting increment affecting the whole of the experimental area. Increased lighting was only applied to the main streets, however. If crimes were only committed within the area directly illuminated by these lights, then the argument presented above would apply. Given the physical form and placement of the houses on the estate, however, there would be many areas that were partly or fully shaded from the street lights. At night, these areas would receive light from natural sources, from artificial skyglow, from escaping room light, and from porch and security lights. All of these could be expected to affect experimental and control areas equally. In both areas, this light would be incremented by scattered and reflected street light. The mean lighting treatment increment in the experimental area would therefore be less than the treatment increment in the relit streets. This would not matter if crime had only taken place in the streets and not in the dimmer areas, but no information about the location of crime within the experimental areas is given in any of the papers by Painter and Farrington. Burglaries at least could be expected to take place sometimes in dimly lit areas. The after/before illuminance ratio quoted for the experimental estates is therefore overstated. Even more of the mean lighting treatments could therefore be fitted into the range from 0.01 lux to 1000 lux. This bolsters the conclusion that the claimed crime reduction results for a single treatment are improbably large.
The largest odds ratio reported for the 13 studies included in the meta-analysis was for the Birmingham Market study. This study is reviewed here to see if the large odds ratio is justifiable.
Poyner and Webb (1997) [94] is an edited version of an earlier paper by Poyner and Webb (1987) [93]. It is about attempts to deter the theft of purses from shopping bags usually carried by women in one of the largest English retail market places, centred on the Birmingham Bull Ring.
Table 2 reproduces key data and results of the study. The number of stalls in each area of the market in April 1985 is shown. These numbers had been approximately steady in the first three parts of the market but the number of stalls in the fourth part grew substantially over the seven years covered. At the end of this time, traders complained that the market overall was less busy than formerly but no data are given about this. A trading decline could have contributed to the falling total crime numbers as a consequence of there being less of the crowding that was used to advantage by the thieves.
| TABLE 2. Recorded thefts from shopping bags, March to August each year | |||||
| YEAR | 1978 | 1982 | 1983 | 1984 | 1985 |
| PLACE | |||||
| Rag Market | 52 | 82 | 54 | 17 | 12 |
| (large shed) | (552 stalls) | ||||
| Open Market | 54 | 21 | 45 | 33 | 12 |
| (158 stalls) | |||||
| Market Hall | 20 | 4 | 11 | 12 | 9 |
| (197 stalls) | |||||
| Flea Market | - | - | 2 | 2 | - |
| (outdoors) | (few stalls) | (231 stalls) | |||
| TOTALS | 126 | 107 | 112 | 64 | 33 |
| EVENTS | Original | Police unit | Wider aisles | New ceiling | |
| study | operating | installed in | and lighting | ||
| until end of | Open Market, | installed in | |||
| 1982. | early 1983 | Rag Market, | |||
| Open Market | late 1983 | ||||
| partly roofed | |||||
The thefts tended to be restricted to midday to 2 pm on Tuesdays and 1 pm to 4 pm on Friday and Saturday, the three busiest days of the week when all four of the markets listed were open. Daylight is bright at the times mentioned, possibly indicating that light was facilitating the thefts rather than deterring them. This possibility is reinforced by the fact that the crimes were mostly committed in the summer months. The authors assumed that police were more active in the Open Market and Market Hall in 1982, thereby displacing crime to the Rag Market. This allowed the 1982 Rag Market crime peak to be ignored and the new lighting in the Rag Market to be identified as the reason for the drop in crime from 1983 through 1985, although this time without apparent displacement.
Poyner and Webb stated that the reduction in crime in the Rag Market in 1983 and 1984 was
``the first clear evidence found by the authors to show that improved illumination levels reduce crime. It is perhaps paradoxical that the crime concerned only occurs during daylight hours.''
It is even more paradoxical that it was mostly during early afternoons in summer. Presumably ingress of daylight through windows, skylights etc. was supplementing the existing indoor artificial lighting system and facilitating crime but putting in a ceiling (a confounding change by itself) and replacement artificial lighting of greater electrical efficiency is supposed to do the opposite. Another explanation is that the Rag Market or the four markets were the location for a crime hotspot that reached its peak in 1982 and 1983 and was in decline or moving laterally thereafter. A third explanation came from the security staff at the market, who believed that congestion was a primary facilitator of the thefts. They had moved stall holders within the Rag Market to reduce congestion and the bumping that was part of the purse-stealing action. The relocation added further confounding to this unplanned and poorly controlled `experiment'.
Farrington and Welsh (2002a,b) assigned an odds ratio of 3.82 to the lighting intervention in the Rag Market. This appears to have been reached by summing the Rag Market figures for 1982 and 1983 as the experimental before value, with the after value likewise as the sum of the 1984 and 1985 figures. Open Market and Market Hall data were pooled and summed similarly for the control values. But data for 1982 are suspect because of the police presence and the confounding change in aisle width introduced in the experimental area in early 1983. Eck (1997) [31] mentioned only the aisle widening as a factor in the reduction of the purse thefts, not lighting, and stated that simultaneous changes in nearby markets made them unsuitable as control places so there was ``no evidence about background trends''.
Poyner and Webb (1997) [94] gave no photometric details beyond mention of ``improved illumination levels''. At a shopping mall described as `dingy' and in need of more lighting, the illuminance was about 40 lux (Horner 2002) [46]. For this discussion, assume that this was the before condition in the Rag Market, and that the treatment doubled the illuminance. Four treatments like this in succession would have brought the illuminance to 640 lux. With an odds ratio of 3.82, four treatments in succession would give an overall odds ratio of 213, reducing crime to less than 0.5 % in the treated area. Even a single additional lighting treatment would produce an overall odds ratio of 14.6, which would mean a crime reduction to 6.9 %, still well beyond any reasonable expectations. Leaving the 1982 data out gives an odds ratio of 2.19. Although this is a more credible value than 3.82, it still greater than the next two largest values of the 13 studies (which were Stoke-on-Trent and Dudley). It leads to an improbably large crime reduction, to 21 %, for just two successive treatments.
Poyner and Webb (1997) [94] concluded that ``the two original hypotheses ... have been proved to a considerable extent.'' Hypotheses about a non-closed system cannot be proved at all, however.
The case for attributing any, let alone all, of the reduction in crime to the lighting change appears to be so problematical that the study should have not have been included in the meta-analysis. The same conclusion could be reached by considering the odds ratio to be so large as to be an outlier justifying exclusion.
The meta-analysis includes the Poyner and Webb (1987) [93] study of theft of and from cars in a multi-storey, long-stay council car park in Dover (UK). By 1983, the authorities realised that their security program, which combined private security officers patrolling the car park at night and random visits from council inspectors during the day, was not working. Vandalism was a problem - graffiti; broken windows; damage to lifts, doors, sand buckets and fire extinguishers; and defecation on stairs.
Geason and Wilson (1990) [40] described the interventions as including:
``... gaps between the low walls around the ground floor of the car park were filled with wire mesh; the pedestrian entrance by the staircase was fitted with a self-closing steel door so that it could be used only as an exit; lighting at the main entrance and the pedestrian exit door was improved; and to provide surveillance, an office was built beside the main entrance and leased to a taxi firm operating 24 hours a day.''
These modifications limited access by pedestrians. Although supervision of the entrances and exits was not particularly thorough, the presence of security guards or manned barriers appeared to reduce the incidence of car theft greatly. Adjacent open-air car parks were used as the control. However, theft of items from cars actually increased by nearly as much as car thefts reduced. Poyner and Webb concluded that car thefts were committed by outsiders with no business in the car park, but thefts from cars were done either by legitimate users tempted by opportunity, or people who drove in specifically to steal. Environmental prevention measures worked against the car thieves, but were [worse than?] useless in dealing with determined petty thieves.
Although Farrington and Welsh recognised that added fencing and entrance supervision had confounded the effect of increased lighting, they ascribed the overall reduction in crime to the effect of the lighting changes. There seems to be no compelling reason why any of the change in crime at all can be claimed to be a result of lighting changes, so it seems that this study should have not have been included in the meta-analysis either. An alternative would be to apportion contributions from all of the interventions and reduce the odds ratio (1.14) to that proportion attributed to lighting. If lighting were estimated to have contributed one-third, say, the odds ratio would be the cube root of 1.14, ie 1.045.
Apportioning all of the crime reduction to lighting has the effect of overestimating the relevant odds ratio, which results in the weighted average being an overestimate by a smaller amount.
The largest odds ratio in the meta-analysis of the five British studies of Farrington and Welsh (2002a,b) is 3.82. This is from the case discussed above in Section 5.2. Not only is this odds ratio suspect because of its size but the inclusion of the study itself is questionable because of the doubt that lighting was responsible for any part of the observed change in crime, let alone all of it. Again this makes the overall odds ratio larger than it should be.
The next largest odds ratios of the five are 1.72 for the Stoke-on-Trent study and 1.44 for the Dudley study. Serious doubts have been raised above about the validity of both of these studies and their inclusion in the meta-analysis is therefore questionable. From the preceding section, the Dover study also looks as though it should not have been included. This leaves just the Bristol study, which is suspect because of the long gap between before and after measures, the hotchpotch of drawn-out lighting changes and the choice of high-crime areas for special lighting treatment. It is therefore hard to give any credence at all to the meta-analysis of the UK studies, which returned an overall odds ratio of 1.42.
The associated lighting increase is not known for three of the five UK studies and its mean for the other two was about 3.5 times. The mean for the five studies is not likely to be much different. The overall odds ratio for the UK is improbably large for any likely value of the mean lighting increase.
The eight US studies produced weighted combined odds ratios of 1.02 for property crimes, 1.07 for violence crimes and 1.08 overall.20Of these studies, only the Indianapolis one of Quinet and Nunn (1998) [95] is reviewed above, and it certainly has unfortunate features. Farrington and Welsh drew attention to a number of shortcomings in the remaining studies. Lab (1997) [55] did also in two cases, giving less favourable assessments as can be seen:
``Improved street lighting was most clearly effective in reducing crimes in the Fort Worth evaluation. Crimes decreased by 21.5 % in the experimental area and increased by 8.8 % in the control area (Lewis & Sullivan, 1979, p. 75 [59]). Since crime in the whole city stayed constant (a decrease of 1.1 %), it may be argued that some crime had been displaced from the experimental to the adjacent control area. In the experimental area, property crime decreased, but violent crime did not. Information about types of crime was not provided for the control area, and information was not provided about nighttime as opposed to daytime crime.'' (Farrington and Welsh 2002a,b)
``Lewis and Sullivan found that a threefold increase in lighting did not appear to reduce crime in areas of Fort Worth, Texas.'' (Lab (1997) [55])
``Improved street lighting was followed by a decrease in robberies and burglaries in Atlanta, whereas the incidence of these crimes increased in the control area (Atlanta Regional Commission, 1974, pp. 11-12). There was an increase in assaults in the experimental area, but the number was relatively small (from 11 to 57). Overall, daytime crime decreased by 16.4 % in the experimental area after the improved lighting, in comparison with an increase of 33.3 % in the control area. Nighttime crime increased considerably in both areas.'' (Farrington and Welsh 2002a,b)
``The City of Atlanta found that the relit areas of a high-crime census tract experienced a greater increase in robbery and assault compared to unrelit areas (Atlanta, 1975).'' (Lab 1997)
Farrington and Welsh (2002a,b) assigned odds ratios of 1.38 (p0.1) to
Fort Worth and 1.39 (p0.05) to Atlanta.
In relation to the British studies, Farrington and Welsh stated:
``In most cases, the experimental area was chosen for relighting because it was a high crime area. This high crime rate raises the problem of `regression to the mean'; an area that has a high crime rate at one time is likely to have a lower crime rate at another time. To investigate this possibility, long time series of crimes before and after the intervention in experimental and control areas are needed.''
Only the Bristol study of the British set includes time-series observations. Crime values were given for nine successive six-month periods but the treatment was staged over 28 months. This study does not meet the usual criteria for a time-series analysis of treatment effects, eg at least five before time periods.
Farrington and Welsh did not mention the regression to the mean problem in relation to the US studies. But the likelihood again is that relighting of particular areas was done to try to control an existing high crime rate, not because the opportunity was available for well-matched experimental and control areas for scientific purposes. Unless there is evidence to show that this confounding effect did not apply or was insignificant, the US results remain suspect.
If the eight US studies are accepted in the absence of further information, Farrington and Welsh's finding of an overall odds ratio of 1.08 for them becomes the result of the meta-analysis. But the problems do not end there.
Farrington and Welsh cited a reference that the reader has to consult to try to reproduce the weighting factors used to combine the various estimates of the odds ratio. One should not have to `second guess' the authors in this way. Explicit details of the weighting factors or other essential features of the process should have been given so that readers could readily check the calculations for themselves, and examine the effect of removing one or more of the studies from the pool used to find the best estimate of the odds ratio.21
The Cochrane Handbook (Clarke and Oxman 2002 [22]) sets out necessary requirements for acceptable scientific quality of healthcare review articles. It was used as a model for the Campbell Collaboration (2002) [17] guidelines. In the case of conflict of interest issues, the two are virtually identical: reviewers should report any conflict of interest capable of influencing their judgements, including personal, political, academic and other possible conflicts, as well as financial conflicts. It is hardly surprising that the rules for good science are consistent across disciplines. Within the healthcare discipline, information about sources of funding is considered a desirable inclusion in trials reports also (Moher, Schultz and Altman 2001 [69]).
The Dudley and Stoke-on-Trent papers both acknowledge the managing director of a lighting company for funding the research. Insofar as the funding provided the means to do the research and thereby benefitted both authors, and that one of the authors was also an author of the review, it would seem that these two potential sources of bias should have been mentioned explicitly in the review, but they were not. Farrington and Petrosino (2001) [33] suggested a solution to the problem of a review author also being an author of one or more of the included papers; this is for the review to have an additional author, one who had not previously worked on the topic. This is understood to be the case for the review and meta-analysis in question, but halving potential bias does not eliminate it. Under the Collaboration guidelines it is up to authors to decide whether to mention potential sources of bias. In this case, its non-mention does not seem justified, particularly in view of the unpalatably large effects of financial and nonfinancial conflicts of interest that have been described recently in leading scientific journals (see Section 2.1 above.)
The odds ratio used by Farrington and Welsh (2002a,b) seems to be inverse of that required. It most directly represents a change in crime in the control area relative to the experimental area, which is not the most convenient arrangement when making practical use of the result. In an ideal quasi-experiment, extraneous influences would be negligible and so would be the actual after/before change in the control area. In further lighting and crime experiments and analysis, it is suggested that the measure of effect should be the reciprocal of the odds ratio used by Farrington and Welsh. Then it would directly describe the effective after/before change in crime in the treated area. A decrease in crime would give an odds ratio of less than 1, and an increase in crime, an odds ratio of greater than 1. To avoid possible confusion, the quantity might usefully be given a different name, say `crime ratio', `crime response', or `crime effect ratio'. It would also seem useful to try to keep direct and indirect effects separate, or at least night and day effects.