​Climate change is shifting species' distribution and phenology. Ecological traits, such as mobility or reproductive mode, explain variation in observed rates of shift for some taxa. However, estimates of relationships between traits and climate responses could be influenced by how responses are measured. We compiled a global data set of 651 published marine species' responses to climate change, from 47 papers on distribution shifts and 32 papers on phenology change. We assessed the relative importance of two classes of predictors of the rate of change, ecological traits of the responding taxa and methodological approaches for quantifying biological responses. Methodological differences explained 22% of the variation in range shifts, more than the 7.8% of the variation explained by ecological traits. For phenology change, methodological approaches accounted for 4% of the variation in measurements, whereas 8% of the variation was explained by ecological traits. Our ability to predict responses from traits was hindered by poor representation of species from the tropics, where temperature isotherms are moving most rapidly. Thus, the mean rate of distribution change may be underestimated by this and other global syntheses. Our analyses indicate that methodological approaches should be explicitly considered when designing, analysing and comparing results among studies. To improve climate impact studies, we recommend that (1) reanalyses of existing time series state how the existing data sets may limit the inferences about possible climate responses; (2) qualitative comparisons of species' responses across different studies be limited to studies with similar methodological approaches; (3) meta-analyses of climate responses include methodological attributes as covariates; and (4) that new time series be designed to include the detection of early warnings of change or ecologically relevant change. Greater consideration of methodological attributes will improve the accuracy of analyses that seek to quantify the role of climate change in species' distribution and phenology changes.