Modern climate models are 'semi-empirical', in the sense that they contain parameters which are not operationally-defined, and must be inferred. This process is rather informal, since the models themselves are enormously expensive to evaluate. However, at its heart is a comparison of model outputs against observations, and the 'tuning' of certain parameters to bring these into better agreement. This process has proceeded over several generations of models, so that it is now very difficult to identify a 'hold-out' dataset. Hence there is a strong interest in developing new sources of climate data: palaeo-data is one such source. Palaeo-data exists in the form of climate proxies: quantities that are influenced by climate, predominantly pollen counts from lake sediments. Therefore we need an additional modelling step, to map the climate model output to the proxy observations: the proxy model, which is typically cheap to evaluate. We then have a statistical inverse problem: to infer the climate model parameters (and, on the way, the palaeo-climate itself) from the proxy observations. As well as uncertainty in the climate model parameters, this inference must take account of structural errors in the climate model and the proxy model, and observational errors in the proxy data. Neither of the two 'standard' approaches will work: the climate model is too expensive to be embedded in a Monte Carlo sampler directly, and too complicated (in the number of its parameters and outputs, and the spatial and temporal interdependence of its outputs) to be statistically-modelled using an emulator. We propose a two step approach which, while not formally coherent, is at least practical. In the first step, we construct a palaeo-version of our climate model, which involves modifying the land-, ice-, and vegetation-maps, and changing the solar forcing. We use this model to construct a Perturbed Physics Ensemble (PPE): a collection of model-evaluations at different sets of parameter values. This PPE can be used to sample palaeo-climates using a Kernel Density Estimator (KDE), and including an extra source of uncertainty to account for the climate model's structural error. On the basis of this sampling mechanism, we can condition palaeo-climate on the proxy data using the proxy model (which is cheap to evaluate) within a Monte Carlo sampler. In the second step, we treat the reconstructed palaeo-climate as 'data', and use it to reweight the members of the PPE in a second conditioning step. This reweighting translates into an updated distribution on the model-parameters.