Make std::map pickable

[alvaro-diaz-valenzuela]

Hello,
I am trying to make an instance of std::map<QCDate, double> pickable, where QCDate is a class of my own which represents a date and is already pickable. This std::map represents a financial time series.
I am trying to accomplish "pickability" using the following:

py::bind_map<std::map<QCDate, double>>(m, "time_series")
    .def(py::pickle(
        [](const std::map<QCDate, double> &p) { // __getstate__
                            /* Return a tuple that fully encodes the state of the object */
                            std::vector<std::pair<QCDate, double>> items;
                            for (const auto& element : p) {
                                items.emplace_back(element);
                            }
                            return py::make_tuple(items);
                        },
                        [](py::tuple t) { // __setstate__
                            if (t.size() != 1)
                                throw std::runtime_error("Invalid state!");

                            std::map<QCDate, double> result;
                            for(const std::pair<QCDate, double>& p:t[0]) {
                                result[p.first].push_back(p.second);
                            }

                            return result;
                        }
                ));

but I am having trouble with the __setstate__ part. I am getting the following error:

error: no viable conversion from 'reference' (aka 'const pybind11::handle') to 'const std::pair<QCDate, double>'
                            for(const std::pair<QCDate, double>& p:t[0]) {
                                                                 ^~ 

Is it possible to make this std::map pickable? If so, Can anyone point me in the right direction? Any help would be much appreciated.

[jiwaszki]

Hi @alvaro-diaz-valenzuela you need to manually cast to desired pair in a loop. Automatic iteration over py::tuple is simply catching py::handle instances -- thus the translation to std::pair is simply unknown, you probably would need type_caster for it. However, there is a simpler way, written below.

Here is what you need to do (I have simplified the QCData):

class QCDate;  // forward declarations

PYBIND11_MAKE_OPAQUE(std::map<QCDate, double>);

namespace py = pybind11;

class QCDate {
public:
	QCDate() {}
	QCDate(int v) : value{v} {}

    bool operator<(const QCDate& other) const
    {
        return value < other.value;
    }

	int value;
};

PYBIND11_MODULE(mymodule, m)
{
    m.doc() = "This is the module test";

    py::class_<QCDate>(m, "QCDate")
        .def(py::init<>())
        .def(py::init<int>())
        .def(py::pickle(
                    [](const QCDate &self) { // __getstate__
                        return py::make_tuple(self.value);
                    },
                    [](py::tuple t) { // __setstate__
                        if (t.size() != 1)
                            throw std::runtime_error("Invalid state!");

                        QCDate obj(t[0].cast<int>());
                        return obj;
                    }
            ))
        .def("__repr__", [](QCDate &self){
            return std::string("QCDate:") + std::to_string(self.value);
        });

    py::bind_map<std::map<QCDate, double>>(m, "time_series")
        .def(py::pickle(
                    [](std::map<QCDate, double> &self) { // __getstate__
                        std::vector<py::tuple> items;
                        for (auto& element : self) {
                            items.emplace_back(py::cast(element));
                        }
                        return py::make_tuple(items);
                    },
                    [](py::tuple t) { // __setstate__
                        if (t.size() != 1)
                            throw std::runtime_error("Invalid state!");

                        std::map<QCDate, double> result;
                        for(auto tuple_item : t[0]) {
                            auto p = tuple_item.cast<std::pair<QCDate, double>>();
                            result[p.first] = p.second;
                        }

                        return result;
                    }
            ));
}

Here is code snippet for Python:

import mymodule
import pickle

ts = mymodule.time_series()
ts[mymodule.QCDate(21)] = 4.2
ts[mymodule.QCDate(37)] = 2.5
ts[mymodule.QCDate(42)] = 9.6

p = pickle.dumps(ts)
loaded = pickle.loads(p)
print(dict(loaded))  # that will print: {QCDate:21: 4.2, QCDate:37: 2.5, QCDate:42: 9.6}

Hope that helps, cheers!

PS I have edited the answer as I missed the custom map in testing:)

[alvaro-diaz-valenzuela]

Hi @jiwaszki , your solution worked like a charm! Thank you very much for taking the time. Cheers.