nekpy_memory_management.tex 24.5 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
\chapter{Memory management in NekPy}

One of the biggest technical challenges of the NekPy project was to ensure the appropriate 
memory management between Python and C++. The computations carried out in \nek{} are 
usually highly demanding on the machines they are run on, therefore any unnecessary data 
duplication or memory leaks would be detrimental to software performance.

This section outlines the basics of memory management in C++ and Python as well as the 
solutions devised to effectively use the resources when passing data between the different 
layers of the bindings.

\section{Memory management in C++ and Python}

\subsection{C++}

In C++ the memory used by the application is divided into five containers \cite{C++Memory}:
\begin{itemize}
    \item text segment -- containing the set of instructions for the program to execute;
    \item data segment -- containing static and global variables initialised with values, 
    e.g. \texttt{float pi = 3.14;} -- the size of the data segment is pre-determined at 
    the time of the compilation of the program and depends on the size of the variables 
    in the source code;
    \item bss (block started by symbol) segment -- containing static and global variables 
    not explicitly initialised with any value, e.g. \texttt{int n;} -- the size of the 
    data segment is pre-determined at the time of the compilation of the program and 
    depends on the size of the variables in the source code;
    \item stack -- a LIFO (last in, first out) structure containing the function calls 
    and variables intialised in the functions -- the size of the stack is pre-determined 
    at the time of compilation of the program and depends on the operating system and 
    development environment;
    \item heap -- containing all dynamically allocated memory (e.g. using instruction 
    \texttt{new} in C++).
\end{itemize}

The access to data on the heap is maintained by pointers which store the memory address of 
said data. If the pointer ceases to exist (e.g. because the function which contained it 
finished running and hence was taken off the stack) the programmer has no way of referencing 
the data on the heap again. The unused, inaccessible data will stay in the memory, consuming 
valuable resources if not properly deallocated using \texttt{delete} instruction. Issues may 
arise if the same memory address is held by two different pointers -- the deallocation of 
memory can render one of the pointers empty and possibly lead to errors.

In order to facilitate memory management, C++11 standard introduced shared pointers 
(\texttt{shared\_ptr}) \cite{C++SharedPtr}. Shared pointers maintain a reference counter 
which is increased when another shared pointer refers to the same memory address. The memory 
will only be deallocated when all shared pointers referencing the memory are destroyed, as shown 
in Figure \ref{fig:shared_ptr}.

\begin{figure}[h!]
    \centering
51
    \includegraphics[width=0.6\textwidth]{img/shared_ptr}
52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
    \caption{Memory management with C++11 \texttt{shared\_ptr}. The two declared shared pointers 
    reference an array of 4 integers. Reference counter shown in green.}
    \label{fig:shared_ptr}
\end{figure}

\subsubsection{Python}

Python manages memory though a private heap containing all Python objects \cite{PythonMemory}. 
An internal memory manager is used to ensure the memory is properly allocated and dealocated 
while the script runs. In contrast to C++, Python Memory Manager tries to optimise the memory 
usage as much as possible. For example, the same object reference is allocated to a new variable 
if the object already exists in memory. Another important feature of Python Memory Manager is 
the use of garbage collector based on reference counting. When the object is no longer referenced 
by any variable the reference counter is set to zero which triggers the garbage collector to free 
the memory. A disadvantage of this solution is slower execution time, since the garbage collector 
routines have to be called frequently. Features of Python Memory Manager are schematically shown 
in Figure \ref{fig:python_gc}

\begin{figure}[h!]
    \centering
72
    \includegraphics[width=0.6\textwidth]{img/python_gc}
73 74 75 76 77 78 79 80 81 82
    \caption{Memory management in Python. Memory is optimised as much as possible and garbage 
    collector deallocates the memory if the reference counter (shown in green) reaches 0.}
    \label{fig:python_gc}
\end{figure}

It is unusual for the Python programmer to manually modify the way Python uses memory resources 
-- however it is sometimes necessary, as is the case with this project. In particular, 
Python enables the programmer to increase or decrease the reference counter of the object 
using \texttt{Py\_XINCREF} and \texttt{Py\_XDECREF} respectively \cite{PythonManualMemory}.

Emilia Juda's avatar
Emilia Juda committed
83
\section{Passing C++ arrays to Python \footnote{The file implementing the below procedure: \texttt{LibUtilities\/Python\/LinearAlgebra\/NekMatrix.cpp}} }
84 85

In many situations arrays created by Nektar++ (usually a \texttt{shared\_ptr} of 
86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105
\texttt{NekMatrix<D, StandardMatrixTag>} type) in C++ need to be passed to Python - 
for instance, when preforming differentiation using e.g. Gauss quadrature rules a 
differentiation matrix must be obtained. In order to keep the program memory efficient 
the data should not be copied into a NumPy array but rather be referenced by the Python 
interface. This, however, complicates the issue of memory management.

Consider a situation, where C++ program no longer needs to work with the generated array 
and the memory dedicated to it is deallocated. Meanwhile, Python interface might still require 
the data contained within the array, only to be faced with an error as the data no longer 
exists in memory. To prevent such situations a solution employing reference counting must be used.

\subsubsection{Converter method}

Boost.Python provides the methods to do both convert a C++ type element to one recognised by 
Python as well as to maintain appropriate reference counting. Listing \ref{lst:c_to_python} 
shows an abridged version of the converter method (for Python 2 only) with comments on 
individual parameters. The object requiring conversion is a \texttt{shared\_ptr} of 
\texttt{NekMatrix<D, StandardMatrixTag>} type (named \texttt{mat}).

\begin{lstlisting}[caption={Converter method for converting the C++ arrays into Python NumPy arrays.}, label={lst:c_to_python}, language=C++]
106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142
#include <LibUtilities/LinearAlgebra/NekMatrix.hpp>
#include <LibUtilities/LinearAlgebra/MatrixStorageType.h>
#include <NekPyConfig.hpp>

using namespace Nektar;
using namespace Nektar::LibUtilities;

template<typename T, typename F>
void NekMatrixCapsuleDestructor(void *ptr) // destructor for the shared_ptr to data
{
    std::shared_ptr<NekMatrix<T, F>> *mat =
        (std::shared_ptr<NekMatrix<T, F>> *)ptr;
    delete mat;
}

template<typename T>
struct NekMatrixToPython
{
    static PyObject *convert(
        std::shared_ptr<NekMatrix<T, StandardMatrixTag>> const &mat)
    {
    py::object capsule(
            py::handle<>(PyCObject_FromVoidPtr(
                             new std::shared_ptr<NekMatrix<T, StandardMatrixTag>>(mat), // new shared_ptr to the data
                             NekMatrixCapsuleDestructor<T, StandardMatrixTag>))); // destructor method
                             
    int nRows = mat->GetRows(), nCols = mat->GetColumns();
    
    return py::incref( // increments Python reference counter
            np::from_data(
                mat->GetRawPtr(), // pointer to data
                np::dtype::get_builtin<T>(), // data type
                py::make_tuple(nRows, nCols), // shape of the array
                py::make_tuple(sizeof(T), nRows * sizeof(T)), // stride of the array
                capsule).ptr()); // capsule - contains the object owning the data (preventing it from being prematurely deallocated)
\end{lstlisting}

143 144 145 146 147 148 149 150
First, a new Python object is created, holding a shared pointer (\texttt{PyCObject\_FromVoidPtr}) 
to the array held in memory, increasing its reference counter and preventing it from being 
prematurely deallocated, as shown in Figure \ref{fig:c_to_python}, as well as destructor which 
contains instructions on what is to be done when the object is no longer needed (in this case, 
the \texttt{shared\_ptr} is to be deleted). It is worth noting that the steps (c) and (d) can 
be reversed and the \texttt{shared\_ptr} created by the Python binding can be removed first. 
In this case the memory will be deallocated only when the \texttt{shared\_ptr} created by C++ 
is also removed.
151

152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199
\begin{figure}[h!]
    \centering
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{c_to_python_a}
        \caption{Nektar++ creates a data array referenced by a \texttt{shared\_ptr}}
        \label{fig:c_to_python_a}
    \end{subfigure}
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{c_to_python_b}
        \caption{Array is passed to Python binding which creates a new \texttt{shared\_ptr} 
        to the data}
        \label{fig:c_to_python_b}
    \end{subfigure}
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{c_to_python_c}
        \caption{Nektar++ no longer needs the data - its \texttt{shared\_ptr} is removed but 
        the memory is not deallocated}
        \label{fig:c_to_python_c}
    \end{subfigure}
    \begin{subfigure}{0.6\textwidth}
        \includegraphics[width=0.99\textwidth]{c_to_python_d}
        \caption{When the data is no longer needed in the Python interface the destructor is 
        called and \texttt{shared\_ptr} is removed.}
        \label{fig:c_to_python_d}
    \end{subfigure}
    \caption{Memory management of data created in C++ using \texttt{shared\_ptr} and passed 
    to Python. Reference counter shown in green.}
    \label{fig:c_to_python}
\end{figure}

Then, a Boost.Python handle is created around the object, allowing Boost.Python to manage references 
correctly. Finally, a Boost.Python object named \texttt{capsule} is created from the handle.

The converter method returns a NumPy array -- this was chosen as it is a well-known datatype used 
in Python's popular \texttt{numpy} package. Additionally, Boost.Python allows the programmer to 
create a NumPy array from a generic C++ data container and to specify the owner of said container 
using the \texttt{np::from\_data} method. 

Care had to be taken to ensure that the array is presented in the correct order, especially as 
Nektar++ stores data in column major order whereas NumPy arrays are traditionally row major. 
The stride of the array has to be passed into the \texttt{np::from\_data} in a form of 
\texttt{(a, b)} tuple, where \texttt{a} is the number of bytes needed to skip to get to the 
same position in the next row and \texttt{b} is the number of bytes needed to skip to get to the 
same position in the next column. Hence, the tuple passed was \texttt{(sizeof(T), nRows * sizeof(T))}, 
where \texttt{T} is the datatype of the array.

In order to stop Python from immediately destroying the resulting NumPy array, its reference counter 
is manually increased before the array is passed on to Boost.Python and eventually returned to the user.
200

201
\subsubsection{Testing}
202

203 204 205 206
The process outlined above requires little manual intervention from the programmer. There are no almost 
no explicit calls to Python C API (aside from creating a \texttt{PyObject} -- \texttt{PyCObject\_FromVoidPtr}) 
as all operations are carried out by Boost.Python. Therefore, the testing focused mostly on correctness 
of returned data, in particular the order of the array.
207

208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242
To this end, the \emph{Differentiation} tutorials were used as tests. In order to correctly run the 
tutorials the Python wrapper needs to retrieve the differentiation matrix which, as mentioned before, 
has to be converted to a datatype Python recognises. The test runs the differentiation tutorials and 
compares the final result to the fixed expected value. The test is automatically run as a part of 
\texttt{ctest} command if both the Python wrapper and the tutorials have been built.

\subsection{Passing Python data to C++ 
\footnote{The files implementing the below procedure are: \texttt{LibUtilities/Python/BasicUtils/SharedArray.cpp} and \texttt{LibUtilities/BasicUtils/SharedArray.hpp}} }

Conversely, a similar problem exists when data is created in Python and has to be passed to 
the C++ program. In this case, as the data is managed by Python, the main reference counter 
should be maintained by the Python object and incremented or decremented as appropriate 
using \texttt{Py\_XINCREF} and \texttt{Py\_XDECREF} methods respectively.

When the array is no longer needed by the C++ program the reference counter on the Python 
side should be decremented in order for Python garbage collection to work appropriately 
-- however this should only happen when the array was created by Python in the first place. 

\subsubsection{Modifications to \texttt{Array<OneD, const DataType>} class template}

In order to perform the operations described above, the C++ array structure should contain 
information on whether or not it was created from data managed by Python. To this end, 
two new attributes were added to C++ \texttt{Array<OneD, const DataType>} class template:

\begin{itemize}
    \item \texttt{m\_memory\_pointer} - holding the pointer to the 
        \texttt{PyObject} containing the data,
    \item \texttt{m\_python\_decrement} - holding the pointer to the 
        callback function decrementing the reference counter of the \texttt{PyObject}.
\end{itemize}

If the two new attributes are not set to \texttt{nullptr} the array has been constructed 
throught the Python to C++ converter. The already existing \texttt{m\_count} attribute 
was retained in order to keep track of how many C++ references to the array there are.

243 244 245 246
Adding new attributes to the arrays might cause a significantly increased memory usage. 
Therefore a preprocessor directive has been added to only include the additional arguments 
if \texttt{NekPy} had been built (using the option \texttt{NEKTAR\_BUILD\_PYTHON}).

247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274
A new constructor has been added to the class template, as seen in Listing \ref{lst:new_const}. 
\texttt{m\_memory\_pointer} and \texttt{m\_python\_decrement} have been set to 
\texttt{nullptr} in the pre-existing constructors. A similar constructor was added for 
\texttt{const} arrays to ensure that these can also be passed between the languages. Note that 
no calls to Nektar++ array initialisation policies are made in this constructor, unlike in 
the pre-existing ones, as there is no need for the new array to copy the data.

\begin{lstlisting}[caption={New constructor for initialising arrays created through the Python to C++ converter method.}, label={lst:new_const}, language=C++]
Array(unsigned int dim1Size, DataType* data, void* memory_pointer, void (*python_decrement)(void *)) :
                m_size( dim1Size ),
                m_capacity( dim1Size ),
                m_data( data ),
                m_count( nullptr ),
                m_offset( 0 ),
                m_memory_pointer( memory_pointer ),
                m_python_decrement( python_decrement )
            {
                m_count = new unsigned int(); 
                *m_count = 1;
            }
\end{lstlisting}

Changes have also been made to the destructor, as shown in Listing \ref{lst:destructor}, 
in order to ensure that if the data was initially created in Python the callback function 
would decrement the reference counter of the NekPy array object. The detailed procedure 
for deleting arrays is described further in this section.

\begin{lstlisting}[caption={The modified destructor for C++ arrays.}, label={lst:destructor}, language=C++]
275
~~Array()
276 277 278 279 280 281 282 283 284
            {
                if( m_count == nullptr )
                {
                    return;
                }

                *m_count -= 1;
                if( *m_count == 0 )
                {
285
                    #ifdef WITH_PYTHON
286 287 288 289 290 291 292 293 294
                    if (m_memory_pointer == nullptr)
                    {
                        ArrayDestructionPolicy<DataType>::Destroy( m_data, m_capacity );
                        MemoryManager<DataType>::RawDeallocate( m_data, m_capacity );
                    }
                    else
                    {
                        m_python_decrement(m_data);
                    }
295 296 297 298 299
                    #else
                    ArrayDestructionPolicy<DataType>::Destroy( m_data, m_capacity );
                    MemoryManager<DataType>::RawDeallocate( m_data, m_capacity );
                    #endif
                  
300 301 302 303 304 305 306 307 308
                    delete m_count; // Clean up the memory used for the reference count.
                }
            }
\end{lstlisting}

\subsubsection{Creation of new arrays}

The following algorithm has been proposed to create new arrays in Python and allow the 
C++ code to access their contents:
309 310

\begin{enumerate}
311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328
    \item A new NumPy array object is created in Python.
    \item The NumPy array object is passed as an argument to a C++ method available in 
        the Python wrapper.
    \item The converter method is called to convert a Python NumPy array into C++ 
        \texttt{Array<OneD, const DataType>} object.
    \item The converter creates a new \texttt{Array<OneD, const DataType>} object with the 
        following attribute values:
    \begin{itemize}
        \item \texttt{m\_data} points to the data contained by the NumPy array,
        \item \texttt{m\_memory\_pointer} points to the NumPy array object,
        \item \texttt{m\_python\_decrement} points to the function decrementing the 
            reference counter of the \texttt{PyObject},
        \item \texttt{m\_count} is equal to 1.
    \end{itemize}
    \item The Python reference counter of the NumPy array object is increased.
    \item If any new references to the array are created in C++ the \texttt{m\_count} 
        attribute is increased accordingly. Likewise, if new references to NumPy array 
        object are made in Python the reference counter increases.
329 330
\end{enumerate}

331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421
The process is schematically shown in Figure \ref{fig:array_creation_deletion_a} 
and \ref{fig:array_creation_deletion_b}.

\subsubsection{Array deletion}

The array deletion process relies on decrementing two reference counters: one 
on the Python side of the program (Python reference counter) and the other one 
on C++ side of the program. The former registers how many Python references to 
the data there are and if there is a C++ reference to the array. The latter 
(represented by \texttt{m\_count} attribute) counts only the number of references 
on the C++ side and as soon as it reaches zero the callback function is triggered to 
decrement the Python reference counter so that registers that the data is no longer 
referred to in C++. Figure \ref{fig:array_deletion} presents the overview of the 
procedure used to delete the data.

In short, the fact that C++ uses the array is represented to Python as just an 
increment to the object reference counter. Even if the Python object goes out 
of scope or is explicitly deleted, the reference counter will always be non-zero 
until the callback function to decrement it is executed, as shown in Figure 
\ref{fig:array_creation_deletion_c}. Similarly, if the C++ array is deleted first, 
the Python object will still exist as the reference counter will be non-zero 
(see Figure \ref{fig:array_creation_deletion_d}).

\subsubsection{Converter method}

As with conversion from C++ to Python, a converter method was registered 
to make Ptyhon NumPy arrays available in C++ with Boost.Python. 
Listing \ref{lst:python_to_c} shows the converter methods with comments on its contents.

In essence, Boost.Python provides the used with a memory segment (all expressions 
containing \linebreak \texttt{rvalue\_from\_python} are to do with doing that). The 
data has to be extracted from PyObject in order to be presented in a format C++ knows 
how to read -- the \texttt{get\_data} method allows the programmer to do it for 
NumPy arrays. Finally, care must be taken to manage memory correctly, thus the 
use of borrowed references when creating Boost.Python object and the incrementation 
of PyObject reference counter at the end of the method.

The callback decrement method is shown below in Listing \ref{lst:callback}. 
When provided with a pointer to PyObject it decrements it reference counter.

\begin{lstlisting}[caption={The decrement method called when the \texttt{m\_count} of C++ array reaches 0.}, label={lst:callback}, language=C++]
static void decrement(void *objPtr) 
    {
        PyObject *pyObjPtr = (PyObject *)objPtr;
        Py_XDECREF(pyObjPtr);
    }
\end{lstlisting}

\begin{figure}[h!]
    \centering
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_a}
        \caption{The NumPy array is created in Python. Note that the NumPy 
        object and the data it contains are represented by two separate memory addresses.}
        \label{fig:array_creation_deletion_a}
    \end{subfigure}
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_b}
        \caption{The C++ array is created through the converter method: its attributes 
        point to the appropriate memory addresses and the reference counter of the 
        memory address of NumPy array object is incremented.}
        \label{fig:array_creation_deletion_b}
    \end{subfigure}
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_c}
        \caption{If the NumPy object is deleted first, the reference counter is 
        decremented, but the data still exists in memory.}
        \label{fig:array_creation_deletion_c}
    \end{subfigure}
    \begin{subfigure}{0.9\textwidth}
        \includegraphics[width=0.99\textwidth]{img/array_creation_deletion_d}
        \caption{If the C++ array is deleted first, the callback function decrements 
        the reference counter of the NumPy object but the data still exists in memory.}
        \label{fig:array_creation_deletion_d}
    \end{subfigure}
    \caption{Diagram showing the process of creation and deletion of arrays. Reference 
    counters shown in green.}
    \label{fig:array_creation_deletion}
\end{figure}

\clearpage

\begin{figure}[h!]
    \centering
    \includegraphics{img/deletion_flowchart}
    \caption{Flowchart describing the procedure for array deletion from either side of 
    the code.}
    \label{fig:array_deletion}
\end{figure}

\clearpage
422

423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454
\begin{lstlisting}[caption={Converter method for converting the Python NumPy arrays into C++ arrays.}, label={lst:python_to_c}, language=C++]
static void construct(
        PyObject *objPtr,
        py::converter::rvalue_from_python_stage1_data* data)
    {
        // Extract data from the PyObject.
        // This has to be a _borrowed_ reference, otherwise at the end of this
        // scope it seems memory gets deallocated
        py::object obj((py::handle<>(py::borrowed(objPtr))));
        np::ndarray array = py::extract<np::ndarray>(obj);
        
        // Get pointer to the memory into which the C++ type will be constructed
        void *storage = (
            (py::converter::rvalue_from_python_storage<Array<OneD, T> >*)data)
            ->storage.bytes;
        data->convertible = storage;
        
        // Grab the address of PyObject for C++ to use.
        void *memory_pointer = objPtr;
        
        // Remedies issues with using const datatypes.
        using nonconst_t = typename std::remove_const<T>::type;
        
        // Construct OneD array from numpy array.        
        new (storage) Array<OneD, T>(array.shape(0), // array size
            (nonconst_t *)array.get_data(), // pointer to the data readable by C++
            memory_pointer, // memory address of the PyObject
            &decrement); // pointer to the callback decrement function
        // Increment the Python reference counter.
        Py_XINCREF(objPtr);
    }
\end{lstlisting}
455

456 457 458 459 460 461 462 463 464 465 466 467
\subsubsection{Testing}

As the process of converting arrays from Python to C++ required making direct calls to 
C API and relying on custom-written methods, more detailed testing was deemed necessary. 
In order to thoroughly check if the conversion works as expected, three tests were 
conducted to determine whether the array is:

\begin{enumerate}
\item referenced (not copied) between the C++ program and the Python wrapper,
\item accessible to C++ program after being deleted in Python code,
\item accessible in Python script after being deleted from C++ objects.
\end{enumerate}
468 469 470 471

Python files containing test scripts are currently located in 
\path{library\Demos\Python\tests}. They should be converted into unit tests 
that should be run when Python components are built.