pypy with numpy

Looks like pypy now can build numpy. Well, a slightly modified numpy.

1. Get default branch of pypy. be careful cause the developers don't maintain default branch compilable. Revision 84341 (c86b42dd7613) works for me.
2. compile using 
./rpython/bin/rpython -O2 ./pypy/goal/targetpypystandalone.py --withoutmod-micronumpy
3. Create package and vitual environment. Something like this:
./pypy/tool/release/package.py --targetdir ./my_builds/build.tar.bz2 --builddir ./tmp/ --nostrip --archive-name pypy_84341
Needed to copy pypy-c and libpypy to pypy/goal beforehand.
4. Clone and follow instructions from
https://github.com/pypy/numpy/commits/cpyext-ext
Revision 3299d0d76fdb831fbcb4429a89c1f53bb36ea07f worked for me

Testing results:
----------------------------------------------------------------------
Ran 5900 tests in 78.216s

FAILED (KNOWNFAIL=3, SKIP=6, errors=218, failures=83)

scipy can be compiled if disabling submodules io/matlab and spatial. Though is still doesn't work.

About python

Cheat Sheet: Writing Python 2-3 compatible code
http://python-future.org/compatible_idioms.html

Google Python Guidelines
https://google.github.io/styleguide/pyguide.html


Inheritance
https://rhettinger.wordpress.com/2011/05/26/super-considered-super/

Debugging numpy (any C code of Python) using gdb

I created a tiny python script that executes some python code, that executes some C code:

# contents of dbg_broadcast.py
import numpy
print list(numpy.broadcast([[1,2]],[[3],[4]]))

> gdb python
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.

.................. bla bla bla ..................................................

# Adding breakpoint to function "arraymultiter_new" (example):

(gdb) break arraymultiter_new

# There is name completition by Tab.

# Run script

(gdb) run dbg_broadcast.py 

Starting program: /home/asdasd/work/bin/python dbg_broadcast.py

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, arraymultiter_new (__NPY_UNUSED_TAGGEDsubtype=0x7ffff6abbf80 <PyArrayMultiIter_Type>, 

    args=0x7ffff32dd050, kwds=0x0) at numpy/core/src/multiarray/iterators.c:1578

1578 {

(gdb) 

And here we are.

TIL about PyPy

Building from source root using command
pypy_src$ rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
produces structure with obsolete pypy-c and libpypy-c.so in
/tmp/usession-release-4.0.1-XXXX/build/pypy-nightly/bin/

Probably pypy compiler places there files integrated in the src distribution. To get fresh versions I had to use pypy-c and libpypy-c.so from sources root.

UPDATE:
Probably I was completely wrong.
pypy/tool/release/package.py has an option for (not) stripping resulting binary file: "--nostrip". By default it is enabled. Looks like it removed something unused from binaries. This operation updates timestamp of the pypy-c and libpypy-c.so. So probably that was the cause of my misunderstanding.


Script for packaging and creating virtual environment:

#!/bin/bash

rm -rf ./my_builds/ || exit 2
mkdir ./my_builds/ || exit 3

DST_NAME=$1
if [ -z "$DST_NAME" ]; then
    echo "DST_NAME is  empty"
    exit 3
fi

# runs packaging
./pypy/tool/release/package.py --builddir /home/pypy/builds/ --nostrip --archive-name $DST_NAME || exit 4

# creates a new virtual environment
virtualenv -p /home/pypy/builds/$DST_NAME/bin/pypy /home/pypy/env/$DST_NAME

# installing nose for numpy testing (optional)
source /home/pypy/env/$DST_NAME/bin/activate
pip install nose

Detector of flying objects in IR video

Implemented using python

https://github.com/serge-m/object_detection_ir_video

Python logging

best practices:
http://victorlin.me/posts/2012/08/26/good-logging-practice-in-python



Logging exceptions logging.exception

code for logging in ipython notebook (jupyter)

import logging
import datetime
import sys, os

def prepare_logger(logger, level=logging.DEBUG, filename_template="logs/notebook_log_{}.txt"):
    def prepare_handler(handler):
        handler.setLevel(level)
        handler.setFormatter(formatter)
        return handler
   
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
   
    path_file = filename_template.format(datetime.datetime.now().isoformat())
   
    path_dir = os.path.dirname(path_file)
    if not os.path.exists(path_dir):
        os.makedirs(path_dir)
     
    del logger.handlers[:]
    logger.handlers.append(prepare_handler(logging.FileHandler(path_file)))
    logger.handlers.append(prepare_handler(logging.StreamHandler(sys.stderr)))
   
    logger.setLevel(level=level)
    return logger

logger = logging.getLogger()
logger = prepare_logger(logger, level=logging.DEBUG)

Installing theano on windows

0. install Anaconda
1. download Theano sources from git (install it using setup.py)
2. Setup NVIDIA GPU Toolkit. I have installed version 6.5
3. Setup Visual Studio Community Edition 2013
4. Create config file .theanorc  in c:\Users\X\:

[global]
floatX = float32
device = gpu

[nvcc]
fastmath = True
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\cl.exe

Without the line with path to cl.exe I got:

nvcc fatal   : Cannot find compiler 'cl.exe' in PATH
['nvcc', '-shared', '-O3', '-use_fast_math', '-Xlinker', '/DEBUG', 

'-D HAVE_ROUND', '-m64', '-Xcompiler',

 '-DCUDA_NDARRAY_CUH=f411a53ee0a470fbaad3b5c4a681ef64,-D 

NPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,/Zi,/MD', 

'-Ic:\\anaconda\\theano\\theano\\sandbox\\cuda', 

'-Ic:\\anaconda\\lib\\site-packages\\numpy\\core\\include', 

'-Ic:\\anaconda\\include', '-o', 

'C:\\Users\\X\\AppData\\Local\\Theano\\compiledir_Windows-7-6.1.7601-SP1-Intel64_Family_6_Model_42_Stepping_7_GenuineIntel-2.7.7-64\\cuda_ndarray\\cuda_ndarray.pyd', 

'mod.cu', '-Lc:\\anaconda\\libs', 

'-Lc:\\anaconda', '-lpython27', '-lcublas', '-lcudart']

Writing simple optical flow in python. Part 3

Today, my imaginary readers, we improve our optical flow dramatically. Lets see, what our algorithm produces for images that have more than 1-pixel shifts.

I0	I1
OF results:

You can see, that the algorithm cannot deal with too large displacements. Even with multiple warps.
Lets apply multiscale scheme now. We need to construct image pyramid:

Writing simple optical flow in python. Part 2

After fixing some errors, it seems my OF is working.

Image I0	Image I1
first iteration OF	1st iteration of warped I1
Second Iteration OF	Second iteration of warped I1
3 iteration Of	3rd iteration of warped I1
Last iteration