Site moved to https://serge-m.github.io/

Archives for 2017

2017-01-09T07:10:00, Mon: Set up HP 1020 printer in Linux Mint 18
2017-01-08T14:10:00, Sun: Pelican Hints
2017-01-08T11:10:00, Sun: Shell commands
2017-01-08T07:10:00, Sun: Add service in ubuntu
2017-01-07T07:10:00, Sat: OpenVPN server in cloud using docker
2017-01-05T07:10:00, Thu: Docker in Scaleway's vc1s Ubuntu Xenial
2017-01-04T07:10:00, Wed: Check server fingerprint for scaleway
2017-01-04T07:10:00, Wed: Operations on images in python

Archives for 2016

2016-12-31T21:30:00, Sat: Raspberry Pi Links
2016-12-19T21:30:00, Mon: Run docker as pytest fixture
2016-12-07T00:30:00, Wed: Installing java 8 on linux mint 17.3
2016-11-27T22:47:00, Sun: Testing json responses in Flask REST apps with pytest
2016-11-27T20:47:00, Sun: Set up Travis CI for building personal page on Github Pages with Pelican
2016-10-08T01:10:00, Sat: Refactoring python code. Extracting variables and other.
2016-10-08T01:10:00, Sat: Useful python links
2016-09-29T14:10:00, Thu: How to fix google search suggestions in Firefox in Linux Mint
2016-08-14T14:16:00, Sun: podcasts
2016-08-14T14:10:00, Sun: git apply patch from another repository
2016-08-07T11:20:00, Sun: terminal setup in linux mint
2016-07-16T20:07:00, Sat: Fix boot record after moving linux mint partitions to another disk
2016-07-16T13:45:00, Sat: Mount yandex webdav on local dir
2016-06-26T23:01:00, Sun: how torch stores images
2016-06-26T22:11:00, Sun: Installing torch for miniconda
2016-06-25T22:43:00, Sat: compile dlib for miniconda
2016-06-19T12:37:00, Sun: Mount windows shares in linux
2016-06-03T23:08:00, Fri: deep learning
2016-06-02T23:24:00, Thu: autocomplete from history in terminal (in linux mint)
2016-06-01T20:47:00, Wed: Working with date and time in Java (russian)
2016-05-16T00:11:00, Mon: pypy with numpy
2016-05-01T12:33:00, Sun: About python
2016-04-10T20:50:00, Sun: Dropbox in linux mint
2016-04-05T00:45:00, Tue: C++ IDE for linux
2016-04-04T00:01:00, Mon: Debugging numpy (any C code of Python) using gdb
2016-03-22T22:12:00, Tue: Scipy in pypy
2016-03-22T21:51:00, Tue: Building Pypy
2016-01-08T08:30:00, Fri: TIL about PyPy

podcasts

http://dontspeak.podster.fm/
http://www.spotlightradio.net/

git apply patch from another repository

$ git --git-dir=../<some_other_repo>/.git format-patch -k -1 --stdout <commit SHA> | git am -3 -k

Source

http://stackoverflow.com/questions/6658313/generate-a-git-patch-for-a-specific-commit

terminal setup in linux mint

Updated version

sudo apt-get install tmux

Commands:

In tmux, hit the prefix ctrl+b (my modified prefix is ctrl+a) and then:

Sessions

:new<CR>  new session
s  list sessions
$  name session

Windows (tabs)

c  create window
w  list windows
n  next window
p  previous window
f  find window
,  name window
&  kill window

Panes (splits)

%  vertical split
"  horizontal split

o  swap panes
q  show pane numbers
x  kill pane
+  break pane into window (e.g. to select text by mouse to copy)
-  restore pane from window
⍽  space - toggle between layouts

Set up scrolling
put this command in your ~/.tmux.conf

setw -g mode-mouse on

# Lower escape timing from 500ms to 50ms for quicker response to scroll-buffer access.
set -s escape-time 50

Fix boot record after moving linux mint partitions to another disk

1. Copying partition is straightforward. Made using gparted.

2. Create bootable USB stick (Startup disk creator in Linux mint), boot from it.
Assume we have following partitions:

boot /sdb1 (don't really know how it works)
root /sdb2
home /sdb3

3. Do

sudo mount /dev/sda2 /mnt

## if you have boot partition:

# sudo mount /dev/sda1 /mnt/boot

sudo grub-install --root-directory=/mnt /dev/sda

Source of GRUB instructions

Mount yandex webdav on local dir

apt-get install davfs2

mkdir /mnt/yandex.disk

mount -t davfs https://webdav.yandex.ru /mnt/yandex.disk/

# check:

df -h /mnt/yandex.disk/

how torch stores images

require('image')

imgPath = "image.jpg"

img = torch.Tensor(1, 3, imgDim, imgDim)

img[1] = image.load(imgPath, 3, byte)

image is stored as float, conversion is (intensity/255.). stored top->bottom, line by line. format RGB.

display

require 'gnuplot'

gnuplot.figure(1)

gnuplot.imagesc(img[1])

Torch<->numpy dictionary
https://github.com/torch/torch7/wiki/Torch-for-Numpy-users

Installing torch for miniconda

I got the error while installing torch for miniconda. Something like

Linking CXX executable mshrable
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `BC'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `tgetnum'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `PC'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `tputs'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `tgetent'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `tgetflag'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `tgoto'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `UP'
~/miniconda/envs/py27/lib/libreadline.so.6: undefined reference to `tgetstr

The same was for python3 and python2 environments. The solution (at least for python 2) was to remove miniconda from path and compile torch with the system python.

Solution from https://github.com/ContinuumIO/anaconda-issues/issues/152

same here, temporay fix for me is to remove conda readline from the environment:

conda remove --force readline

and install the python bindings with pip install readline.

That way, I presume it is using the system readline.

compile dlib for miniconda

Activate miniconda environment (my environment is called py3):

source activate py3

Produce initial steps from the readme:

cd examples
mkdir build
cd build
cmake ..
cmake --build . --config Release

inside function build_dlib() of file setup.py add the highlighted code:

if platform_arch == '64bit' and sys.platform == "win32":

# 64bit build on Windows

.................. bla bla ...................

for ext in [py_ver.replace(".", "") + '.lib', py_ver + 'mu.lib', py_ver + 'm.lib', py_ver + 'u.lib']:

py_lib = os.path.abspath(os.path.join(inc_dir, '../libs/', 'python' + ext))

if os.path.exists(py_lib):

cmake_extra_arch += ['-DPYTHON_LIBRARY={lib}'.format(lib=py_lib)]

break

else:

cmake_extra_arch += ['-DPYTHON_LIBRARY=/home/user/miniconda/envs/py3/lib/libpython3.so']

cmake_extra_arch += ['-DPYTHON_INCLUDE_DIR=/home/user/miniconda/envs/py3/include/python3.5m']

build_dir = os.path.join(script_dir, "./tools/python/build")

......................bla bla.....................................

Replace /home/user/miniconda by your path to miniconda

Run python setup.py install (as usual)

compile dlib for miniconda

Activate miniconda environment (my environment is called py3):

source activate py3

Produce initial steps from the readme:

cd examples
mkdir build
cd build
cmake ..
cmake --build . --config Release

inside function build_dlib() of file setup.py add the highlighted code:

if platform_arch == '64bit' and sys.platform == "win32":

# 64bit build on Windows

.................. bla bla ...................

for ext in [py_ver.replace(".", "") + '.lib', py_ver + 'mu.lib', py_ver + 'm.lib', py_ver + 'u.lib']:

py_lib = os.path.abspath(os.path.join(inc_dir, '../libs/', 'python' + ext))

if os.path.exists(py_lib):

cmake_extra_arch += ['-DPYTHON_LIBRARY={lib}'.format(lib=py_lib)]

break

else:

cmake_extra_arch += ['-DPYTHON_LIBRARY=/home/user/miniconda/envs/py3/lib/libpython3.so']

cmake_extra_arch += ['-DPYTHON_INCLUDE_DIR=/home/user/miniconda/envs/py3/include/python3.5m']

build_dir = os.path.join(script_dir, "./tools/python/build")

......................bla bla.....................................

Replace /home/user/miniconda by your path to miniconda

Run python setup.py install (as usual)

Mount

mkdir /mnt/share
mount -t cifs //windowsmachineip/sharename -o username=user,password=urPassword /mnt/share

Unmount

umount /mnt/shares

deep learning

Inception-like CNN model based on 1d convolutions http://arxiv.org/pdf/1512.00567v3.pdf

Chat (in russian) http://closedcircles.com/?invite=99b1ac08509c560137b2e3c54d4398b0fa4c175e

autocomplete from history in terminal (in linux mint)

create a file .inputrc

in your home directory and put there
"\e[A": history-search-backward
"\e[B": history-search-forward
set show-all-if-ambiguous on
set completion-ignore-case on
TAB: menu-complete

Restart your terminal.

How you can autocomplete from history using Up and down keys.

source http://osxdaily.com/2013/04/24/improve-command-line-history-search/

Working with date and time in Java (russian)

Часть 1 https://habrahabr.ru/post/274811/
Часть 2 https://habrahabr.ru/post/274905/

pypy with numpy

Looks like pypy now can build numpy. Well, a slightly modified numpy.

1. Get default branch of pypy. be careful cause the developers don't maintain default branch compilable. Revision 84341 (c86b42dd7613) works for me.
2. compile using
./rpython/bin/rpython -O2 ./pypy/goal/targetpypystandalone.py --withoutmod-micronumpy
3. Create package and vitual environment. Something like this:
./pypy/tool/release/package.py --targetdir ./my_builds/build.tar.bz2 --builddir ./tmp/ --nostrip --archive-name pypy_84341
Needed to copy pypy-c and libpypy to pypy/goal beforehand.
4. Clone and follow instructions from
https://github.com/pypy/numpy/commits/cpyext-ext
Revision 3299d0d76fdb831fbcb4429a89c1f53bb36ea07f worked for me

Testing results:
----------------------------------------------------------------------
Ran 5900 tests in 78.216s

FAILED (KNOWNFAIL=3, SKIP=6, errors=218, failures=83)

scipy can be compiled if disabling submodules io/matlab and spatial. Though is still doesn't work.

About python

Cheat Sheet: Writing Python 2-3 compatible code
http://python-future.org/compatible_idioms.html

Google Python Guidelines
https://google.github.io/styleguide/pyguide.html

Inheritance
https://rhettinger.wordpress.com/2011/05/26/super-considered-super/

Dropbox in linux mint

Hmm. It doesn't work.
I see the icon when i do
dropbox stop
dropbox start
but I cannot clic on it.

Installed nemo-dropbox. At least now I have "Copy dropbox link" command in menu.
Don't forget to do

$ killall nemo

after installation to make it work.

Final Solution

source

Use

dropbox stop && dbus-launch dropbox start

Or

Do: opening 'Preferences' -> 'Startup Applications' and editing the dropbox entry so the command now reads:

CODE: SELECT ALL
dbus-launch dropbox start

 update might overwrite the change. I think it's better to leave the existing entry alone (only disabled) and create a new entry (e.g. "Launch DropBox") with the new start command (`dbus-launch dropbox start`)

C++ IDE for linux

CLion is awesome but expensive.

Found Codelite http://www.codelite.org on http://stackoverflow.com/a/1775460.
Looks good. No disgust after 10 minutes of work.

I was able to debug C code of Numpy in Virtualenv.
1. Launched from console after activating of virtual environment
> source ./<path to activate>/activate
2. add breakpoints
3. Run (F5)

Debugging numpy (any C code of Python) using gdb

I created a tiny python script that executes some python code, that executes some C code:

# contents of dbg_broadcast.py
import numpy
print list(numpy.broadcast([[1,2]],[[3],[4]]))

> gdb python
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.

.................. bla bla bla ..................................................

# Adding breakpoint to function "arraymultiter_new" (example):

(gdb) break arraymultiter_new

# There is name completition by Tab.

# Run script

(gdb) run dbg_broadcast.py

Starting program: /home/asdasd/work/bin/python dbg_broadcast.py

[Thread debugging using libthread_db enabled]

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, arraymultiter_new (__NPY_UNUSED_TAGGEDsubtype=0x7ffff6abbf80 <PyArrayMultiIter_Type>,

args=0x7ffff32dd050, kwds=0x0) at numpy/core/src/multiarray/iterators.c:1578

1578 {

(gdb)

And here we are.

Scipy in pypy

Looks like it is too early for using scipy in Pypy. There is a plenty of dependencies on C-code there.
I was able to install scipy 0.17 in pypy. I disabled all failed dependencies. Unfortunately it is completely useless. Almost everything doesn't work.

https://github.com/serge-m/scipy/blob/v0.17.0_for_pypy_bin/bin/scipy_0.17.0_for_pypy_draft.tar.gz

Building Pypy

Pypy builds faster if using -O2 option.

To build faster (according to pypy documentation) use prebuilt pypy from http://buildbot.pypy.org/nightly/trunk/

Using virtualenv to create virtual environment for it.

Build script (to be placed in pypy source directory):

#!/bin/bash
cd pypy/goal || exit 1
source <path to existing pypy environment>/bin/activate || exit 2
pypy ../../rpython/bin/rpython --batch -O2 targetpypystandalone

TIL about PyPy

Building from source root using command
pypy_src$ rpython/bin/rpython -Ojit pypy/goal/targetpypystandalone.py
produces structure with obsolete pypy-c and libpypy-c.so in
/tmp/usession-release-4.0.1-XXXX/build/pypy-nightly/bin/

Probably pypy compiler places there files integrated in the src distribution. To get fresh versions I had to use pypy-c and libpypy-c.so from sources root.

UPDATE:
Probably I was completely wrong.
pypy/tool/release/package.py has an option for (not) stripping resulting binary file: "--nostrip". By default it is enabled. Looks like it removed something unused from binaries. This operation updates timestamp of the pypy-c and libpypy-c.so. So probably that was the cause of my misunderstanding.

Script for packaging and creating virtual environment:

#!/bin/bash

rm -rf ./my_builds/ || exit 2
mkdir ./my_builds/ || exit 3

DST_NAME=$1
if [ -z "$DST_NAME" ]; then
    echo "DST_NAME is  empty"
    exit 3
fi

# runs packaging
./pypy/tool/release/package.py --builddir /home/pypy/builds/ --nostrip --archive-name $DST_NAME || exit 4

# creates a new virtual environment
virtualenv -p /home/pypy/builds/$DST_NAME/bin/pypy /home/pypy/env/$DST_NAME

# installing nose for numpy testing (optional)
source /home/pypy/env/$DST_NAME/bin/activate
pip install nose