Saturday, May 15, 2010

How to start python IDLE in virtualenv on Windows

IDLE is an integrated development environment for Python. It is more convenient than plain python command prompt. In order to start IDLE in virtualenv environment you have to:
  • Copy tcl directory from your python installation folder to your environment directory.
  • Create shortcut with name IDLE to pythonw.exe that is located in Scripts directory of your environment.
  • Append "%PYTHON_HOME%\Lib\idlelib\idle.pyw" to the shortcut Target.
Once you are done you should be able start IDLE from within your environment.

Friday, May 14, 2010

Python IDLE themes

It is pretty easy customize python IDLE font, syntax highlight. In order to change the font to consolas add the following to ~/.idlerc/config-main.cfg file (or download it from here):
[EditorWindow]
font-size = 11
font = consolas
There are two themes available desert and steam. Just download config-highlight.cfg into ~/.idlerc folder and add this to ~/.idlerc/config-main.cfg file:
[Theme]
default = 0
name = desert
The changes take place the next time you start IDLE.

Python development environment on Windows

Download Python msi installer from here. It is recommended to run the installer with administrator privileges. Point your installation directory to program files:
C:\Program Files\Python26
Create a system variable PYTHON_HOME and point it to the python installation directory. Append the following to PATH system variable:
;%PYTHON_HOME%;%PYTHON_HOME%\Scripts
Restart your computer so the environment changes take place.

Setuptools and easy_install

Setuptools available here. Download exe version and launch it. Once installer completes you will have easy_install available in %PYTHON_HOME%\Scripts directory.

MinGW

The easy_install needs C++ compiler in order to compile the C extensions that likely be distributed as source along with the python source. MinGW - is a native Windows port of the GNU Compiler Collection (GCC). You can download it from here. You need to select two options during install:
  • g++ compiler
  • MinGW Make
Point your installation directory to program files:
C:\Program Files\MinGW
Add the following to %PYTHON_HOME%\Lib\distutils\distutils.cfg file:
[build]
compiler=mingw32

Virtualenv

Virtual python environment builder can be installed this way:
easy_install virtualenv
Once you are done you can create isolated environment:
virtualenv --no-site-packages devenv
Please note that you must to run python, easy_install from your environment Scripts directory. You must manually copy %PYTHON_HOME%\Lib\distutils\distutils.cfg file to your environment e.g. devenv\Lib\destutils in order to be able compile C extension in your isolated environment.

Wednesday, May 12, 2010

Using TortoiseSVN SSH

TortoiseSVN is a windows shell extension for subversion. Here we are going access svn repository over ssh. You can read how to install and configure svn in this post, how to configure svnserve here and take a look at password-less ssh login here.
  • TortoiseSVN > Settings > Network > SSH Client, browse for TortoisePlink.exe, typical path is "C:\Program Files\TortoiseSVN\bin\TortoisePlink.exe"
  • In Checkout dialog enter path to the remote repository, e.g. svn+ssh://user1@deby/project1
  • In popup window type password.
  • If you setup password-less ssh login you need to add the private key to pageant (you can download it here). In this case authentication will go transparently.
  • If you already have open ssh session via PuTTY, you can use tunneling feature. In PuTTY configuration, under Category Connection > SSH > Tunnels set Source port to 22, Destination to localhost:22. Click Add, Apply. In this case URL to repository will be svn+ssh://user1@localhost/project1
Read more about subversion here.

Combining port knocking and password-less ssh login to a single click

You need to follow previous posts related to port knocking and password-less ssh. Here is a script that combines both:
@echo off

set ip=XXX.XXX.XXX.XXX
cd nmap-5.00
cmd /c knockin.cmd %ip% AAA BBB CCC DDD

cd ..\putty
start putty.exe -file deby %ip%
Here are few comments to the script:
  • Both nmap-5.00 and putty are sub directories of the script location.
  • Replace XXX.XXX.XXX.XXX with your remote host ip address
  • Replace AAA BBB CCC DDD with your knockin code
  • Putty uses file session (settings) stored in file deby.
The only thing you have to do is create a shortcut to your quick launch toolbar and you are done.

Tuesday, May 11, 2010

Password-less ssh login

SSH is often used to login without requiring passwords. It requires you generate your own personal set of private/public pair.

RSA security key

Generate personal set of private/public pair (do not use a passphrase):
user1@deby:~$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user1/.ssh/id_rsa):
Created directory '/home/user1/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user1/.ssh/id_rsa.
Your public key has been saved in /home/user1/.ssh/id_rsa.pub.
The key fingerprint is:
81:95:1a:bd:32:89:3b:c7:34:da:a2:a0:14:24:26:73 user1@deby
The key's randomart image is:
+--[ RSA 2048]----+
|       ...       |
|+oE   .oo        |
|=o   ..+..       |
| .  . B ..       |
|  .  * +S        |
|..  = +          |
|o. . +           |
|. .              |
|                 |
+-----------------+
Let ssh know your public key (here we are copy public ssh key from the client to remote server):
cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
Secure .ssh directory so nobody except you can get access there:
user1@deby:~$ chmod -R go-rwx .ssh/
user1@deby:~$ ls -la .ssh
total 20
drwx------ 2 user1 user1 4096 2010-06-09 15:33 .
drwxr-xr-x 4 user1 user1 4096 2010-06-09 15:22 ..
-rw------- 1 user1 user1 393  2010-06-09 15:33 authorized_keys
-rw------- 1 user1 user1 1675 2010-06-09 15:22 id_rsa
-rw------- 1 user1 user1 393  2010-06-09 15:22 id_rsa.pub

Troubleshooting ssh localhost login

You might need this while using existing ssh tunneling feature, e.g. svn+ssh access.
user1@deby:~$ ssh deby
ssh_exchange_identification: Connection closed by remote host
You need to add localhost to /etc/hosts.allow, e.g.
sshd: localhost
Here is another issue that is related to pam_access module (if it configured to prohibit local logins):
user1@deby:~$ ssh deby
Connection closed by 127.0.0.1
Here is a rule that prohibit local logins except from localhost (file /etc/security/access.conf):
# Disallow console logins
- : ALL : LOCAL EXCEPT 127.0.0.1

Windows client

If you are using a windows machine to connect to your remote ssh server with PuTTY you need few extra steps to import private key.
  • You need PuTTYgen. Download it from here.
  • Import the key. Menu Conversions > Import key.
  • Save private key (so PuTTY can understand it): Menu File > Save private key (do not set password).
  • Load previously saved session in PuTTY
  • In Category select Connection > Data, enter your remote username into Auto-login username
  • In Category select Connection > SSH, choose SSH2 as your preferred protocol version
  • In Category select Connection > SSH > Auth, browse the private key that you saved with PuTTYgen previously.
  • Save your session

ssh-copy-id

Mac OS X doesn't come with ssh-copy-id, here is a single line command:
cat ~/.ssh/id_rsa.pub | ssh user@machine \
  "mkdir ~/.ssh; cat >> ~/.ssh/authorized_keys"
You can download script here.

Monday, May 10, 2010

Accessing subversion repository via svnserve

The svnserve program is a lightweight server, capable of speaking to clients over TCP/IP using a custom, stateful protocol. Clients contact an svnserve server by using URLs that begin with the svn:// scheme.

Wrapper script

Using the svn protocol discloses the absolute path of subversion project repository. This is not good due to security reasons. Find where is svnserve:
deby:~# whereis svnserve
svnserve: /usr/bin/svnserve
We are going to hide the root of the repository tree by using a wrapper script.
mv /usr/bin/svnserve /usr/bin/svnserve.bin
touch /usr/bin/svnserve
chmod +x /usr/bin/svnserve
Add the following to /usr/bin/svnserve (assuming /var/svn/repos is the root of the repository tree):
#!/bin/sh

exec /usr/bin/svnserve.bin -r /var/svn/repos "$@"

svnserve as daemon

It is easy to run svnserve as a standalone “daemon” process. Use the -d option for this:
svnserve -d
Here is how you can checkout files now:
master@deby:~$ svn co svn://localhost/project1
A    project1/trunk
     ...
Checked out revision 4.
Stop svnserve:
deby:~# ps -A | grep svnserve
 2144 ?        00:00:00 svnserve.bin
deby:~# kill 2144

svnserve via ssh

You need configure password-less ssh login (see this). Here is how you can access the svn repository now (note, you do not have to start svnserve daemon, ssh does that for you):
svn co svn+ssh://localhost/project1
Read more about subversion here.

Working with subversion

Here is a quick summary of svn commands.

Status

Let make few changes to the trunk:
user1@deby:~/project1$ cd trunk/
user1@deby:~/project1/trunk$ mkdir tests
user1@deby:~/project1/trunk$ touch readme.txt tests/readme.txt
Here is how you can status changes made to the current directory:
user1@deby:~/project1/trunk$ svn status
?      tests
?      readme.txt

Add

Now let add these changes to repository:
user1@deby:~/project1/trunk$ svn add *
A         readme.txt
A         tests
A         tests/readme.txt
In order to add folder non recursively, use -N option

Commit

Added doesn't mean it now available to everyone to checkout. You need commit your work.
user1@deby:~/project1/trunk$ svn ci -m 'Testing add svn command'
Adding         trunk/readme.txt
Adding         trunk/tests
Adding         trunk/tests/readme.txt
Transmitting file data ..
Committed revision 2.

Delete

Deleting files is easy.
user1@deby:~/project1/trunk$ svn rm tests/readme.txt
D         tests/readme.txt
user1@deby:~/project1/trunk$ svn ci -m 'Testing delete svn command'
Deleting       trunk/tests/readme.txt

Committed revision 3.

Revert

Reverting deleted files:
user1@deby:~/project1/trunk$ svn rm readme.txt
D         readme.txt
user1@deby:~/project1/trunk$ svn revert readme.txt
Reverted 'readme.txt'

Make changes

Make some changes to readme.txt file and commit your work.
user1@deby:~/project1/trunk$ svn ci -m 'Testing modifications'
Sending        trunk/readme.txt
Transmitting file data .
Committed revision 4.

Update

You can update your local copy with changes made by other team members:
user1@deby:~/project1/trunk$ svn update
At revision 4.

Ignore

Often your working folder has some files or folders that you would like exclude from svn related operations (ignore), here is how you can get this done:
master@deby:~/project1/trunk$ svn propedit svn:ignore .
Set new value for property 'svn:ignore' on '.'
The above command will open your default editor to setup ignore properties for the current folder (notice '.' in command line). You need to enter file names, folders, etc you need to ignore (one item per line). Once you save changes and exit the editor, svn will update appropriate properties. You can also add settings to svn config file (~/.subversion/config or /etc/subversion/config) to ignore certain file types:
[miscellany]
global-ignores = build dist *.pyc *.pyo *.mo

Changes history

If you need to see a history log since revision 3:
user1@deby:~/project1/trunk$ svn log -r 3:HEAD
--------------------------------------------------------------------
r3 | user1 | 2010-05-09 01:07:12 +0300 (Sun, 09 May 2010) | 1 line

Testing delete svn command
--------------------------------------------------------------------
r4 | user1 | 2010-05-09 01:11:56 +0300 (Sun, 09 May 2010) | 1 line

Testing modifications
--------------------------------------------------------------------

File defferences

Open readme.txt file and add world to the end. Now we can see a difference we made:
user1@deby:~/project1/trunk$ svn diff readme.txt
Index: readme.txt
===================================================================
--- readme.txt  (revision 4)
+++ readme.txt  (working copy)
@@ -1 +1 @@
-hello
+hello world
Read more about subversion here.

Revision control with subversion

Subversion is a revision control system that stores your code on a central server in a repository.

Install

Let install subversion (if you followed previous post on how to mount ext3 partitions securily, you will need remount /usr and /var so they can be used by apt):
apt-get -y install subversion

Create repository

Once Subversion is installed, the next step is create your repository. We are going to keep all our repositories below /var/lib/svn/repos directory.
deby:~# mkdir -p /var/lib/svn/repos
The only users of security group svnusers can access repositories.
groupadd -r svnusers
chgrp svnusers /var/lib/svn/repos
chmod o-rwx /var/lib/svn/repos
ls -l /var/svn
Add users to group svnusers:
usermod -a -G svnusers user1
There are two common conventions for organizing projects. One is project major, and the other is project minor. In project major, each project has its own repository; in project minor - the repository is top-level. Here is a command to create svn repository for project1 using project major convention.
svnadmin create /var/lib/svn/repos/project1
Let secure project repository (consider use per project group assignment).
chgrp -R svnusers /var/lib/svn/repos/project1
find /var/lib/svn/repos/project1 -type d | xargs chmod g+x
chmod -R g+w,o-rwx /var/lib/svn/repos/project1
Now login as user1 and checkout:
user1@deby:~$ svn co file:///var/lib/svn/repos/project1/
Checked out revision 0.

Repository top directories

It is recommended create the following top level directories: trunk, branches, tags.
user1@deby:~$ cd project1/
user1@deby:~/project1$ svn mkdir trunk branches tags
A         trunk
A         branches
A         tags
user1@deby:~/project1$ svn commit -m 'added top directories'
Adding         branches
Adding         tags
Adding         trunk

Committed revision 1.