Changes between Initial Version and Version 1 of PyMw


Ignore:
Timestamp:
Aug 13, 2009, 2:33:26 PM (15 years ago)
Author:
davea
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • PyMw

    v1 v1  
     1= PyMW: Python master/worker system =
     2
     3The [http://pymw.sourceforge.net/ PyMW framework] is a master-worker computing framework for Python. Using PyMW in conjunction with a special interpreter (PyBOINC) allows you to send Python scripts as work units and does not require the client to have Python installed. PyMW can also be used without the portable Python interpreter, however, the client is responsible for installing the correct version of Python (2.4 or greater) and adding the interpreter to their system PATH environment variable.
     4
     5In addition to BOINC, PyMW also provides non-BOINC interfaces for Condor, MPI, and simple multi-core processing which can be selected using a command line switch. The multi-core interface is particularly useful for debugging BOINC applications before submitting them to your project.
     6
     7The following is a brief explanation of how to setup PyMW using the [VmServer BOINC virtual server].
     8
     9== Installing and Testing PyMW ==
     10Before starting this tutorial you should setup the [VmServer BOINC virtual server] following the instructions in the [QuickStart Quick Start] guide. It's also highly recommended that you ssh into your VM instead of using a VM viewer so that you can copy and paste commands. You can start the ssh daemon with the following command:
     11
     12{{{
     13 $ su -c "/usr/sbin/sshd"
     14}}}
     15
     16Once your server is setup, login to your VM as "boincadm" (with password "boincadmpw" if you haven't changed it). To install PyMW, first download the PyMW package and unzip it into your home folder on the BOINC VM and run the setup script to install it into your local Python "site-packages" directory. This can be done with the following commmands:
     17
     18{{{
     19$ cd ~
     20$ wget http://downloads.sourceforge.net/project/pymw/pymw/PyMW%200.2/pymw-0.2.tar.gz
     21$ tar -xzf pymw-0.2.tar.gz
     22$ rm pymw-0.2.tar.gz
     23$ cd pymw-0.2
     24$ su -c "python setup.py install"
     25
     26(the password for root is rootpw, unless you changed it)
     27}}}
     28
     29''Note that the URL for PyMW was current as of the time of this writing (v0.2), but you should check for and use the newest available release.''
     30
     31At this point, PyMW is installed and so it's a good idea to test the installation. While still in the PyMW directory that you extracted above, try the following example distributed with PyMW:
     32
     33{{{
     34$ cd examples
     35$ python monte_pi.py
     36
     37(some output should appear)
     38
     39$ cd ../../
     40}}}
     41
     42After a moment, you should see some vague estimate of PI printed out to the screen. This means that PyMW is installed and working properly. By default, the example that was run above uses the "generic" interface and does not interact with the BOINC server.
     43
     44Next, you should install PyBOINC to allow your PyMW apps to run without Python being installed on the BOINC client's machines. The following downloads the zip file, extracts it and moves it into a convenient location:
     45
     46{{{
     47$ wget http://bitbucket.org/jeremycowles/pyboinc/downloads/pyboinc-0.01.zip
     48$ unzip pyboinc-0.01.zip
     49$ mv pyboinc-0.01/bin-python26 pyboinc
     50$ rm -r pyboinc-0.01
     51$ rm pyboinc-0.01.zip
     52}}}
     53
     54With PyMW and PyBOINC setup, it's time to run a real BOINC application with PyMW.
     55
     56== Running PyMW Applications ==
     57
     58'''''Note:''' the following assumes that you have a folder in your home directory called "pyboinc" containing the Python 2.6 binaries and that you have installed PyMW as described in the previous section.''
     59
     60First, you must setup a project for testing, I will call mine "sandbox" and it will be run on the local address I have given my VM, 192.168.1.190, but you should use whatever address or domain name that is setup on your VM:
     61
     62{{{
     63$ cd ~/boinc/tools
     64$ ./make_project --url_base http://192.168.1.190 sandbox
     65
     66(answer yes to all questions/warnings)
     67}}}
     68
     69This may take a while to generate the server keys. After it's completed, you should go into your project directory and add it to your cron jobs and to the Apache server:
     70
     71{{{
     72$ cd ~/projects/sandbox
     73$ su -c 'cat sandbox.httpd.conf >> /etc/apache2/httpd.conf'
     74  (you'll be prompted for the root password, which is "rootpw")
     75
     76$ su -c 'apache2ctl -k restart'
     77$ crontab sandbox.cronjob
     78}}}
     79
     80The project should not be created, but there is no application installed (notice, we never called xadd or update_versions). Luckily, PyMW will do this for us. Now all we have to do is run our application with the boinc interface and point it to our home directory:
     81
     82{{{
     83$ cd ~/pymw-0.2/examples
     84$ python monte_pi.py -n 1 -i boinc -p /home/boincadm/projects/sandbox -c /home/boincadm/pyboinc -a 1.00_python26.zip
     85
     86(lots of information will scroll out and then it will eventually wait)
     87}}}
     88
     89The Monte Carlo PI estimation application is waiting for BOINC clients to process it's work units. So go back to your main PC (not the VM) and open up the BOINC client (install it if needed).
     90
     91Once you have the BOINC manager open, click on "advanced view", when debugging it's best if you stay in this view. Next, click "Tools->Attach to project" from the main application menu. Click "Next" and then enter your project URL ("http://192.168.1.190/sandbox" if you are following this exactly) and then click next again.
     92
     93Now that you are attached to the project, it should be listed in the project view. Click on the project and click the "update" button to force it to request work from the server. The PyBOINC interpreter will download and then a handful of tasks should appear. Once these tasks complete you can hit "update" on the project again to force the client to send them back to the server.
     94
     95Back at the BOINC VM, you should now see the results of the computation.
     96
     97== Talking to BOINC from Python ==
     98
     99The PyBOINC interpreter also includes a custom module for interfacing with the BOINC client API under the namespace "boinc". Most common BOINC API functions have been implemented, to see a full list, run the following on the BOINC VM:
     100
     101{{{
     102$ cd ~/pyboinc/linux
     103
     104(you may or may not need the next line)
     105
     106$ chmod +x pymw_1.03_i686-pc-linux-gnu
     107$ echo -e "import boinc\n""print dir(boinc)" > boinc_test.py
     108$ ./pymw_1.03_i686-pc-linux-gnu boinc_test.py 1.00_python26.zip
     109}}}
     110
     111Accessing the boinc API in your code is as easy as importing the boinc namespace and then using the function you need, for example:
     112
     113{{{
     114import boinc
     115print boinc.time_to_checkpoint()
     116}}}
     117
     118There are two things worth noting here, first, all boinc functions have the "boinc_" part of the C name removed, which is redundant when using the "boinc" namespace as a prefix. Secondly, the function boinc.get_app_name() is not actually part of the BOINC API, but is provided by PyBOINC to let you discover the true application name that was launched and not just your script name.
     119
     120See the PyMW /examples folder for more examples on using PyMW or go to the [http://pymw.sourceforge.net/ PyMW] web site for official documentation.