parallel processing in python in slicer? plus python path weirdness.

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

parallel processing in python in slicer? plus python path weirdness.

Lauren O'Donnell-2
Hi slicers,
I am trying to find a way to do parallel processing in the slicer
python environment.  It seems this is somehow disabled by slicer?

I installed the joblib package and from the slicer python shell I get
the following error in Mac OS X 10.5.8.  This test code runs fine if I
only use 1 job, however 2 jobs fail.

>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
Traceback (most recent call last):
 File "<console>", line 1, in <module>
 File "./joblib/parallel.py", line 287, in __call__
   pool.join()
 File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
line 342, in join
   p.join()
 File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
line 119, in join
   res = self._popen.wait(timeout)
 File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
line 117, in wait
   return self.poll(0)
 File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
line 106, in poll
   pid, sts = os.waitpid(self.pid, flag)
OSError: [Errno 4] Interrupted system call


On Mac OS X 10.5.8 the above code actually runs correctly if I use the
same slicer python build but I run it outside of slicer itself (run
python from the command line).  I had a different experience with my
mac laptop where it did not work in or out of slicer, but it was also
a type of forking-related error message.

This is with Slicer4, a recent build from the last week or so.

I have one more question/issue. Running the Slicer python from an
xterm launched by slicer, versus using the python shell within slicer,
gives different python package loading behavior. Specifically, the
within-slicer python shell did not find a package that was installed
using easy_install from an egg, while the same python run from the
xterm did find the package.  A specific difference in the environment
is that the len(sys.path)  is different when python is run in the two
ways, however the variable PYTHONPATH is the same.

So overall it seems the python inside of slicer does not have the same
multiprocessing functionality or path-related startup as the python run from the
command line.  Is this the expected behavior?

Please advise, thanks!  It would be really helpful to be able to multiprocess...
Lauren
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

pieper
Administrator
Hi Lauren -

Can you send the version of the joblib package you are installing
(preferably a url that can pass to wget) and the command you use to
install it?  I can try to replicate this on linux.

-Steve

On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:

> Hi slicers,
> I am trying to find a way to do parallel processing in the slicer
> python environment.  It seems this is somehow disabled by slicer?
>
> I installed the joblib package and from the slicer python shell I get
> the following error in Mac OS X 10.5.8.  This test code runs fine if I
> only use 1 job, however 2 jobs fail.
>
>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
> Traceback (most recent call last):
>   File "<console>", line 1, in<module>
>   File "./joblib/parallel.py", line 287, in __call__
>     pool.join()
>   File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
> line 342, in join
>     p.join()
>   File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
> line 119, in join
>     res = self._popen.wait(timeout)
>   File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
> line 117, in wait
>     return self.poll(0)
>   File "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
> line 106, in poll
>     pid, sts = os.waitpid(self.pid, flag)
> OSError: [Errno 4] Interrupted system call
>
>
> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
> same slicer python build but I run it outside of slicer itself (run
> python from the command line).  I had a different experience with my
> mac laptop where it did not work in or out of slicer, but it was also
> a type of forking-related error message.
>
> This is with Slicer4, a recent build from the last week or so.
>
> I have one more question/issue. Running the Slicer python from an
> xterm launched by slicer, versus using the python shell within slicer,
> gives different python package loading behavior. Specifically, the
> within-slicer python shell did not find a package that was installed
> using easy_install from an egg, while the same python run from the
> xterm did find the package.  A specific difference in the environment
> is that the len(sys.path)  is different when python is run in the two
> ways, however the variable PYTHONPATH is the same.
>
> So overall it seems the python inside of slicer does not have the same
> multiprocessing functionality or path-related startup as the python run from the
> command line.  Is this the expected behavior?
>
> Please advise, thanks!  It would be really helpful to be able to multiprocess...
> Lauren
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Lauren O'Donnell-2
Hi Steve,
Here are the source and egg versions. Please let me know how it goes. Thanks!
Lauren

http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg

http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz


On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper <[hidden email]> wrote:

> Hi Lauren -
>
> Can you send the version of the joblib package you are installing
> (preferably a url that can pass to wget) and the command you use to install
> it?  I can try to replicate this on linux.
>
> -Steve
>
> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>
>> Hi slicers,
>> I am trying to find a way to do parallel processing in the slicer
>> python environment.  It seems this is somehow disabled by slicer?
>>
>> I installed the joblib package and from the slicer python shell I get
>> the following error in Mac OS X 10.5.8.  This test code runs fine if I
>> only use 1 job, however 2 jobs fail.
>>
>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>
>> Traceback (most recent call last):
>>  File "<console>", line 1, in<module>
>>  File "./joblib/parallel.py", line 287, in __call__
>>    pool.join()
>>  File
>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>> line 342, in join
>>    p.join()
>>  File
>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>> line 119, in join
>>    res = self._popen.wait(timeout)
>>  File
>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>> line 117, in wait
>>    return self.poll(0)
>>  File
>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>> line 106, in poll
>>    pid, sts = os.waitpid(self.pid, flag)
>> OSError: [Errno 4] Interrupted system call
>>
>>
>> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
>> same slicer python build but I run it outside of slicer itself (run
>> python from the command line).  I had a different experience with my
>> mac laptop where it did not work in or out of slicer, but it was also
>> a type of forking-related error message.
>>
>> This is with Slicer4, a recent build from the last week or so.
>>
>> I have one more question/issue. Running the Slicer python from an
>> xterm launched by slicer, versus using the python shell within slicer,
>> gives different python package loading behavior. Specifically, the
>> within-slicer python shell did not find a package that was installed
>> using easy_install from an egg, while the same python run from the
>> xterm did find the package.  A specific difference in the environment
>> is that the len(sys.path)  is different when python is run in the two
>> ways, however the variable PYTHONPATH is the same.
>>
>> So overall it seems the python inside of slicer does not have the same
>> multiprocessing functionality or path-related startup as the python run
>> from the
>> command line.  Is this the expected behavior?
>>
>> Please advise, thanks!  It would be really helpful to be able to
>> multiprocess...
>> Lauren
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to
>> [hidden email] with unsubscribe as the
>> subject
>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

pieper
Administrator
Hi Lauren -

I was able to replicate this - on linux I saw this message on the output:

> Process PoolWorker-1:
> Traceback (most recent call last):
>   File "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py", line 223, in _bootstrap
>     sys.stdin.close()

so I went to that file and line and made the following change (take out
the sys.stdin and add 'pass')

             try:
                 #sys.stdin.close()
                 #sys.stdin = open(os.devnull)
                 pass

now it works for me :)  Can you try the same fix?

The issue is probably due to the fact that the pythonqt console has
already taken over the stdin/stdout/stderr so joblib can't work with it
the way it normally would.  I don't know what side effects there might
be, but the output of the one-line test was shown fine in the slicer
console (looked the same as running it from python in an xterm).

Regarding the paths, it's possible that when you start your xterm there
is another .bashrc (or similar) that changes the environment after the
slicer launcher has configured it.  I haven't used easy_install myself -
probably best to install with the setup.py method.

Best,
Steve


On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:

> Hi Steve,
> Here are the source and egg versions. Please let me know how it goes. Thanks!
> Lauren
>
> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>
> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>
>
> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>  wrote:
>> Hi Lauren -
>>
>> Can you send the version of the joblib package you are installing
>> (preferably a url that can pass to wget) and the command you use to install
>> it?  I can try to replicate this on linux.
>>
>> -Steve
>>
>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>
>>> Hi slicers,
>>> I am trying to find a way to do parallel processing in the slicer
>>> python environment.  It seems this is somehow disabled by slicer?
>>>
>>> I installed the joblib package and from the slicer python shell I get
>>> the following error in Mac OS X 10.5.8.  This test code runs fine if I
>>> only use 1 job, however 2 jobs fail.
>>>
>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>
>>> Traceback (most recent call last):
>>>   File "<console>", line 1, in<module>
>>>   File "./joblib/parallel.py", line 287, in __call__
>>>     pool.join()
>>>   File
>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>> line 342, in join
>>>     p.join()
>>>   File
>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>> line 119, in join
>>>     res = self._popen.wait(timeout)
>>>   File
>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>> line 117, in wait
>>>     return self.poll(0)
>>>   File
>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>> line 106, in poll
>>>     pid, sts = os.waitpid(self.pid, flag)
>>> OSError: [Errno 4] Interrupted system call
>>>
>>>
>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
>>> same slicer python build but I run it outside of slicer itself (run
>>> python from the command line).  I had a different experience with my
>>> mac laptop where it did not work in or out of slicer, but it was also
>>> a type of forking-related error message.
>>>
>>> This is with Slicer4, a recent build from the last week or so.
>>>
>>> I have one more question/issue. Running the Slicer python from an
>>> xterm launched by slicer, versus using the python shell within slicer,
>>> gives different python package loading behavior. Specifically, the
>>> within-slicer python shell did not find a package that was installed
>>> using easy_install from an egg, while the same python run from the
>>> xterm did find the package.  A specific difference in the environment
>>> is that the len(sys.path)  is different when python is run in the two
>>> ways, however the variable PYTHONPATH is the same.
>>>
>>> So overall it seems the python inside of slicer does not have the same
>>> multiprocessing functionality or path-related startup as the python run
>>> from the
>>> command line.  Is this the expected behavior?
>>>
>>> Please advise, thanks!  It would be really helpful to be able to
>>> multiprocess...
>>> Lauren
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to
>>> [hidden email] with unsubscribe as the
>>> subject
>>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Halle, Michael Wilfred,Ph.D.
Not sure you want to make this change in general -- a process needs to dissociate it's streams from a terminal or its parent process to properly fork (otherwise you can end up with zombie processes).

--Mike




On Feb 14, 2011, at 11:16 AM, Steve Pieper <[hidden email]> wrote:

> Hi Lauren -
>
> I was able to replicate this - on linux I saw this message on the output:
>
>> Process PoolWorker-1:
>> Traceback (most recent call last):
>>  File "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py", line 223, in _bootstrap
>>    sys.stdin.close()
>
> so I went to that file and line and made the following change (take out
> the sys.stdin and add 'pass')
>
>             try:
>                 #sys.stdin.close()
>                 #sys.stdin = open(os.devnull)
>                 pass
>
> now it works for me :)  Can you try the same fix?
>
> The issue is probably due to the fact that the pythonqt console has
> already taken over the stdin/stdout/stderr so joblib can't work with it
> the way it normally would.  I don't know what side effects there might
> be, but the output of the one-line test was shown fine in the slicer
> console (looked the same as running it from python in an xterm).
>
> Regarding the paths, it's possible that when you start your xterm there
> is another .bashrc (or similar) that changes the environment after the
> slicer launcher has configured it.  I haven't used easy_install myself -
> probably best to install with the setup.py method.
>
> Best,
> Steve
>
>
> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>> Hi Steve,
>> Here are the source and egg versions. Please let me know how it goes. Thanks!
>> Lauren
>>
>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>
>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>
>>
>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>  wrote:
>>> Hi Lauren -
>>>
>>> Can you send the version of the joblib package you are installing
>>> (preferably a url that can pass to wget) and the command you use to install
>>> it?  I can try to replicate this on linux.
>>>
>>> -Steve
>>>
>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>
>>>> Hi slicers,
>>>> I am trying to find a way to do parallel processing in the slicer
>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>
>>>> I installed the joblib package and from the slicer python shell I get
>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if I
>>>> only use 1 job, however 2 jobs fail.
>>>>
>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> Traceback (most recent call last):
>>>>  File "<console>", line 1, in<module>
>>>>  File "./joblib/parallel.py", line 287, in __call__
>>>>    pool.join()
>>>>  File
>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>> line 342, in join
>>>>    p.join()
>>>>  File
>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>> line 119, in join
>>>>    res = self._popen.wait(timeout)
>>>>  File
>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 117, in wait
>>>>    return self.poll(0)
>>>>  File
>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 106, in poll
>>>>    pid, sts = os.waitpid(self.pid, flag)
>>>> OSError: [Errno 4] Interrupted system call
>>>>
>>>>
>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
>>>> same slicer python build but I run it outside of slicer itself (run
>>>> python from the command line).  I had a different experience with my
>>>> mac laptop where it did not work in or out of slicer, but it was also
>>>> a type of forking-related error message.
>>>>
>>>> This is with Slicer4, a recent build from the last week or so.
>>>>
>>>> I have one more question/issue. Running the Slicer python from an
>>>> xterm launched by slicer, versus using the python shell within slicer,
>>>> gives different python package loading behavior. Specifically, the
>>>> within-slicer python shell did not find a package that was installed
>>>> using easy_install from an egg, while the same python run from the
>>>> xterm did find the package.  A specific difference in the environment
>>>> is that the len(sys.path)  is different when python is run in the two
>>>> ways, however the variable PYTHONPATH is the same.
>>>>
>>>> So overall it seems the python inside of slicer does not have the same
>>>> multiprocessing functionality or path-related startup as the python run
>>>> from the
>>>> command line.  Is this the expected behavior?
>>>>
>>>> Please advise, thanks!  It would be really helpful to be able to
>>>> multiprocess...
>>>> Lauren
>>>> _______________________________________________
>>>> slicer-devel mailing list
>>>> [hidden email]
>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>> To unsubscribe: send email to
>>>> [hidden email] with unsubscribe as the
>>>> subject
>>>
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Lauren O'Donnell-2
Thank you Steve for your suggestion to test what is going on with the
python parallelization.  I commented out those lines and now, I can
run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
otherwise Slicer would hang. Now Slicer does not hang and there is an
error message displayed in the python console (see below).

This is on Mac OS X 10.6.6.

>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
line 287, in __call__
    pool.join()
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
line 342, in join
    p.join()
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
line 119, in join
    res = self._popen.wait(timeout)
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
line 117, in wait
    return self.poll(0)
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
line 106, in poll
    pid, sts = os.waitpid(self.pid, flag)
OSError: [Errno 4] Interrupted system call
>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
line 287, in __call__
    pool.join()
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
line 342, in join
    p.join()
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
line 119, in join
    res = self._popen.wait(timeout)
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
line 117, in wait
    return self.poll(0)
  File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
line 106, in poll
    pid, sts = os.waitpid(self.pid, flag)
OSError: [Errno 4] Interrupted system call



On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle <[hidden email]> wrote:

> Not sure you want to make this change in general -- a process needs to dissociate it's streams from a terminal or its parent process to properly fork (otherwise you can end up with zombie processes).
>
> --Mike
>
>
>
>
> On Feb 14, 2011, at 11:16 AM, Steve Pieper <[hidden email]> wrote:
>
>> Hi Lauren -
>>
>> I was able to replicate this - on linux I saw this message on the output:
>>
>>> Process PoolWorker-1:
>>> Traceback (most recent call last):
>>>  File "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py", line 223, in _bootstrap
>>>    sys.stdin.close()
>>
>> so I went to that file and line and made the following change (take out
>> the sys.stdin and add 'pass')
>>
>>             try:
>>                 #sys.stdin.close()
>>                 #sys.stdin = open(os.devnull)
>>                 pass
>>
>> now it works for me :)  Can you try the same fix?
>>
>> The issue is probably due to the fact that the pythonqt console has
>> already taken over the stdin/stdout/stderr so joblib can't work with it
>> the way it normally would.  I don't know what side effects there might
>> be, but the output of the one-line test was shown fine in the slicer
>> console (looked the same as running it from python in an xterm).
>>
>> Regarding the paths, it's possible that when you start your xterm there
>> is another .bashrc (or similar) that changes the environment after the
>> slicer launcher has configured it.  I haven't used easy_install myself -
>> probably best to install with the setup.py method.
>>
>> Best,
>> Steve
>>
>>
>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>> Hi Steve,
>>> Here are the source and egg versions. Please let me know how it goes. Thanks!
>>> Lauren
>>>
>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>
>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>
>>>
>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>  wrote:
>>>> Hi Lauren -
>>>>
>>>> Can you send the version of the joblib package you are installing
>>>> (preferably a url that can pass to wget) and the command you use to install
>>>> it?  I can try to replicate this on linux.
>>>>
>>>> -Steve
>>>>
>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>
>>>>> Hi slicers,
>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>
>>>>> I installed the joblib package and from the slicer python shell I get
>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if I
>>>>> only use 1 job, however 2 jobs fail.
>>>>>
>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>
>>>>> Traceback (most recent call last):
>>>>>  File "<console>", line 1, in<module>
>>>>>  File "./joblib/parallel.py", line 287, in __call__
>>>>>    pool.join()
>>>>>  File
>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>> line 342, in join
>>>>>    p.join()
>>>>>  File
>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>> line 119, in join
>>>>>    res = self._popen.wait(timeout)
>>>>>  File
>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>> line 117, in wait
>>>>>    return self.poll(0)
>>>>>  File
>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>> line 106, in poll
>>>>>    pid, sts = os.waitpid(self.pid, flag)
>>>>> OSError: [Errno 4] Interrupted system call
>>>>>
>>>>>
>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>> python from the command line).  I had a different experience with my
>>>>> mac laptop where it did not work in or out of slicer, but it was also
>>>>> a type of forking-related error message.
>>>>>
>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>
>>>>> I have one more question/issue. Running the Slicer python from an
>>>>> xterm launched by slicer, versus using the python shell within slicer,
>>>>> gives different python package loading behavior. Specifically, the
>>>>> within-slicer python shell did not find a package that was installed
>>>>> using easy_install from an egg, while the same python run from the
>>>>> xterm did find the package.  A specific difference in the environment
>>>>> is that the len(sys.path)  is different when python is run in the two
>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>
>>>>> So overall it seems the python inside of slicer does not have the same
>>>>> multiprocessing functionality or path-related startup as the python run
>>>>> from the
>>>>> command line.  Is this the expected behavior?
>>>>>
>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>> multiprocess...
>>>>> Lauren
>>>>> _______________________________________________
>>>>> slicer-devel mailing list
>>>>> [hidden email]
>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>> To unsubscribe: send email to
>>>>> [hidden email] with unsubscribe as the
>>>>> subject
>>>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

pieper
Administrator
Hi Folks -

As a follow up to this thread: Lauren, Demian, and other of us met
yesterday and decided that rather than make the embedded console support
the parallelism constructs we're going to try importing the slicer base
and modules logics into a standard python process - it should work and
avoid these issues plus make it much easier to use slicer code together
with packages like matplotlib, ipython, etc.

-Steve

On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:

> Thank you Steve for your suggestion to test what is going on with the
> python parallelization.  I commented out those lines and now, I can
> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
> otherwise Slicer would hang. Now Slicer does not hang and there is an
> error message displayed in the python console (see below).
>
> This is on Mac OS X 10.6.6.
>
>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
> Traceback (most recent call last):
>    File "<console>", line 1, in<module>
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
> line 287, in __call__
>      pool.join()
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
> line 342, in join
>      p.join()
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
> line 119, in join
>      res = self._popen.wait(timeout)
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
> line 117, in wait
>      return self.poll(0)
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
> line 106, in poll
>      pid, sts = os.waitpid(self.pid, flag)
> OSError: [Errno 4] Interrupted system call
>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
> Traceback (most recent call last):
>    File "<console>", line 1, in<module>
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
> line 287, in __call__
>      pool.join()
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
> line 342, in join
>      p.join()
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
> line 119, in join
>      res = self._popen.wait(timeout)
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
> line 117, in wait
>      return self.poll(0)
>    File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
> line 106, in poll
>      pid, sts = os.waitpid(self.pid, flag)
> OSError: [Errno 4] Interrupted system call
>
>
>
> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>  wrote:
>> Not sure you want to make this change in general -- a process needs to dissociate it's streams from a terminal or its parent process to properly fork (otherwise you can end up with zombie processes).
>>
>> --Mike
>>
>>
>>
>>
>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>  wrote:
>>
>>> Hi Lauren -
>>>
>>> I was able to replicate this - on linux I saw this message on the output:
>>>
>>>> Process PoolWorker-1:
>>>> Traceback (most recent call last):
>>>>   File "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py", line 223, in _bootstrap
>>>>     sys.stdin.close()
>>>
>>> so I went to that file and line and made the following change (take out
>>> the sys.stdin and add 'pass')
>>>
>>>              try:
>>>                  #sys.stdin.close()
>>>                  #sys.stdin = open(os.devnull)
>>>                  pass
>>>
>>> now it works for me :)  Can you try the same fix?
>>>
>>> The issue is probably due to the fact that the pythonqt console has
>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>> the way it normally would.  I don't know what side effects there might
>>> be, but the output of the one-line test was shown fine in the slicer
>>> console (looked the same as running it from python in an xterm).
>>>
>>> Regarding the paths, it's possible that when you start your xterm there
>>> is another .bashrc (or similar) that changes the environment after the
>>> slicer launcher has configured it.  I haven't used easy_install myself -
>>> probably best to install with the setup.py method.
>>>
>>> Best,
>>> Steve
>>>
>>>
>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>> Hi Steve,
>>>> Here are the source and egg versions. Please let me know how it goes. Thanks!
>>>> Lauren
>>>>
>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>
>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>
>>>>
>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>    wrote:
>>>>> Hi Lauren -
>>>>>
>>>>> Can you send the version of the joblib package you are installing
>>>>> (preferably a url that can pass to wget) and the command you use to install
>>>>> it?  I can try to replicate this on linux.
>>>>>
>>>>> -Steve
>>>>>
>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>
>>>>>> Hi slicers,
>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>
>>>>>> I installed the joblib package and from the slicer python shell I get
>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if I
>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>
>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>   File "<console>", line 1, in<module>
>>>>>>   File "./joblib/parallel.py", line 287, in __call__
>>>>>>     pool.join()
>>>>>>   File
>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>> line 342, in join
>>>>>>     p.join()
>>>>>>   File
>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>> line 119, in join
>>>>>>     res = self._popen.wait(timeout)
>>>>>>   File
>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>> line 117, in wait
>>>>>>     return self.poll(0)
>>>>>>   File
>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>> line 106, in poll
>>>>>>     pid, sts = os.waitpid(self.pid, flag)
>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>
>>>>>>
>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>> python from the command line).  I had a different experience with my
>>>>>> mac laptop where it did not work in or out of slicer, but it was also
>>>>>> a type of forking-related error message.
>>>>>>
>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>
>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>> xterm launched by slicer, versus using the python shell within slicer,
>>>>>> gives different python package loading behavior. Specifically, the
>>>>>> within-slicer python shell did not find a package that was installed
>>>>>> using easy_install from an egg, while the same python run from the
>>>>>> xterm did find the package.  A specific difference in the environment
>>>>>> is that the len(sys.path)  is different when python is run in the two
>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>
>>>>>> So overall it seems the python inside of slicer does not have the same
>>>>>> multiprocessing functionality or path-related startup as the python run
>>>>>> from the
>>>>>> command line.  Is this the expected behavior?
>>>>>>
>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>> multiprocess...
>>>>>> Lauren
>>>>>> _______________________________________________
>>>>>> slicer-devel mailing list
>>>>>> [hidden email]
>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>> To unsubscribe: send email to
>>>>>> [hidden email] with unsubscribe as the
>>>>>> subject
>>>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

pieper
Administrator
I got a chance to work on importing slicer libraries into a stand-alone
python shell a bit so I wrote up a wiki page on it:

http://www.slicer.org/slicerWiki/index.php/Slicer4:Python

it works pretty slick for me on linux.  I'll be curious to hear how it
works for Lauren on mac (if it works - no doubt some platform-specific
tweaking will be needed).  Also, although I'm sure it's possible to do,
I pity the person who works on getting this to run correctly on windows.

-Steve

On 02/23/2011 11:16 AM, Steve Pieper wrote:

> Hi Folks -
>
> As a follow up to this thread: Lauren, Demian, and other of us met
> yesterday and decided that rather than make the embedded console support
> the parallelism constructs we're going to try importing the slicer base
> and modules logics into a standard python process - it should work and
> avoid these issues plus make it much easier to use slicer code together
> with packages like matplotlib, ipython, etc.
>
> -Steve
>
> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>> Thank you Steve for your suggestion to test what is going on with the
>> python parallelization.  I commented out those lines and now, I can
>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>> error message displayed in the python console (see below).
>>
>> This is on Mac OS X 10.6.6.
>>
>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>> Traceback (most recent call last):
>>     File "<console>", line 1, in<module>
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>> line 287, in __call__
>>       pool.join()
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>> line 342, in join
>>       p.join()
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>> line 119, in join
>>       res = self._popen.wait(timeout)
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>> line 117, in wait
>>       return self.poll(0)
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>> line 106, in poll
>>       pid, sts = os.waitpid(self.pid, flag)
>> OSError: [Errno 4] Interrupted system call
>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>> Traceback (most recent call last):
>>     File "<console>", line 1, in<module>
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>> line 287, in __call__
>>       pool.join()
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>> line 342, in join
>>       p.join()
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>> line 119, in join
>>       res = self._popen.wait(timeout)
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>> line 117, in wait
>>       return self.poll(0)
>>     File "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>> line 106, in poll
>>       pid, sts = os.waitpid(self.pid, flag)
>> OSError: [Errno 4] Interrupted system call
>>
>>
>>
>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>   wrote:
>>> Not sure you want to make this change in general -- a process needs to dissociate it's streams from a terminal or its parent process to properly fork (otherwise you can end up with zombie processes).
>>>
>>> --Mike
>>>
>>>
>>>
>>>
>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>   wrote:
>>>
>>>> Hi Lauren -
>>>>
>>>> I was able to replicate this - on linux I saw this message on the output:
>>>>
>>>>> Process PoolWorker-1:
>>>>> Traceback (most recent call last):
>>>>>    File "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py", line 223, in _bootstrap
>>>>>      sys.stdin.close()
>>>>
>>>> so I went to that file and line and made the following change (take out
>>>> the sys.stdin and add 'pass')
>>>>
>>>>               try:
>>>>                   #sys.stdin.close()
>>>>                   #sys.stdin = open(os.devnull)
>>>>                   pass
>>>>
>>>> now it works for me :)  Can you try the same fix?
>>>>
>>>> The issue is probably due to the fact that the pythonqt console has
>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>> the way it normally would.  I don't know what side effects there might
>>>> be, but the output of the one-line test was shown fine in the slicer
>>>> console (looked the same as running it from python in an xterm).
>>>>
>>>> Regarding the paths, it's possible that when you start your xterm there
>>>> is another .bashrc (or similar) that changes the environment after the
>>>> slicer launcher has configured it.  I haven't used easy_install myself -
>>>> probably best to install with the setup.py method.
>>>>
>>>> Best,
>>>> Steve
>>>>
>>>>
>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>> Hi Steve,
>>>>> Here are the source and egg versions. Please let me know how it goes. Thanks!
>>>>> Lauren
>>>>>
>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>
>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>
>>>>>
>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>     wrote:
>>>>>> Hi Lauren -
>>>>>>
>>>>>> Can you send the version of the joblib package you are installing
>>>>>> (preferably a url that can pass to wget) and the command you use to install
>>>>>> it?  I can try to replicate this on linux.
>>>>>>
>>>>>> -Steve
>>>>>>
>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>
>>>>>>> Hi slicers,
>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>
>>>>>>> I installed the joblib package and from the slicer python shell I get
>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if I
>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>
>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>>    File "<console>", line 1, in<module>
>>>>>>>    File "./joblib/parallel.py", line 287, in __call__
>>>>>>>      pool.join()
>>>>>>>    File
>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>> line 342, in join
>>>>>>>      p.join()
>>>>>>>    File
>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>> line 119, in join
>>>>>>>      res = self._popen.wait(timeout)
>>>>>>>    File
>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>> line 117, in wait
>>>>>>>      return self.poll(0)
>>>>>>>    File
>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>> line 106, in poll
>>>>>>>      pid, sts = os.waitpid(self.pid, flag)
>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>
>>>>>>>
>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use the
>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>> python from the command line).  I had a different experience with my
>>>>>>> mac laptop where it did not work in or out of slicer, but it was also
>>>>>>> a type of forking-related error message.
>>>>>>>
>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>
>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>> xterm launched by slicer, versus using the python shell within slicer,
>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>> xterm did find the package.  A specific difference in the environment
>>>>>>> is that the len(sys.path)  is different when python is run in the two
>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>
>>>>>>> So overall it seems the python inside of slicer does not have the same
>>>>>>> multiprocessing functionality or path-related startup as the python run
>>>>>>> from the
>>>>>>> command line.  Is this the expected behavior?
>>>>>>>
>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>> multiprocess...
>>>>>>> Lauren
>>>>>>> _______________________________________________
>>>>>>> slicer-devel mailing list
>>>>>>> [hidden email]
>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>> To unsubscribe: send email to
>>>>>>> [hidden email] with unsubscribe as the
>>>>>>> subject
>>>>>>
>>>> _______________________________________________
>>>> slicer-devel mailing list
>>>> [hidden email]
>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>>>
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Lauren O'Donnell-2
Hi Steve,
I've almost gotten it to work on a Mac...  I can import vtk but not
yet slicer, any idea what these errors mean?  Thanks!
-Lauren

brage:bin odonnell$ ./python
Python 2.6.6 (r266:84292, Feb 15 2011, 13:35:35)
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import vtk
>>> import slicer
No module named libFreeSurferPython
No module named libMRMLPython
No module named libMRMLCLIPython
No module named libMRMLLogicPython
No module named libRemoteIOPython
No module named libvtkTeemPython
No module named PythonQt.qMRMLWidgets
No module named libSlicerBaseLogicPython
No module named PythonQt.qSlicerBaseQTCore
No module named PythonQt.qSlicerBaseQTGUI
No module named libvtkSlicerAtlasCreatorModuleLogicPython
No module named libvtkSlicerCropVolumeModuleLogicPython
No module named libvtkSlicerEMSegmentModuleLogicPython
No module named libvtkSlicerFetchMIModuleLogicPython
No module named libvtkSlicerModelMirrorModuleLogicPython
No module named libvtkSlicerModelTransformModuleLogicPython
No module named libvtkSlicerRDTractSelectModuleLogicPython
No module named libvtkSlicerTractographyDisplayModuleLogicPython
No module named libvtkSlicerTractographyModuleLogicPython
No module named libvtkSlicerVolumesModuleLogicPython
>>> mrml = slicer.vtkMRMLScene()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'vtkMRMLScene'



On Sat, Feb 26, 2011 at 5:30 PM, Steve Pieper <[hidden email]> wrote:

> I got a chance to work on importing slicer libraries into a stand-alone
> python shell a bit so I wrote up a wiki page on it:
>
> http://www.slicer.org/slicerWiki/index.php/Slicer4:Python
>
> it works pretty slick for me on linux.  I'll be curious to hear how it works
> for Lauren on mac (if it works - no doubt some platform-specific tweaking
> will be needed).  Also, although I'm sure it's possible to do, I pity the
> person who works on getting this to run correctly on windows.
>
> -Steve
>
> On 02/23/2011 11:16 AM, Steve Pieper wrote:
>>
>> Hi Folks -
>>
>> As a follow up to this thread: Lauren, Demian, and other of us met
>> yesterday and decided that rather than make the embedded console support
>> the parallelism constructs we're going to try importing the slicer base
>> and modules logics into a standard python process - it should work and
>> avoid these issues plus make it much easier to use slicer code together
>> with packages like matplotlib, ipython, etc.
>>
>> -Steve
>>
>> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>>>
>>> Thank you Steve for your suggestion to test what is going on with the
>>> python parallelization.  I commented out those lines and now, I can
>>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>>> error message displayed in the python console (see below).
>>>
>>> This is on Mac OS X 10.6.6.
>>>
>>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>>
>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>
>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>
>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>
>>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>>>
>>> Traceback (most recent call last):
>>>    File "<console>", line 1, in<module>
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>> line 287, in __call__
>>>      pool.join()
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>> line 342, in join
>>>      p.join()
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>> line 119, in join
>>>      res = self._popen.wait(timeout)
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>> line 117, in wait
>>>      return self.poll(0)
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>> line 106, in poll
>>>      pid, sts = os.waitpid(self.pid, flag)
>>> OSError: [Errno 4] Interrupted system call
>>>>>>
>>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>>>
>>> Traceback (most recent call last):
>>>    File "<console>", line 1, in<module>
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>> line 287, in __call__
>>>      pool.join()
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>> line 342, in join
>>>      p.join()
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>> line 119, in join
>>>      res = self._popen.wait(timeout)
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>> line 117, in wait
>>>      return self.poll(0)
>>>    File
>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>> line 106, in poll
>>>      pid, sts = os.waitpid(self.pid, flag)
>>> OSError: [Errno 4] Interrupted system call
>>>
>>>
>>>
>>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>
>>> wrote:
>>>>
>>>> Not sure you want to make this change in general -- a process needs to
>>>> dissociate it's streams from a terminal or its parent process to properly
>>>> fork (otherwise you can end up with zombie processes).
>>>>
>>>> --Mike
>>>>
>>>>
>>>>
>>>>
>>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>
>>>> wrote:
>>>>
>>>>> Hi Lauren -
>>>>>
>>>>> I was able to replicate this - on linux I saw this message on the
>>>>> output:
>>>>>
>>>>>> Process PoolWorker-1:
>>>>>> Traceback (most recent call last):
>>>>>>   File
>>>>>> "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>> line 223, in _bootstrap
>>>>>>     sys.stdin.close()
>>>>>
>>>>> so I went to that file and line and made the following change (take out
>>>>> the sys.stdin and add 'pass')
>>>>>
>>>>>              try:
>>>>>                  #sys.stdin.close()
>>>>>                  #sys.stdin = open(os.devnull)
>>>>>                  pass
>>>>>
>>>>> now it works for me :)  Can you try the same fix?
>>>>>
>>>>> The issue is probably due to the fact that the pythonqt console has
>>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>>> the way it normally would.  I don't know what side effects there might
>>>>> be, but the output of the one-line test was shown fine in the slicer
>>>>> console (looked the same as running it from python in an xterm).
>>>>>
>>>>> Regarding the paths, it's possible that when you start your xterm there
>>>>> is another .bashrc (or similar) that changes the environment after the
>>>>> slicer launcher has configured it.  I haven't used easy_install myself
>>>>> -
>>>>> probably best to install with the setup.py method.
>>>>>
>>>>> Best,
>>>>> Steve
>>>>>
>>>>>
>>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>>>
>>>>>> Hi Steve,
>>>>>> Here are the source and egg versions. Please let me know how it goes.
>>>>>> Thanks!
>>>>>> Lauren
>>>>>>
>>>>>>
>>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>>
>>>>>>
>>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>
>>>>>>     wrote:
>>>>>>>
>>>>>>> Hi Lauren -
>>>>>>>
>>>>>>> Can you send the version of the joblib package you are installing
>>>>>>> (preferably a url that can pass to wget) and the command you use to
>>>>>>> install
>>>>>>> it?  I can try to replicate this on linux.
>>>>>>>
>>>>>>> -Steve
>>>>>>>
>>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>>
>>>>>>>> Hi slicers,
>>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>>
>>>>>>>> I installed the joblib package and from the slicer python shell I
>>>>>>>> get
>>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if
>>>>>>>> I
>>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>>
>>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>>
>>>>>>>> Traceback (most recent call last):
>>>>>>>>   File "<console>", line 1, in<module>
>>>>>>>>   File "./joblib/parallel.py", line 287, in __call__
>>>>>>>>     pool.join()
>>>>>>>>   File
>>>>>>>>
>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>>> line 342, in join
>>>>>>>>     p.join()
>>>>>>>>   File
>>>>>>>>
>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>> line 119, in join
>>>>>>>>     res = self._popen.wait(timeout)
>>>>>>>>   File
>>>>>>>>
>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>> line 117, in wait
>>>>>>>>     return self.poll(0)
>>>>>>>>   File
>>>>>>>>
>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>> line 106, in poll
>>>>>>>>     pid, sts = os.waitpid(self.pid, flag)
>>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use
>>>>>>>> the
>>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>>> python from the command line).  I had a different experience with my
>>>>>>>> mac laptop where it did not work in or out of slicer, but it was
>>>>>>>> also
>>>>>>>> a type of forking-related error message.
>>>>>>>>
>>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>>
>>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>>> xterm launched by slicer, versus using the python shell within
>>>>>>>> slicer,
>>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>>> xterm did find the package.  A specific difference in the
>>>>>>>> environment
>>>>>>>> is that the len(sys.path)  is different when python is run in the
>>>>>>>> two
>>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>>
>>>>>>>> So overall it seems the python inside of slicer does not have the
>>>>>>>> same
>>>>>>>> multiprocessing functionality or path-related startup as the python
>>>>>>>> run
>>>>>>>> from the
>>>>>>>> command line.  Is this the expected behavior?
>>>>>>>>
>>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>>> multiprocess...
>>>>>>>> Lauren
>>>>>>>> _______________________________________________
>>>>>>>> slicer-devel mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>> To unsubscribe: send email to
>>>>>>>> [hidden email] with unsubscribe as
>>>>>>>> the
>>>>>>>> subject
>>>>>>>
>>>>> _______________________________________________
>>>>> slicer-devel mailing list
>>>>> [hidden email]
>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>> To unsubscribe: send email to
>>>>> [hidden email] with unsubscribe as the
>>>>> subject
>>>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to
>> [hidden email] with unsubscribe as the
>> subject
>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

pieper
Administrator
Hmm, that's odd - I just tried and it works on my mac build.

Did you update slicer4 svn?  There was a small change to the launcher to
set the slicer python path.

Regarding the messages, they are probably coming from the
Base/Python/*.py scripts which import the shared libraries.  Maybe by
looking in there you can see what's going on...

-Steve

On 03/01/2011 06:17 PM, Lauren O'Donnell wrote:

> Hi Steve,
> I've almost gotten it to work on a Mac...  I can import vtk but not
> yet slicer, any idea what these errors mean?  Thanks!
> -Lauren
>
> brage:bin odonnell$ ./python
> Python 2.6.6 (r266:84292, Feb 15 2011, 13:35:35)
> [GCC 4.0.1 (Apple Inc. build 5490)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import vtk
>>>> import slicer
> No module named libFreeSurferPython
> No module named libMRMLPython
> No module named libMRMLCLIPython
> No module named libMRMLLogicPython
> No module named libRemoteIOPython
> No module named libvtkTeemPython
> No module named PythonQt.qMRMLWidgets
> No module named libSlicerBaseLogicPython
> No module named PythonQt.qSlicerBaseQTCore
> No module named PythonQt.qSlicerBaseQTGUI
> No module named libvtkSlicerAtlasCreatorModuleLogicPython
> No module named libvtkSlicerCropVolumeModuleLogicPython
> No module named libvtkSlicerEMSegmentModuleLogicPython
> No module named libvtkSlicerFetchMIModuleLogicPython
> No module named libvtkSlicerModelMirrorModuleLogicPython
> No module named libvtkSlicerModelTransformModuleLogicPython
> No module named libvtkSlicerRDTractSelectModuleLogicPython
> No module named libvtkSlicerTractographyDisplayModuleLogicPython
> No module named libvtkSlicerTractographyModuleLogicPython
> No module named libvtkSlicerVolumesModuleLogicPython
>>>> mrml = slicer.vtkMRMLScene()
> Traceback (most recent call last):
>    File "<stdin>", line 1, in<module>
> AttributeError: 'module' object has no attribute 'vtkMRMLScene'
>
>
>
> On Sat, Feb 26, 2011 at 5:30 PM, Steve Pieper<[hidden email]>  wrote:
>> I got a chance to work on importing slicer libraries into a stand-alone
>> python shell a bit so I wrote up a wiki page on it:
>>
>> http://www.slicer.org/slicerWiki/index.php/Slicer4:Python
>>
>> it works pretty slick for me on linux.  I'll be curious to hear how it works
>> for Lauren on mac (if it works - no doubt some platform-specific tweaking
>> will be needed).  Also, although I'm sure it's possible to do, I pity the
>> person who works on getting this to run correctly on windows.
>>
>> -Steve
>>
>> On 02/23/2011 11:16 AM, Steve Pieper wrote:
>>>
>>> Hi Folks -
>>>
>>> As a follow up to this thread: Lauren, Demian, and other of us met
>>> yesterday and decided that rather than make the embedded console support
>>> the parallelism constructs we're going to try importing the slicer base
>>> and modules logics into a standard python process - it should work and
>>> avoid these issues plus make it much easier to use slicer code together
>>> with packages like matplotlib, ipython, etc.
>>>
>>> -Steve
>>>
>>> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>>>>
>>>> Thank you Steve for your suggestion to test what is going on with the
>>>> python parallelization.  I commented out those lines and now, I can
>>>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>>>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>>>> error message displayed in the python console (see below).
>>>>
>>>> This is on Mac OS X 10.6.6.
>>>>
>>>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>
>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>
>>>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> Traceback (most recent call last):
>>>>     File "<console>", line 1, in<module>
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>> line 287, in __call__
>>>>       pool.join()
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>> line 342, in join
>>>>       p.join()
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>> line 119, in join
>>>>       res = self._popen.wait(timeout)
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 117, in wait
>>>>       return self.poll(0)
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 106, in poll
>>>>       pid, sts = os.waitpid(self.pid, flag)
>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>
>>>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> Traceback (most recent call last):
>>>>     File "<console>", line 1, in<module>
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>> line 287, in __call__
>>>>       pool.join()
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>> line 342, in join
>>>>       p.join()
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>> line 119, in join
>>>>       res = self._popen.wait(timeout)
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 117, in wait
>>>>       return self.poll(0)
>>>>     File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 106, in poll
>>>>       pid, sts = os.waitpid(self.pid, flag)
>>>> OSError: [Errno 4] Interrupted system call
>>>>
>>>>
>>>>
>>>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>
>>>> wrote:
>>>>>
>>>>> Not sure you want to make this change in general -- a process needs to
>>>>> dissociate it's streams from a terminal or its parent process to properly
>>>>> fork (otherwise you can end up with zombie processes).
>>>>>
>>>>> --Mike
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Hi Lauren -
>>>>>>
>>>>>> I was able to replicate this - on linux I saw this message on the
>>>>>> output:
>>>>>>
>>>>>>> Process PoolWorker-1:
>>>>>>> Traceback (most recent call last):
>>>>>>>    File
>>>>>>> "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>> line 223, in _bootstrap
>>>>>>>      sys.stdin.close()
>>>>>>
>>>>>> so I went to that file and line and made the following change (take out
>>>>>> the sys.stdin and add 'pass')
>>>>>>
>>>>>>               try:
>>>>>>                   #sys.stdin.close()
>>>>>>                   #sys.stdin = open(os.devnull)
>>>>>>                   pass
>>>>>>
>>>>>> now it works for me :)  Can you try the same fix?
>>>>>>
>>>>>> The issue is probably due to the fact that the pythonqt console has
>>>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>>>> the way it normally would.  I don't know what side effects there might
>>>>>> be, but the output of the one-line test was shown fine in the slicer
>>>>>> console (looked the same as running it from python in an xterm).
>>>>>>
>>>>>> Regarding the paths, it's possible that when you start your xterm there
>>>>>> is another .bashrc (or similar) that changes the environment after the
>>>>>> slicer launcher has configured it.  I haven't used easy_install myself
>>>>>> -
>>>>>> probably best to install with the setup.py method.
>>>>>>
>>>>>> Best,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>>>>
>>>>>>> Hi Steve,
>>>>>>> Here are the source and egg versions. Please let me know how it goes.
>>>>>>> Thanks!
>>>>>>> Lauren
>>>>>>>
>>>>>>>
>>>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>>>
>>>>>>>
>>>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>
>>>>>>>      wrote:
>>>>>>>>
>>>>>>>> Hi Lauren -
>>>>>>>>
>>>>>>>> Can you send the version of the joblib package you are installing
>>>>>>>> (preferably a url that can pass to wget) and the command you use to
>>>>>>>> install
>>>>>>>> it?  I can try to replicate this on linux.
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>>
>>>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>>>
>>>>>>>>> Hi slicers,
>>>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>>>
>>>>>>>>> I installed the joblib package and from the slicer python shell I
>>>>>>>>> get
>>>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if
>>>>>>>>> I
>>>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>>>
>>>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>>>
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>    File "<console>", line 1, in<module>
>>>>>>>>>    File "./joblib/parallel.py", line 287, in __call__
>>>>>>>>>      pool.join()
>>>>>>>>>    File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>>>> line 342, in join
>>>>>>>>>      p.join()
>>>>>>>>>    File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>> line 119, in join
>>>>>>>>>      res = self._popen.wait(timeout)
>>>>>>>>>    File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>> line 117, in wait
>>>>>>>>>      return self.poll(0)
>>>>>>>>>    File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>> line 106, in poll
>>>>>>>>>      pid, sts = os.waitpid(self.pid, flag)
>>>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use
>>>>>>>>> the
>>>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>>>> python from the command line).  I had a different experience with my
>>>>>>>>> mac laptop where it did not work in or out of slicer, but it was
>>>>>>>>> also
>>>>>>>>> a type of forking-related error message.
>>>>>>>>>
>>>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>>>
>>>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>>>> xterm launched by slicer, versus using the python shell within
>>>>>>>>> slicer,
>>>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>>>> xterm did find the package.  A specific difference in the
>>>>>>>>> environment
>>>>>>>>> is that the len(sys.path)  is different when python is run in the
>>>>>>>>> two
>>>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>>>
>>>>>>>>> So overall it seems the python inside of slicer does not have the
>>>>>>>>> same
>>>>>>>>> multiprocessing functionality or path-related startup as the python
>>>>>>>>> run
>>>>>>>>> from the
>>>>>>>>> command line.  Is this the expected behavior?
>>>>>>>>>
>>>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>>>> multiprocess...
>>>>>>>>> Lauren
>>>>>>>>> _______________________________________________
>>>>>>>>> slicer-devel mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>>> To unsubscribe: send email to
>>>>>>>>> [hidden email] with unsubscribe as
>>>>>>>>> the
>>>>>>>>> subject
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> slicer-devel mailing list
>>>>>> [hidden email]
>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>> To unsubscribe: send email to
>>>>>> [hidden email] with unsubscribe as the
>>>>>> subject
>>>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to
>>> [hidden email] with unsubscribe as the
>>> subject
>>
_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Demian Wassermann-2
In reply to this post by Lauren O'Donnell-2
Hey Lauren, my bet is that there is some environment variable that should point the DYLD_LIBRARY_PATH and the PYTHONPATH to the lib folder where libMRMLPython and the other modules are.

You can find that folder and try doing

import sys
sys.path.append('<here goes that folder>')
import slicer

HTH
D

--
Demian Wassermann, PhD
[hidden email]
LMI / PNL / SPL Labs
Harvard Medical School
Brigham and Women's Hospital
1249 Boylston, Boston, MA, USA

On Mar 1, 2011, at 6:17 PM, Lauren O'Donnell wrote:

> Hi Steve,
> I've almost gotten it to work on a Mac...  I can import vtk but not
> yet slicer, any idea what these errors mean?  Thanks!
> -Lauren
>
> brage:bin odonnell$ ./python
> Python 2.6.6 (r266:84292, Feb 15 2011, 13:35:35)
> [GCC 4.0.1 (Apple Inc. build 5490)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import vtk
>>>> import slicer
> No module named libFreeSurferPython
> No module named libMRMLPython
> No module named libMRMLCLIPython
> No module named libMRMLLogicPython
> No module named libRemoteIOPython
> No module named libvtkTeemPython
> No module named PythonQt.qMRMLWidgets
> No module named libSlicerBaseLogicPython
> No module named PythonQt.qSlicerBaseQTCore
> No module named PythonQt.qSlicerBaseQTGUI
> No module named libvtkSlicerAtlasCreatorModuleLogicPython
> No module named libvtkSlicerCropVolumeModuleLogicPython
> No module named libvtkSlicerEMSegmentModuleLogicPython
> No module named libvtkSlicerFetchMIModuleLogicPython
> No module named libvtkSlicerModelMirrorModuleLogicPython
> No module named libvtkSlicerModelTransformModuleLogicPython
> No module named libvtkSlicerRDTractSelectModuleLogicPython
> No module named libvtkSlicerTractographyDisplayModuleLogicPython
> No module named libvtkSlicerTractographyModuleLogicPython
> No module named libvtkSlicerVolumesModuleLogicPython
>>>> mrml = slicer.vtkMRMLScene()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> AttributeError: 'module' object has no attribute 'vtkMRMLScene'
>
>
>
> On Sat, Feb 26, 2011 at 5:30 PM, Steve Pieper <[hidden email]> wrote:
>> I got a chance to work on importing slicer libraries into a stand-alone
>> python shell a bit so I wrote up a wiki page on it:
>>
>> http://www.slicer.org/slicerWiki/index.php/Slicer4:Python
>>
>> it works pretty slick for me on linux.  I'll be curious to hear how it works
>> for Lauren on mac (if it works - no doubt some platform-specific tweaking
>> will be needed).  Also, although I'm sure it's possible to do, I pity the
>> person who works on getting this to run correctly on windows.
>>
>> -Steve
>>
>> On 02/23/2011 11:16 AM, Steve Pieper wrote:
>>>
>>> Hi Folks -
>>>
>>> As a follow up to this thread: Lauren, Demian, and other of us met
>>> yesterday and decided that rather than make the embedded console support
>>> the parallelism constructs we're going to try importing the slicer base
>>> and modules logics into a standard python process - it should work and
>>> avoid these issues plus make it much easier to use slicer code together
>>> with packages like matplotlib, ipython, etc.
>>>
>>> -Steve
>>>
>>> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>>>>
>>>> Thank you Steve for your suggestion to test what is going on with the
>>>> python parallelization.  I commented out those lines and now, I can
>>>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>>>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>>>> error message displayed in the python console (see below).
>>>>
>>>> This is on Mac OS X 10.6.6.
>>>>
>>>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>
>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>
>>>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> Traceback (most recent call last):
>>>>    File "<console>", line 1, in<module>
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>> line 287, in __call__
>>>>      pool.join()
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>> line 342, in join
>>>>      p.join()
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>> line 119, in join
>>>>      res = self._popen.wait(timeout)
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 117, in wait
>>>>      return self.poll(0)
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 106, in poll
>>>>      pid, sts = os.waitpid(self.pid, flag)
>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>
>>>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>>>>
>>>> Traceback (most recent call last):
>>>>    File "<console>", line 1, in<module>
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>> line 287, in __call__
>>>>      pool.join()
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>> line 342, in join
>>>>      p.join()
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>> line 119, in join
>>>>      res = self._popen.wait(timeout)
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 117, in wait
>>>>      return self.poll(0)
>>>>    File
>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>> line 106, in poll
>>>>      pid, sts = os.waitpid(self.pid, flag)
>>>> OSError: [Errno 4] Interrupted system call
>>>>
>>>>
>>>>
>>>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>
>>>> wrote:
>>>>>
>>>>> Not sure you want to make this change in general -- a process needs to
>>>>> dissociate it's streams from a terminal or its parent process to properly
>>>>> fork (otherwise you can end up with zombie processes).
>>>>>
>>>>> --Mike
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>
>>>>> wrote:
>>>>>
>>>>>> Hi Lauren -
>>>>>>
>>>>>> I was able to replicate this - on linux I saw this message on the
>>>>>> output:
>>>>>>
>>>>>>> Process PoolWorker-1:
>>>>>>> Traceback (most recent call last):
>>>>>>>   File
>>>>>>> "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>> line 223, in _bootstrap
>>>>>>>     sys.stdin.close()
>>>>>>
>>>>>> so I went to that file and line and made the following change (take out
>>>>>> the sys.stdin and add 'pass')
>>>>>>
>>>>>>              try:
>>>>>>                  #sys.stdin.close()
>>>>>>                  #sys.stdin = open(os.devnull)
>>>>>>                  pass
>>>>>>
>>>>>> now it works for me :)  Can you try the same fix?
>>>>>>
>>>>>> The issue is probably due to the fact that the pythonqt console has
>>>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>>>> the way it normally would.  I don't know what side effects there might
>>>>>> be, but the output of the one-line test was shown fine in the slicer
>>>>>> console (looked the same as running it from python in an xterm).
>>>>>>
>>>>>> Regarding the paths, it's possible that when you start your xterm there
>>>>>> is another .bashrc (or similar) that changes the environment after the
>>>>>> slicer launcher has configured it.  I haven't used easy_install myself
>>>>>> -
>>>>>> probably best to install with the setup.py method.
>>>>>>
>>>>>> Best,
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>>>>
>>>>>>> Hi Steve,
>>>>>>> Here are the source and egg versions. Please let me know how it goes.
>>>>>>> Thanks!
>>>>>>> Lauren
>>>>>>>
>>>>>>>
>>>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>>>
>>>>>>>
>>>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>
>>>>>>>     wrote:
>>>>>>>>
>>>>>>>> Hi Lauren -
>>>>>>>>
>>>>>>>> Can you send the version of the joblib package you are installing
>>>>>>>> (preferably a url that can pass to wget) and the command you use to
>>>>>>>> install
>>>>>>>> it?  I can try to replicate this on linux.
>>>>>>>>
>>>>>>>> -Steve
>>>>>>>>
>>>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>>>
>>>>>>>>> Hi slicers,
>>>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>>>
>>>>>>>>> I installed the joblib package and from the slicer python shell I
>>>>>>>>> get
>>>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if
>>>>>>>>> I
>>>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>>>
>>>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>>>
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>   File "<console>", line 1, in<module>
>>>>>>>>>   File "./joblib/parallel.py", line 287, in __call__
>>>>>>>>>     pool.join()
>>>>>>>>>   File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>>>> line 342, in join
>>>>>>>>>     p.join()
>>>>>>>>>   File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>> line 119, in join
>>>>>>>>>     res = self._popen.wait(timeout)
>>>>>>>>>   File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>> line 117, in wait
>>>>>>>>>     return self.poll(0)
>>>>>>>>>   File
>>>>>>>>>
>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>> line 106, in poll
>>>>>>>>>     pid, sts = os.waitpid(self.pid, flag)
>>>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use
>>>>>>>>> the
>>>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>>>> python from the command line).  I had a different experience with my
>>>>>>>>> mac laptop where it did not work in or out of slicer, but it was
>>>>>>>>> also
>>>>>>>>> a type of forking-related error message.
>>>>>>>>>
>>>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>>>
>>>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>>>> xterm launched by slicer, versus using the python shell within
>>>>>>>>> slicer,
>>>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>>>> xterm did find the package.  A specific difference in the
>>>>>>>>> environment
>>>>>>>>> is that the len(sys.path)  is different when python is run in the
>>>>>>>>> two
>>>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>>>
>>>>>>>>> So overall it seems the python inside of slicer does not have the
>>>>>>>>> same
>>>>>>>>> multiprocessing functionality or path-related startup as the python
>>>>>>>>> run
>>>>>>>>> from the
>>>>>>>>> command line.  Is this the expected behavior?
>>>>>>>>>
>>>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>>>> multiprocess...
>>>>>>>>> Lauren
>>>>>>>>> _______________________________________________
>>>>>>>>> slicer-devel mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>>> To unsubscribe: send email to
>>>>>>>>> [hidden email] with unsubscribe as
>>>>>>>>> the
>>>>>>>>> subject
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> slicer-devel mailing list
>>>>>> [hidden email]
>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>> To unsubscribe: send email to
>>>>>> [hidden email] with unsubscribe as the
>>>>>> subject
>>>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to
>>> [hidden email] with unsubscribe as the
>>> subject
>>
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with unsubscribe as the subject

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Luca Antiga
Hey (sometimes they come back :-) ),
 or rather

import os
os.environ["DYLD_LIBRARY_PATH"] = '<here goes that folder>'
os.environ["PYTHONPATH"] = '<here goes that folder>'
import slicer

Cheers

Luca



On Mar 2, 2011, at 4:00 PM, Demian Wassermann wrote:

> Hey Lauren, my bet is that there is some environment variable that should point the DYLD_LIBRARY_PATH and the PYTHONPATH to the lib folder where libMRMLPython and the other modules are.
>
> You can find that folder and try doing
>
> import sys
> sys.path.append('<here goes that folder>')
> import slicer
>
> HTH
> D
>
> --
> Demian Wassermann, PhD
> [hidden email]
> LMI / PNL / SPL Labs
> Harvard Medical School
> Brigham and Women's Hospital
> 1249 Boylston, Boston, MA, USA
>
> On Mar 1, 2011, at 6:17 PM, Lauren O'Donnell wrote:
>
>> Hi Steve,
>> I've almost gotten it to work on a Mac...  I can import vtk but not
>> yet slicer, any idea what these errors mean?  Thanks!
>> -Lauren
>>
>> brage:bin odonnell$ ./python
>> Python 2.6.6 (r266:84292, Feb 15 2011, 13:35:35)
>> [GCC 4.0.1 (Apple Inc. build 5490)] on darwin
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import vtk
>>>>> import slicer
>> No module named libFreeSurferPython
>> No module named libMRMLPython
>> No module named libMRMLCLIPython
>> No module named libMRMLLogicPython
>> No module named libRemoteIOPython
>> No module named libvtkTeemPython
>> No module named PythonQt.qMRMLWidgets
>> No module named libSlicerBaseLogicPython
>> No module named PythonQt.qSlicerBaseQTCore
>> No module named PythonQt.qSlicerBaseQTGUI
>> No module named libvtkSlicerAtlasCreatorModuleLogicPython
>> No module named libvtkSlicerCropVolumeModuleLogicPython
>> No module named libvtkSlicerEMSegmentModuleLogicPython
>> No module named libvtkSlicerFetchMIModuleLogicPython
>> No module named libvtkSlicerModelMirrorModuleLogicPython
>> No module named libvtkSlicerModelTransformModuleLogicPython
>> No module named libvtkSlicerRDTractSelectModuleLogicPython
>> No module named libvtkSlicerTractographyDisplayModuleLogicPython
>> No module named libvtkSlicerTractographyModuleLogicPython
>> No module named libvtkSlicerVolumesModuleLogicPython
>>>>> mrml = slicer.vtkMRMLScene()
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> AttributeError: 'module' object has no attribute 'vtkMRMLScene'
>>
>>
>>
>> On Sat, Feb 26, 2011 at 5:30 PM, Steve Pieper <[hidden email]> wrote:
>>> I got a chance to work on importing slicer libraries into a stand-alone
>>> python shell a bit so I wrote up a wiki page on it:
>>>
>>> http://www.slicer.org/slicerWiki/index.php/Slicer4:Python
>>>
>>> it works pretty slick for me on linux.  I'll be curious to hear how it works
>>> for Lauren on mac (if it works - no doubt some platform-specific tweaking
>>> will be needed).  Also, although I'm sure it's possible to do, I pity the
>>> person who works on getting this to run correctly on windows.
>>>
>>> -Steve
>>>
>>> On 02/23/2011 11:16 AM, Steve Pieper wrote:
>>>>
>>>> Hi Folks -
>>>>
>>>> As a follow up to this thread: Lauren, Demian, and other of us met
>>>> yesterday and decided that rather than make the embedded console support
>>>> the parallelism constructs we're going to try importing the slicer base
>>>> and modules logics into a standard python process - it should work and
>>>> avoid these issues plus make it much easier to use slicer code together
>>>> with packages like matplotlib, ipython, etc.
>>>>
>>>> -Steve
>>>>
>>>> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>>>>>
>>>>> Thank you Steve for your suggestion to test what is going on with the
>>>>> python parallelization.  I commented out those lines and now, I can
>>>>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>>>>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>>>>> error message displayed in the python console (see below).
>>>>>
>>>>> This is on Mac OS X 10.6.6.
>>>>>
>>>>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>>>>
>>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>>
>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>
>>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>>
>>>>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File "<console>", line 1, in<module>
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>>> line 287, in __call__
>>>>>     pool.join()
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>> line 342, in join
>>>>>     p.join()
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>> line 119, in join
>>>>>     res = self._popen.wait(timeout)
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>> line 117, in wait
>>>>>     return self.poll(0)
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>> line 106, in poll
>>>>>     pid, sts = os.waitpid(self.pid, flag)
>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>
>>>>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>>>>>
>>>>> Traceback (most recent call last):
>>>>>   File "<console>", line 1, in<module>
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>>> line 287, in __call__
>>>>>     pool.join()
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>> line 342, in join
>>>>>     p.join()
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>> line 119, in join
>>>>>     res = self._popen.wait(timeout)
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>> line 117, in wait
>>>>>     return self.poll(0)
>>>>>   File
>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>> line 106, in poll
>>>>>     pid, sts = os.waitpid(self.pid, flag)
>>>>> OSError: [Errno 4] Interrupted system call
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>
>>>>> wrote:
>>>>>>
>>>>>> Not sure you want to make this change in general -- a process needs to
>>>>>> dissociate it's streams from a terminal or its parent process to properly
>>>>>> fork (otherwise you can end up with zombie processes).
>>>>>>
>>>>>> --Mike
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Lauren -
>>>>>>>
>>>>>>> I was able to replicate this - on linux I saw this message on the
>>>>>>> output:
>>>>>>>
>>>>>>>> Process PoolWorker-1:
>>>>>>>> Traceback (most recent call last):
>>>>>>>>  File
>>>>>>>> "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>> line 223, in _bootstrap
>>>>>>>>    sys.stdin.close()
>>>>>>>
>>>>>>> so I went to that file and line and made the following change (take out
>>>>>>> the sys.stdin and add 'pass')
>>>>>>>
>>>>>>>             try:
>>>>>>>                 #sys.stdin.close()
>>>>>>>                 #sys.stdin = open(os.devnull)
>>>>>>>                 pass
>>>>>>>
>>>>>>> now it works for me :)  Can you try the same fix?
>>>>>>>
>>>>>>> The issue is probably due to the fact that the pythonqt console has
>>>>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>>>>> the way it normally would.  I don't know what side effects there might
>>>>>>> be, but the output of the one-line test was shown fine in the slicer
>>>>>>> console (looked the same as running it from python in an xterm).
>>>>>>>
>>>>>>> Regarding the paths, it's possible that when you start your xterm there
>>>>>>> is another .bashrc (or similar) that changes the environment after the
>>>>>>> slicer launcher has configured it.  I haven't used easy_install myself
>>>>>>> -
>>>>>>> probably best to install with the setup.py method.
>>>>>>>
>>>>>>> Best,
>>>>>>> Steve
>>>>>>>
>>>>>>>
>>>>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>>>>>
>>>>>>>> Hi Steve,
>>>>>>>> Here are the source and egg versions. Please let me know how it goes.
>>>>>>>> Thanks!
>>>>>>>> Lauren
>>>>>>>>
>>>>>>>>
>>>>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>>>>
>>>>>>>>
>>>>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>
>>>>>>>>    wrote:
>>>>>>>>>
>>>>>>>>> Hi Lauren -
>>>>>>>>>
>>>>>>>>> Can you send the version of the joblib package you are installing
>>>>>>>>> (preferably a url that can pass to wget) and the command you use to
>>>>>>>>> install
>>>>>>>>> it?  I can try to replicate this on linux.
>>>>>>>>>
>>>>>>>>> -Steve
>>>>>>>>>
>>>>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>>>>
>>>>>>>>>> Hi slicers,
>>>>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>>>>
>>>>>>>>>> I installed the joblib package and from the slicer python shell I
>>>>>>>>>> get
>>>>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if
>>>>>>>>>> I
>>>>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>>>>
>>>>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>>>>
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>  File "<console>", line 1, in<module>
>>>>>>>>>>  File "./joblib/parallel.py", line 287, in __call__
>>>>>>>>>>    pool.join()
>>>>>>>>>>  File
>>>>>>>>>>
>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>>>>> line 342, in join
>>>>>>>>>>    p.join()
>>>>>>>>>>  File
>>>>>>>>>>
>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>>> line 119, in join
>>>>>>>>>>    res = self._popen.wait(timeout)
>>>>>>>>>>  File
>>>>>>>>>>
>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>>> line 117, in wait
>>>>>>>>>>    return self.poll(0)
>>>>>>>>>>  File
>>>>>>>>>>
>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>>> line 106, in poll
>>>>>>>>>>    pid, sts = os.waitpid(self.pid, flag)
>>>>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use
>>>>>>>>>> the
>>>>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>>>>> python from the command line).  I had a different experience with my
>>>>>>>>>> mac laptop where it did not work in or out of slicer, but it was
>>>>>>>>>> also
>>>>>>>>>> a type of forking-related error message.
>>>>>>>>>>
>>>>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>>>>
>>>>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>>>>> xterm launched by slicer, versus using the python shell within
>>>>>>>>>> slicer,
>>>>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>>>>> xterm did find the package.  A specific difference in the
>>>>>>>>>> environment
>>>>>>>>>> is that the len(sys.path)  is different when python is run in the
>>>>>>>>>> two
>>>>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>>>>
>>>>>>>>>> So overall it seems the python inside of slicer does not have the
>>>>>>>>>> same
>>>>>>>>>> multiprocessing functionality or path-related startup as the python
>>>>>>>>>> run
>>>>>>>>>> from the
>>>>>>>>>> command line.  Is this the expected behavior?
>>>>>>>>>>
>>>>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>>>>> multiprocess...
>>>>>>>>>> Lauren
>>>>>>>>>> _______________________________________________
>>>>>>>>>> slicer-devel mailing list
>>>>>>>>>> [hidden email]
>>>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>>>> To unsubscribe: send email to
>>>>>>>>>> [hidden email] with unsubscribe as
>>>>>>>>>> the
>>>>>>>>>> subject
>>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> slicer-devel mailing list
>>>>>>> [hidden email]
>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>> To unsubscribe: send email to
>>>>>>> [hidden email] with unsubscribe as the
>>>>>>> subject
>>>>>>
>>>> _______________________________________________
>>>> slicer-devel mailing list
>>>> [hidden email]
>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>> To unsubscribe: send email to
>>>> [hidden email] with unsubscribe as the
>>>> subject
>>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>
> _______________________________________________
> slicer-devel mailing list
> [hidden email]
> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
> To unsubscribe: send email to [hidden email] with unsubscribe as the subject

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Demian Wassermann-2
Hey!!! You do come back, actually I would advice more:
> import os
> os.environ["DYLD_LIBRARY_PATH"] += ':'+'<here goes that folder>'
> os.environ["PYTHONPATH"] += ':'+'<here goes that folder>'
> import slicer


but the PYTHONPATH is redundant with the sys.path.append

D

--
Demian Wassermann, PhD
[hidden email]
LMI / PNL / SPL Labs
Harvard Medical School
Brigham and Women's Hospital
1249 Boylston, Boston, MA, USA

On Mar 2, 2011, at 10:18 AM, Luca Antiga wrote:

> Hey (sometimes they come back :-) ),
> or rather
>
> import os
> os.environ["DYLD_LIBRARY_PATH"] = '<here goes that folder>'
> os.environ["PYTHONPATH"] = '<here goes that folder>'
> import slicer
>
> Cheers
>
> Luca
>
>
>
> On Mar 2, 2011, at 4:00 PM, Demian Wassermann wrote:
>
>> Hey Lauren, my bet is that there is some environment variable that should point the DYLD_LIBRARY_PATH and the PYTHONPATH to the lib folder where libMRMLPython and the other modules are.
>>
>> You can find that folder and try doing
>>
>> import sys
>> sys.path.append('<here goes that folder>')
>> import slicer
>>
>> HTH
>> D
>>
>> --
>> Demian Wassermann, PhD
>> [hidden email]
>> LMI / PNL / SPL Labs
>> Harvard Medical School
>> Brigham and Women's Hospital
>> 1249 Boylston, Boston, MA, USA
>>
>> On Mar 1, 2011, at 6:17 PM, Lauren O'Donnell wrote:
>>
>>> Hi Steve,
>>> I've almost gotten it to work on a Mac...  I can import vtk but not
>>> yet slicer, any idea what these errors mean?  Thanks!
>>> -Lauren
>>>
>>> brage:bin odonnell$ ./python
>>> Python 2.6.6 (r266:84292, Feb 15 2011, 13:35:35)
>>> [GCC 4.0.1 (Apple Inc. build 5490)] on darwin
>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>> import vtk
>>>>>> import slicer
>>> No module named libFreeSurferPython
>>> No module named libMRMLPython
>>> No module named libMRMLCLIPython
>>> No module named libMRMLLogicPython
>>> No module named libRemoteIOPython
>>> No module named libvtkTeemPython
>>> No module named PythonQt.qMRMLWidgets
>>> No module named libSlicerBaseLogicPython
>>> No module named PythonQt.qSlicerBaseQTCore
>>> No module named PythonQt.qSlicerBaseQTGUI
>>> No module named libvtkSlicerAtlasCreatorModuleLogicPython
>>> No module named libvtkSlicerCropVolumeModuleLogicPython
>>> No module named libvtkSlicerEMSegmentModuleLogicPython
>>> No module named libvtkSlicerFetchMIModuleLogicPython
>>> No module named libvtkSlicerModelMirrorModuleLogicPython
>>> No module named libvtkSlicerModelTransformModuleLogicPython
>>> No module named libvtkSlicerRDTractSelectModuleLogicPython
>>> No module named libvtkSlicerTractographyDisplayModuleLogicPython
>>> No module named libvtkSlicerTractographyModuleLogicPython
>>> No module named libvtkSlicerVolumesModuleLogicPython
>>>>>> mrml = slicer.vtkMRMLScene()
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> AttributeError: 'module' object has no attribute 'vtkMRMLScene'
>>>
>>>
>>>
>>> On Sat, Feb 26, 2011 at 5:30 PM, Steve Pieper <[hidden email]> wrote:
>>>> I got a chance to work on importing slicer libraries into a stand-alone
>>>> python shell a bit so I wrote up a wiki page on it:
>>>>
>>>> http://www.slicer.org/slicerWiki/index.php/Slicer4:Python
>>>>
>>>> it works pretty slick for me on linux.  I'll be curious to hear how it works
>>>> for Lauren on mac (if it works - no doubt some platform-specific tweaking
>>>> will be needed).  Also, although I'm sure it's possible to do, I pity the
>>>> person who works on getting this to run correctly on windows.
>>>>
>>>> -Steve
>>>>
>>>> On 02/23/2011 11:16 AM, Steve Pieper wrote:
>>>>>
>>>>> Hi Folks -
>>>>>
>>>>> As a follow up to this thread: Lauren, Demian, and other of us met
>>>>> yesterday and decided that rather than make the embedded console support
>>>>> the parallelism constructs we're going to try importing the slicer base
>>>>> and modules logics into a standard python process - it should work and
>>>>> avoid these issues plus make it much easier to use slicer code together
>>>>> with packages like matplotlib, ipython, etc.
>>>>>
>>>>> -Steve
>>>>>
>>>>> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>>>>>>
>>>>>> Thank you Steve for your suggestion to test what is going on with the
>>>>>> python parallelization.  I commented out those lines and now, I can
>>>>>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>>>>>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>>>>>> error message displayed in the python console (see below).
>>>>>>
>>>>>> This is on Mac OS X 10.6.6.
>>>>>>
>>>>>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>
>>>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>>>
>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>
>>>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>>>
>>>>>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>  File "<console>", line 1, in<module>
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>>>> line 287, in __call__
>>>>>>    pool.join()
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>> line 342, in join
>>>>>>    p.join()
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>> line 119, in join
>>>>>>    res = self._popen.wait(timeout)
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>> line 117, in wait
>>>>>>    return self.poll(0)
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>> line 106, in poll
>>>>>>    pid, sts = os.waitpid(self.pid, flag)
>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>
>>>>>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>>  File "<console>", line 1, in<module>
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>>>> line 287, in __call__
>>>>>>    pool.join()
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>> line 342, in join
>>>>>>    p.join()
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>> line 119, in join
>>>>>>    res = self._popen.wait(timeout)
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>> line 117, in wait
>>>>>>    return self.poll(0)
>>>>>>  File
>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>> line 106, in poll
>>>>>>    pid, sts = os.waitpid(self.pid, flag)
>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>> Not sure you want to make this change in general -- a process needs to
>>>>>>> dissociate it's streams from a terminal or its parent process to properly
>>>>>>> fork (otherwise you can end up with zombie processes).
>>>>>>>
>>>>>>> --Mike
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Lauren -
>>>>>>>>
>>>>>>>> I was able to replicate this - on linux I saw this message on the
>>>>>>>> output:
>>>>>>>>
>>>>>>>>> Process PoolWorker-1:
>>>>>>>>> Traceback (most recent call last):
>>>>>>>>> File
>>>>>>>>> "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>> line 223, in _bootstrap
>>>>>>>>>   sys.stdin.close()
>>>>>>>>
>>>>>>>> so I went to that file and line and made the following change (take out
>>>>>>>> the sys.stdin and add 'pass')
>>>>>>>>
>>>>>>>>            try:
>>>>>>>>                #sys.stdin.close()
>>>>>>>>                #sys.stdin = open(os.devnull)
>>>>>>>>                pass
>>>>>>>>
>>>>>>>> now it works for me :)  Can you try the same fix?
>>>>>>>>
>>>>>>>> The issue is probably due to the fact that the pythonqt console has
>>>>>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>>>>>> the way it normally would.  I don't know what side effects there might
>>>>>>>> be, but the output of the one-line test was shown fine in the slicer
>>>>>>>> console (looked the same as running it from python in an xterm).
>>>>>>>>
>>>>>>>> Regarding the paths, it's possible that when you start your xterm there
>>>>>>>> is another .bashrc (or similar) that changes the environment after the
>>>>>>>> slicer launcher has configured it.  I haven't used easy_install myself
>>>>>>>> -
>>>>>>>> probably best to install with the setup.py method.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Steve
>>>>>>>>
>>>>>>>>
>>>>>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>>>>>>
>>>>>>>>> Hi Steve,
>>>>>>>>> Here are the source and egg versions. Please let me know how it goes.
>>>>>>>>> Thanks!
>>>>>>>>> Lauren
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>
>>>>>>>>>   wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Lauren -
>>>>>>>>>>
>>>>>>>>>> Can you send the version of the joblib package you are installing
>>>>>>>>>> (preferably a url that can pass to wget) and the command you use to
>>>>>>>>>> install
>>>>>>>>>> it?  I can try to replicate this on linux.
>>>>>>>>>>
>>>>>>>>>> -Steve
>>>>>>>>>>
>>>>>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi slicers,
>>>>>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>>>>>
>>>>>>>>>>> I installed the joblib package and from the slicer python shell I
>>>>>>>>>>> get
>>>>>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if
>>>>>>>>>>> I
>>>>>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>>>>>
>>>>>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>>>>>
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> File "<console>", line 1, in<module>
>>>>>>>>>>> File "./joblib/parallel.py", line 287, in __call__
>>>>>>>>>>>   pool.join()
>>>>>>>>>>> File
>>>>>>>>>>>
>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>>>>>> line 342, in join
>>>>>>>>>>>   p.join()
>>>>>>>>>>> File
>>>>>>>>>>>
>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>>>> line 119, in join
>>>>>>>>>>>   res = self._popen.wait(timeout)
>>>>>>>>>>> File
>>>>>>>>>>>
>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>>>> line 117, in wait
>>>>>>>>>>>   return self.poll(0)
>>>>>>>>>>> File
>>>>>>>>>>>
>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>>>> line 106, in poll
>>>>>>>>>>>   pid, sts = os.waitpid(self.pid, flag)
>>>>>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use
>>>>>>>>>>> the
>>>>>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>>>>>> python from the command line).  I had a different experience with my
>>>>>>>>>>> mac laptop where it did not work in or out of slicer, but it was
>>>>>>>>>>> also
>>>>>>>>>>> a type of forking-related error message.
>>>>>>>>>>>
>>>>>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>>>>>
>>>>>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>>>>>> xterm launched by slicer, versus using the python shell within
>>>>>>>>>>> slicer,
>>>>>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>>>>>> xterm did find the package.  A specific difference in the
>>>>>>>>>>> environment
>>>>>>>>>>> is that the len(sys.path)  is different when python is run in the
>>>>>>>>>>> two
>>>>>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>>>>>
>>>>>>>>>>> So overall it seems the python inside of slicer does not have the
>>>>>>>>>>> same
>>>>>>>>>>> multiprocessing functionality or path-related startup as the python
>>>>>>>>>>> run
>>>>>>>>>>> from the
>>>>>>>>>>> command line.  Is this the expected behavior?
>>>>>>>>>>>
>>>>>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>>>>>> multiprocess...
>>>>>>>>>>> Lauren
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> slicer-devel mailing list
>>>>>>>>>>> [hidden email]
>>>>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>>>>> To unsubscribe: send email to
>>>>>>>>>>> [hidden email] with unsubscribe as
>>>>>>>>>>> the
>>>>>>>>>>> subject
>>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> slicer-devel mailing list
>>>>>>>> [hidden email]
>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>> To unsubscribe: send email to
>>>>>>>> [hidden email] with unsubscribe as the
>>>>>>>> subject
>>>>>>>
>>>>> _______________________________________________
>>>>> slicer-devel mailing list
>>>>> [hidden email]
>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>> To unsubscribe: send email to
>>>>> [hidden email] with unsubscribe as the
>>>>> subject
>>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>>
>> _______________________________________________
>> slicer-devel mailing list
>> [hidden email]
>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject
Reply | Threaded
Open this post in threaded view
|

Re: parallel processing in python in slicer? plus python path weirdness.

Luca Antiga
Hey Demian!
 you're right (what's new?)! Just make sure that the two keys do actually exist
in the dictionary prior to appending stuff.
Also, you can use os.path.pathsep instead of ':' to make it cross-platform.

Luca


On Mar 2, 2011, at 7:43 PM, Demian Wassermann wrote:

> Hey!!! You do come back, actually I would advice more:
>> import os
>> os.environ["DYLD_LIBRARY_PATH"] += ':'+'<here goes that folder>'
>> os.environ["PYTHONPATH"] += ':'+'<here goes that folder>'
>> import slicer
>
>
> but the PYTHONPATH is redundant with the sys.path.append
>
> D
>
> --
> Demian Wassermann, PhD
> [hidden email]
> LMI / PNL / SPL Labs
> Harvard Medical School
> Brigham and Women's Hospital
> 1249 Boylston, Boston, MA, USA
>
> On Mar 2, 2011, at 10:18 AM, Luca Antiga wrote:
>
>> Hey (sometimes they come back :-) ),
>> or rather
>>
>> import os
>> os.environ["DYLD_LIBRARY_PATH"] = '<here goes that folder>'
>> os.environ["PYTHONPATH"] = '<here goes that folder>'
>> import slicer
>>
>> Cheers
>>
>> Luca
>>
>>
>>
>> On Mar 2, 2011, at 4:00 PM, Demian Wassermann wrote:
>>
>>> Hey Lauren, my bet is that there is some environment variable that should point the DYLD_LIBRARY_PATH and the PYTHONPATH to the lib folder where libMRMLPython and the other modules are.
>>>
>>> You can find that folder and try doing
>>>
>>> import sys
>>> sys.path.append('<here goes that folder>')
>>> import slicer
>>>
>>> HTH
>>> D
>>>
>>> --
>>> Demian Wassermann, PhD
>>> [hidden email]
>>> LMI / PNL / SPL Labs
>>> Harvard Medical School
>>> Brigham and Women's Hospital
>>> 1249 Boylston, Boston, MA, USA
>>>
>>> On Mar 1, 2011, at 6:17 PM, Lauren O'Donnell wrote:
>>>
>>>> Hi Steve,
>>>> I've almost gotten it to work on a Mac...  I can import vtk but not
>>>> yet slicer, any idea what these errors mean?  Thanks!
>>>> -Lauren
>>>>
>>>> brage:bin odonnell$ ./python
>>>> Python 2.6.6 (r266:84292, Feb 15 2011, 13:35:35)
>>>> [GCC 4.0.1 (Apple Inc. build 5490)] on darwin
>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>> import vtk
>>>>>>> import slicer
>>>> No module named libFreeSurferPython
>>>> No module named libMRMLPython
>>>> No module named libMRMLCLIPython
>>>> No module named libMRMLLogicPython
>>>> No module named libRemoteIOPython
>>>> No module named libvtkTeemPython
>>>> No module named PythonQt.qMRMLWidgets
>>>> No module named libSlicerBaseLogicPython
>>>> No module named PythonQt.qSlicerBaseQTCore
>>>> No module named PythonQt.qSlicerBaseQTGUI
>>>> No module named libvtkSlicerAtlasCreatorModuleLogicPython
>>>> No module named libvtkSlicerCropVolumeModuleLogicPython
>>>> No module named libvtkSlicerEMSegmentModuleLogicPython
>>>> No module named libvtkSlicerFetchMIModuleLogicPython
>>>> No module named libvtkSlicerModelMirrorModuleLogicPython
>>>> No module named libvtkSlicerModelTransformModuleLogicPython
>>>> No module named libvtkSlicerRDTractSelectModuleLogicPython
>>>> No module named libvtkSlicerTractographyDisplayModuleLogicPython
>>>> No module named libvtkSlicerTractographyModuleLogicPython
>>>> No module named libvtkSlicerVolumesModuleLogicPython
>>>>>>> mrml = slicer.vtkMRMLScene()
>>>> Traceback (most recent call last):
>>>> File "<stdin>", line 1, in <module>
>>>> AttributeError: 'module' object has no attribute 'vtkMRMLScene'
>>>>
>>>>
>>>>
>>>> On Sat, Feb 26, 2011 at 5:30 PM, Steve Pieper <[hidden email]> wrote:
>>>>> I got a chance to work on importing slicer libraries into a stand-alone
>>>>> python shell a bit so I wrote up a wiki page on it:
>>>>>
>>>>> http://www.slicer.org/slicerWiki/index.php/Slicer4:Python
>>>>>
>>>>> it works pretty slick for me on linux.  I'll be curious to hear how it works
>>>>> for Lauren on mac (if it works - no doubt some platform-specific tweaking
>>>>> will be needed).  Also, although I'm sure it's possible to do, I pity the
>>>>> person who works on getting this to run correctly on windows.
>>>>>
>>>>> -Steve
>>>>>
>>>>> On 02/23/2011 11:16 AM, Steve Pieper wrote:
>>>>>>
>>>>>> Hi Folks -
>>>>>>
>>>>>> As a follow up to this thread: Lauren, Demian, and other of us met
>>>>>> yesterday and decided that rather than make the embedded console support
>>>>>> the parallelism constructs we're going to try importing the slicer base
>>>>>> and modules logics into a standard python process - it should work and
>>>>>> avoid these issues plus make it much easier to use slicer code together
>>>>>> with packages like matplotlib, ipython, etc.
>>>>>>
>>>>>> -Steve
>>>>>>
>>>>>> On 02/22/2011 01:33 PM, Lauren O'Donnell wrote:
>>>>>>>
>>>>>>> Thank you Steve for your suggestion to test what is going on with the
>>>>>>> python parallelization.  I commented out those lines and now, I can
>>>>>>> run 2 jobs, however not 3 or 4.  Previously I could only run 1 job,
>>>>>>> otherwise Slicer would hang. Now Slicer does not hang and there is an
>>>>>>> error message displayed in the python console (see below).
>>>>>>>
>>>>>>> This is on Mac OS X 10.6.6.
>>>>>>>
>>>>>>>>>> Parallel(n_jobs=1)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>
>>>>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>>>>
>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>
>>>>>>> [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]
>>>>>>>>>>
>>>>>>>>>> Parallel(n_jobs=4)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>> File "<console>", line 1, in<module>
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>>>>> line 287, in __call__
>>>>>>>   pool.join()
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>> line 342, in join
>>>>>>>   p.join()
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>> line 119, in join
>>>>>>>   res = self._popen.wait(timeout)
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>> line 117, in wait
>>>>>>>   return self.poll(0)
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>> line 106, in poll
>>>>>>>   pid, sts = os.waitpid(self.pid, flag)
>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>>
>>>>>>>>>> Parallel(n_jobs=3)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>> File "<console>", line 1, in<module>
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/site-packages/joblib/parallel.py",
>>>>>>> line 287, in __call__
>>>>>>>   pool.join()
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>> line 342, in join
>>>>>>>   p.join()
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>> line 119, in join
>>>>>>>   res = self._popen.wait(timeout)
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>> line 117, in wait
>>>>>>>   return self.poll(0)
>>>>>>> File
>>>>>>> "/Users/lauren/Software/Slicer4Dec2010/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>> line 106, in poll
>>>>>>>   pid, sts = os.waitpid(self.pid, flag)
>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 14, 2011 at 8:55 PM, Michael Halle<[hidden email]>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Not sure you want to make this change in general -- a process needs to
>>>>>>>> dissociate it's streams from a terminal or its parent process to properly
>>>>>>>> fork (otherwise you can end up with zombie processes).
>>>>>>>>
>>>>>>>> --Mike
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Feb 14, 2011, at 11:16 AM, Steve Pieper<[hidden email]>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Lauren -
>>>>>>>>>
>>>>>>>>> I was able to replicate this - on linux I saw this message on the
>>>>>>>>> output:
>>>>>>>>>
>>>>>>>>>> Process PoolWorker-1:
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> File
>>>>>>>>>> "/home/pieper/slicer4/latest/Slicer4-superbuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>>> line 223, in _bootstrap
>>>>>>>>>>  sys.stdin.close()
>>>>>>>>>
>>>>>>>>> so I went to that file and line and made the following change (take out
>>>>>>>>> the sys.stdin and add 'pass')
>>>>>>>>>
>>>>>>>>>           try:
>>>>>>>>>               #sys.stdin.close()
>>>>>>>>>               #sys.stdin = open(os.devnull)
>>>>>>>>>               pass
>>>>>>>>>
>>>>>>>>> now it works for me :)  Can you try the same fix?
>>>>>>>>>
>>>>>>>>> The issue is probably due to the fact that the pythonqt console has
>>>>>>>>> already taken over the stdin/stdout/stderr so joblib can't work with it
>>>>>>>>> the way it normally would.  I don't know what side effects there might
>>>>>>>>> be, but the output of the one-line test was shown fine in the slicer
>>>>>>>>> console (looked the same as running it from python in an xterm).
>>>>>>>>>
>>>>>>>>> Regarding the paths, it's possible that when you start your xterm there
>>>>>>>>> is another .bashrc (or similar) that changes the environment after the
>>>>>>>>> slicer launcher has configured it.  I haven't used easy_install myself
>>>>>>>>> -
>>>>>>>>> probably best to install with the setup.py method.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Steve
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 02/11/2011 11:33 AM, Lauren O'Donnell wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Steve,
>>>>>>>>>> Here are the source and egg versions. Please let me know how it goes.
>>>>>>>>>> Thanks!
>>>>>>>>>> Lauren
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://pypi.python.org/packages/2.6/j/joblib/joblib-0.4.6.dev-py2.6.egg
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> http://pypi.python.org/packages/source/j/joblib/joblib-0.4.6.dev.tar.gz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 10, 2011 at 6:43 PM, Steve Pieper<[hidden email]>
>>>>>>>>>>  wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Lauren -
>>>>>>>>>>>
>>>>>>>>>>> Can you send the version of the joblib package you are installing
>>>>>>>>>>> (preferably a url that can pass to wget) and the command you use to
>>>>>>>>>>> install
>>>>>>>>>>> it?  I can try to replicate this on linux.
>>>>>>>>>>>
>>>>>>>>>>> -Steve
>>>>>>>>>>>
>>>>>>>>>>> On 02/10/2011 05:00 PM, Lauren O'Donnell wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi slicers,
>>>>>>>>>>>> I am trying to find a way to do parallel processing in the slicer
>>>>>>>>>>>> python environment.  It seems this is somehow disabled by slicer?
>>>>>>>>>>>>
>>>>>>>>>>>> I installed the joblib package and from the slicer python shell I
>>>>>>>>>>>> get
>>>>>>>>>>>> the following error in Mac OS X 10.5.8.  This test code runs fine if
>>>>>>>>>>>> I
>>>>>>>>>>>> only use 1 job, however 2 jobs fail.
>>>>>>>>>>>>
>>>>>>>>>>>>>>> Parallel(n_jobs=2)(delayed(sqrt)(i**2) for i in range(10))
>>>>>>>>>>>>
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>> File "<console>", line 1, in<module>
>>>>>>>>>>>> File "./joblib/parallel.py", line 287, in __call__
>>>>>>>>>>>>  pool.join()
>>>>>>>>>>>> File
>>>>>>>>>>>>
>>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/pool.py",
>>>>>>>>>>>> line 342, in join
>>>>>>>>>>>>  p.join()
>>>>>>>>>>>> File
>>>>>>>>>>>>
>>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/process.py",
>>>>>>>>>>>> line 119, in join
>>>>>>>>>>>>  res = self._popen.wait(timeout)
>>>>>>>>>>>> File
>>>>>>>>>>>>
>>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>>>>> line 117, in wait
>>>>>>>>>>>>  return self.poll(0)
>>>>>>>>>>>> File
>>>>>>>>>>>>
>>>>>>>>>>>> "/Users/odonnell/Software/SlicerVersions/FOUR/Slicer4-SuperBuild/python-build/lib/python2.6/multiprocessing/forking.py",
>>>>>>>>>>>> line 106, in poll
>>>>>>>>>>>>  pid, sts = os.waitpid(self.pid, flag)
>>>>>>>>>>>> OSError: [Errno 4] Interrupted system call
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mac OS X 10.5.8 the above code actually runs correctly if I use
>>>>>>>>>>>> the
>>>>>>>>>>>> same slicer python build but I run it outside of slicer itself (run
>>>>>>>>>>>> python from the command line).  I had a different experience with my
>>>>>>>>>>>> mac laptop where it did not work in or out of slicer, but it was
>>>>>>>>>>>> also
>>>>>>>>>>>> a type of forking-related error message.
>>>>>>>>>>>>
>>>>>>>>>>>> This is with Slicer4, a recent build from the last week or so.
>>>>>>>>>>>>
>>>>>>>>>>>> I have one more question/issue. Running the Slicer python from an
>>>>>>>>>>>> xterm launched by slicer, versus using the python shell within
>>>>>>>>>>>> slicer,
>>>>>>>>>>>> gives different python package loading behavior. Specifically, the
>>>>>>>>>>>> within-slicer python shell did not find a package that was installed
>>>>>>>>>>>> using easy_install from an egg, while the same python run from the
>>>>>>>>>>>> xterm did find the package.  A specific difference in the
>>>>>>>>>>>> environment
>>>>>>>>>>>> is that the len(sys.path)  is different when python is run in the
>>>>>>>>>>>> two
>>>>>>>>>>>> ways, however the variable PYTHONPATH is the same.
>>>>>>>>>>>>
>>>>>>>>>>>> So overall it seems the python inside of slicer does not have the
>>>>>>>>>>>> same
>>>>>>>>>>>> multiprocessing functionality or path-related startup as the python
>>>>>>>>>>>> run
>>>>>>>>>>>> from the
>>>>>>>>>>>> command line.  Is this the expected behavior?
>>>>>>>>>>>>
>>>>>>>>>>>> Please advise, thanks!  It would be really helpful to be able to
>>>>>>>>>>>> multiprocess...
>>>>>>>>>>>> Lauren
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> slicer-devel mailing list
>>>>>>>>>>>> [hidden email]
>>>>>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>>>>>> To unsubscribe: send email to
>>>>>>>>>>>> [hidden email] with unsubscribe as
>>>>>>>>>>>> the
>>>>>>>>>>>> subject
>>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> slicer-devel mailing list
>>>>>>>>> [hidden email]
>>>>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>>>>> To unsubscribe: send email to
>>>>>>>>> [hidden email] with unsubscribe as the
>>>>>>>>> subject
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> slicer-devel mailing list
>>>>>> [hidden email]
>>>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>>>> To unsubscribe: send email to
>>>>>> [hidden email] with unsubscribe as the
>>>>>> subject
>>>>>
>>>> _______________________________________________
>>>> slicer-devel mailing list
>>>> [hidden email]
>>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>>>
>>> _______________________________________________
>>> slicer-devel mailing list
>>> [hidden email]
>>> http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
>>> To unsubscribe: send email to [hidden email] with unsubscribe as the subject
>>
>

_______________________________________________
slicer-devel mailing list
[hidden email]
http://massmail.spl.harvard.edu/mailman/listinfo/slicer-devel
To unsubscribe: send email to [hidden email] with unsubscribe as the subject