File Conveyor: A thing that does stuff to things in places

A couple weeks ago I had the opportunity (and pleasure) to mess around with File Conveyor. File Conveyor is the bachelor's thesis project by well know Drupal contributor Wim Leers. It can do a lot of different things, so it's a bit hard to summarize, but essentially:

  • It monitors a directory (remote or local) for new or changed files.
  • It sends those files through various processesors which may rename, resize, or otherwise act on the file.
  • It then sends the finished files to various destinations which could be local, s3, ftp, and many others.

The primary motivation for the project was to be able to feed Content Delivery Networks (CDN) with files from your site and resize images, compress Javascript, etc on the way. You can see the docs about Drupal integration in the CDN project. The code is written in Python and makes use of several other open source libraries, in particular Django Storages (does not require Django) While File conveyor supports a lot of storage types (s3, ftp, etc) through Storages, there was no way to send files via SFTP. One of Acquia's clients urgently needed sftp / ssh support to send files up to Akamai.

Being the resident Python expert at Acquia (which isn't saying much), I volunteered to help out. The code was fun to write, and it seems to work well, despite being a bit rough in terms of missing tests, etc. If you need SFTP support for Django storages or for FileConveyor, check out my my fork.

Tags: 

Comments

Thanks!

Thanks for writing about File Conveyor :)

I hope to merge in your code soon!

Assembling is nicely end and

jgurma's picture

Assembling is nicely end and it contains zesty echt things for me. I am rapturous to ascertain your awing way of essay the set. Now it evince promiscuous for me to hap and essay the secern. Thanks for condiment the earthborn.

free ecards

 

The Transporter module 'transporters.transporter_sftp' could not

Rene's picture

When I try to use your sftp transporter with the sample config.xml that you provided I get the following error:

I receive the following error:

ERROR    - The Transporter module 'transporters.transporter_sftp' could not be found.

But it is installed as you can see below.

[root@nyc-h837x9 code]# ls -al transporters/
total 116
drwxr-xr-x 2 root root 4096 Apr 13 13:50 .
drwxr-xr-x 5 root root 4096 Apr 13 13:52 ..
-rw-r--r-- 1 root root    0 Mar 23 03:06 __init__.py
-rw-r--r-- 1 root root  153 Apr 13 13:49 __init__.pyc
-rw-r--r-- 1 root root 1822 Mar 23 03:06 transporter_cf.py
-rw-r--r-- 1 root root 1224 Mar 23 03:06 transporter_ftp.py
-rw-r--r-- 1 root root 1428 Mar 23 03:06 transporter_mosso.py
-rw-r--r-- 1 root root 5666 Mar 23 03:06 transporter.py
-rw-r--r-- 1 root root 5962 Apr 13 13:49 transporter.pyc
-rw-r--r-- 1 root root 1545 Mar 23 03:06 transporter_s3.py
-rw-r--r-- 1 root root 3710 Mar 23 03:06 transporter_sample.py
-rw-r--r-- 1 root root 1569 Mar 23 03:06 transporter_sftp.py
-rw-r--r-- 1 root root 1644 Apr 13 13:49 transporter_sftp.pyc
-rw-r--r-- 1 root root  857 Mar 23 03:06 transporter_symlink_or_copy.py

Any suggestions would be much appriciated.

Network Camera

Network Camera's picture

You made some good points there. I did a search on the topic and found most people will agree with your views.
<a href="http://www.wansview.com">Network Camera</a>

IP Camera

IP Camera's picture

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts into the stuff Network Camera you post!! Thumbs up!!!

This is some helpful material

larserim's picture

This is some helpful material. It took me a while to locate this web site but it was worth the time. I noticed this page was hidden in bing and not the number one spot. This web page has a ton of first-class material and it does not deserve to be burried in the search engines like that. By the way I am going to save this site to my list of favorites. @ Spotting Between Periods

 

Pressure Washers

 

Best Kitchen Faucets

 

Kitchen Faucet Reviews

 

zhpjaseqf

humanqsepk's picture

comment_body[und][0][value]

Your website is so cool. I’m

looton's picture

Your website is so cool. I’m impressed by the details that you have on this blog. It reveals how nicely you perceive this subject. Bookmarked this website page, will come back for extra articles. www.getmyloan.net