File Conveyor: A thing that does stuff to things in places

Submitted by Jacob Singh on

A couple weeks ago I had the opportunity (and pleasure) to mess around with File Conveyor. File Conveyor is the bachelor's thesis project by well know Drupal contributor Wim Leers. It can do a lot of different things, so it's a bit hard to summarize, but essentially:

  • It monitors a directory (remote or local) for new or changed files.
  • It sends those files through various processesors which may rename, resize, or otherwise act on the file.
  • It then sends the finished files to various destinations which could be local, s3, ftp, and many others.

The primary motivation for the project was to be able to feed Content Delivery Networks (CDN) with files from your site and resize images, compress Javascript, etc on the way. You can see the docs about Drupal integration in the CDN project. The code is written in Python and makes use of several other open source libraries, in particular Django Storages (does not require Django) While File conveyor supports a lot of storage types (s3, ftp, etc) through Storages, there was no way to send files via SFTP. One of Acquia's clients urgently needed sftp / ssh support to send files up to Akamai.

Being the resident Python expert at Acquia (which isn't saying much), I volunteered to help out. The code was fun to write, and it seems to work well, despite being a bit rough in terms of missing tests, etc. If you need SFTP support for Django storages or for FileConveyor, check out my my fork.

Tags: 

Comments

Thanks!

Thanks for writing about File Conveyor :)

I hope to merge in your code soon!

Assembling is nicely end and

jgurma's picture

Submitted by jgurma (not verified) on

Assembling is nicely end and it contains zesty echt things for me. I am rapturous to ascertain your awing way of essay the set. Now it evince promiscuous for me to hap and essay the secern. Thanks for condiment the earthborn.

free ecards

 

The Transporter module 'transporters.transporter_sftp' could not

Rene's picture

Submitted by Rene (not verified) on

When I try to use your sftp transporter with the sample config.xml that you provided I get the following error:

I receive the following error:

ERROR    - The Transporter module 'transporters.transporter_sftp' could not be found.

But it is installed as you can see below.

[[email protected] code]# ls -al transporters/
total 116
drwxr-xr-x 2 root root 4096 Apr 13 13:50 .
drwxr-xr-x 5 root root 4096 Apr 13 13:52 ..
-rw-r--r-- 1 root root    0 Mar 23 03:06 __init__.py
-rw-r--r-- 1 root root  153 Apr 13 13:49 __init__.pyc
-rw-r--r-- 1 root root 1822 Mar 23 03:06 transporter_cf.py
-rw-r--r-- 1 root root 1224 Mar 23 03:06 transporter_ftp.py
-rw-r--r-- 1 root root 1428 Mar 23 03:06 transporter_mosso.py
-rw-r--r-- 1 root root 5666 Mar 23 03:06 transporter.py
-rw-r--r-- 1 root root 5962 Apr 13 13:49 transporter.pyc
-rw-r--r-- 1 root root 1545 Mar 23 03:06 transporter_s3.py
-rw-r--r-- 1 root root 3710 Mar 23 03:06 transporter_sample.py
-rw-r--r-- 1 root root 1569 Mar 23 03:06 transporter_sftp.py
-rw-r--r-- 1 root root 1644 Apr 13 13:49 transporter_sftp.pyc
-rw-r--r-- 1 root root  857 Mar 23 03:06 transporter_symlink_or_copy.py

Any suggestions would be much appriciated.

Network Camera

Network Camera's picture

Submitted by Network Camera (not verified) on

You made some good points there. I did a search on the topic and found most people will agree with your views.
<a href="http://www.wansview.com">Network Camera</a>

IP Camera

IP Camera's picture

Submitted by IP Camera (not verified) on

Valuable information and excellent design you got here! I would like to thank you for sharing your thoughts into the stuff Network Camera you post!! Thumbs up!!!

This is some helpful material

larserim's picture

Submitted by larserim (not verified) on

This is some helpful material. It took me a while to locate this web site but it was worth the time. I noticed this page was hidden in bing and not the number one spot. This web page has a ton of first-class material and it does not deserve to be burried in the search engines like that. By the way I am going to save this site to my list of favorites. @ Spotting Between Periods

 

@ Pressure Washers

 

@ Best Kitchen Faucets

 

@ Kitchen Faucet Reviews

 

zhpjaseqf

humanqsepk's picture

Submitted by humanqsepk (not verified) on

comment_body[und][0][value]

Your website is so cool. I’m

looton's picture

Submitted by looton (not verified) on

Your website is so cool. I’m impressed by the details that you have on this blog. It reveals how nicely you perceive this subject. Bookmarked this website page, will come back for extra articles. www.getmyloan.net