<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Parisista &#187; jets3t</title>
	<atom:link href="http://www.khaitan.org/blog/tag/jets3t/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.khaitan.org/blog</link>
	<description>2. Etymology: Sanskrit. Supplementary information usually at the end of a writing</description>
	<lastBuildDate>Thu, 09 Feb 2012 15:40:07 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.8.4</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Using JetS3t to upload larger number of files to S3</title>
		<link>http://www.khaitan.org/blog/2008/02/using-jets3t-to-upload-larger-number-of-files-to-s3/</link>
		<comments>http://www.khaitan.org/blog/2008/02/using-jets3t-to-upload-larger-number-of-files-to-s3/#comments</comments>
		<pubDate>Mon, 04 Feb 2008 04:12:42 +0000</pubDate>
		<dc:creator>indus</dc:creator>
				<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[S3]]></category>
		<category><![CDATA[jets3t]]></category>
		<category><![CDATA[aws]]></category>

		<guid isPermaLink="false">http://www.khaitan.org/blog/?p=207</guid>
		<description><![CDATA[I was looking for a tool to upload large number of files to S3. While I have been a great fan of the bash tools for browsing and accessing s3 objects and buckets and a managing a limited number of files &#8212; I could not find an easy way of uploading a large number of [...]]]></description>
			<content:encoded><![CDATA[<p>I was looking for a tool to upload large number of files to S3. While I have been a great fan of the bash tools for browsing and accessing s3 objects and buckets and a managing a limited number of files &#8212; I could not find an easy way of uploading a large number of files (the first batch being around 800K).</p>
<p>Then I downloaded <a href="http://jets3t.s3.amazonaws.com/index.html" title="JetS3t">JetS3t</a>. It has a nice gui called Cockpit for managing the files on S3. The GUI is pretty neat. However, for simple upload/download <a href="https://addons.mozilla.org/en-US/firefox/addon/3247">S3 organizer</a>, a simple Firefox plugin does the job. If you need to extensively manage your files then JetS3t&#8217;s cockpit is the way-to-go.</p>
<p>For uploading a large number of files, I was looking for something which is multi-threaded and configurable. JetS3t S3 suite has a &#8220;synchronize&#8221; application which is meant to synchronize files between a local PC and S3. JetS3t allows you to configure the number of threads and connections to the S3 service. Without reinventing the wheel, I got what I wanted. However, one additional thing I needed was the ability to delete the local files once the upload was complete. On tinkering with the java src, I modded the Synchronize.java and added the following code fragments:<br />
<code><br />
public void uploadLocalDirectoryToS3(FileComparerResults disrepancyResults, Map filesMap,Map s3ObjectsMap, S3Bucket bucket, String rootObjectPath, String aclString) throws Exception  {<br />
...<br />
List filesToDelete = new ArrayList();<br />
...<br />
if (file.isDirectory() != true){<br />
  filesToDelete.add(file.getPath());<br />
}<br />
...</code></p>
<p><code>// delete files once objects are S3d<br />
for (Iterator ite = filesToDelete.iterator(); ite.hasNext();){<br />
 String fName = (String)ite.next();<br />
 File f = new File(fName);<br />
f.delete();<br />
}<br />
}<br />
</code></p>
]]></content:encoded>
			<wfw:commentRss>http://www.khaitan.org/blog/2008/02/using-jets3t-to-upload-larger-number-of-files-to-s3/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
	</channel>
</rss>

