I was looking for a tool to upload large number of files to S3. While I have been a great fan of the bash tools for browsing and accessing s3 objects and buckets and a managing a limited number of files — I could not find an easy way of uploading a large number of files (the first batch being around 800K).<\/p>\n
Then I downloaded JetS3t<\/a>. It has a nice gui called Cockpit for managing the files on S3. The GUI is pretty neat. However, for simple upload\/download S3 organizer<\/a>, a simple Firefox plugin does the job. If you need to extensively manage your files then JetS3t’s cockpit is the way-to-go.<\/p>\n For uploading a large number of files, I was looking for something which is multi-threaded and configurable. JetS3t S3 suite has a “synchronize” application which is meant to synchronize files between a local PC and S3. JetS3t allows you to configure the number of threads and connections to the S3 service. Without reinventing the wheel, I got what\u00a0I wanted. However, one additional thing I needed was the ability to delete the local files once the upload was complete. On tinkering with the java src, I modded the Synchronize.java and added the following code fragments: I was looking for a tool to upload large number of files to S3. While I have been a great fan of the bash tools for browsing and accessing s3 objects and buckets and a managing a limited number of files — I could not find an easy way of uploading a large number of […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[51,96,53],"tags":[123,326,312],"_links":{"self":[{"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/posts\/207"}],"collection":[{"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/comments?post=207"}],"version-history":[{"count":0,"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/posts\/207\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/media?parent=207"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/categories?post=207"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.khaitan.org\/blog\/wp-json\/wp\/v2\/tags?post=207"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n
\npublic void uploadLocalDirectoryToS3(FileComparerResults disrepancyResults, Map filesMap,Map s3ObjectsMap, S3Bucket bucket, String rootObjectPath, String aclString) throws Exception\u00a0 {
\n...
\nList filesToDelete = new ArrayList();
\n...
\nif (file.isDirectory() != true){
\n\u00a0 filesToDelete.add(file.getPath());
\n}
\n...<\/code><\/p>\n\/\/ delete files once objects are S3d
\nfor (Iterator ite = filesToDelete.iterator(); ite.hasNext();){
\n\u00a0String fName = (String)ite.next();
\n\u00a0File f = new File(fName);
\nf.delete();
\n}
\n}
\n<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"