Transfer Data With Webhdfs

Write the following script transferFromCluster1ToCluster2.sh

for i in $(cat $1);
do echo $i;
curl  -k --location-trusted -u 'login:passwd' -X GET "https://web_hdfs_of_source_cluster:8443/gateway/default/webhdfs/v1${i}?op=OPEN" | curl -v -k -T - --location-trusted -u 'login:passwd' -X PUT "https://web_hdfs_of_target_cluster:8443/gateway/default/webhdfs/v1${i}?op=CREATE&overwrite=true" ;
done;

Launch your script with

./transferFromCluster1ToCluster2.sh ./[file_which_contains_files_to_transfer.txt]
comments powered by Disqus

You May Also Enjoy

Hdfs Admin Troubleshooting Corrupted Missing Blocks hortonworks hadoop

Connect To Kerberized Thrift Server Dbvis hortonworks hadoop

Dispatch Topic Partitions On New Brokers With Kafka hortonworks hadoop

Hdfs Fix Under Replicated Block hortonworks hadoop tips