How to migrate data from a bucket to another cross region

Technical additional documentation

This guide will help you to migrate data from a bucket in one region to a  bucket in another region using “obsutil” Flexible Engine command line tool.

This tool can used from a Linux ECS hosted in any Flexible Engine or from a Windows workstation connected to internet.

Installation on Linux

Prerequisites:

  • A Linux ECS is instantiated in Flexible Engine in eu-west-0 or eu-west-1 with outgoing internet connectivity (EIP or NAT Gateway)
  • Flexible Engine user has OBS access rights (user belong to default “admin’ group” or belong to a group with OBS service IAM policy allowing read/write access

Steps:

  • Connect to ECS using SSH or console “Remote Login”
  • Download “obsutil” tools:
wget https://oss.eu-west-0.prod-cloud-ocb.orange-business.com/obsutil/obsutil_ocb_linux_amd64_5.3.4.tar.gz
  • Extract archive:
tar -xzvf obsutil_ocb_linux_amd64_5.3.4.tar.gz
  • Make the tool executable:
chmod 755 obsutil_linux_amd64_5.3.4/obsutil
  • Check the tool is working:
./obsutil_linux_amd64_5.3.4/obsutil version
  • Configure obsutil for destination region:
./obsutil_linux_amd64_5.3.4/obsutil config -interactive
Please input your ak:
xxxxxxxxxxxxxxxxxxxxxxxxx
Please input your sk:
xxxxxxxxxxxxxxxxxxxxxxxxx
Please input your endpoint:
oss.eu-west-0.prod-cloud-ocb.orange-business.com
Please input your token:
 
Config file url:
  /home/cloud/.obsutilconfig
 
Update config file successfully!
  • Check the access to bucket list in the region:
./obsutil_linux_amd64_5.3.4/obsutil ls -limit=5
Bucket                        CreationDate                  Location         BucketType  
obs://bucket001               2018-09-03T01:53:02Z          example          OBJECT  
 
obs://bucket002               2018-11-01T01:40:01Z          example          OBJECT     
 
obs://bucket003               2018-10-25T11:45:45Z          example          OBJECT      
 
obs://bucket004               2018-10-26T02:33:09Z          example          OBJECT     
 
obs://bucket005               2018-10-26T02:34:50Z          example          OBJECT     
          
Bucket number is: 5
  • Configure obsutil for source region:
./obsutil_linux_amd64_5.3.4/obsutil config -interactive -crr
Please input your akCrr:
xxxxxxxxxxxxxxxxxxxxxxxxx
Please input your skCrr:
xxxxxxxxxxxxxxxxxxxxxxxxx
Please input your endpointCrr:
oss.eu-west-1.prod-cloud-ocb.orange-business.com
Please input your tokenCrr:
 
Config file url:
  /home/cloud/.obsutilconfig
 
Update config file successfully!

Installation on Windows workstation

Prerequisites:

  • A Windows workstation with outgoing internet connectivity

Steps:

Enter "exit" or "quit" to logout
Enter "help" or "help command" to show help docs
Input your command:
-->config -interactive
Please input your ak:
xxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your sk:
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your endpoint:
oss.eu-west-0.prod-cloud-ocb.orange-business.com
Please input your token:
 
Config file url:
  C:\Users\xxxxx\.obsutilconfig
 
Update config file successfully!
  • Check the access to bucket list in the region:
Input your command:
-->ls -limit=5
Bucket                        CreationDate                  Location         BucketType  
obs://bucket001               2018-09-03T01:53:02Z          example          OBJECT  
 
obs://bucket002               2018-11-01T01:40:01Z          example          OBJECT     
 
obs://bucket003               2018-10-25T11:45:45Z          example          OBJECT      
 
obs://bucket004               2018-10-26T02:33:09Z          example          OBJECT     
 
obs://bucket005               2018-10-26T02:34:50Z          example          OBJECT     
          
Bucket number is: 5
  • Configure “obsutil” for source region:
Input your command:
-->config -interactive -crr
Please input your akCrr:
xxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your skCrr:
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
Please input your endpointCrr:
oss.eu-west-1.prod-cloud-ocb.orange-business.com
Please input your tokenCrr:
 
Config file url:
  C:\Users\xxxxx\.obsutilconfig
 
Update config file successfully!
  • If needed, configure your proxy by editing the “.obsutilconfig” file located in your user directory:
proxyUrl=http://username:password@your-proxy:8080

Migration

Steps:

  • Create the new bucket:
mb obs://bucket-destination -location=eu-west-0
  • Synchronize objects from source bucket with destination bucket:

sync -crr obs://bucket-source obs://bucket-destination

  • Delete objects inside source bucket:
rm obs://bucket-source -r -f
  • Delete source bucket:
rm obs://bucket-source -f

Usefull link:

https://docs.prod-cloud-ocb.orange-business.com/utiltg/obs/obs_11_0001.html