GUYS!!
Cheap file storage online?? I think AWS S3 immediately jumps to mind for most people :P
I’m going to be using it to store my training images for GAN. I’ve never set up one before, so I was pleasantly surprised to find that it wasn’t too painful at all.
For people who want to start your own, read on!
So I began here, with the in-house AWS tutorial - it’s pretty good!
http://docs.aws.amazon.com/AmazonS3/latest/gsg/SigningUpforS3.html
Once your bucket is up and running, how do you upload your files via code (not the console)?
In comes boto3!
Use the quickstart guide here:
http://boto3.readthedocs.io/en/latest/guide/quickstart.html
Do your pip install awscli, AND install AWS CLI (AWS command line). I wanted to see if there were other options without using AWS CLI, but… nope. so just install it haha.
Type in aws configure, input your access key id and access key, and for people who selected Singapore as their region, input ap-southeast-1 (you can find the relevant names for other regions here if you need).
There was a next field (I can’t remember!!) but just press enter to skip it if you don’t need it (that’s what I did) and you’re done with the configuration!!
Get yourself into jupyter notebook, and here we go:
import boto3
s3 = boto3.resource('s3') # tells boto which aws resource you are using
for bucket in s3.buckets.all(): # prints out all the bucket names in the s3
print(bucket.name)
s3.Bucket('picture_bucket').upload_file('/mypics/picture.jpg','picture.jpg') # uploads the file into the bucket from the given file path, and saves it as the name specified.
DONE DONE DONEEEEE
now to get that spider going…