AWS Glacier – Ingo's Blog

What I want to do today

As next step to proceed further with my AWS experiences I would like to create a data storage, where I can upload some files programatically and retrive an email every time a file was uploaded.
Within all this activities some metrics should be generated, so I can see them in CloudWatch service and retrive data with my First Test Application for AWS.

Create data storage

Amazon Simple Storage Service (Amazon S3) is an object storage service and Amazon S3 Glacier is an extremely low-cost storage service, ex. for backup. So I decided to go with Glacier, because I like it cheap for my tests.

First step is to create a Vault, which is a container for storing archives.
A Vault is created with a region (EU Frankfurt) and a name ("MyFirstSampleGlacierVault") and some useful information is shown in creation screen:

Data is stored in S3 Glacier in "archives." An archive can be any data such as a photo, video, or document. You can upload a single file as an archive or aggregate multiple files into a TAR or ZIP file and upload as one archive.
A single archive can be as large as 40 terabytes. You can store an unlimited number of archives and an unlimited amount of data in S3 Glacier. Each archive is assigned a unique archive ID at the time of creation, and the content of the archive is immutable, meaning that after an archive is created it cannot be updated.
Vaults allow you to organize your archives and set access policies and notification policies.

In the second step I "Enable notifications and create a new SNS topic" and set the topic name to "MyFirstSampleGlacierVaultSNS" in the third step. and I have to "Select the job type(s) you want to trigger your notifications". As I do not know what this practically means by now, I select both: "Archive Retrieval Job Complete" and "Vault Inventory Retrieval Job Complete".
In the settings of the created Vault I can check, that the Retrieval policies is set to "Free Tier Only", which is great, becaus it means:

Data retrieval requests that exceed the free tier will not be accepted.
Retrieval Cost: Free

IAM Access

To access programatically to my S3 Glacier Vault I create a new user: "MyFirstSampleGlacierVaultTestUser" with Programmatic access and attach the existing "AmazonGlacierFullAccess" policy directly.
As per my current understanding, this allows this user to do everything on every Glacier Vault? I need to check later, if/how I can restrict access to my Test Vault only.

Java Code

I continue with my test project from my last post.

Maven

I have added the entire AWS SDK, I thought. But as I tried to create an AmazonGlacierClient I figured out, that I had to add the Glacier Service SKD to the "entire" AWS SDK:

	<dependencies>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk-glacier</artifactId>
			<version>1.11.852</version>
		</dependency>
	</dependencies>

Credentials

For my first test I added the credentials to system environment properties and created the /.aws/crendentials file. But this was with credentials for the CloudWatch user. Now I need to use the credentials of my Glacier user.

I found min. three ways to provide the Glacier user credentials.

For the first way I have to add a new section to the credentials file and select this profile:

[glacierUser]
aws_access_key_id = the_Access_Key_Id
aws_secret_access_key = the_Secret_Access_Key

System.setProperty("aws.profile", "glacierUser");

For the second way I have to set the properties directly in Java code:

System.setProperty("aws.accessKeyId", "the_Access_Key_Id");
System.setProperty("aws.secretAccessKey", "the_Secret_Access_Key");

I guess, both ways should work. But I only tested the third way, to build an AWSCredentials object:

AWSCredentials awsCredentials = new AWSCredentials() {
			
			public String getAWSSecretKey() {
				return "the_Secret_Access_Key";
			}
			
			public String getAWSAccessKeyId() {
				return "the_Access_Key_Id";
			}
		};

I don't think, it is a good idea to store credentials in code, but I am just testing to get things working.

Create a Glacier client and test

First create a Glacier client and then test to create and delete a new Vault.

Create a Glacier client with both deprecated Constructor and setEndpoint Method; maybe I search for an un-deprecated way later:

AmazonGlacierClient client = new AmazonGlacierClient(awsCredentials);
client.setEndpoint("https://glacier.eu-central-1.amazonaws.com/");

Test to create a new Vault:

CreateVaultRequest request = new CreateVaultRequest()
	.withVaultName("HURZ");
CreateVaultResult result = client.createVault(request);

System.out.println("Created vault successfully: " + result.getLocation());

I checked the Vault creation in S3 Glacier Vaults overview:

It worked, so I can delete it:

DeleteVaultRequest deleteRequest = new DeleteVaultRequest()
    .withVaultName("HURZ");
DeleteVaultResult deleteResult = client.deleteVault(deleteRequest);
System.out.println("Deleted vault with HTTP status code: " + deleteResult.getSdkHttpMetadata().getHttpStatusCode());

Returned a HTTP status code 204, doublechecked in Vaults overview:

YAY! It works! Next test:

File up- and download

To upload a file to my Vault I need a ArchiveTransfer Manager:

String vaultName  = "MyFirstSampleGlacierVault";
String fileToUpload = "src/main/resources/cute_kitty.jpg";

ArchiveTransferManager atm = new ArchiveTransferManager(client, awsCredentials);
try {
	String archiveId = atm.upload(vaultName, "Cute Kitty Pic", new File(fileToUpload)).getArchiveId();
	System.out.println("Kitties archive ID: " + archiveId);
} catch (AmazonClientException | FileNotFoundException e) {
	// TODO Auto-generated catch block
	e.printStackTrace();
}

It seems to work, becaus there is no Error but an Archive ID of my cute kitty pic.

Let's try to download the file from Glacier; just add one line into the try-block:

atm.download(vaultName, archiveId, new File(fileToUpload+"_fromGlacier"));

Unfortunatly this ends in an ERROR/WARNING but no file is downloaded:

com.amazonaws.services.sqs.model.AmazonSQSException: Access to the resource https://sqs.us-east-1.amazonaws.com/ is denied. (Service: AmazonSQS; Status Code: 403; Error Code: AccessDenied; Request ID: 3eb8ca32-120b-520a-8383-9dfbb53cb96e; Proxy: null)

Strange: "the resource https://sqs.us-east-1.amazonaws.com/". My Vault is in Europe!
I will change the code to explicite set this to Europe:

AmazonGlacierClient glacierClient = new AmazonGlacierClient(awsCredentials);
AmazonSQSClient sqsClient = new AmazonSQSClient(awsCredentials);
AmazonSNSClient snsClient = new AmazonSNSClient(awsCredentials);

glacierClient.setEndpoint("glacier.eu-central-1.amazonaws.com");
sqsClient.setEndpoint("sqs.eu-central-1.amazonaws.com");
snsClient.setEndpoint("sns.eu-central-1.amazonaws.com");

ArchiveTransferManager atm = new ArchiveTransferManager(glacierClient, sqsClient, snsClient);

String vaultName  = "MyFirstSampleGlacierVault";
String fileToUpload = "src/main/resources/cute_kitty.jpg";

try {
	String archiveId = atm.upload(vaultName, "Cute Kitty Pic", new File(fileToUpload)).getArchiveId();
	System.out.println("Kitties archive ID: " + archiveId);
	atm.download(vaultName, archiveId, new File(fileToUpload+"_fromGlacier"));
} catch (AmazonClientException | FileNotFoundException e) {
	e.printStackTrace();
}

Lots of deprecated warnings; I'll ignore them all.
Result stays the same, only difference that the access is now denied for Europe:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.amazonaws.util.XpathUtils (file:/C:/Users/i-kau/.m2/repository/com/amazonaws/aws-java-sdk-core/1.11.852/aws-java-sdk-core-1.11.852.jar) to constructor com.sun.org.apache.xpath.internal.XPathContext()
WARNING: Please consider reporting this to the maintainers of com.amazonaws.util.XpathUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
com.amazonaws.services.sqs.model.AmazonSQSException: Access to the resource https://sqs.eu-central-1.amazonaws.com/ is denied. (Service: AmazonSQS; Status Code: 403; Error Code: AccessDenied; Request ID: 1d9d83e6-f301-5137-940d-d42f58994ce4; Proxy: null)

Unfortunatly there is no console (browser) support for Glacier, so I cannot test right now, if this is a problem with the permissions or with the XpathUtils library.

As this is all just for testing, I can live with this error and proceed with testing other services.