Categories
Java Linux

Setup Tomcat application server

Setup files

su myUser
 
# Java
cd /app/java
tar -xzf /app/files/OpenJDK8U-jre_x64_linux_hotspot_8u265b01.tar.gz
tar -xzf /app/files/OpenJDK11U-jre_x64_linux_hotspot_11.0.8_10.tar.gz
tar -xzf /app/files/OpenJDK15U-jre_x64_linux_hotspot_15_36.tar.gz
 
# myApp Tomcat
cd /app/myApp
tar -xzf /app/files/apache-tomcat-8.5.59.tar.gz
tar -xzf /app/files/apache-tomcat-9.0.39.tar.gz
 
# Certificate
cp /app/files/corporate_truststore.jks /app/certs/
 
# SAP JCO
cp /app/files/_sapjco3-64/3.0.19/linuxx86/libsapjco3.so /app/library/
cp /app/files/_sapjco3-64/3.0.19/linuxx86/sapjco3.jar /app/library/

Setup Tomcat

su myUser
 
# Symlink to actual Tomcat version
ln -s /app/ccp/apache-tomcat-8.5.59 /app/myApp/tomcat
 
# remove sample application
# but keep the Tomcat Manager app for deployment
rm -rf /app/myApp/tomcat/webapps/docs
rm -rf /app/myApp/tomcat/webapps/examples
rm -rf /app/myApp/tomcat/webapps/ROOT
 
# configure Tomcat
vim /app/myApp/tomcat/bin/setenv.sh
mv /app/myApp/tomcat/conf/server.xml /app/myApp/tomcat/conf/server.xml_original
vim /app/myApp/tomcat/conf/server.xml
 
# expand Classpath
vim /app/myApp/tomcat/conf/catalina.properties
# tomcat/bin/setenv.sh
CATALINA_HOME=/app/myApp/tomcat
CATALINA_BASE=/app/myApp/tomcat
CATALINA_PID=/app/myApp/tomcat/tomcat.pid
JAVA_HOME=/app/java/jdk8u265-b01-jre
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/app/library"
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/library
export LD_LIBRARY_PATH
JAVA_OPTS="${JAVA_OPTS} -Djavax.net.ssl.trustStore=/app/certs/corporate_truststore.jks -Djavax.net.ssl.trustStorePassword=notchangeit -Xms512M -Xmx2048M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/myApp/dump"
# tomcat/conf/catalina.properties
common.loader= [...] ,"/app/library/sapjco3.jar"
# server.xml
<Server port="7010" shutdown="SHUTDOWN">
  <Service name="Catalina">
    <Connector port="7011" />
   
    <!-- Define an AJP 1.3 Connector on port 7012 -->
    <Connector port="7012" protocol="AJP/1.3" secretRequired="false" />
    <Engine name="Catalina" defaultHost="localhost" jvmRoute="myApp-dev">
      <Host name="localhost" appBase="webapps" />
    </Engine>
  </Service>
</Server>

Tomcat as a Service

# As root
vim /etc/systemd/system/tomcat.service
# enable script:
systemctl enable tomcat.service
# tomcat.service

# Systemd unit file for myApp tomcat
#
# To create clones of this service:
   
# Systemd unit file for tomcat
 [Unit]
 Description=myApp Tomcat Web Application Container
 After=syslog.target network.target
   
   
 [Service]
 Type=forking
   
 ExecStart=/app/myApp/tomcat/bin/startup.sh
 ExecStop=/app/myApp/tomcat/bin/shutdown.sh
   
 User=myUser
 Group=myUser
   
 [Install]
 WantedBy=multi-user.target

Service control

Enable user myUser to control Tomcat services:

visudo -f /etc/sudoers
##################################################
## Allow user myUser to control (apache & tomcat) services
%myUser ALL=(root) NOPASSWD: /bin/systemctl
%myUser ALL=(root) NOPASSWD: /usr/sbin/service

Test Tomcat

For Tomcat testing I use my ShowHeaders app (GitHub).
ShowHeaders is a minimalistic webapp that is not much more than a “Hello World”, but it shows the HTTP headers, what is quite useful when testing reverse proxy integration.

# copy ShowHeaders App for Testing (also for Reverse Proxy Configuration Testing)
[myUser@DEV ~]$ cp /app/files/ShowHeaders/ROOT.war /app/myApp/tomcat/webapps/
# start tomcat:
[myUser@DEV ~]$ sudo systemctl start tomcat

#
curl localhost:7011

Test connection from outside the server itself: http://myAppServer:7011/

Categories
Java

Git & SSL

In my first post about Git I wrote about the problem with non-public CA signed certitificates and how to handle it. I did not mention the easierst (and unsecured) way to handle this, so I write this post to have all possibilities in one place.

Add certificate to truststore

I download the public certificate of the CA from webbrowser and add it to the truststore of Git.

Where is the cert store of git?

git config --system --list
http.sslcainfo={PathToGit}/mingw64/ssl/certs/ca-bundle.crt

To add the non-public CA cert to Git cert store just open ca-bundle.crt and the downloaded certificate with an text editor and copy the content of the certificate to the ca-bundle.

Use Windows Networking Layer

I configured the sslBackend to the Windows Networking Layer:

# use SChannel, the built-in Windows networking layer. This means that it will use the Windows certificate storage mechanism and you do not need to explicitly configure the curl CA storage mechanism.
git config --global http.sslBackend schannel

Disable SSL Verify

The easierst and unsecure way is to simply disable SSL validation:

git config --global http.sslVerify false

This also works with the system configuration (–system instead of –global). I did this an a project with a very short time budget, we had to configure Git on the Linux system and this Git installation was used by a Jenkins. Both servers, Git & Jenkins, are in the same corporate intranet.

Categories
Java

Git

Short installation guide for Git with GitBash, SourceTree for a simple visual user interface and Git Staging View in Eclipse.

Git and GitBash

Download: https://gitforwindows.org
Install with defaults.

As there was no HOME environment variable set, my Git took some other HOME-like variable (like HOMEPATH, not sure), and this is a network share, so my Git performance was sometimes very poor.
To fix this, just set a HOME variable to your ‘normal’ profile folder:

Set a persistent environment variable from cmd.exe

setx HOME %USERPROFILE%

‘set’ sets your environment variable in your current shell only, persist it with ‘setx’.

Another problem is, that the Git reporitory I tried to connect, has a SSL key signed by a non-public CA. This results to a “ssl pkix path validation failed” error.
To resolve this, I download the public certificate of the CA from webbrowser and add it to the truststore of Git.

Where is the cert store of git?

git config --system --list
http.sslcainfo={PathToGit}/mingw64/ssl/certs/ca-bundle.crt

To add the non-public CA cert to Git cert store just open ca-bundle.crt and the downloaded certificate with an text editor and copy the content of the certificate to the ca-bundle.

Additional I configured the sslBackend to the Windows Networking Layer:

# use SChannel, the built-in Windows networking layer. This means that it will use the Windows certificate storage mechanism and you do not need to explicitly configure the curl CA storage mechanism.
git config --global http.sslBackend schannel

Configure my user details:

git config --global user.name "Ingo Kaulbach"
git config --global user.email "ingo.kaulbach@covestro.com"

Some Git commands for Git configuration:

git config --local --list
git config --global --list
git config --system --list
 
git config --local --edit

Checkout / Clone of a project:

cd /path/to/my/workspace
git clone https://{Username}:{PersonalAccessToken}@gitlab.myserver.biz/project/project.git

SourceTree

Download: https://www.sourcetreeapp.com
Current Version: 3.3.9 (Windows)

Install with default settings, without Mercurial.

Open /path/to/my/workspace/project.

Eclipse

Open Windows -> Perspective -> Open Perspective -> Other -> Git.

In Git Repositories View: Add an existing local Git Repository to this view and open /path/to/my/workspace/project.

To Commit and Push changes only use the Git Staging view!!!

Categories
AWS Java

Credentials

What I want to achieve

In my past experiments the AWS credentials were ‘magically’ set in the background. To learn more about AWS credentials I will remove step by step the ‘magic’ and set credentials explicit in my code.

Cleanup

In my first experiment I set up the credentials on my Windows machine.
To ensure, that they are provided I test with my SNS-Test Program from my last post:

package aws;

import software.amazon.awssdk.services.sns.SnsClient;
import software.amazon.awssdk.services.sns.model.ListTopicsRequest;
import software.amazon.awssdk.services.sns.model.ListTopicsResponse;

public class CredentialsTest {

	public static void main(String[] args) {		
		SnsClient snsClient = SnsClient.builder().build();
		ListTopicsRequest request = ListTopicsRequest.builder().build();
		ListTopicsResponse result = snsClient.listTopics(request);
		System.out.println("Status was " + result.sdkHttpResponse().statusCode() + "\n\nTopics\n\n" + result.topics());
	}
}

Result: A list of my SNS topics

To remove the ‘magic’ I rename the files credentials and config in C:\Users\USERNAME\.aws folder to credentials_backup and config_backup.

Start CredentialsTest and the result: A list of my SNS topics.
So the credentials are provided by another mechanism.

Next try to remove the ‘magic’ I remove environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION.
As I have started my IDE with this environment variables set, I need to restart IDE first.

Start CredentialsTest and the result:

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@4372b9b6: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@3e6f3f28: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@4816278d: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@1ecee32c: Unable to contact EC2 metadata service.]
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)

Provide region:

SnsClient snsClient = SnsClient.builder().region(Region.EU_CENTRAL_1).build();

Result:

Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Unable to load credentials from service endpoint.]
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)

OK, looks good so far.
Remove region from code and restore files credentials and config in C:\Users\USERNAME\.aws folder.
Run CredentialsTest, Result: A list of my SNS topics.

Rename the files credentials and config in C:\Users\USERNAME\.aws folder to credentials_backup and config_backup again.
Run CredentialsTest, Result: Unable to load region error again.

The ‘magic’ has been removed,

ProfileCredentialsProvider

Restore files credentials and config in C:\Users\USERNAME\.aws folder.
Empty [default] block and create a new [CredentialsTest] block:

[default]

[CredentialsTest]
aws_access_key_id = My_AWS_Access_Key_Id
aws_secret_access_key = My_AWS_Secret_Access_Key
[default]

[CredentialsTest]
region = eu-central-1

Run CredentialsTest, Result:

2020-09-09 21:13:17 [main] WARN  software.amazon.awssdk.profiles.internal.ProfileFileReader:105 - Ignoring profile 'CredentialsTest' on line 3 because it did not start with 'profile ' and it was not 'default'.
Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@260e86a1: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@59e505b2: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@8e50104: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@43b6123e: Unable to contact EC2 metadata service.]

So I used to try to work with the ProfileCredentialsProvider this way:

import com.amazonaws.auth.profile.ProfileCredentialsProvider;
SnsClient snsClient = SnsClient.builder().credentialsProvider(new ProfileCredentialsProvider("CredentialsTest")).build();

Unfortunatly this won’t compile because:

The method credentialsProvider(AwsCredentialsProvider) in the type AwsClientBuilder<SnsClientBuilder,
 SnsClient> is not applicable for the arguments (ProfileCredentialsProvider)

Refactor to:

import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider;
AwsCredentialsProvider credentialsProvider = ProfileCredentialsProvider.builder().profileName("CredentialsTest").build();
SnsClient snsClient = SnsClient.builder().credentialsProvider(credentialsProvider).build();

Result:

2020-09-09 21:22:40 [main] WARN  software.amazon.awssdk.profiles.internal.ProfileFileReader:105 - Ignoring profile 'CredentialsTest' on line 3 because it did not start with 'profile ' and it was not 'default'.
Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@78f5c518: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@f107c50: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@4ebff610: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@8692d67: Unable to contact EC2 metadata service.]

Hmkay, enhance the code with a region; need to set this explicit, could not find any way to read this from the config file.

SnsClient snsClient = SnsClient.builder().credentialsProvider(credentialsProvider).region(Region.EU_CENTRAL_1).build();

Result: A list of my SNS topics.

Rename the files credentials and config in C:\Users\USERNAME\.aws folder to credentials_backup and config_backup again.
Run CredentialsTest, Result: Profile file contained no credentials for profile ‘CredentialsTest’ error.

Remove ProfileCredentialsProvider and Region from code.
Run CredentialsTest, Result: Unable to load region error again.

Own AwsCredentialsProvider implementation

Write an own credential provider, the simplest way:

package aws;

import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;

public class IngosCredentialProvider implements AwsCredentialsProvider {

	public AwsCredentials resolveCredentials() {
		System.out.println("IngosCredentialProvider::resolveCredentials called");
		AwsCredentials credentials = new AwsCredentials() {
			
			public String secretAccessKey() {
				return "My_AWS_Secret_Access_Key";
			}
			
			public String accessKeyId() {
				return "My_AWS_Access_Key_Id";
			}
		};
		return credentials;
	}
}

Use your own credentials provider in code, don’t forget the region:

package aws;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.sns.SnsClient;
import software.amazon.awssdk.services.sns.model.ListTopicsRequest;
import software.amazon.awssdk.services.sns.model.ListTopicsResponse;

public class CredentialsTest {

	public static void main(String[] args) {
		SnsClient snsClient = SnsClient.builder().credentialsProvider(new IngosCredentialProvider()).region(Region.EU_CENTRAL_1).build();
		ListTopicsRequest request = ListTopicsRequest.builder().build();
		ListTopicsResponse result = snsClient.listTopics(request);
		System.out.println("Status was " + result.sdkHttpResponse().statusCode() + "\n\nTopics\n\n" + result.topics());
	}
}

Run CredentialsTest, Result: A list of my SNS topics.

Categories
AWS Java

Simple Notification Service

What I want to do today

Create a SNS, send and receive messages.

Create SNS

Just go to Amazon SNS -> Topics -> Create topic and set a name for the topic:

In the next screen I create a subscription with Protocol Email and my email address. Immediately I got an email with a link to subscribe to the topic. After Confirmation I can check in the Subsriptions view that the status has changed to “Confirmed”.

There is a Amazon SNS -> Topics -> MyFirstTestTopic -> Publish message function in the AWS Console to publish a message to topic, which is a good way to test the service.

Java Code

I continue with my test project from my last post.

List SNS Topics

SnsClient snsClient = SnsClient.builder().region(Region.EU_CENTRAL_1).build();
ListTopicsRequest request = ListTopicsRequest.builder().build();
ListTopicsResponse result = snsClient.listTopics(request);
System.out.println("Status was " + result.sdkHttpResponse().statusCode() + "\n\nTopics\n\n" + result.topics());

Lists all topics of my AWS account.

Publish message to topic

SnsClient snsClient = SnsClient.builder().region(Region.EU_CENTRAL_1).build();
String topicArn = "arn:aws:sns:eu-central-1:175335015168:MyFirstTestTopic";
String message = "This is a test (c)DerIngo";
PublishRequest request = PublishRequest.builder().message(message).topicArn(topicArn).build();
PublishResponse result = snsClient.publish(request);
System.out.println(result.messageId() + " Message sent. Status was " + result.sdkHttpResponse().statusCode());

Test email received, it works, YAY!

I

Categories
AWS Java

AWS Glacier

What I want to do today

As next step to proceed further with my AWS experiences I would like to create a data storage, where I can upload some files programatically and retrive an email every time a file was uploaded.
Within all this activities some metrics should be generated, so I can see them in CloudWatch service and retrive data with my First Test Application for AWS.

Create data storage

Amazon Simple Storage Service (Amazon S3) is an object storage service and Amazon S3 Glacier is an extremely low-cost storage service, ex. for backup. So I decided to go with Glacier, because I like it cheap for my tests.

First step is to create a Vault, which is a container for storing archives.
A Vault is created with a region (EU Frankfurt) and a name (“MyFirstSampleGlacierVault”) and some useful information is shown in creation screen:

Data is stored in S3 Glacier in “archives.” An archive can be any data such as a photo, video, or document. You can upload a single file as an archive or aggregate multiple files into a TAR or ZIP file and upload as one archive.

A single archive can be as large as 40 terabytes. You can store an unlimited number of archives and an unlimited amount of data in S3 Glacier. Each archive is assigned a unique archive ID at the time of creation, and the content of the archive is immutable, meaning that after an archive is created it cannot be updated.

Vaults allow you to organize your archives and set access policies and notification policies.

In the second step I “Enable notifications and create a new SNS topic” and set the topic name to “MyFirstSampleGlacierVaultSNS” in the third step. and I have to “Select the job type(s) you want to trigger your notifications”. As I do not know what this practically means by now, I select both: “Archive Retrieval Job Complete” and “Vault Inventory Retrieval Job Complete”.
In the settings of the created Vault I can check, that the Retrieval policies is set to “Free Tier Only”, which is great, becaus it means:

Data retrieval requests that exceed the free tier will not be accepted.

Retrieval Cost: Free

IAM Access

To access programatically to my S3 Glacier Vault I create a new user: “MyFirstSampleGlacierVaultTestUser” with Programmatic access and attach the existing “AmazonGlacierFullAccess” policy directly.
As per my current understanding, this allows this user to do everything on every Glacier Vault? I need to check later, if/how I can restrict access to my Test Vault only.

Java Code

I continue with my test project from my last post.

Maven

I have added the entire AWS SDK, I thought. But as I tried to create an AmazonGlacierClient I figured out, that I had to add the Glacier Service SKD to the “entire” AWS SDK:

	<dependencies>
		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk-glacier</artifactId>
			<version>1.11.852</version>
		</dependency>
	</dependencies>

Credentials

For my first test I added the credentials to system environment properties and created the /.aws/crendentials file. But this was with credentials for the CloudWatch user. Now I need to use the credentials of my Glacier user.

I found min. three ways to provide the Glacier user credentials.

For the first way I have to add a new section to the credentials file and select this profile:

[glacierUser]
aws_access_key_id = the_Access_Key_Id
aws_secret_access_key = the_Secret_Access_Key
System.setProperty("aws.profile", "glacierUser");

For the second way I have to set the properties directly in Java code:

System.setProperty("aws.accessKeyId", "the_Access_Key_Id");
System.setProperty("aws.secretAccessKey", "the_Secret_Access_Key");

I guess, both ways should work. But I only tested the third way, to build an AWSCredentials object:

AWSCredentials awsCredentials = new AWSCredentials() {
			
			public String getAWSSecretKey() {
				return "the_Secret_Access_Key";
			}
			
			public String getAWSAccessKeyId() {
				return "the_Access_Key_Id";
			}
		};

I don’t think, it is a good idea to store credentials in code, but I am just testing to get things working.

Create a Glacier client and test

First create a Glacier client and then test to create and delete a new Vault.

Create a Glacier client with both deprecated Constructor and setEndpoint Method; maybe I search for an un-deprecated way later:

AmazonGlacierClient client = new AmazonGlacierClient(awsCredentials);
client.setEndpoint("https://glacier.eu-central-1.amazonaws.com/");

Test to create a new Vault:

CreateVaultRequest request = new CreateVaultRequest()
	.withVaultName("HURZ");
CreateVaultResult result = client.createVault(request);

System.out.println("Created vault successfully: " + result.getLocation());

I checked the Vault creation in S3 Glacier Vaults overview:

It worked, so I can delete it:

DeleteVaultRequest deleteRequest = new DeleteVaultRequest()
    .withVaultName("HURZ");
DeleteVaultResult deleteResult = client.deleteVault(deleteRequest);
System.out.println("Deleted vault with HTTP status code: " + deleteResult.getSdkHttpMetadata().getHttpStatusCode());

Returned a HTTP status code 204, doublechecked in Vaults overview:

YAY! It works! Next test:

File up- and download

To upload a file to my Vault I need a ArchiveTransfer Manager:

String vaultName  = "MyFirstSampleGlacierVault";
String fileToUpload = "src/main/resources/cute_kitty.jpg";

ArchiveTransferManager atm = new ArchiveTransferManager(client, awsCredentials);
try {
	String archiveId = atm.upload(vaultName, "Cute Kitty Pic", new File(fileToUpload)).getArchiveId();
	System.out.println("Kitties archive ID: " + archiveId);
} catch (AmazonClientException | FileNotFoundException e) {
	// TODO Auto-generated catch block
	e.printStackTrace();
}

It seems to work, becaus there is no Error but an Archive ID of my cute kitty pic.

Let’s try to download the file from Glacier; just add one line into the try-block:

atm.download(vaultName, archiveId, new File(fileToUpload+"_fromGlacier"));

Unfortunatly this ends in an ERROR/WARNING but no file is downloaded:

com.amazonaws.services.sqs.model.AmazonSQSException: Access to the resource https://sqs.us-east-1.amazonaws.com/ is denied. (Service: AmazonSQS; Status Code: 403; Error Code: AccessDenied; Request ID: 3eb8ca32-120b-520a-8383-9dfbb53cb96e; Proxy: null)

Strange: “the resource https://sqs.us-east-1.amazonaws.com/”. My Vault is in Europe!
I will change the code to explicite set this to Europe:

AmazonGlacierClient glacierClient = new AmazonGlacierClient(awsCredentials);
AmazonSQSClient sqsClient = new AmazonSQSClient(awsCredentials);
AmazonSNSClient snsClient = new AmazonSNSClient(awsCredentials);

glacierClient.setEndpoint("glacier.eu-central-1.amazonaws.com");
sqsClient.setEndpoint("sqs.eu-central-1.amazonaws.com");
snsClient.setEndpoint("sns.eu-central-1.amazonaws.com");

ArchiveTransferManager atm = new ArchiveTransferManager(glacierClient, sqsClient, snsClient);

String vaultName  = "MyFirstSampleGlacierVault";
String fileToUpload = "src/main/resources/cute_kitty.jpg";

try {
	String archiveId = atm.upload(vaultName, "Cute Kitty Pic", new File(fileToUpload)).getArchiveId();
	System.out.println("Kitties archive ID: " + archiveId);
	atm.download(vaultName, archiveId, new File(fileToUpload+"_fromGlacier"));
} catch (AmazonClientException | FileNotFoundException e) {
	e.printStackTrace();
}

Lots of deprecated warnings; I’ll ignore them all.
Result stays the same, only difference that the access is now denied for Europe:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.amazonaws.util.XpathUtils (file:/C:/Users/i-kau/.m2/repository/com/amazonaws/aws-java-sdk-core/1.11.852/aws-java-sdk-core-1.11.852.jar) to constructor com.sun.org.apache.xpath.internal.XPathContext()
WARNING: Please consider reporting this to the maintainers of com.amazonaws.util.XpathUtils
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
com.amazonaws.services.sqs.model.AmazonSQSException: Access to the resource https://sqs.eu-central-1.amazonaws.com/ is denied. (Service: AmazonSQS; Status Code: 403; Error Code: AccessDenied; Request ID: 1d9d83e6-f301-5137-940d-d42f58994ce4; Proxy: null)

Unfortunatly there is no console (browser) support for Glacier, so I cannot test right now, if this is a problem with the permissions or with the XpathUtils library.

As this is all just for testing, I can live with this error and proceed with testing other services.

Categories
AWS Java

Getting Started with AWS

Create an IAM user

I want to start with some practical experiences in AWS, so I go to https://aws.amazon.com, login with my Root user and open the Identity and Access Management (IAM ), where I create aa new IAM user, that I call “MyFirstProgrammaticAccessTestUser”, because the user is of access type Programmatic access. For now, I do not add the user to any group and add only one tag (that I name tag-key) to the user.
AWS is warning, that this user has not permissons, but this is fine for now, I will add any permission as soon as the user needs one.
Finally I note down the user name, Access key ID and the secret access key.

Set up AWS credentials and region

I am working on a Windows machine, so I create a folder .aws in C:\Users\USERNAME. In this folder I create a file credentials:

[default]
aws_access_key_id = your_access_key_id
aws_secret_access_key = your_secret_access_key

To set the default AWS Region I have to create another file in .aws folder: config:

[default]
region = eu-central-1

Additionally I have to set this information as environment variables.

I am really not sure, if this is the correct way to set this environment variables, but hey, this is only a test.

AWS SDK

I have to go to https://github.com/aws/aws-sdk-java-v2 to get the Clone with HTTPS URL.
Then open Eclipse and use the IMPORT dialog to import the project from GIT.
After checkout use the Configure -> Convert to Maven project dialog.
Then I tried Run as -> Maven install. But this results in a Build Failure:

[ERROR] Failed to execute goal com.github.spotbugs:spotbugs-maven-plugin:3.1.11:spotbugs (spotbugs) on project annotations: Execution spotbugs of goal com.github.spotbugs:spotbugs-maven-plugin:3.1.11:spotbugs failed: java.lang.IllegalArgumentException: Unsupported class file major version 57 -> [Help 1]

I found a clue, that I have to use Java 11 instead of my Java 13. So I downloaded a Java 11 JDK and added it to my Eclipse.
But unfortunately I have no clue, how to tell the embedded Eclipse Maven to use this Java 11 instead of Java 13. Great….NOT

Next try: Start a WSL Bash. Need to install Java and Maven first:

sudo apt install -y openjdk-11-jre maven
cd /mnt/[...]/aws-sdk-java-v2
mvn clean install

Now it took 15 minutes to run until it ends wit an ERROR: There are test failures.

While I was waiting for the WSL-Maven to finish, I figured out, how to tell the Eclipse-Maven to run with the Java 11: I have to create a new Run Configuration where I explicite select the JRE:

The Eclipse-Maven also ends with an ERROR: There are test failures.

But for today I am fine with this result.

Create an AWS Maven Project

I create a new Maven project in Eclipse where I pull in the entire AWS SDK. This is not a good choice for a real world application, where you should only pull in components you need, but for a test project it’s a good start.
This is my pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>test</groupId>
	<artifactId>aws</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<packaging>war</packaging>

	<properties>
		<java.version>1.8</java.version>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<aws.java.sdk.version>2.14.7</aws.java.sdk.version>
	</properties>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>software.amazon.awssdk</groupId>
				<artifactId>bom</artifactId>
				<version>${aws.java.sdk.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<dependencies>
		<dependency>
			<groupId>software.amazon.awssdk</groupId>
			<artifactId>aws-sdk-java</artifactId>
			<version>${aws.java.sdk.version}</version>
		</dependency>
		<dependency>
			<groupId>org.apache.tomcat</groupId>
			<artifactId>tomcat-catalina</artifactId>
			<version>8.5.33</version>
			<scope>provided</scope>
		</dependency>
	</dependencies>

	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>3.8.0</version>
				<configuration>
					<source>${java.version}</source>
					<target>${java.version}</target>
				</configuration>
			</plugin>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-war-plugin</artifactId>
				<version>3.2.2</version>
				<configuration>
					<failOnMissingWebXml>false</failOnMissingWebXml>
					<warName>ROOT</warName>
					<wtpContextName>ROOT</wtpContextName>
				</configuration>
			</plugin>
		</plugins>
	</build>
</project>

Add AWS SDK Logging

I wanted to add some logging so I put Log4J Libs dependencies to pom.xml and create a log4j2.xml file for configuration in src/main/resources folder.

log4j2.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
  <Appenders>
    <Console name="ConsoleAppender" target="SYSTEM_OUT">
      <PatternLayout pattern="%d{YYYY-MM-dd HH:mm:ss} [%t] %-5p %c:%L - %m%n" />
    </Console>
  </Appenders>

  <Loggers>
    <Root level="WARN">
     <AppenderRef ref="ConsoleAppender"/>
    </Root>
    <Logger name="software.amazon.awssdk" level="WARN" />
    <Logger name="software.amazon.awssdk.request" level="DEBUG" />
    <Logger name="org.apache.http.wire" level="DEBUG" />
  </Loggers>
</Configuration>

pom.xml:

	<properties>
		<org.apache.logging.log4j.version>2.13.3</org.apache.logging.log4j.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-core</artifactId>
			<version>${org.apache.logging.log4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-api</artifactId>
			<version>${org.apache.logging.log4j.version}</version>
		</dependency>
		<dependency>
			<groupId>org.apache.logging.log4j</groupId>
			<artifactId>log4j-slf4j-impl</artifactId>
			<version>${org.apache.logging.log4j.version}</version>
		</dependency>
	</dependencies>

First Test Application

A fist simple Test application to get some CloudeWatch metrics:

package aws;

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.cloudwatch.CloudWatchClient;
import software.amazon.awssdk.services.cloudwatch.model.ListMetricsRequest;
import software.amazon.awssdk.services.cloudwatch.model.ListMetricsResponse;
import software.amazon.awssdk.services.cloudwatch.model.Metric;

public class TestMain {

	public static void main(String[] args) {
		String namespace = "<metric-namespace>";
		Region region = Region.EU_CENTRAL_1;
		CloudWatchClient cw = CloudWatchClient.builder()
                .region(region)
                .build();
		listMets(cw, namespace) ;
	}
	public static void listMets( CloudWatchClient cw, String namespace) {

        boolean done = false;
        String nextToken = null;

        while(!done) {

            ListMetricsResponse response;

            if (nextToken == null) {
                ListMetricsRequest request = ListMetricsRequest.builder()
                        .namespace(namespace)
                        .build();

                response = cw.listMetrics(request);
            } else {
                ListMetricsRequest request = ListMetricsRequest.builder()
                        .namespace(namespace)
                        .nextToken(nextToken)
                        .build();

                response = cw.listMetrics(request);
            }

            for (Metric metric : response.metrics()) {
                System.out.printf(
                        "Retrieved metric %s", metric.metricName());
                System.out.println();
            }

            if(response.nextToken() == null) {
                done = true;
            } else {
                nextToken = response.nextToken();
            }
        }
    }
}

Result:

020-09-01 19:23:07 [main] DEBUG software.amazon.awssdk.request:84 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=monitoring.eu-central-1.amazonaws.com, encodedPath=, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[])
2020-09-01 19:23:08 [main] DEBUG software.amazon.awssdk.request:84 - Received error response: software.amazon.awssdk.services.cloudwatch.model.CloudWatchException: User: arn:aws:iam::175335015168:user/MyFirstProgrammaticAccessTestUser is not authorized to perform: cloudwatch:ListMetrics (Service: CloudWatch, Status Code: 403, Request ID: 75f02535-28c7-49c8-930a-b8d8449c625a, Extended Request ID: null)
Exception in thread "main" software.amazon.awssdk.services.cloudwatch.model.CloudWatchException: User: arn:aws:iam::175335015168:user/MyFirstProgrammaticAccessTestUser is not authorized to perform: cloudwatch:ListMetrics (Service: CloudWatch, Status Code: 403, Request ID: 75f02535-28c7-49c8-930a-b8d8449c625a, Extended Request ID: null)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
	at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.cloudwatch.DefaultCloudWatchClient.listMetrics(DefaultCloudWatchClient.java:1877)
	at aws.TestMain.listMets(TestMain.java:33)
	at aws.TestMain.main(TestMain.java:17)

So the Error is:

user/MyFirstProgrammaticAccessTestUser is not authorized to perform: cloudwatch:ListMetrics

I try to solve this by going back to the IAM console and add the user to a new created group with attached policy “CloudWatchFullAccess”.

Result:

2020-09-01 19:38:27 [main] DEBUG software.amazon.awssdk.request:84 - Sending Request: DefaultSdkHttpFullRequest(httpMethod=POST, protocol=https, host=monitoring.eu-central-1.amazonaws.com, encodedPath=, headers=[amz-sdk-invocation-id, Content-Length, Content-Type, User-Agent], queryParameters=[])
2020-09-01 19:38:28 [main] DEBUG software.amazon.awssdk.request:84 - Received successful response: 200

So this worked, this was quite intuitive 🙂
The result is empty, I guess because of the metric-namespace that I initaly set with placeholder name. I looked into my CloudWatch Dashboard, but could not find any metric with data. I guess, I have to create a metric and find a way to create data for the metric. TBC