Author: Ingo

Twitter Developer Account

Post author By Ingo
Post date 2020-December-07
No Comments on Twitter Developer Account

Everyone is talking about and on Twitter. Mostly every Tweet is public and pullable, this makes Twitter a gold mine of data (or a pile of mud, reading some of the Tweets).

I heard, with a Twitter Developer Account you can do a thing or two, so let's try this.

Twitter Developer Account

Login or create a Twitter account at: https://apps.twitter.com/.
Create a new app (button on the top right):

Apply for a Twitter developer account

You have to answer some questions, but finally:

After confirming the email you can create your first Twitter App.
It starts to give it a name

You will be honoured with three secret keys: API key, API secret key and the Bearer token

Give it a try! (really, do it)

Next try: type in the curl command in your command line:

curl -X GET -H "Authorization: Bearer " "https://api.twitter.com/2/tweets/440322224407314432?expansions=attachments.media_keys,author_id"

{"data":{"text":"If only Bradley's arm was longer. Best photo ever. #oscars http://t.co/C9U5NOtGap","id":"440322224407314432","attachments":{"media_keys":["3_440322224092745728"]},"author_id":"15846407"},"includes":{"media":[{"media_key":"3_440322224092745728","type":"photo"}],"users":[{"id":"15846407","name":"Ellen DeGeneres","username":"TheEllenShow"}]}}

Next stop: The Developer Portal Dashboard

Documentation

The official Twitter API v2 Documentation seems to be a good point to start: https://developer.twitter.com/en/docs/twitter-api/early-access

Playground

Try to use the User lookup from the command line:

curl -X GET -H "Authorization: Bearer " "https://api.twitter.com/2/users/by?usernames=twitterdev,twitterapi,adsapi&user.fields=created_at&expansions=pinned_tweet_id&tweet.fields=author_id,created_at"
{"data":[{"name":"Twitter Dev","pinned_tweet_id":"1293593516040269825","created_at":"2013-12-14T04:35:55.000Z","username":"TwitterDev","id":"2244994945"},{"name":"Twitter API","pinned_tweet_id":"1293595870563381249","created_at":"2007-05-23T06:01:13.000Z","username":"TwitterAPI","id":"6253282"},{"name":"Twitter Ads API","created_at":"2013-02-27T20:01:12.000Z","username":"AdsAPI","id":"1225933934"}],"includes":{"tweets":[{"author_id":"2244994945","id":"1293593516040269825","created_at":"2020-08-12T17:01:42.000Z","text":"It's finally here! \uD83E\uDD41 Say hello to the new #TwitterAPI.\n\nWe're rebuilding the Twitter API v2 from the ground up to better serve our developer community. And today's launch is only the beginning.\n\nhttps://t.co/32VrwpGaJw https://t.co/KaFSbjWUA8"},{"author_id":"6253282","id":"1293595870563381249","created_at":"2020-08-12T17:11:04.000Z","text":"Twitter API v2: Early Access released\n\nToday we announced Early Access to the first endpoints of the new Twitter API!\n\n#TwitterAPI #EarlyAccess #VersionBump https://t.co/g7v3aeIbtQ"}]}}

Try to use the Tweets lookup from the command line:

curl -X GET -H "Authorization: Bearer " "https://api.twitter.com/2/tweets?ids=1228393702244134912,1227640996038684673,1199786642791452673&tweet.fields=created_at&expansions=author_id&user.fields=created_at"
{"data":[{"created_at":"2020-02-14T19:00:55.000Z","text":"What did the developer write in their Valentine's card?\n  \nwhile(true) {\n    I = Love(You);  \n}","id":"1228393702244134912","author_id":"2244994945"},{"created_at":"2020-02-12T17:09:56.000Z","text":"Doctors: Googling stuff online does not make you a doctor\n\nDevelopers: https://t.co/mrju5ypPkb","id":"1227640996038684673","author_id":"2244994945"},{"created_at":"2019-11-27T20:26:41.000Z","text":"C#","id":"1199786642791452673","author_id":"2244994945"}],"includes":{"users":[{"id":"2244994945","username":"TwitterDev","name":"Twitter Dev","created_at":"2013-12-14T04:35:55.000Z"}]}}

Making things a little bit more comfortable:

export BEARER_TOKEN=

Search for the recently posted Tweets with Hashtag Corona in german language:

curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/1.1/search/tweets.json?q=%23Corona&lang=de&result_type=recent"

Tags Twitter

Linux

NFS4

Post author By Ingo
Post date 2020-December-07
1 Comment on NFS4

In my post to NFS I decribed how to create and attach a NFS folder.

Last week a colleague asked me to double check the version of this NS folder on the client. It turns out, that it was NFS3 only and not NFS4.

Interestingly this depends on the file permissions of the parent folder. The parend folder has to be read- and executable for others.

myAppServer1 – Server

chmod o+rx /app/myApp
exportfs -r
exportfs -s

myAppServer2 – Client

umount /app/myApp/uploads
mount /app/myApp/uploads
mount
-> myAppServer1.mynetwork.net:/app/myApp/uploads on /misc/myapp_uploads type nfs4

Tags NFS

Database

Search for lost data

A simplified description of a workflow in one of our supported application is:

AnotherServer is sending data through a middleware to MyAppServer. We store the received data in a MS-SQL database an process the data.

Now there is a problem in the AnotherServer system and they did not know, which data they have send to MyAppServer. So they send us a list of data IDs from that day and asked us to check in our application, which data was not received or received but not processed.

The list contains 600 IDs and it would take ~ 1 minute to check a single ID. So this would be work for ~10 hours. And not one single hour would be fun. After a short discussion we decided to do the data analytics directly in the MyApp database.

List of values

First problem is, that they send us an excel file containing the data IDs and we have to transform this in a way, so we can use this IDs in a SQL statement:

SELECT TempTable.Field1
FROM ( VALUES
(10),
(11),
(12)
      ) AS TempTable (Field1)

Connect list to table

LEFT OUTER JOIN [MyAppSchema].[ReceivedDataTable]
ON TempTable.Field1 = ReceivedDataTable.Data_ID

GROUP BY Field1, Data_ID, is_Processed
ORDER BY is_Processed DESC

Filter processed data

WHERE is_Processed != 1 
   OR is_Processed IS NULL

Tags MS-SQL, SQL

Java

Remote Debugging

I need to directly debug on the application Tomcat server, not only on my local Tomcat instance.

Compile Java code

For debugging we need to keep the line numbers while compiling.
To build the war file we use an ANT script and we have to add the debug and debuglevel attributes in the javac tag:


  [...]

Configure Tomcat server

To enable remote debugging on the Tomcat server add some arguments to CATALINA_OPTS in setenv.sh file:

ATALINA_HOME=/app/myApp/tomcat
CATALINA_BASE=/app/myApp/tomcat
CATALINA_PID=/app/myApp/tomcat/tomcat.pid
JAVA_HOME=/app/java/jdk8u265-b01-jre
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/app/library -Xdebug -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n"
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/library
export LD_LIBRARY_PATH
JAVA_OPTS="${JAVA_OPTS} -Djavax.net.ssl.trustStore=/app/certs/corporate_truststore.jks -Djavax.net.ssl.trustStorePassword=secret -Xms256M -Xmx1024M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/myApp/dump"

Configure Eclipse

Create a new debug configuration:

Tags Ant, Eclipse, Tomcat

Linux

NFS

Post author By Ingo
Post date 2020-November-12
1 Comment on NFS

NFS - Network File System

We have two Linux servers, each with one Tomcat application server. Both application servers must have access to one folder to up- and download files.

We will create a folder on the first server and enable access through the network on this server for the second server.

Operating System: RedHat Enterprise Linux 7

myAppServer1 - Server

vim /etc/exports

systemctl list-units *nfs-server*
# if 0 loaded units, you have to enable and start

systemctl enable nfs-server.service
systemctl start nfs-server.service
systemctl list-units *nfs-server*
exportfs -r
exportfs -s

/app/myApp/uploads myAppServer2.mynetwork.net(rw,sync)

myAppServer2 - Client

vim /etc/fstab
vim /etc/auto.master
vim /etc/auto.misc

systemctl enable autofs.service
systemctl start autofs.service

su tomcat
ls -lisah /app/myApp/uploads

# add:
myAppServer1.mynetwork.net:/app/myApp/uploads   /app/myApp/uploads     nfs     user,timeo=3,bg 0       0

# add:
/misc   /etc/auto.misc  --timeout 0 --negative-timeout 5

# add:
myapp_uploads     -fstype=nfs,bg,timeout=-1,timeo=5,rw,tcp,soft,intr,nosuid myAppServer1.mynetwork.net:/app/myApp/uploads

Test

Reboot both servers and doublecheck that the folder is still available and accessible from both servers.

reboot now

su tomcat
ls -lisah /app/myApp/uploads

Open webapplication in your browser on myAppServer1 and upload a file. Test download of the file.
Switch to myAppServer2 and download file.

Open webapplication in your browser on myAppServer2 and upload a file. Test download of the file.
Switch to myAppServer1 and download file.

NFS4 - Update

Please also note this post: NFS4

Tags NFS

Java

Deploy to Tomcat with Maven

Post author By Ingo
Post date 2020-November-10
No Comments on Deploy to Tomcat with Maven

In my last post I decribed the way to build and deploy a war file to Tomcat application server with Ant.

In this post I show how to deploy the build war file to Tomcat application server with Maven.

Create inside the "ant" folder this Maven file:


    4.0.0
    deringo
    myApp
    0.0.1-SNAPSHOT
    war
    myApp
    myApp Maven deployment

    
        1.8
        8.5.33
        UTF-8
    

    
        
            org.apache.tomcat
            tomcat-catalina
            ${tomcat.version}
            provided
        
    

    
        
            
                org.apache.maven.plugins
                maven-compiler-plugin
                3.8.0
                
                    ${java.version}
                    ${java.version}
                
            
            
                org.apache.maven.plugins
                maven-war-plugin
                3.2.2
                
                    false
                    myApp
                    myApp
                
            
            
                org.apache.tomcat.maven
                tomcat7-maven-plugin
                2.2
                
                    http://myAppServer:7011/manager/text
                    tomcat
                    tomcat
                    dist/myApp.war
                    /myApp
                    true

Execute with goal: tomcat7:deploy

Use Properties file

In my ANT file I have used a properties file for tomcat username/password etc. In my Maven script I want also to use this properties file.

Unfortunatly Maven can not handle property files out of the box. But there is a Plugin we can use:

        
            org.codehaus.mojo
            properties-maven-plugin
            1.0-alpha-1
            
                
                    initialize
                    
                        read-project-properties
                    
                    
                        
                            tomcat.properties

This is the same, as writing this directly into pom.xml:


    tomcat
    tomcat

The complete pom.xml with properties:


  4.0.0
  deringo
  myApp
  0.0.1-SNAPSHOT
  war
  myApp
  myApp Maven deployment

  
    1.8
    8.5.33
    UTF-8
  

  
    
      org.apache.tomcat
      tomcat-catalina
      ${tomcat.version}
      provided
    
  

  
    
      
        org.codehaus.mojo
        properties-maven-plugin
        1.0-alpha-1
        
          
            initialize
            
              read-project-properties
            
            
              
                tomcat.properties
              
            
          
        
      
      
        org.apache.maven.plugins
        maven-compiler-plugin
        3.8.0
        
          ${java.version}
          ${java.version}
        
      
      
        org.apache.maven.plugins
        maven-war-plugin
        3.2.2
        
          false
          myApp
          myApp
        
      
      
        org.apache.tomcat.maven
        tomcat7-maven-plugin
        2.2
        
          ${tomcat.manager.url}
          ${tomcat.username}
          ${tomcat.password}
          dist/myApp.war
          ${webapp.name}
          true

Update 2021-03-24

I tried the configuration in a project but got this error:

Plugin execution not covered by lifecycle configuration: org.codehaus.mojo:properties-maven-plugin:1.0-alpha-1:read-project-properties (execution: default, phase: initialize)

I ~~solved~~ the error adding a pluginManagement tag:

But this solution seems more like a workaround when reading the discussion on stackoverflow.
...need to search further...

Use in Jenkins

I have not tried it, but it should be possible to set an environment variable in Jenkins build step to execute with a configurable filename.

For example TOMCAT_PROPERTIES=jenkins-home/secret/tomcat-dev.properties.
And we can use it in Maven script this way:

            
              
                ${env.TOMCAT_PROPERTIES}

Tags Maven, Tomcat

Java

Deploy to Tomcat

Deploy myApp.war file to Tomcat application server, using Tomcat manager app and an ant script.

Make a folder "ant" in your project and this it the folder for all other actions below.

Make a folder "dist" and generate your myApp.war file into this folder with the "build.xml" script.

Make a folder "tomcat-libs" and copy following files from your tomcat installation:

catalina-ant.jar
catalina.jar
servlet-api.jar
jsp-api.jar

Create file "tomcat.properties" with properties for tomcat server and tomcat manager app.

Create "deploy.xml" file for deployment.

Execute deploy.xml script with ant, it must show the info sections with properties of your tomcat.properties file.

Execute deploy-webapp target from deploy.xml script to deploy your myApp.war file to tomcat server through the tomcat manager app.

If you are using for example Jenkins, you can set the "secprops.location" from outside the script, so you can use the same script for different tomcat installations.





  
    myApp ant deploy file.

tomcat.manager.url=http://myAppServer:7011/manager/text
tomcat.username=tomcat
tomcat.password=tomcat
webapp.name=myApp




  
    myApp ant build file.

Tags Ant, Tomcat

Java Linux

Setup Tomcat manager app

In my last post I set up a Tomcat application server in general, now I enable Tomcat manager app for deployment.

# Tomcat Users
mv /app/myApp/tomcat/conf/tomcat-users.xml /app/myApp/tomcat/conf/tomcat-users.xml_original
vim /app/myApp/tomcat/conf/tomcat-users.xml

vim/app/myApp/tomcat/conf/server.xml

# By default the Manager is only accessible from a browser running on the same machine as Tomcat. If you wish to modify this restriction, you'll need to edit the Manager's context.xml file.
vim /app/myApp/tomcat/webapps/manager/META-INF/context.xml

Test Tomcat manager app

Open http://myAppServer:7011/manager and login with username: tomcat and password: tomcat.

Tags Tomcat

Java Linux

Setup Tomcat application server

Post author By Ingo
Post date 2020-November-10
No Comments on Setup Tomcat application server

Setup files

su myUser

# Java
cd /app/java
tar -xzf /app/files/OpenJDK8U-jre_x64_linux_hotspot_8u265b01.tar.gz
tar -xzf /app/files/OpenJDK11U-jre_x64_linux_hotspot_11.0.8_10.tar.gz
tar -xzf /app/files/OpenJDK15U-jre_x64_linux_hotspot_15_36.tar.gz

# myApp Tomcat
cd /app/myApp
tar -xzf /app/files/apache-tomcat-8.5.59.tar.gz
tar -xzf /app/files/apache-tomcat-9.0.39.tar.gz

# Certificate
cp /app/files/corporate_truststore.jks /app/certs/

# SAP JCO
cp /app/files/_sapjco3-64/3.0.19/linuxx86/libsapjco3.so /app/library/
cp /app/files/_sapjco3-64/3.0.19/linuxx86/sapjco3.jar /app/library/

Setup Tomcat

su myUser

# Symlink to actual Tomcat version
ln -s /app/ccp/apache-tomcat-8.5.59 /app/myApp/tomcat

# remove sample application
# but keep the Tomcat Manager app for deployment
rm -rf /app/myApp/tomcat/webapps/docs
rm -rf /app/myApp/tomcat/webapps/examples
rm -rf /app/myApp/tomcat/webapps/ROOT

# configure Tomcat
vim /app/myApp/tomcat/bin/setenv.sh
mv /app/myApp/tomcat/conf/server.xml /app/myApp/tomcat/conf/server.xml_original
vim /app/myApp/tomcat/conf/server.xml

# expand Classpath
vim /app/myApp/tomcat/conf/catalina.properties

# tomcat/bin/setenv.sh
CATALINA_HOME=/app/myApp/tomcat
CATALINA_BASE=/app/myApp/tomcat
CATALINA_PID=/app/myApp/tomcat/tomcat.pid
JAVA_HOME=/app/java/jdk8u265-b01-jre
CATALINA_OPTS="$CATALINA_OPTS -Djava.library.path=/app/library"
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/app/library
export LD_LIBRARY_PATH
JAVA_OPTS="${JAVA_OPTS} -Djavax.net.ssl.trustStore=/app/certs/corporate_truststore.jks -Djavax.net.ssl.trustStorePassword=notchangeit -Xms512M -Xmx2048M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/app/myApp/dump"

# tomcat/conf/catalina.properties
common.loader= [...] ,"/app/library/sapjco3.jar"

# server.xml

Tomcat as a Service

# As root
vim /etc/systemd/system/tomcat.service
# enable script:
systemctl enable tomcat.service

# tomcat.service

# Systemd unit file for myApp tomcat
#
# To create clones of this service:

# Systemd unit file for tomcat
 [Unit]
 Description=myApp Tomcat Web Application Container
 After=syslog.target network.target

 [Service]
 Type=forking

 ExecStart=/app/myApp/tomcat/bin/startup.sh
 ExecStop=/app/myApp/tomcat/bin/shutdown.sh

 User=myUser
 Group=myUser

 [Install]
 WantedBy=multi-user.target

Service control

Enable user myUser to control Tomcat services:

visudo -f /etc/sudoers
##################################################
## Allow user myUser to control (apache & tomcat) services
%myUser ALL=(root) NOPASSWD: /bin/systemctl
%myUser ALL=(root) NOPASSWD: /usr/sbin/service

Test Tomcat

For Tomcat testing I use my ShowHeaders app (GitHub).
ShowHeaders is a minimalistic webapp that is not much more than a "Hello World", but it shows the HTTP headers, what is quite useful when testing reverse proxy integration.

# copy ShowHeaders App for Testing (also for Reverse Proxy Configuration Testing)
[myUser@DEV ~]$ cp /app/files/ShowHeaders/ROOT.war /app/myApp/tomcat/webapps/
# start tomcat:
[myUser@DEV ~]$ sudo systemctl start tomcat

#
curl localhost:7011

Test connection from outside the server itself: http://myAppServer:7011/

Tags Tomcat

Linux

Copy files with rsync

I have to setup and configure a DEV server and afterwards the QA/staging and PROD servers.

I will download required software with my laptop and upload to DEV server into folder /app/files. For other servers I just want to copy this folder from the DEV server.

Setup server

DEV

# install software
yum install rsync -y

# create user
adduser myUser

# create folder
mkdir /app/files
chown myUser:myUser /app/files

# switch to myUser
su myUser

# Create Private & Public keys
ssh-keygen -t rsa -b 4096            # generates id_rsa & id_rsa.pub in  /home/myUser/.ssh/

# authorize yourself
cat /home/myUser/.ssh/id_rsa.pub >> /home/myUser/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

QA / others

nearly the same setup, but we copy the keys from the DEV user, as he is the same as on the other servers.

# install software
yum install rsync -y

# create user
adduser myUser

# create folder
mkdir /app/files
chown myUser:myUser /app/files

# switch to myUser
su myUser

# create .ssh folder
mkdir ~/.ssh
chmod 0700 ~/.ssh

# create files, copy content from DEV server!
vi ~/.ssh/id_rsa
chmod 0600 ~/.ssh/id_rsa
vi ~/.ssh/id_rsa.pub
chmod 0600 ~/.ssh/id_rsa.pub
vi ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys

Test SSH login

# login from DEV to QA
[myUser@DEV] ssh QA
# login from QA to DEV
[myUser@QA] ssh DEV

# loggin with explicit port
[myUser@DEV] ssh QA -p 22

Download software

Copy all files to /app/files

Java

Open URL https://adoptopenjdk.net/

Click on "Andere Plattformen"

Select for each:

JVM: HotSpot
Operating System: Linux
Architecture: x64

Java Versions:

OpenJDK 8 (LTS)
OpenJDK 11 (LTS)
OpenJDK 15 (Latest)

Download the JRE files.

Tomcat

Open URL: http://tomcat.apache.org/

Download Binary Distribution → Core → tar.gz-Version of:

Tomcat 8.5
Tomcat 9

SAP JCO

Copy the latest version from the corporate repository.

Path: _sapjco3-64/versions/3.0.19

Corporate Root CA

Copy the password ("notchangeit") protected truststore: corporate_truststore.jks

Copy from server to server

[myUser@QA ~]$ rsync --progress -avhe 'ssh -p 22' myUser@DEV:/app/files/ /app/files