Tuesday, 3 September 2019

How to configure Implicit context in Talend With Encrypt and Decrypt the password values


Context variables:

Context describes the user-defined parameters that are passed to your Job at runtime. Context Variables are the values that may change as you promote your Job from Development, through to Test and Production. Values may also change as your environment changes, for example, passwords may change from time to time.

In Talend context variables can create two ways
  •  Local variables:  create in job level that are only used in the corresponding job. 
  • Global variables: create in Talend Repository that are used in throughout project

How can we pass the Values for Context variables? 
  •   We can hardcode the values in Talend.
  •   We can use tContextLoad component in each job t get the values from file.

And we have one more way to pass the values for context variables in Talend Project. i.e. implicit context


How to configure implicit context in Talend?

We can configure implicit context with files or Database. If you use file or database we need to have two Main columns i.e. key and value.

If you use table for maintain context variables. Just create a table with two columns i.e. key and values.

CREATE TABLE `tbl_implici_context` (`key` varchar (255) DEFAULT NULL, ‘value` varchar (255) DEFAULT NULL) ENGINE=InnoDB DEFAULT CHARSET=latin1;

The charset should be latin1 because it should accept all special characters.

 Generally the most of the customer Expect to maintain security on their values. So they don’t want expose their application passwords. So they need to encrypt all the passwords.

So while loading all variables and values all values for password fields should be encrypted. For this Encryption I have used database encryption algorithm.

i.e. AES_ENCRYPT and AES_DECRYPT

Create a Talend job to populate the implicit context table. For this use case I have taken tFixedflowInput component as source.



 Use tJavaRow1 component to generate insert statement

if(input_row.key.matches(".*password.*"))
{
context.username= "insert into tbl_implici_context values('"+input_row.key+"',AES_ENCRYPT('"+input_row.value+"','mykeystring'));";
System.out.println(context.username);
}
else
{
context.username= "insert into tbl_implici_context values('"+input_row.key+"','"+input_row.value+"');";
System.out.println(context.username);
}


Use tMysqlRow component to execute this insert statement



Run the Job



Check the Results


Configure implicit context in project settings with Decrypt the password values.

In Talend StudioàProject settingsà Job settings


In the Query section we should use the Decrypt methods. Here I have written one Query to decrypt the passwords.

Generally the Query condition option only take where clause.

"`key` not like '%password%'  union  select `key`, case when `key` like '%password%' then AES_DECRYPT(value,'mykeystring') else value  END value  from test.tbl_implici_context"
****************************THANK YOU******************************

Thursday, 9 May 2019

GC Overhead limit Exceeded in TALEND JOBS

In Many Talend jobs we face this issue when our job reach maximum memory.
we can resolve this issue by adding the JVM parameter(-XX:-UseGCOverheadLimit) in Talend run Job settings.

refer the below screens



Saturday, 9 February 2019

How to install Talend BPM on Tomcat


 How to install Talend BPM on Tomcat

1.       Download ApacheTomcat6  zip folder from the following web site.
Unzip into a folder and use this path as <TOMCAT_HOME>
2.       Download server files
In the bundle section on the page, download the deploy zip for your version
3.       For Talend 5.3 we need to download  BOSVersion5.10 i.e BOS-5.10-deploy.
4.       Unzip the file into the folder of your choice.
This location path will be referenced as <Talend BPM-DEPLOY> in the installation guides.
5.       Copy Talend conf folders
copy <Talend BPM-DEPLOY>\conf\bonita to <TOMCAT_HOME>\bonita
copy <Talend BPM-DEPLOY>\conf\external to <TOMCAT_HOME>\external
6.       Copy BPM Execution Engine libraries
·         Create a folder <TOMCAT_HOME>\lib\bonita.
·         Copy all *.jar files from
·         <Talend BPM-deploy>\bonita_execution_engine\engine\libs
·         <Talend BPM-deploy>\bonita_execution_engine\bonita_client\libs
·         NOTE: Please overwrite duplicate files if any.
7.    Modify <TOMCAT_HOME>\conf\catalina.properties  by adding ${catalina.home}/lib/bonita/*.jar to the property common.loader
Result:
common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/
lib/*.jar,${catalina.home}/lib/bonita/*.jar

8.       Deploy User Experience web application (*.war file)
Copy
<Talend BPM-5.3 DEPLOY>\bonita_user_experience\without_execution_engine_without_client\bonita.war
to
<TOMCAT_HOME>\webapps

Important for Talend BPM: Remove the jdtcore*.jar file
To do this, open the bonita.war file (using an unzip tool) and remove jdt-core*.jar from the WEB-INF/lib folder.
9.        Define system variables
To configure Talend, system variables need to be defined. To do that, use a setenv script.
                For Windows:
 create <TOMCAT_HOME>\bin\setenv.bat to set variables:
Copy the following content into setenv.bat file and save it.
@echo on
rem Sets some variables
set BONITA_HOME="-DBONITA_HOME=%CATALINA_HOME%\bonita"
set LOG_OPTS="-Djava.util.logging.config.file=%CATALINA_HOME%\external\logging\logging.properties"
set SECURITY_OPTS="-Djava.security.auth.login.config=%CATALINA_HOME%\external\security\jaas-standard.cfg"
set JAVA_OPTS=%JAVA_OPTS% %LOG_OPTS% %SECURITY_OPTS% %BONITA_OPTS% %BONITA_HOME% -Dfile.encoding=UTF-8 -Xshare:auto
-Xms512m -Xmx1024m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError



10.   Finally Go to <TOMCAT_HOME>\bin  click on startup.bat to start the tomcat server.


No w the Tomcat server started .
While starting  tomcat first time all the bonita tables created respective Datasource like h2or mysql or sqlserver.


11.   Default it integrated with h2 databse.
If we want we can configure any one of the Datasource by configuring properties files
12.   Login
Access the login page at http://localhost:8080/bonita and the use the admin default account ("admin"/"bpm") to login.



Tuesday, 17 April 2018

JDBC Java Program to connect HIVE Database with Kerbose authentication

Hi

This article Explains to you how to write a JDBC program to connect Hive with Kerbose Authentication.

Here writing Java program is very simple and easy you can see the java code Below.
But I have struggled to get all the support jar files. here I have attached the lib file folder you can use directly.


Java Code:
=====================

import java.sql.*;
import java.sql.Statement;
import org.apache.hadoop.security.UserGroupInformation;

public class test {
  public static void main (String args[]) {
    try {
      org.apache.hadoop.conf.Configuration conf = new org.apache.hadoop.conf.Configuration();
      conf.set("hadoop.security.authentication", "kerberos");
     // conf.set(name, value);
   
      UserGroupInformation.setConfiguration(conf);
      System.out.println("started");
      UserGroupInformation.loginUserFromKeytab("cloudera@quickstart.cloudera", "keytabpath");
      Class.forName("org.apache.hive.jdbc.HiveDriver");
      System.out.println("getting connection");
      Connection con = DriverManager.getConnection("jdbc:hive2://quickstart.cloudera/default;principal=hive/_HOST@quickstart.cloudera;");
      System.out.println("got connection");
   
      Statement stmt = con.createStatement();
    String tableName = "testHiveDriverTable";
    stmt.execute("drop table if exists " + tableName);
    stmt.execute("create table " + tableName + " (key int, value string)");
      con.close();
    }
    catch (Exception e) {
      e.printStackTrace();
    }
  }
}
Download the source code and jar

Wednesday, 4 October 2017

Dynamic Schema Load in Talend

Scenario

I want to Load 10 table from database 1 to database 2. these 10 tables having different schema .
I need to read table schema dynamically from database 1 and create table and insert the data into database 2.

Here i am going to Read schema from each table Dynamically. Dynamic schema we can use for only for insert the data and we cant  perform any validation on data.

Please see the job design




  • Take mysqlinput connection component for source and target DB's and configure the settings like hostname,server,user,password and db name.
tMysqlTablelist
  • Take TmysqlTablelist component to retrieve all tables from the Source DB.To retrieve all table write query as 'table_name' like '%' in the component and configure the settings as below.



tMysqlinput
  • tMysqlInput reads a database and extracts fields based on a query.
Mention the query as below
" select  *  from  " +((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE"))+";" in the tmysqlinput component.

((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE")  variable which retrieves all tables in the Database.
  • click on the edit schema of  tMysqlInput component add a field like data and mention schema type as dynamic as mentioned  in the below screenshot.


tMysqloutput
  • Take tMysqloutput component and mention ((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE") in the table name slot. 
  • click on Sync columns.
Run the Job
job will read the table by table dynamically and load the data into target database.


***********************THANK YOU*****************************




Tuesday, 19 September 2017

Reading and inserting Arabic using Talend

·         Arabic Data supports UTF-8 characters
·         UTF-8 encodes each Unicode character as a variable number of 1 to 4 octets, where the number of octets depends on the integer value assigned to the Unicode character. It is an efficient encoding of Unicode documents that use mostly US-ASCII characters because it represents each character in the range U+0000 through U+007F as a single octet
Steps
·         First we need create Table to accept UTF-8 encoding format field.
·         For example I need to load the Arabic names into Name filed so the create statement will be like
o   “create table test.L_Arabic (Id int,Name Varchar(255)CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL);”
·         Design a Talend job
·         tFixedFlowInPut-àtMysqloutput


Configure tMysqloutput with required parameter.
We need to put in Advance setting tab"useUnicode=true&characterEncoding=UTF-8"


Save and Run the job.
Output


Tuesday, 21 March 2017

Sqoop Import Data from MYSQL to HDFS By using Talend(Error: customer_sqoop : Unsupported major.minor version 52.0)

I have configured the tsqoopImport component as given in Talend document.

ISSUE:

Error: customer_sqoop : Unsupported major.minor version 52.0
Error: customer_sqoop : Unsupported major.minor version 52.0
Error: customer_sqoop : Unsupported major.minor version 52.0

[WARN ]: mapreduce.Counters - Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
Exception in component tSqoopImport_1
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
at mdm_basics.sqoopimport_0_1.SqoopImport.tSqoopImport_1Process(SqoopImport.java:515)
at mdm_basics.sqoopimport_0_1.SqoopImport.runJobInTOS(SqoopImport.java:819)
at mdm_basics.sqoopimport_0_1.SqoopImport.main(SqoopImport.java:653)
[ERROR]: org.apache.sqoop.tool.ImportTool - Error during import: Import job failed!
[FATAL]: mdm_basics.sqoopimport_0_1.SqoopImport - tSqoopImport_1 The Sqoop import job has failed. Please check the logs.
[statistics] disconnected
Job SqoopImport ended at 12:26 21/03/2017. [exit code=1]

Solution:
I am runnig Talend6.1.1 studio on JAVA8 version but my hadoop cluster is on JAVA7.

so i have changed the JDK path in talend studio i.e Window--->Preferences-->Talend.



Make sure to run the sqoop import job the java interpreter points to the JDK. Default the interpreter points to the JRE.





java.io.IOException: org.eclipse.aether.deployment.DeploymentException: Failed to deploy artifacts: Could not transfer artifact Return code is: 400, ReasonPhrase: Repository does not allow updating assets: releases.

 HI, by Default in Nexus release branch will not allow redeploy of the Same version of Talend job. if we need to redeploy same job again int...