Wednesday, 4 October 2017

Dynamic Schema Load in Talend

Scenario

I want to Load 10 table from database 1 to database 2. these 10 tables having different schema .
I need to read table schema dynamically from database 1 and create table and insert the data into database 2.

Here i am going to Read schema from each table Dynamically. Dynamic schema we can use for only for insert the data and we cant  perform any validation on data.

Please see the job design




  • Take mysqlinput connection component for source and target DB's and configure the settings like hostname,server,user,password and db name.
tMysqlTablelist
  • Take TmysqlTablelist component to retrieve all tables from the Source DB.To retrieve all table write query as 'table_name' like '%' in the component and configure the settings as below.



tMysqlinput
  • tMysqlInput reads a database and extracts fields based on a query.
Mention the query as below
" select  *  from  " +((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE"))+";" in the tmysqlinput component.

((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE")  variable which retrieves all tables in the Database.
  • click on the edit schema of  tMysqlInput component add a field like data and mention schema type as dynamic as mentioned  in the below screenshot.


tMysqloutput
  • Take tMysqloutput component and mention ((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE") in the table name slot. 
  • click on Sync columns.
Run the Job
job will read the table by table dynamically and load the data into target database.


***********************THANK YOU*****************************




Tuesday, 19 September 2017

Reading and inserting Arabic using Talend

·         Arabic Data supports UTF-8 characters
·         UTF-8 encodes each Unicode character as a variable number of 1 to 4 octets, where the number of octets depends on the integer value assigned to the Unicode character. It is an efficient encoding of Unicode documents that use mostly US-ASCII characters because it represents each character in the range U+0000 through U+007F as a single octet
Steps
·         First we need create Table to accept UTF-8 encoding format field.
·         For example I need to load the Arabic names into Name filed so the create statement will be like
o   “create table test.L_Arabic (Id int,Name Varchar(255)CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL);”
·         Design a Talend job
·         tFixedFlowInPut-àtMysqloutput


Configure tMysqloutput with required parameter.
We need to put in Advance setting tab"useUnicode=true&characterEncoding=UTF-8"


Save and Run the job.
Output


Tuesday, 21 March 2017

Sqoop Import Data from MYSQL to HDFS By using Talend(Error: customer_sqoop : Unsupported major.minor version 52.0)

I have configured the tsqoopImport component as given in Talend document.

ISSUE:

Error: customer_sqoop : Unsupported major.minor version 52.0
Error: customer_sqoop : Unsupported major.minor version 52.0
Error: customer_sqoop : Unsupported major.minor version 52.0

[WARN ]: mapreduce.Counters - Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
Exception in component tSqoopImport_1
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
at mdm_basics.sqoopimport_0_1.SqoopImport.tSqoopImport_1Process(SqoopImport.java:515)
at mdm_basics.sqoopimport_0_1.SqoopImport.runJobInTOS(SqoopImport.java:819)
at mdm_basics.sqoopimport_0_1.SqoopImport.main(SqoopImport.java:653)
[ERROR]: org.apache.sqoop.tool.ImportTool - Error during import: Import job failed!
[FATAL]: mdm_basics.sqoopimport_0_1.SqoopImport - tSqoopImport_1 The Sqoop import job has failed. Please check the logs.
[statistics] disconnected
Job SqoopImport ended at 12:26 21/03/2017. [exit code=1]

Solution:
I am runnig Talend6.1.1 studio on JAVA8 version but my hadoop cluster is on JAVA7.

so i have changed the JDK path in talend studio i.e Window--->Preferences-->Talend.



Make sure to run the sqoop import job the java interpreter points to the JDK. Default the interpreter points to the JRE.





Sunday, 15 June 2014

Swing Programme for Generate Sitcker with Barcode

import java.awt.EventQueue;
import java.awt.Font;
import java.awt.Graphics;
import java.awt.Graphics2D;
import javax.imageio.ImageIO;
import javax.swing.JFrame;
import javax.swing.JPanel;
import javax.swing.JButton;
import java.awt.event.ActionListener;
import java.awt.event.ActionEvent;
import java.awt.image.BufferedImage;
import java.awt.print.PageFormat;
import java.awt.print.Printable;
import java.awt.print.PrinterException;
import java.awt.print.PrinterJob;
import javax.swing.JTextField;
import java.awt.Color;
import javax.swing.ImageIcon;
import javax.swing.JLabel;
import javax.swing.UIManager;
import java.awt.SystemColor;
import org.eclipse.wb.swing.FocusTraversalOnArray;
import java.awt.Component;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.UUID;
import javax.swing.JTextArea;
import javax.swing.SwingConstants;

public class Sticker extends JFrame {

    private static final long serialVersionUID = 1L;
    private JTextField textField;
    private JTextField textField_1;
    private JTextField textField_2;
    private JTextField textField_3;
    private JTextField textField_4;
    private JTextField textField_5;
  

    /**
     * Launch the application.
     */
    public static void main(String[] args) {
        EventQueue.invokeLater(new Runnable() {
            public void run() {
                try {
                    Sticker frame = new Sticker();
                    frame.setVisible(true);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        });
    }

    /**
     * Create the frame.
     */
    public Sticker() {
        getContentPane().setBackground(new Color(169, 169, 169));
        getContentPane().setForeground(SystemColor.menu);
        setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
        setBounds(200, 200, 450, 348);
        getContentPane().setLayout(null);

        textField = new JTextField();
        textField.setBounds(155, 39, 109, 17);
        getContentPane().add(textField);
        textField.setColumns(10);

        JLabel lblPanNo = new JLabel("ASK NO :");
        lblPanNo.setBounds(10, 41, 84, 14);
        getContentPane().add(lblPanNo);

        JLabel lblNewLabel = new JLabel("Date :");
        lblNewLabel.setBounds(10, 78, 46, 14);
        getContentPane().add(lblNewLabel);

        textField_1 = new JTextField();
        textField_1.setBounds(155, 75, 109, 20);
        getContentPane().add(textField_1);
        textField_1.setColumns(10);

        JLabel lblNewLabel_1 = new JLabel("PAN :");
        lblNewLabel_1.setBounds(10, 115, 46, 14);
        getContentPane().add(lblNewLabel_1);

        textField_2 = new JTextField();
        textField_2.setColumns(10);
        textField_2.setBounds(155, 112, 109, 20);
        getContentPane().add(textField_2);

        JLabel lblName = new JLabel("NAME :");
        lblName.setBounds(10, 154, 46, 14);
        getContentPane().add(lblName);

        textField_3 = new JTextField();
        textField_3.setColumns(10);
        textField_3.setBounds(155, 151, 109, 20);
        getContentPane().add(textField_3);

        JLabel lblRrbin = new JLabel("RR/BIN:");
        lblRrbin.setBounds(10, 193, 46, 14);
        getContentPane().add(lblRrbin);

        textField_4 = new JTextField();
        textField_4.setColumns(10);
        textField_4.setBounds(155, 190, 109, 20);
        getContentPane().add(textField_4);

        JLabel lblAo = new JLabel("AO :");
        lblAo.setBounds(10, 237, 46, 14);
        getContentPane().add(lblAo);

        textField_5 = new JTextField();
        textField_5.setColumns(10);
        textField_5.setBounds(155, 234, 109, 20);
        getContentPane().add(textField_5);

        // ADD Submit Button

        JButton btnSubmit = new JButton("Submit");
        btnSubmit.setBackground(new Color(123, 104, 238));
        btnSubmit.setBounds(79, 276, 89, 23);
        getContentPane().add(btnSubmit);
      
        JLabel lblWelcome = new JLabel("WELCOME");
        lblWelcome.setForeground(new Color(153, 51, 204));
        lblWelcome.setFont(new Font("Tahoma", Font.PLAIN, 15));
        lblWelcome.setBounds(163, 11, 179, 14);
        getContentPane().add(lblWelcome);
        getContentPane().setFocusTraversalPolicy(
                new FocusTraversalOnArray(
                        new Component[] { textField, lblPanNo }));

        btnSubmit.addActionListener(new ActionListener() {

            public void actionPerformed(ActionEvent e) {

                JFrame frame2 = new JFrame("Details Page");
                frame2.setLocationByPlatform(true);

                JPanel contentPane2 = new JPanel();

                contentPane2.setBackground(Color.GRAY);
                JTextArea ta = new JTextArea("Total Information", 5, 20);
                frame2.getContentPane().add(contentPane2);
                contentPane2.add(ta);

                frame2.getContentPane().add(contentPane2);
                frame2.setSize(300, 300);
                frame2.setVisible(true);
                ta.setEditable(false);
                ta.setBackground(Color.GRAY);
                ta.setForeground(Color.white);
                try {

                    System.out.println("you clicked Button");

                    int width, height;
                    String temp = "";
                    File saveFile = new File(
                            "E:/Sticker/Input/barcode_image.jpg");
                    String format = new String("jpg");
                    width = 250;
                    height = 50;

                    BufferedImage bufimg = new BufferedImage(width, height,
                            BufferedImage.TYPE_INT_RGB);
                    Graphics graphicsobj = bufimg.createGraphics();

                    File file1 = new File("E:/Sticker/Input/3of9.TTF");
                    FileInputStream fin = new FileInputStream(file1);
                    Font font = Font.createFont(Font.TRUETYPE_FONT, fin);
                    Font font1 = font.deriveFont(46f);

                    graphicsobj.setFont(font1);
                    graphicsobj.setFont(Font.getFont("3 of 9 Barcode"));
                    graphicsobj.setColor(Color.WHITE);

                    graphicsobj.fillRect(1, 1, 248, 48);
                    graphicsobj.setColor(Color.BLACK);
                  
                    UUID uid = UUID
                            .fromString("38400000-8cf0-11bd-b23e-10b96e4ef00d");

                    ((Graphics2D) graphicsobj).drawString(
                            "" + uid.randomUUID(), 0, 40);

                    ImageIO.write(bufimg, format, saveFile);

                    String imgstr = "E:/Sticker/Input/barcode_image.jpg";
                    ImageIcon image1 = new ImageIcon(imgstr);
                    JLabel lable2 = new JLabel("", image1, JLabel.CENTER);
                    contentPane2.add(lable2);
                  
                    // Generate Sticker Formate

                    int width1, height1;
                    File saveFile1 = new File(
                            "E:/Sticker/Output/final_bar_image.jpg");
                    String format1 = new String("jpg");
                  
                    width1 = 275;
                    height1 = 225;

                    BufferedImage bufimg1 = new BufferedImage(width1, height1,
                            BufferedImage.TYPE_INT_RGB);
                    Graphics graphicsobj1 = bufimg1.createGraphics();

                    graphicsobj1.setColor(Color.WHITE);
                    graphicsobj1.fillRect(0, 0, width1, height1);
                    graphicsobj1.setColor(Color.BLACK);
                    StringBuffer br = new StringBuffer();
                    temp = "AskNo     : " + textField.getText() + "\n";
                    br.append(temp);
                    textField.setText("");
                    ((Graphics2D) graphicsobj1).drawString(temp, 5, 20);
                    temp = "Date        : " + textField_1.getText() + "\n";
                    br.append(temp);
                    textField_1.setText("");
                    ((Graphics2D) graphicsobj1).drawString(temp, 5, 40);
                    temp = "PANNO   : " + textField_2.getText() + "\n";
                    br.append(temp);
                    textField_2.setText("");
                    ((Graphics2D) graphicsobj1).drawString(temp, 5, 60);
                    temp = "NAME      : " + textField_3.getText() + "\n";
                    br.append(temp);
                    textField_3.setText("");
                    ((Graphics2D) graphicsobj1).drawString(temp, 5, 80);
                    temp = "RR/BIN   : " + textField_4.getText() + "\n";
                    br.append(temp);
                    textField_4.setText("");
                    ((Graphics2D) graphicsobj1).drawString(temp, 5, 100);
                    temp = "AO           : " + textField_5.getText() + "\n";
                    br.append(temp);
                    textField_5.setText("");
                    ((Graphics2D) graphicsobj1).drawString(temp, 5, 120);
              
                    BufferedImage read = ImageIO.read(new File(
                            "E:/Sticker/Input/barcode_image.jpg"));
                    ((Graphics2D) graphicsobj1).drawImage(read, 10, 140, null);
                    ta.setText(br.toString());
                  
                  
                  
                    graphicsobj1.dispose();

                    ImageIO.write(bufimg1, format1, saveFile1);

                    UIManager.setLookAndFeel(UIManager
                            .getSystemLookAndFeelClassName());
                    JButton btnPrint = new JButton("print");
                    contentPane2.add(btnPrint);
                    btnPrint.setBounds(5, 5, 100, 25);
              
                    btnPrint.addActionListener(new btnPrintAction());

                } catch (Exception Exp) {
                    Exp.printStackTrace();
                }
            }
        });
    }

    public static class btnPrintAction implements ActionListener, Printable {
      
        public int print(Graphics gx, PageFormat pf, int page)
                throws PrinterException {

            if (page > 0) {
                return NO_SUCH_PAGE;
            } // Only one page

            Graphics2D g = (Graphics2D) gx; // Cast to Graphics2D object

            g.translate(pf.getImageableX(), pf.getImageableY()); // Match
                                                                    // origins
                                                                    // to
                                                                    // imageable
                                                                    // area

            // Print Hello World at offset (100, 100)
            try {
                BufferedImage read = ImageIO.read(new File(
                        "E:/Sticker/Output/final_bar_image.jpg"));
                g.drawImage(read, 20, 100, null);
            } catch (IOException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }

            return PAGE_EXISTS; // Page exists (offsets start at zero!)
        }

        @Override
        public void actionPerformed(ActionEvent e) {
            PrinterJob job = PrinterJob.getPrinterJob(); // Get the printer's
                                                            // job list

            job.setPrintable(this); // We print with this class (btnPrintAction,
                                    // which implements Printable)

            if (job.printDialog() == true) { // If we clicked OK in the print
                                                // dialog

                try {
                    job.print();
                } catch (PrinterException ex) {

                    // It did not work (PrinterException thrown), so add any
                    // error handling routines.

                }
            }
        }
    }
}

java.io.IOException: org.eclipse.aether.deployment.DeploymentException: Failed to deploy artifacts: Could not transfer artifact Return code is: 400, ReasonPhrase: Repository does not allow updating assets: releases.

 HI, by Default in Nexus release branch will not allow redeploy of the Same version of Talend job. if we need to redeploy same job again int...