Wednesday, 4 October 2017

Dynamic Schema Load in Talend

Scenario

I want to Load 10 table from database 1 to database 2. these 10 tables having different schema .
I need to read table schema dynamically from database 1 and create table and insert the data into database 2.

Here i am going to Read schema from each table Dynamically. Dynamic schema we can use for only for insert the data and we cant  perform any validation on data.

Please see the job design




  • Take mysqlinput connection component for source and target DB's and configure the settings like hostname,server,user,password and db name.
tMysqlTablelist
  • Take TmysqlTablelist component to retrieve all tables from the Source DB.To retrieve all table write query as 'table_name' like '%' in the component and configure the settings as below.



tMysqlinput
  • tMysqlInput reads a database and extracts fields based on a query.
Mention the query as below
" select  *  from  " +((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE"))+";" in the tmysqlinput component.

((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE")  variable which retrieves all tables in the Database.
  • click on the edit schema of  tMysqlInput component add a field like data and mention schema type as dynamic as mentioned  in the below screenshot.


tMysqloutput
  • Take tMysqloutput component and mention ((String)globalMap.get("tMysqlTableList_2_CURRENT_TABLE") in the table name slot. 
  • click on Sync columns.
Run the Job
job will read the table by table dynamically and load the data into target database.


***********************THANK YOU*****************************




15 comments:

  1. It is asking for primary key.. How to pass primary key as well ?

    ReplyDelete
    Replies
    1. Talend Data Integration And Talend Mdm And Talend Big Data : Dynamic Schema Load In Talend >>>>> Download Now

      >>>>> Download Full

      Talend Data Integration And Talend Mdm And Talend Big Data : Dynamic Schema Load In Talend >>>>> Download LINK

      >>>>> Download Now

      Talend Data Integration And Talend Mdm And Talend Big Data : Dynamic Schema Load In Talend >>>>> Download Full

      >>>>> Download LINK XW

      Delete
  2. Hi Chiranjeevi,

    i think you have selected update option in the output component as only insert works for the above scenario.
    please change it to insert and try to run the job.

    ReplyDelete
  3. This solution is for enterprise version. Is there any solution for Open Source Application?

    ReplyDelete
    Replies
    1. I am not sure in open source i have did in Enterprise version

      Delete
  4. How does this feature works for Big data Batch job? I cannot see for tFileOutputParquet , the dynamic feature. Any workaround for this?

    ReplyDelete
  5. I will really appreciate the writer's choice for choosing this excellent article appropriate to my matter.Here is deep description about the article matter which helped me more.
    data analytics course in hyderabad
    business analytics course in hyderabad
    data science course in hyderabad

    ReplyDelete
  6. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own Blog Engine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it.
    data science course in guwahati

    ReplyDelete
  7. We play a small role in upskilling people providing the latest tech courses. Join us to upgrade on Talend Big Data training

    ReplyDelete
  8. Great read! Thank you for such useful insights. Visit here for latest tech courses on Talend Big Data training

    ReplyDelete
  9. "I am looking for and I love to post a comment thatExcelR Online data science training
    ""The content of your post is awesome"" Great work!"

    ReplyDelete
  10. Talend Data Integration And Talend Mdm And Talend Big Data : Dynamic Schema Load In Talend >>>>> Download Now

    >>>>> Download Full

    Talend Data Integration And Talend Mdm And Talend Big Data : Dynamic Schema Load In Talend >>>>> Download LINK

    >>>>> Download Now

    Talend Data Integration And Talend Mdm And Talend Big Data : Dynamic Schema Load In Talend >>>>> Download Full

    >>>>> Download LINK

    ReplyDelete

java.io.IOException: org.eclipse.aether.deployment.DeploymentException: Failed to deploy artifacts: Could not transfer artifact Return code is: 400, ReasonPhrase: Repository does not allow updating assets: releases.

 HI, by Default in Nexus release branch will not allow redeploy of the Same version of Talend job. if we need to redeploy same job again int...