Uploaded image for project: 'Teiid'
  1. Teiid
  2. TEIID-6017

Not able to connect GCP bucket using S3 translator

    • Icon: Feature Request Feature Request
    • Resolution: Not a Bug
    • Icon: Major Major
    • None
    • None
    • None
    • None

      As suggested in the ticket TEIID-6015

      We have tried to connect the GCP bucket using the S3 translator(not amazon-s3).

      As mentioned, S3  doesn't support google authentication,

      so we have created access key and secret key of Google bucket by logging into the GCP accoudnt.

      We have followed below steps to create access key and secret key.

      Storage->Setting->INTEROPERABILITY-> Service account HMAC->Create a key for service account"

       

      We have also created below adapter file and VDB but still, we are not able to connect to GCP.

        

      <resource-adapter id="amazon-s3-custom">
      <module slot="main" id="org.jboss.teiid.resource-adapter.s3"/>
      <transaction-support>NoTransaction</transaction-support>
      <connection-definitions>
      <connection-definition class-name="org.teiid.resource.adapter.s3.S3ManagedConnectionFactory" jndi-name="java:/s3DS" enabled="true" use-java-context="true" pool-name="teiid-s3-ds">
      <config-property name="bucket">abc</config-property>
      <config-property name="endpoint">https://storage.googleapis.com</config-property>
      <config-property name="secretKey">xxx</config-property>
      <config-property name="accessKey">xxx</config-property>
      </connection-definition>
      </connection-definitions>
      </resource-adapter>

       

       

       

      VDB


       

      <?xml version="1.0"?>
      <vdb name="gcp" version="1">
      <connection-type>BY_VERSION</connection-type>
      <model name="s3">
      <source name="web-connector" translator-name="amazon-s3-custom" connection-jndi-name="java:/s3DS"/>
      </model>

      <translator name="amazon-s3-custom" type="amazon-s3">
      <property name="accesskey" value="xxx"/>
      <property name="secretkey" value="xxx"/>
      <property name="bucket" value="abc"/>
      </translator>
      </vdb

       

       

      as we tried to connect we are getting below error.

       

      "org.teiid.jdbc.TeiidSQLException: TEIID20018 Unable to find a component used authenticate on to Teiid"

       

      In the above configuration, where should we mention the File name?

      As I have read the properties in the documentation, I could see bucket property, where we can add a bucket name. but where should we add the file name which is present in the bucket?

        

      Could you please help us with it?

       

       

       

       

            [TEIID-6017] Not able to connect GCP bucket using S3 translator

            Ramesh Reddy added a comment - - edited

            1) I copied your VDB to a file name "gcp-vdb.ddl"

            2) I edited the "standalone/configuration/standalone-teiid.xml" added the resource adapter in there in the resource adapters section.

            3) Then I started a CLI console by running "bin/jboss-cli.sh --connect"

            4) Then deployed the vdb using the command

            deploy /path/gcp-vdb.ddl 

            That deployed the vdb without any errors. Then you can use the SQL client like SQuirrel to connect to "gcp" as the VDB name and issue queries. I did not do this step as I do not have the passwords to connect to GCP.

             

            Ramesh Reddy added a comment - - edited 1) I copied your VDB to a file name "gcp-vdb.ddl" 2) I edited the "standalone/configuration/standalone-teiid.xml" added the resource adapter in there in the resource adapters section. 3) Then I started a CLI console by running "bin/jboss-cli.sh --connect" 4) Then deployed the vdb using the command deploy /path/gcp-vdb.ddl That deployed the vdb without any errors. Then you can use the SQL client like SQuirrel to connect to "gcp" as the VDB name and issue queries. I did not do this step as I do not have the passwords to connect to GCP.  

            Nayan Bija (Inactive) added a comment - - edited

            I have put the same content in the VDB then try to start the server.

            Please find the attached files.

            I have included VDB along with a resource adapter.

             

             

            teiid-data.zip

            Nayan Bija (Inactive) added a comment - - edited I have put the same content in the VDB then try to start the server. Please find the attached files. I have included VDB along with a resource adapter.     teiid-data.zip

            provide the complete vdb for examination, attach as file to the JIRA

            Ramesh Reddy added a comment - provide the complete vdb for examination, attach as file to the JIRA

            Hi,

             

            I have tried to deploy the VDB mentioned by you. by replacing our GCP file content but getting a parsing error.

            VDB

            =====

            CREATE DATABASE gcp OPTIONS (ANNOTATION 'GCP VDB');
            USE DATABASE gcp;
            CREATE SERVER mysite FOREIGN DATA WRAPPER file OPTIONS ( "resource-name" 'java:/s3DS');
            CREATE SCHEMA accounts SERVER mysite;

            IMPORT FROM SERVER mysite INTO accounts;
            SET SCHEMA accounts;

            CREATE VIEW stock_price (
            e1 string(255) NOT NULL,
            e2 string(255) NOT NULL
            )
            AS
            SELECT tt.e1, tt.e2
            FROM (EXEC accounts.getTextFiles('stock.txt')) AS f,
            TEXTTABLE(f.file COLUMNS e1 string, e2 string HEADER) AS tt;

             

             

            While starting the server getting below error.

             

             

            nus ONE +2' 9.4.18.Final
            0:22:28,048 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.deployment.unit."SvcSourceVdb_amazo
            -vdb.xml".PARSE: org.jboss.msc.service.StartException in service jboss.deployment.unit."SvcSourceVdb_amazon-vdb.xml".PARSE: WFLYSRV0153: Failed to pr
            cess phase PARSE of deployment "SvcSourceVdb_amazon-vdb.xml"
            at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:183)
            at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739)
            at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701)
            at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559)
            at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
            at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
            at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
            at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
            at java.lang.Thread.run(Thread.java:745)
            Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: TEIID50017
            at org.teiid.jboss.VDBParserDeployer.parseVDBXML(VDBParserDeployer.java:141)
            at org.teiid.jboss.VDBParserDeployer.deploy(VDBParserDeployer.java:68)
            at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:176)
            ... 8 more
            Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog.
            at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:201)
            at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:175)
            at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:398)
            at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:325)
            at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:282)
            at org.apache.xerces.impl.XMLScanner.reportFatalError(XMLScanner.java:1496)
            at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(XMLDocumentScannerImpl.java:870)
            at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324)
            at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:875)
            at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:798)
            at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(StreamValidatorHelper.java:186)
            at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(ValidatorImpl.java:129)
            at javax.xml.validation.Validator.validate(Validator.java:124)
            at org.teiid.adminapi.impl.VDBMetadataParser.validate(VDBMetadataParser.java:98)
            at org.teiid.jboss.VDBParserDeployer.parseVDBXML(VDBParserDeployer.java:111)
            ... 10 more

             

             

             

             

            Could you please suggest I am doing it in a proper way or not.

             

             

             

            Nayan Bija (Inactive) added a comment - Hi,   I have tried to deploy the VDB mentioned by you. by replacing our GCP file content but getting a parsing error. VDB ===== CREATE DATABASE gcp OPTIONS (ANNOTATION 'GCP VDB'); USE DATABASE gcp; CREATE SERVER mysite FOREIGN DATA WRAPPER file OPTIONS ( "resource-name" 'java:/s3DS'); CREATE SCHEMA accounts SERVER mysite; IMPORT FROM SERVER mysite INTO accounts; SET SCHEMA accounts; CREATE VIEW stock_price ( e1 string(255) NOT NULL, e2 string(255) NOT NULL ) AS SELECT tt.e1, tt.e2 FROM (EXEC accounts.getTextFiles('stock.txt')) AS f, TEXTTABLE(f.file COLUMNS e1 string, e2 string HEADER) AS tt;     While starting the server getting below error.     nus ONE +2' 9.4.18.Final 0:22:28,048 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.deployment.unit."SvcSourceVdb_amazo -vdb.xml".PARSE: org.jboss.msc.service.StartException in service jboss.deployment.unit."SvcSourceVdb_amazon-vdb.xml".PARSE: WFLYSRV0153: Failed to pr cess phase PARSE of deployment "SvcSourceVdb_amazon-vdb.xml" at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:183) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1739) at org.jboss.msc.service.ServiceControllerImpl$StartTask.execute(ServiceControllerImpl.java:1701) at org.jboss.msc.service.ServiceControllerImpl$ControllerTask.run(ServiceControllerImpl.java:1559) at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) at java.lang.Thread.run(Thread.java:745) Caused by: org.jboss.as.server.deployment.DeploymentUnitProcessingException: TEIID50017 at org.teiid.jboss.VDBParserDeployer.parseVDBXML(VDBParserDeployer.java:141) at org.teiid.jboss.VDBParserDeployer.deploy(VDBParserDeployer.java:68) at org.jboss.as.server.deployment.DeploymentUnitPhaseService.start(DeploymentUnitPhaseService.java:176) ... 8 more Caused by: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Content is not allowed in prolog. at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:201) at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:175) at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:398) at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:325) at org.apache.xerces.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:282) at org.apache.xerces.impl.XMLScanner.reportFatalError(XMLScanner.java:1496) at org.apache.xerces.impl.XMLDocumentScannerImpl$PrologDispatcher.dispatch(XMLDocumentScannerImpl.java:870) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:324) at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:875) at org.apache.xerces.parsers.XML11Configuration.parse(XML11Configuration.java:798) at org.apache.xerces.jaxp.validation.StreamValidatorHelper.validate(StreamValidatorHelper.java:186) at org.apache.xerces.jaxp.validation.ValidatorImpl.validate(ValidatorImpl.java:129) at javax.xml.validation.Validator.validate(Validator.java:124) at org.teiid.adminapi.impl.VDBMetadataParser.validate(VDBMetadataParser.java:98) at org.teiid.jboss.VDBParserDeployer.parseVDBXML(VDBParserDeployer.java:111) ... 10 more         Could you please suggest I am doing it in a proper way or not.      

            I understand, but failure to follow the directions there is only so much we can help!

             

            We are saying that forget that there is an "amazon-s3" translator, that has been deprecated. For your concern, there is no "amazon-s3" only "s3" resource adapter that works with the "file" translator. A resource-adapter makes the connection and translator let you use a resource-adapter to fetch data and convert data what you want. Then the "file" translator uses the "TEXTTABLE" semantics to convert a CSV data to a Table that can be queried.

             

            For "s3" resource-adapter, I have given the above how to configure it. Throw away your current VDB, and use the one I suggested and modify to meet your needs. You only need to add the "View" by replacing "stock_price" with your own that suits to your own CSV file format to make it work. Once you have the modifications, deploy it, and connect using Squirrel as you did for the DUMMY vdb and issue queries.

             

            Ramesh Reddy added a comment - I understand, but failure to follow the directions there is only so much we can help!   We are saying that forget that there is an "amazon-s3" translator, that has been deprecated. For your concern, there is no "amazon-s3" only "s3" resource adapter that works with the "file" translator. A resource-adapter makes the connection and translator let you use a resource-adapter to fetch data and convert data what you want. Then the "file" translator uses the "TEXTTABLE" semantics to convert a CSV data to a Table that can be queried.   For "s3" resource-adapter, I have given the above how to configure it. Throw away your current VDB, and use the one I suggested and modify to meet your needs. You only need to add the "View" by replacing "stock_price" with your own that suits to your own CSV file format to make it work. Once you have the modifications, deploy it, and connect using Squirrel as you did for the DUMMY vdb and issue queries.  

            Actually I am completely new to teiid so I am getting confused about the Amazon-s3 translator and S3 source translator. I have designed vdb and adapter as you suggested by you.

            I am not getting about the S3 source
            Could you please help me into it

            Nayan Bija (Inactive) added a comment - Actually I am completely new to teiid so I am getting confused about the Amazon-s3 translator and S3 source translator. I have designed vdb and adapter as you suggested by you. I am not getting about the S3 source Could you please help me into it

            Yes I am using Teiid version 15.

            Nayan Bija (Inactive) added a comment - Yes I am using Teiid version 15.

            nayanbijagare I have given you all the complete instructions above along with sample VDB, however it looks like you are not following the directions and asking the same questions.

             

            You used different VDB, and are using Teiid version 15?

            Ramesh Reddy added a comment - nayanbijagare I have given you all the complete instructions above along with sample VDB, however it looks like you are not following the directions and asking the same questions.   You used different VDB, and are using Teiid version 15?

            Could you please tell me how to use S3 source.
            I want to read TXT and CSV file here.
            Actually I couldn't find S3 source translator in Teiid.

            Nayan Bija (Inactive) added a comment - Could you please tell me how to use S3 source. I want to read TXT and CSV file here. Actually I couldn't find S3 source translator in Teiid.

            The amazon-s3 translator is now only for legacy / backwards compatibility. You do not use the amazon-s3 translator with the s3 source. An s3 source supports the file translator types - currently file and excel types.

            Steven Hawkins added a comment - The amazon-s3 translator is now only for legacy / backwards compatibility. You do not use the amazon-s3 translator with the s3 source. An s3 source supports the file translator types - currently file and excel types.

              rhn-engineering-shawkins Steven Hawkins
              nayanbijagare Nayan Bija (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: