https://pprakash.me/tags/glue/ Recent content in Glue on Technology Enlightened Hugo en &copy; <a href="https://github.com/p-prakash/">Prakash P</a> 2015 - 2023 Thu, 06 Sep 2018 07:55:46 +0000 https://pprakash.me/tech/2018/09/06/running-aws-glue-jobs-connecting-to-database-in-vpc-with-custom-dns/ Thu, 06 Sep 2018 07:55:46 +0000 https://pprakash.me/tech/2018/09/06/running-aws-glue-jobs-connecting-to-database-in-vpc-with-custom-dns/ <p>AWS Glue is a fully managed ETL service that makes it simple and cost-effective to categorize your data, clean it, enrich it, and move it reliably between various data stores. Glue job is the business logic that automate the extract, transform, and transfer data to different locations.</p> <p>Glue job creates a Hadoop task in the backend and since it uses Hadoop ecosystem all the nodes in the cluster must be able to find using the FQDN and communicate with each other. To satisfy Hadoop DNS lookup, both forward and reverse lookups must succeed. When you use the default DHCP option set with AmazonProvidedDNS, it automatically creates the DNS record for the nodes that’s getting created in the VPC.</p>