Monday, January 24, 2022

Azure Data Integration Pipelines – A Deep Dive

Must read

Paul Andrew
Group Manager & Analytics Architect specialising in big data solutions on the Microsoft Azure cloud platform. Data engineering competencies include Azure Synapse Analytics, Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack. Many years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry leading methods and technical design patterns. STEM ambassador and very active member of the data platform community delivering training and technical sessions at conferences both nationally and internationally. Father, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!

Having worked with Azure Data Factory and now Azure Synapse Analytics data integration pipelines ever since the resources first appeared in the Microsoft cloud platform, I’ve definitley earned (more than) my fair share of experience using the products. Both good and bad. As a data engineer and data archtiect.

To help soften those bad experiences, battle scares and developer pain. Part of what I enjoy in my job is sharing that knowledge with others so we can grow and learn together. Ultimately making all our experiences good. To that end, I’m consolidating everything I know about data pipelines and packaging it up into a complete 2 day training course Azure Data Integration Pipelines A Deep Dive. Which in December 2021 will be hosted by Orange Man.

Training course direct link here:

As you know if you’ve read my blog or seen me speak at community events, I don’t believe in talking about theoretical things in abstract ways. Everything I know and say on this topic comes from real world, hands on experience… Designing solutions and leading delivery teams for various multinational organisations.

If you’d like to take a trip to Denmark in December, you can expect two intense days covering the following areas.

  • Module 1: Pipeline Fundamentals
  • Module 2: Integration Runtime Design Patterns
  • Module 3: Data Transformation
  • Module 4: Metadata Driven Dynamic Pipelines
  • Module 5: Execution Parallelism
  • Module 6: Pipeline Extensibility
  • Module 7: VNet Integration
  • Module 8: Security
  • Module 9: Monitoring & Alerting
  • Module 10: CI/CD
  • Module 11: Solution Testing
  • Module 12: Wrap Up and Final Thoughts

As you’ll see, this is going to be a deep dive session, so I’m recommending watching my complete introduction session from a recent community MeetUp as a pre-requisitic, blog link below.

Hope to see you in December 🙂

More articles

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Latest articles