Assistant Manager

Here's an overview of what it typically involves, along with the qualifications and skills needed

Responsibilities

• Role and Responsibilities:

  • Data Pipeline Development: Design, develop, and maintain scalable batch and streaming ETL pipelines using Databricks, PySpark, Python, and SQL.
  • Lakehouse Implementation: Build and manage data solutions leveraging Delta Lake and medallion architecture principles to support trusted and reusable data products.
  • Data Integration: Integrate data from enterprise applications, databases, APIs, files, and cloud platforms to enable unified analytics and reporting.
  • Data Modeling & Transformation: Develop efficient data models and transformation frameworks that support business intelligence, advanced analytics, and AI/ML use cases.
  • Platform Optimization: Monitor and optimize Databricks workloads, cluster configurations, and data processing performance to ensure scalability and cost efficiency.
  • Data Quality & Governance: Implement data quality checks, monitoring, lineage, security controls, and governance practices in alignment with enterprise standards.
  • DevOps & Automation: Contribute to CI/CD implementation, automated deployments, testing frameworks, and release management for data engineering solutions.
  • Cross-functional Collaboration: Partner with business stakeholders, data architects, analytics teams, and AI engineers to deliver data products that support business outcomes.
  • AI & Analytics Enablement: Enable downstream AI, machine learning, and Generative AI initiatives by building reliable, high-quality, and governed data foundations.
  • Continuous Improvement & Innovation: Stay current with advancements in Databricks, cloud technologies, and modern data engineering practices to continuously enhance platform capabilities.

Qualifications

• Qualifications:

  • Experience: 4 - 6 years of experience in data engineering, ETL development, and cloud-based data platforms.
  • Technical Skills: 
    • Proficiency in Python, PySpark and SQL
    • Strong expertise in Databricks Data Engineering capabilities, including: Databricks Lakehouse Architecture, ETL/ELT Design and Development, Delta Lake, Data Modeling, Batch and Streaming Data Pipelines, Spark Optimization and Performance Tuning
    • Experience in designing and implementing: Medallion Architecture, Data Quality Frameworks, Data Governance Controls, Metadata and Lineage Solutions
    • Experience with orchestration and workflow management tools such as Databricks Workflows, Azure Data Factory, or equivalent.
    • Experience implementing CI/CD pipelines, Git-based source control, automated testing, and deployment automation.
    • Strong understanding of data warehousing, dimensional modeling, and modern Lakehouse architectures. 
    • Exposure to Databricks Mosaic AI, Vector Search, MLflow, feature engineering, or GenAI data preparation workflows (Preferred).
    • Familiarity with data preparation for RAG, LLM, and AI/ML use cases (Preferred).
    • Experience with infrastructure-as-code and containerization technologies (Preferred).
  • Certifications:  Relevant certifications in data engineering, such as Databricks Certified Data Engineer Associate/Professional, or similar. (Preferred not mandatory)
  • Problem-Solving:  Strong analytical and problem-solving skills, with a proven ability to design scalable data solutions and resolve complex data engineering challenges.
  • Communication:  Excellent communication skills with experience working directly with clients and business stakeholders to deliver data and analytics solutions.
  • Stakeholder Management: Experience engaging with business stakeholders to understand data and analytics requirements and translate them into scalable data engineering solutions.