Description: As the Enterprise transitions to utilizing Google Cloud Platform (GCP) for all its AI / ML workloads, it is essential to migrate all production pipelines to this new AI / ML platform on GCP. While the objective is to migrate every pipeline, it is crucial to monitor production executions and derive insights regarding resource utilization and costs associated with each pipeline in operation. This approach will facilitate resource management, minimize production failures, and enable the execution of pipelines with optimized cost efficiency.

The candidate will focus on developing a utility that monitors all production pipelines and generates easily interpretable insights (summaries) that correlate resource utilization key performance indicators (KPIs) with costs through statistical analysis techniques. The tool should provide an oversight of the platform, including success and failure rates, identify optimization opportunities based on historical pipeline executions, and generate alerts for any anomalies.

A stretch goal would be to support customizable rules by the utility that would assist in generating alerts tailored to specific business or use case requirements.

Skill sets: GCP platform fundamentals, log analysis, Python, Linux / Unix understanding, Big Data fundamentals such as Hadoop, Spark, Algorithms. Understanding of Boosting algorithm, HPT, AI / ML GPU compatibility aspects would be a plus.

Expected learning: Big Data framework for Analytical and AI / ML workloads on GCP, Dask & rapids framework support for compute intensive tasks, Optimization recommendations for users based on the pipeline running on production by different modelling teams in Amex.

Specific courses required for project execution: Cloud Architecture, fundamentals of Hive/ Spark/ Big Query, Programming in Python. Acquaintance with Big Data environments will be an added advantage, visualizing big data platform logs using Splunk / Kibana will be a plus.

American Express - AiDa, Gurugram

Station Overview

Project 1

Title: Enterprise Ai / ML Platform production pipeline Monitoring on Google Cloud and Insights (Cost, Resource utilization etc.)