Data Engineer Intern
StoneX Group
Chicago, Illinois, United States
Data as a Service Team
Aug 2022 - Dec 2022

demo

  • Enhanced and optimized a Streamlit web application through security integration and ETL pipeline improvements
  • Researched solutions to integrate Okta authentication into the web application, deploying Envoy Proxy microservice, achieving user access control using bearer tokens and MS SQL Server, enhancing data confidentiality
  • Developed a dynamic usage analysis dashboard in Splunk, delivering associate usage and category-specific insights
  • Migrated the data curation ETL pipeline from Apache Airflow to a continuous Databricks workflow, efficiently managing staging data with Azure Blob Storage, and updated the application using Docker to reduce load times by 30x to 5 seconds
  • Managed microservices using Rancher for Kubernetes management and documented the overall system design for clarity
  • Utilized Agile methodologies to ensure project flexibility and timely completion, adapting to evolving project needs