Data Engineer Intern
StoneX Group
Chicago, Illinois, United States
Data as a Service Team
Aug 2022 - Dec 2022
- Enhanced and optimized a
Streamlit
web application through security integration andETL
pipeline improvements - Researched solutions to integrate
Okta authentication
into the web application, deployingEnvoy Proxy
microservice, achieving user access control using bearer tokens andMS SQL Server
, enhancing data confidentiality - Developed a dynamic usage analysis dashboard in
Splunk
, delivering associate usage and category-specific insights - Migrated the data curation
ETL
pipeline fromApache Airflow
to a continuousDatabricks workflow
, efficiently managing staging data withAzure Blob Storage
, and updated the application usingDocker
to reduce load times by 30x to 5 seconds - Managed microservices using
Rancher
forKubernetes
management and documented the overall system design for clarity - Utilized
Agile methodologies
to ensure project flexibility and timely completion, adapting to evolving project needs