udemy-writing-production-ready-etl-pipelines-in-python-pandas-2022-4

\ 0:00 Course Introduction 3:26 Task Description 7:27 Production Environment 9:31 Task Steps and Dirty Solution\ 13:20 Why to use a virtual environment_ 17:23 Virtual Environment Setup 23:33 AWS Setup 30:26 Understanding the source data 40:37 Quick and Dirty_ Read multiple files 53:05 Quick and Dirty_ Transformations 1:08:54 Quick and Dirty_ Argument Date 1:18:51 Quick and Dirty_ Save to S3 1:27:33 Quick and Dirty_ Code Improvements Approach\ 1:36:00 Why a code design is needed_ 1:38:42 Functional vs. Object Oriented Programming 1:45:11 Why Software Testing_ 1:49:43 Quick and Dirty to Functions_ Architecture Design 1:50:41 Quick and Dirty to Functions_ Restructure Part 1 2:06:20 Quick and Dirty to Functions_ Restructure Part 2 2:18:52 Restructure get_objects Intro 2:20:42 Restructure get_objects Implementation Oriented Approach\ 2:32:18 Design Principles OOP 2:36:40 More Requirements - Configuration, Meta Data, Logging, Exceptions, Entrypoint 2:48:10 Meta Data_ return_date_list Quick and Dirty 3:06:01 Meta Data_ return_date_list Function 3:20:18 Meta Data_ update_meta_file 3:32:30 Code Design - Class design, methods, attributes, arguments 3:46:12 Comparison Functional Programming and OOP and Class Frame Implementation\ 3:47:17 Setting up Git Repository 3:52:11 Setting up Python Project - Folder Structure 3:56:51 Installation Visual Studio Code 3:59:22 Setting up class frame - Task Description 4:01:17 Setting up class frame - Solution S3 4:13:15 Setting up class frame - Solution meta_process 4:14:16 Setting up class frame - Solution constants 4:15:16 Setting up class frame - Solution custom_exceptions 4:15:39 Setting up class frame - Solution xetra_transformer 4:18:34 Setting up class frame - Solution run 4:19:08 Logging in Python - Intro 4:20:37 Logging in Python - Implementation 4:33:28 Create Pythonpath 4:39:28 Python Clean Coding Implementation\ 4:43:03 list_files_in_prefix - Thoughts 4:47:14 list_files_in_prefix - Implementation 4:49:36 list_files_in_prefix - Linting Intro 4:50:51 list_files_in_prefix - Pylint 4:55:37 list_files_in_prefix - Unit Testing Intro 4:58:35 list_files_in_prefix - Unit Test Specification 5:01:47 list_files_in_prefix - Unit Test Implementation 1 5:16:37 list_files_in_prefix - Unit Test Implementation 2 5:31:07 Task Description - Writing Methods 5:33:10 Solution - read_csv_to_df - Implementation 5:33:49 Solution - read_csv_to_df - Unit Test Implementation 5:36:04 Solution - write_df_to_s3 - Implementation 5:38:46 Solution - write_df_to_s3 - Unit Test Implementation 5:42:25 Solution - update_meta_file - Implementation 5:44:26 Solution - update_meta_file - Unit Test Implementation 5:50:56 Solution - return_date_list - Implementation 5:51:20 Solution - return_date_list - Unit Test Implementation 5:56:11 Solution - extract - Implementation 5:57:38 Solution - extract - Unit Test Implementation 6:02:09 Solution - transform_report1 - Implementation 6:02:44 Solution - transform_report1 - Unit Test Implementation 6:05:04 Solution - load - Implementation 6:05:48 Solution - load - Unit Test Implementation 6:07:20 Solution - etl_report1 - Implementation 6:07:54 Solution - etl_report1 - Unit Test Implementation 6:09:43 Integration Tests - Intro 6:10:11 Integration Tests - Test Specification 6:11:16 Integration Tests - Implementation 6:23:26 Entrypoint run - Implementation the ETL Job\ 6:35:48 Dependency Management - Intro 6:43:22 pipenv Implementation 6:47:35 Profiling and Timing - Intro 6:49:41 Mem-Profiler 6:52:49 Dockerization 6:57:43 Run in Production Environment \ 7:02:15 Summary