Difference between revisions of "Data Preprocessing"
m |
m |
||
| Line 5: | Line 5: | ||
|description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools | |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools | ||
}} | }} | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Data+Preprocessing+machine+learning+ML YouTube search...] |
| − | [ | + | [https://www.google.com/search?q=Data+Preprocessing+machine+learning+ML ...Google search] |
* [[AI Governance]] / [[Algorithm Administration]] | * [[AI Governance]] / [[Algorithm Administration]] | ||
** [[Data Science]] / [[Data Governance]] | ** [[Data Science]] / [[Data Governance]] | ||
*** [[Benchmarks]] | *** [[Benchmarks]] | ||
| − | *** | + | *** Data Preprocessing |
**** [[Feature Exploration/Learning]] | **** [[Feature Exploration/Learning]] | ||
**** [[Data Quality]] ...[[AI Verification and Validation|validity]], [[Evaluation - Measures#Accuracy|accuracy]], [[Data Quality#Data Cleaning|cleaning]], [[Data Quality#Data Completeness|completeness]], [[Data Quality#Data Consistency|consistency]], [[Data Quality#Data Encoding|encoding]], [[Data Quality#Zero Padding|padding]], [[Data Quality#Data Augmentation, Data Labeling, and Auto-Tagging|augmentation, labeling, auto-tagging]], [[Data Quality#Batch Norm(alization) & Standardization| normalization, standardization]], and [[Data Quality#Imbalanced Data|imbalanced data]] | **** [[Data Quality]] ...[[AI Verification and Validation|validity]], [[Evaluation - Measures#Accuracy|accuracy]], [[Data Quality#Data Cleaning|cleaning]], [[Data Quality#Data Completeness|completeness]], [[Data Quality#Data Consistency|consistency]], [[Data Quality#Data Encoding|encoding]], [[Data Quality#Zero Padding|padding]], [[Data Quality#Data Augmentation, Data Labeling, and Auto-Tagging|augmentation, labeling, auto-tagging]], [[Data Quality#Batch Norm(alization) & Standardization| normalization, standardization]], and [[Data Quality#Imbalanced Data|imbalanced data]] | ||
| Line 27: | Line 27: | ||
* [[Train, Validate, and Test]] | * [[Train, Validate, and Test]] | ||
* [[Python]] | * [[Python]] | ||
| − | * [ | + | * [https://scikit-learn.org/stable/modules/preprocessing.html Sklearn.preprocessing] |
| − | * The Passenger Screening Kaggle challenge [ | + | * The Passenger Screening Kaggle challenge [https://www.kaggle.com/c/passenger-screening-algorithm-challenge/discussion/45805 1st place solution] was won in part due to data preparation/generation. |
| − | * [ | + | * [https://towardsdatascience.com/data-pre-processing-techniques-you-should-know-8954662716d6 Data Pre Processing Techniques You Should Know | Maneesha Rajaratne - Towards Data Science] |
| − | * [ | + | * [https://medium.com/datadriveninvestor/machine-learning-ml-data-preprocessing-5b346766fc48 Machine Learning(ML) — Data Preprocessing | Raji Adam Bifola] |
| − | * [ | + | * [https://sci2s.ugr.es/most-influential-preprocessing Most Influential Data Preprocessing Algorithms | S. García, J. Luengo, F. Herrera] |
| − | * [ | + | * [https://www.kdnuggets.com/2019/05/fix-unbalanced-dataset.html How to fix an Unbalanced Dataset | Will Badr -] [[Amazon | Amazon Web Services]] |
| − | * [ | + | * [https://docs.aws.amazon.com/machine-learning/latest/dg/creating-and-using-datasources.html Creating and Using Datasources |] [[Amazon | Amazon Web Services]] |
| − | * [ | + | * [https://github.com/jontupitza Jon Tupitza Famous Jupyter Notebooks:] |
| − | ** [ | + | ** [https://github.com/JonTupitza/Data-Science-Process/blob/master/01-Data-Preparation.ipynb Data Preparation 01] |
| − | ** [ | + | ** [https://github.com/JonTupitza/Data-Science-On-Ramp/blob/master/03-Data-Preparation.ipynb Data Preparation 02] |
| − | * [ | + | * [https://covidtracking.com/software/ The COVID Tracking Project - software used] |
| − | + | https://www.researchgate.net/profile/Martin_Beibel/publication/49849827/figure/fig1/AS:601681616183296@1520463484026/Overview-of-the-data-preprocessing-pipeline-The-data-preprocessing-consists-of-1_W640.jpg | |
| − | [ | + | [https://www.researchgate.net/publication/49849827_Comparison_of_Multivariate_Data_Analysis_Strategies_for_High-Content_Screening/figures?lo=1 Article] |
<youtube>cw2LvVkmtkQ</youtube> | <youtube>cw2LvVkmtkQ</youtube> | ||
| Line 59: | Line 59: | ||
== [[Time]]-Series Data == | == [[Time]]-Series Data == | ||
* [[Backtesting]] | * [[Backtesting]] | ||
| − | * [ | + | * [https://primo.ai/index.php?title=PRIMO.ai&action=edit§ion=19 Time-based Algorithms] |
| − | * [ | + | * [https://blog.netsil.com/a-comparison-of-time-series-databases-and-netsils-use-of-druid-db805d471206 A Comparison of Time Series Databases and Netsil’s Use of Druid | Netsil] |
| − | * [ | + | * [https://azure.microsoft.com/en-us/blog/microsoft-announces-the-general-availability-of-azure-time-series-insights/ Microsoft announces the general availability of Azure Time Series Insights | Ryan Waite - Microsoft] |
| − | * [ | + | * [https://www.outlyer.com/blog/top10-open-source-time-series-databases/ Top 10 Time Series Databases | Outlyer] |
<youtube>HYvAPjukKic</youtube> | <youtube>HYvAPjukKic</youtube> | ||
| Line 71: | Line 71: | ||
<youtube>2SUBRE6wGiA</youtube> | <youtube>2SUBRE6wGiA</youtube> | ||
| − | + | https://azurecomcdn.azureedge.net/mediahandler/acomblog/media/Default/blog/578a09a1-f144-4a62-98cb-e6e3ed774817.png | |
== Categorical Variables == | == Categorical Variables == | ||
Revision as of 19:24, 28 January 2023
YouTube search... ...Google search
- AI Governance / Algorithm Administration
- Data Science / Data Governance
- Benchmarks
- Data Preprocessing
- Bias and Variances
- Master Data Management (MDM)
- Privacy in Data Science
- Data Interoperability
- Excel - Data Analysis
- Data Science / Data Governance
- Visualization
- Hyperparameters
- Evaluation
- Train, Validate, and Test
- Python
- Sklearn.preprocessing
- The Passenger Screening Kaggle challenge 1st place solution was won in part due to data preparation/generation.
- Data Pre Processing Techniques You Should Know | Maneesha Rajaratne - Towards Data Science
- Machine Learning(ML) — Data Preprocessing | Raji Adam Bifola
- Most Influential Data Preprocessing Algorithms | S. García, J. Luengo, F. Herrera
- How to fix an Unbalanced Dataset | Will Badr - Amazon Web Services
- Creating and Using Datasources | Amazon Web Services
- Jon Tupitza Famous Jupyter Notebooks:
- The COVID Tracking Project - software used
Contents
Splitting Data - training and testing sets
Time-Series Data
- Backtesting
- Time-based Algorithms
- A Comparison of Time Series Databases and Netsil’s Use of Druid | Netsil
- Microsoft announces the general availability of Azure Time Series Insights | Ryan Waite - Microsoft
- Top 10 Time Series Databases | Outlyer
Categorical Variables
Categorical variables require special attention in regression analysis because, unlike dichotomous or continuous variables, they cannot by entered into the regression equation just as they are. Instead, they need to be recoded into a series of variables which can then be entered into the regression model. There are a variety of coding systems that can be used when recoding categorical variables. Coding Systems for Categorical Variables In Regression Analysis | UCLA institute for Digital Research & Education Statistical Consulting
SQL Database Optimization
