Difference between revisions of "Python"
m |
|||
| (127 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
|title=PRIMO.ai | |title=PRIMO.ai | ||
|titlemode=append | |titlemode=append | ||
| − | |keywords=artificial, intelligence, machine, learning, models | + | |keywords=ChatGPT, artificial, intelligence, machine, learning, GPT-4, GPT-5, NLP, NLG, NLC, NLU, models, data, singularity, moonshot, Sentience, AGI, Emergence, Moonshot, Explainable, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Hugging Face, OpenAI, Tensorflow, OpenAI, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, LLM, metaverse, assistants, agents, digital twin, IoT, Transhumanism, Immersive Reality, Generative AI, Conversational AI, Perplexity, Bing, You, Bard, Ernie, prompt Engineering LangChain, Video/Image, Vision, End-to-End Speech, Synthesize Speech, Speech Recognition, Stanford, MIT |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools |
| − | |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools | + | |
| + | <!-- Google tag (gtag.js) --> | ||
| + | <script async src="https://www.googletagmanager.com/gtag/js?id=G-4GCWLBVJ7T"></script> | ||
| + | <script> | ||
| + | window.dataLayer = window.dataLayer || []; | ||
| + | function gtag(){dataLayer.push(arguments);} | ||
| + | gtag('js', new Date()); | ||
| + | |||
| + | gtag('config', 'G-4GCWLBVJ7T'); | ||
| + | </script> | ||
}} | }} | ||
| − | [ | + | [https://www.youtube.com/results?search_query=ai+python YouTube] |
| − | [ | + | [https://www.quora.com/search?q=ai%20python ... Quora] |
| + | [https://www.google.com/search?q=ai+python ...Google search] | ||
| + | [https://news.google.com/search?q=ai+python ...Google News] | ||
| + | [https://www.bing.com/news/search?q=ai+python&qft=interval%3d%228%22 ...Bing News] | ||
| − | * [[Natural Language Tools & Services#Capability (other)|Natural Language libraries]], e.g. [[SpaCy]] | + | * [[Python]] ... [[Generative AI with Python|GenAI w/ Python]] ... [[JavaScript]] ... [[Generative AI with JavaScript|GenAI w/ JavaScript]] ... [[TensorFlow]] ... [[PyTorch]] |
| + | * [[Libraries & Frameworks Overview]] ... [[Libraries & Frameworks]] ... [[Git - GitHub and GitLab]] ... [[Other Coding options]] | ||
| + | * [[Agents]] ... [[Robotic Process Automation (RPA)|Robotic Process Automation]] ... [[Assistants]] ... [[Personal Companions]] ... [[Personal Productivity|Productivity]] ... [[Email]] ... [[Negotiation]] ... [[LangChain]] Python library | ||
| + | * [[Natural Language Tools & Services#Capability (other)|Natural Language libraries]], e.g. [[SpaCy]], [[Natural Language Toolkit (NLTK)]], [https://stanfordnlp.github.io/CoreNLP/ CoreNLP], [https://textblob.readthedocs.io/en/dev/ TextBlob], [[Python#scikit-learn|scikit-learn]] NLP toolkit, [https://fasttext.cc/ fastText], [https://www.intel.ai/nlp-architect-by-intel-ai-lab-release-0-2/ Intel NLP Architect], [[Gensim]] | ||
* Other Python-related pages: | * Other Python-related pages: | ||
** [[TensorFlow]] for machine learning model building | ** [[TensorFlow]] for machine learning model building | ||
| − | ** [[PyTorch]] authored by Facebook | + | ** [[PyTorch]] authored by [[Meta|Facebook]] |
** [[Google AutoML]] automatically build and deploy state-of-the-art machine learning models | ** [[Google AutoML]] automatically build and deploy state-of-the-art machine learning models | ||
** [[Ludwig]] - a Python toolbox from Uber that allows to train and test deep learning models | ** [[Ludwig]] - a Python toolbox from Uber that allows to train and test deep learning models | ||
| Line 17: | Line 32: | ||
** [[AWS Lambda & Python]] | ** [[AWS Lambda & Python]] | ||
** [[Notebooks]]; [[Jupyter]] and [[Notebooks#R Markdown |R Markdown]] | ** [[Notebooks]]; [[Jupyter]] and [[Notebooks#R Markdown |R Markdown]] | ||
| − | * [ | + | * [https://medium.com/applied-data-science/how-to-build-your-own-alphazero-ai-using-python-and-keras-7f664945c188 How to build your own AlphaZero AI using Python and Keras] |
| − | * [ | + | * [https://automatetheboringstuff.com/ Automate the Boring Stuff with Python] |
| − | * [ | + | * [https://www.fullstackpython.com/best-python-resources.html Best Python Resources | Full Stack Python] |
| − | * [ | + | * [https://realpython.com/start-here/ Learn Python Programming, By Example | Real Python] |
| − | * [ | + | * [https://www.kdnuggets.com/2018/02/top-20-python-ai-machine-learning-open-source-projects.html Top 20 Python AI and Machine Learning Open Source Projects] |
| − | * [ | + | * [https://startupsventurecapital.com/essential-cheat-sheets-for-machine-learning-and-deep-learning-researchers-efb6a8ebd2e5 Essential Cheat Sheets for Machine Learning and Deep Learning Engineers] |
| − | + | * [https://www.kdnuggets.com/2019/02/setup-python-environment-machine-learning.html How to Setup a Python Environment for Machine Learning | George Seif - KDnuggets] | |
* [[Creatives#Guido Van Rossum |Guido Van Rossum]]; author of Python | * [[Creatives#Guido Van Rossum |Guido Van Rossum]]; author of Python | ||
| − | * [ | + | * [https://www.sphinx-doc.org/en/master/ Sphinx] is a tool that makes it easy to create intelligent and beautiful documentation | Georg Brandl |
| − | * [ | + | * [https://www.techrepublic.com/article/python-programming-language-a-cheat-sheet/ Python programming language: A cheat sheet | James Sanders - TechRepublic] explores what it is used for, how it compares to other languages, and building skills resources |
* References: | * References: | ||
| − | ** [ | + | ** [https://docs.python.org/3/reference/index.html#reference-index The Python Language Reference] |
| − | ** [ | + | ** [https://docs.python.org/3/library/ The Python Standard Library] |
* [[Quantum#Quantum Development Algorithms & Kits|Quantum Development Algorithms & Kits]] | * [[Quantum#Quantum Development Algorithms & Kits|Quantum Development Algorithms & Kits]] | ||
* [[Autonomous Drones]] | * [[Autonomous Drones]] | ||
* Environments: | * Environments: | ||
| − | + | * [[Development]] ... [[Notebooks]] ... [[Development#AI Pair Programming Tools|AI Pair Programming]] ... [[Codeless Options, Code Generators, Drag n' Drop|Codeless, Generators, Drag n' Drop]] ... [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] ... [[Platforms: AI/Machine Learning as a Service (AIaaS/MLaaS)|AIaaS/MLaaS]] | |
*** [[Colaboratory]] | *** [[Colaboratory]] | ||
** [[Local]] Machine | ** [[Local]] Machine | ||
| − | * Code completion: works with the top Python editors: [ | + | *** [[Anaconda]] |
| − | ** [ | + | * Code completion: works with the top Python editors: [https://atom.io/packages/ide-python Atom], [https://www.jetbrains.com/pycharm/ PyCharm], [https://realpython.com/setting-up-sublime-text-3-for-full-stack-python-development/ Sublime],[https://code.visualstudio.com/docs/languages/python Visual Studio (VS) Code] and [https://www.fullstackpython.com/vim.html Vim] |
| − | ** [ | + | ** [https://kite.com/ Kite] uses machine learning to give useful code completions |
| − | * [ | + | ** [https://tabnine.com/ TabNine] utilizes GPT-2 |
| − | ** [ | + | * [https://wiki.python.org/moin/PythonImplementations Alternative implementations and extensions of Python] to address speed & [[memory]] usage... |
| − | ** [ | + | ** [https://thenewstack.io/python-gets-its-mojo-working-for-ai/ Python Gets Its Mojo Working for AI | Jessica Wachtel - The New Stack] ... Combining the usability of Python with the performance of C, Mojo is a new programming language designed specifically for AI developers. |
| + | ** [https://github.com/python/cpython CPython] written in C and Python is [https://en.wikipedia.org/wiki/Guido_van_Rossum Guido van Rossum]'s reference version of the Python computing language | ||
| + | ** [https://www.pypy.org/ PyPy] uses just-in-time compilation | ||
** [https://cython.org/ Cython] an optimizing static compiler | ** [https://cython.org/ Cython] an optimizing static compiler | ||
| + | * [[Attention]] Mechanism ... [[Transformer]] ... [[Generative Pre-trained Transformer (GPT)]] ... [[Generative Adversarial Network (GAN)|GAN]] ... [[Bidirectional Encoder Representations from Transformers (BERT)|BERT]] | ||
| + | * [[Artificial General Intelligence (AGI) to Singularity]] ... [[Inside Out - Curious Optimistic Reasoning| Curious Reasoning]] ... [[Emergence]] ... [[Moonshots]] ... [[Explainable / Interpretable AI|Explainable AI]] ... [[Algorithm Administration#Automated Learning|Automated Learning]] | ||
| + | * [[Conversational AI]] ... [[ChatGPT]] | [[OpenAI]] ... [[Bing/Copilot]] | [[Microsoft]] ... [[Gemini]] | [[Google]] ... [[Claude]] | [[Anthropic]] ... [[Perplexity]] ... [[You]] ... [[phind]] ... [[Ernie]] | [[Baidu]] | ||
| + | ** [[ChatGPT#Integration| ChatGPT Integration]] | ||
| − | + | = Using Python = | |
| − | + | {|<!-- T --> | |
| − | + | | valign="top" | | |
| − | + | {| class="wikitable" style="width: 550px;" | |
| + | || | ||
| + | <youtube>_uQrJ0TkZlc</youtube> | ||
| + | <b>Python Tutorial - Python for Beginners [Full Course] | ||
| + | </b><br>Learn Python programming for a career in machine learning, data science & web development. Get My FREE Python Cheat Sheet: https://bit.ly/2Gp80s6 | ||
| + | |} | ||
| + | |<!-- M --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
| + | <youtube>7lmCu8wz8ro</youtube> | ||
| + | <b>What Does It Take To Be An Expert At Python? | ||
| + | </b><br>If you want to become an expert in Python, you should definitely watch this PyData talk from James Powell. | ||
| + | |} | ||
| + | |}<!-- B --> | ||
| + | {|<!-- T --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>6KM8HOQi5Xk</youtube> | <youtube>6KM8HOQi5Xk</youtube> | ||
| − | < | + | <b>Introduction to Python | SciPy 2019 Tutorial | Matt Davis |
| + | </b><br>This tutorial is a gentle introduction to Python for folks who are completely new to it and may not have much experience programming. We’ll work in a [[Jupyter]] Notebook, one of the most popular tools in scientific Python. You’ll learn how to write beautiful Python while practicing loops, if’s, functions, and usage of Python’s built-in features in a series of fun, interactive exercises. By the end of the tutorial we think you’ll be ready to write your own basic Python -- but most importantly, we want you to learn the form and vocabulary of Python so that you can understand Python documentation and interpret code written by others. To continue learning more about Python as it’s applied to data and science, join our companion tutorial titled Introduction to Scientific Python. | ||
| + | |} | ||
| + | |<!-- M --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>8OixQrWRiXo</youtube> | <youtube>8OixQrWRiXo</youtube> | ||
| + | <b>Top 5 Python Libraries For Data Science | Python Libraries Explained | Python Tutorial | Simplilearn | ||
| + | </b><br>Python is the most widely used programming language today. When it comes to solving Data Science tasks and challenges, Python never ceases to surprise its audience. Most data scientists are already leveraging the power of Python programming every day. Python is easy to learn, easier to debug, widely used, object-oriented, open source, high-performance language and there are many more benefits of using Python programming. Python has been built with extraordinary libraries which are used by programmers everyday in solving the problems. So, now let us talk about the Top 5 Python libraries for Data Science. 1. [[Tensorflow]] 2. Numpy 3. Scipy 4. Pandas 5. Matplotlib | ||
| + | |} | ||
| + | |}<!-- B --> | ||
| + | {|<!-- T --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>azfWrlxVxDU</youtube> | <youtube>azfWrlxVxDU</youtube> | ||
| + | <b>Python For Deep Learning - I | Python Basics | Python Tutorial | Python Training | Edureka | ||
| + | </b><br>In this Edureka tutorial on Code-Basics For Deep Learning, we will be covering all the fundamentals of Python required for implementing deep learning models. This is the first video in the series, 1. What is Python? 2. Variables, Data Types, Operators 3. Conditional Statements, Loops, Functions 4. Pandas, Numpy and Matplotlib | ||
| + | |} | ||
| + | |<!-- M --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>OGxgnH8y2NM</youtube> | <youtube>OGxgnH8y2NM</youtube> | ||
| + | <b>Practical Machine Learning Tutorial with Python Intro p.1 | ||
| + | </b><br>The objective of this course is to give you a holistic understanding of machine learning, covering theory, application, and inner workings of supervised, unsupervised, and deep learning algorithms. In this series, we'll be covering linear regression, K Nearest Neighbors, Support Vector Machines (SVM), flat clustering, hierarchical clustering, and neural networks. | ||
| + | For each major algorithm that we cover, we will discuss the high level intuitions of the algorithms and how they are logically meant to work. Next, we'll apply the algorithms in code using real world data sets along with a module, such as with Scikit-Learn. Finally, we'll be diving into the inner workings of each of the algorithms by recreating them in code, from scratch, ourselves, including all of the math involved. This should give you a complete understanding of exactly how the algorithms work, how they can be tweaked, what advantages are, and what their disadvantages are. In order to follow along with the series, I suggest you have at the very least a basic understanding of Python. If you do not, I suggest you at least follow the Python 3 Basics tutorial until the module installation with pip tutorial. If you have a basic understanding of Python, and the willingness to learn/ask questions, you will be able to follow along here with no issues. | ||
| + | |} | ||
| + | |}<!-- B --> | ||
| + | {|<!-- T --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>nyrjl97WZhA</youtube> | <youtube>nyrjl97WZhA</youtube> | ||
| + | <b>Learn Python Programming - Full Course for Beginners [2019] | ||
| + | </b><br>Watch this Python tutorial for absolute beginners to learn Python programming from scratch. Includes full exercises and projects. | ||
| + | |} | ||
| + | |<!-- M --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
| + | <youtube>belS2Ek4-ow</youtube> | ||
| + | <b>Best feature of Python | List Comprehensions | ||
| + | </b><br>Python has lots of amazing features, but to me, list comprehensions are #1. | ||
| + | |} | ||
| + | |}<!-- B --> | ||
== Python Data Science Handbook == | == Python Data Science Handbook == | ||
| − | * [ | + | * [https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/Index.ipynb Python Data Science Handbook (Jupyter notebook)] | [[Creatives#Jake VanderPlas |Jake VanderPlas]] - O'Reilly |
| − | ** [ | + | ** [https://tanthiamhuat.files.wordpress.com/2018/04/pythondatasciencehandbook.pdf Python Data Science Handbook (PDF)] | [[Creatives#Jake VanderPlas |Jake VanderPlas]] |
| + | {|<!-- T --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>DifMYH3iuFw</youtube> | <youtube>DifMYH3iuFw</youtube> | ||
| + | <b>Keynote [[Creatives#Jake VanderPlas|Jake VanderPlas]] | ||
| + | </b><br>Jul 24, 2017 www.pydata.org PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States | ||
| + | |} | ||
| + | |}<!-- B --> | ||
| + | |||
| + | == <span id="SentenceTransformers"></span>SentenceTransformers == | ||
| + | [https://www.youtube.com/results?search_query=SentenceTransformers Youtube search...] | ||
| + | [https://www.google.com/search?q=python+SentenceTransformers ...Google search] | ||
| + | |||
| + | <b>SentenceTransformers</b> is a Python framework for state-of-the-art sentence, text, and image [[embedding]]s. It is based on PyTorch and [[Transformer]]s and offers a large collection of pre-trained models tuned for various tasks. You can use this framework to compute sentence/text [[embedding]]s for more than 100 languages. These [[embedding]]s can then be compared, for example, with cosine similarity to find sentences with a similar meaning. This can be useful for semantic textual similarity, semantic search, or paraphrase mining. You can install the Sentence Transformers library using pip: pip install -U sentence-transformers | ||
| + | |||
| + | == <span id="PyScript"></span>PyScript == | ||
| + | [https://www.youtube.com/results?search_query=PyScript Youtube search...] | ||
| + | [https://www.google.com/search?q=python+PyScript ...Google search] | ||
| + | |||
| + | * [https://pyscript.net/ py]... run python in your HTML; a framework that allows users to create rich Python applications in the browser using HTML's interface | ||
| + | * [https://medium.com/analytics-vidhya/pyscript-use-python-code-in-html-f7c8b49486a4 PyScript-Use Python Code in HTML | Senthil E - Analytics Vidhya] | ||
| + | |||
| + | <youtube>vxqBm6_0vyk</youtube> | ||
| + | <youtube>MJvCeKwr4z4</youtube> | ||
== <span id="NumPy"></span>NumPy == | == <span id="NumPy"></span>NumPy == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=NumPy Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=python+NumPy ...Google search] |
| − | * [ | + | * [https://numpy.org/ NumPy] -manipulation of numerical arrays. NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases. |
| − | * [ | + | * [https://cs231n.github.io/python-numpy-tutorial/ Python Numpy Tutorial | Justin Johnson] |
<youtube>EEUXKG97YRw</youtube> | <youtube>EEUXKG97YRw</youtube> | ||
| Line 72: | Line 177: | ||
== <span id="Pandas"></span>Pandas == | == <span id="Pandas"></span>Pandas == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=python+pandas Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=python+pandas ...Google search] |
| − | * [ | + | * [https://pandas.pydata.org/ Python Data Analysis] library - data structures and data analysis tools for the Python programming language. Pandas is a newer package built on top of [[Python#NumPy |NumPy]], and provides an efficient implementation of a [[Python#Pandas DataFrame |Pandas DataFrame]]. [[Python#Pandas DataFrame |Pandas DataFrame]]s are essentially multidimensional arrays with attached row and column labels, and often with heterogeneous types and/or missing data. As well as offering a convenient storage interface for labeled data, Pandas implements a number of powerful data operations familiar to users of both database frameworks and spreadsheet programs. |
| − | * [ | + | * [https://www.oreilly.com/library/view/python-for-data/9781491957653/ Python for Data Analysis | Wes McKinney] |
| + | * [https://github.com/modin-project/modin Modin] accelerates Pandas by automatically distributing the computation across all of the system’s available CPU cores | ||
<youtube>5rNu16O3YNE</youtube> | <youtube>5rNu16O3YNE</youtube> | ||
| Line 84: | Line 190: | ||
=== <span id="Pandas DataFrame"></span>Pandas DataFrame === | === <span id="Pandas DataFrame"></span>Pandas DataFrame === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=python+pandas+Dataframe Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=python+pandas+Dataframe ...Google search] |
| − | * [ | + | * [https://pandas.pydata.org/ Pandas DataFrame] is two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). A Data frame is a two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and columns. |
| − | * [ | + | * [https://towardsdatascience.com/pandas-dataframe-a-lightweight-intro-680e3a212b96 Pandas DataFrame: A lightweight Intro | Daksh Deepak - Towards Data Science] |
| − | * [ | + | * [https://www.datacamp.com/community/tutorials/joining-dataframes-pandas Joining DataFrames in Pandas | Manish Pathak - Data Camp] |
<youtube>e60ItwlZTKM</youtube> | <youtube>e60ItwlZTKM</youtube> | ||
| Line 95: | Line 201: | ||
== <span id="SciPy"></span>SciPy == | == <span id="SciPy"></span>SciPy == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=SciPy+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=SciPy+python ...Google search] |
| − | * [ | + | * [https://www.scipy.org// SciPy] library - one of the core packages that make up the SciPy stack. It provides many user-friendly and efficient numerical routines such as routines for numerical integration, interpolation, optimization, linear algebra and statistics. |
<youtube>oYTs9HwFGbY</youtube> | <youtube>oYTs9HwFGbY</youtube> | ||
== <span id="SymPy"></span>SymPy == | == <span id="SymPy"></span>SymPy == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=SymPy+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=SymPy+python ...Google search] |
| − | * [ | + | * [https://www.sympy.org SymPy] library - a Python library for symbolic mathematics aiming to become a full-featured computer algebra system (CAS) |
| − | * [ | + | * [https://mpmath.org/ mpmath | Fredrik Johansson] library for real and complex floating-point arithmetic with arbitrary precision |
<youtube>AqnpuGbM6-Q</youtube> | <youtube>AqnpuGbM6-Q</youtube> | ||
=== <span id="mpmath"></span>mpmath === | === <span id="mpmath"></span>mpmath === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=mpmath+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=mpmathy+python ...Google search] |
| − | * [ | + | * [https://mpmath.org/ mpmath | Fredrik Johansson] library for real and complex floating-point arithmetic with arbitrary precision. can be used as a library, interactively via the Python interpreter, or from within the SymPy or [https://www.sagemath.org/ Sage] computer algebra systems which include mpmath as standard component. [https://cocalc.com/ CoCalc] lets you use mpmath directly in the browser. Cocalc or "Collaborative Calculation in the Cloud" enables programming online without the need to install any software. |
== <span id="NetworkX"></span>NetworkX == | == <span id="NetworkX"></span>NetworkX == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=NetworkX Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=NetworkX ...Google search] |
| − | * [ | + | * [https://networkx.github.io/ NetworkX] a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. Video: Connected: A Social Network Analysis Tutorial with [https://networkx.github.io/ NetworkX] ...[https://en.wikipedia.org/wiki/Social_network_analysis Social Network Analysis (SNA)], the study of the relational structure between actors, is used throughout the social and natural sciences to discover insight from connected entities. |
* [[Social Network Analysis (SNA)]] | * [[Social Network Analysis (SNA)]] | ||
| − | * [ | + | * [https://github.com/PyCon/2015-slides Twitter Network Analysis with NetworkX - PyCon 2015 | Sarah Guido, Celia La - GitHub] |
| − | * [ | + | * [https://ep2016.europython.eu//conference/talks/networkx-visualization-powered-by-bokeh NetworkX Visualization Powered By Bokeh | Björn Meier] |
| + | * [https://wiki.sagemath.org/graph_survey SageMath graph tools survey] ...[https://www.sagemath.org/links-components.html components] | ||
| + | * [[Network Pattern]] | ||
<youtube>RlVtbuRpKTw</youtube> | <youtube>RlVtbuRpKTw</youtube> | ||
| Line 132: | Line 240: | ||
== <span id="scikit-learn"></span>scikit-learn == | == <span id="scikit-learn"></span>scikit-learn == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=scikit-learn Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=scikit-learn ...Google search] |
| − | * [ | + | * [https://scikit-learn.org/stable/ Scikit-learn] library for machine learning in Python built on NumPy, SciPy, and matplotlib. A toolkit implement a wide variety of algorithms for un/supervised machine learning tasks, including regressions, [[clustering]], [[Manifold Hypothesis|manifold learning]], principal components, density estimation, and much more. It also provides many useful tools to help build “[[Algorithm Administration#AIOps/MLOps|pipelines]]” for managing modeling tasks such as data processing /normalization, feature engineering, cross-validation, fitting, and prediction. The package scikit-learn is recommended to be installed using pip install scikit-learn but in your code imported using import sklearn. |
<youtube>rvVkVsG49uU</youtube> | <youtube>rvVkVsG49uU</youtube> | ||
<youtube>4PXAztQtoTg</youtube> | <youtube>4PXAztQtoTg</youtube> | ||
| − | == <span id="Python | + | === <span id="LazyPredict"></span>LazyPredict === |
| − | + | [https://www.youtube.com/results?search_query=LazyPredict Youtube search...] | |
| + | [https://www.google.com/search?q=LazyPredict ...Google search] | ||
| + | |||
| + | * [https://github.com/shankarpandala/lazypredict shankarpandala/lazypredict | Shankar Rao Pandala] | ||
| + | |||
| + | Lazy Predict help build a lot of basic models without much code and helps understand which models works better without any parameter tuning | ||
| + | |||
| + | LazyPredict is a low-code machine learning library that allows you to run up to 40 baseline models with two lines of code. LazyPredict uses Sklearn, which allows you to get the models, see what works best for you, and hypertune it as you would usually do. To install, you can type pip install lazypredict in your Terminal. [https://towardsdatascience.com/3-awesome-python-libraries-that-you-should-know-about-e2485e6e1cbe 3 Awesome Python Libraries That You Should Know About | Ismael Araujo - Towards Data Science] | ||
| + | |||
| + | <youtube>ZdDUwlwJNi0</youtube> | ||
| + | <youtube>IUUpPL5JRUU</youtube> | ||
| + | |||
| + | == <span id="Graphical User Interface (GUI)"></span>Graphical User Interface (GUI) == | ||
| + | |||
| + | <youtube>YJ8PTCcU3J0</youtube> | ||
| + | |||
| + | === <span id="HiPlot"></span>HiPlot === | ||
| + | [https://www.youtube.com/results?search_query=HiPlot Youtube search...] | ||
| + | [https://www.google.com/search?q=HiPlot ...Google search] | ||
| + | |||
| + | * [https://ai.facebook.com/blog/hiplot-high-dimensional-interactive-plots-made-easy#u_0_1v HiPlot: High-dimensional interactive plots made easy] | ||
| + | ** [https://github.com/facebookresearch/hiplot GitHub] ...or install with pip install HiPlot. | ||
| + | * [https://towardsdatascience.com/introduction-to-best-parallel-plot-python-library-hiplot-8387f5786d97 Introduction to Best Parallel Plot Python Library: “HiPlot” | Moto DEI - Towards Data Science] | ||
| + | |||
| + | HiPlot is [[Meta|Facebook]]’s interactive visualization tool to help AI researchers discover correlations and patterns in high-dimensional data | ||
| + | |||
| + | === <span id="Tkinter"></span>Tkinter === | ||
| + | [https://www.youtube.com/results?search_query=Tkinter Youtube search...] | ||
| + | [https://www.google.com/search?q=Tkinter ...Google search] | ||
| + | |||
| + | * [https://wiki.python.org/moin/TkInter TkInter] ... comes with Python already. Tkinter is a Python binding to the Tk GUI toolkit. It is the standard Python interface to the Tk GUI toolkit, and is Python's de facto standard GUI. Tkinter is included with standard Linux, Microsoft Windows and Mac OS X installs of Python. The name Tkinter comes from Tk interface. | ||
| + | * [https://github.com/flatplanet/Intro-To-TKinter-Youtube-Course flatplanet/Intro-To-TKinter-Youtube-Course] | ||
| + | |||
| + | <youtube>YXPyB4XeYLA</youtube> | ||
| + | <youtube>xuXYKhdoTsw</youtube> | ||
| + | |||
| + | |||
| + | === <span id="Kivy"></span>Kivy === | ||
| + | [https://www.youtube.com/results?search_query=Kivy Youtube search...] | ||
| + | [https://www.google.com/search?q=Kivy ...Google search] | ||
| + | |||
| + | * [https://kivy.org/#home Kivy] ... open source Python library for rapid development of applications that make use of innovative user interfaces, such as multi-touch apps. Kivy bassed on OpenGL, draw in 2D, 3D, meshes, and shaders, on runs on Linux, Windows, OS X, Android, iOS, and Raspberry Pi. You can run the same code on all supported platforms. Kivy is 100% free to use, under an MIT license (starting from 1.7.2) and LGPL 3 for the previous versions. | ||
| + | |||
| + | <youtube>GXP8O4dSS3E</youtube> | ||
| + | <youtube>PaaVaBNIFnM</youtube> | ||
| + | |||
| + | === <span id="PyQt"></span>PyQt === | ||
| + | [https://www.youtube.com/results?search_query=PyQt5 Youtube search...] | ||
| + | [https://www.google.com/search?q=PyQt5 ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/PyQt5/ Kivy] ... Qt is set of cross-platform C++ libraries that implement high-level APIs for accessing many aspects of modern desktop and mobile systems. Library implements the QT application development framework and has QTDesigner: drag and drop interface. These include location and positioning services, multimedia, NFC and Bluetooth connectivity, a Chromium based web browser, as well as traditional UI development. PyQt5 is a comprehensive set of Python bindings for Qt v5. It is implemented as more than 35 extension modules and enables Python to be used as an alternative application development language to C++ on all supported platforms including iOS and Android. PyQt5 is released under the GPL v3 license and under a commercial license that allows for the development of proprietary applications. | ||
| + | |||
| + | <youtube>kK5_zTbnEDU</youtube> | ||
| + | <youtube>M7UdAX77kpY</youtube> | ||
| + | |||
| + | === <span id="wxPython"></span>wxPython === | ||
| + | [https://www.youtube.com/results?search_query=wxPython Youtube search...] | ||
| + | [https://www.google.com/search?q=wxPython ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/PyQt5/ wxPython] a cross-platform GUI toolkit for the Python programming language. It allows Python programmers to create programs with a robust, highly functional graphical user interface, simply and easily. It is implemented as a set of Python extension modules that wrap the GUI components of the popular <b>wxWidgets</b> cross platform library, which is written in C++.Like Python and wxWidgets, wxPython is Open Source, which means that it is free for anyone to use and the source code is available for anyone to look at and modify. And anyone can contribute fixes or enhancements to the project.wxPython is a cross-platform toolkit. This means that the same program will run on multiple platforms without modification. Currently Supported platforms are [[Microsoft]] Windows, Mac OS X and macOS, and Linux or other unix-like systems with GTK2 or GTK3 libraries. In most cases the native widgets are used on each platform to provide a 100% native look and feel for the application. | ||
| + | |||
| + | <youtube>PHxAau5-dUc</youtube> | ||
| + | <youtube>7g3V7CsQpDA</youtube> | ||
| + | |||
| + | === <span id="Pyside2"></span>Pyside2 === | ||
| + | [https://www.youtube.com/results?search_query=Pyside2 Youtube search...] | ||
| + | [https://www.google.com/search?q=Pyside2 ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/PyQt5/ Pyside2] ...a Python binding of the cross-platform GUI toolkit Qt, currently developed by The Qt Company under the Qt for Python project on porting PySide to work with Qt 5 instead of Qt 4. It is one of the alternatives to the standard library package Tkinter. Like Qt, PySide2 is free software. | ||
| + | |||
| + | <youtube>vs8nDye8XvM</youtube> | ||
| + | <youtube>Gpw6BygkUCw</youtube> | ||
| + | |||
| + | == Spreadsheets == | ||
| + | |||
| + | === Python & Google Sheets === | ||
| + | * [https://developers.google.com/sheets/ Google Sheets API] | ||
| + | * [[Google Sheets#Google Sheets| Google Sheets and Python]] | ||
| + | * [https://developers.google.com/sheets/api/quickstart/python Python Quickstart] | ||
| + | |||
| + | <youtube>7I2s81TsCnc</youtube> | ||
| + | |||
| + | ==== <span id="gsheets"></span>gsheets ==== | ||
| + | [https://www.youtube.com/results?search_query=gsheets Youtube search...] | ||
| + | [https://www.google.com/search?q=gsheets ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/gsheets/ gsheets] - small wrapper around the [https://developers.google.com/sheets/ Google Sheets API] to provide more convenient access to Google Sheets from Python scripts. | ||
| + | |||
| + | ==== <span id="gsheets.py"></span>gsheets.py ==== | ||
| + | [https://www.youtube.com/results?search_query=gsheets.py Youtube search...] | ||
| + | [https://www.google.com/search?q=gsheets.py ...Google search] | ||
| + | |||
| + | * [https://gist.github.com/xflr6/57508d28adec1cd3cd047032e8d81266 gsheets] - self-containd script to dump all worksheets of a Google Spreadsheet to CSV or convert any subsheet to a pandas DataFrame | ||
| + | |||
| + | ==== <span id="gspread"></span>gspread ==== | ||
| + | [https://www.youtube.com/results?search_query=gspread Youtube search...] | ||
| + | [https://www.google.com/search?q=gspread ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/gspread/ gspread] ...Google Sheets Python API wrapper | ||
| + | * [https://gist.github.com/egradman/3b8140930aef97f9b0e4 example Jupyter notebook] using gspread to fetch a sheet into a Pandas DataFrame | ||
| + | |||
| + | ==== <span id="df2gspread"></span>df2gspread ==== | ||
| + | [https://www.youtube.com/results?search_query=df2gspread Youtube search...] | ||
| + | [https://www.google.com/search?q=df2gspread ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/df2gspread/ df2gspread] ...transfer data between Google Sheets and Pandas | ||
| − | + | ==== <span id="pygsheets"></span>pygsheets ==== | |
| + | [https://www.youtube.com/results?search_query=pygsheets Youtube search...] | ||
| + | [https://www.google.com/search?q=pygsheets ...Google search] | ||
| − | * [ | + | * [https://pypi.org/project/pygsheets/ pygsheets] ...Google Sheets Python API v4 (v4 port of gspread providing further extensions) |
| − | * Building Interactive Python tools with Excel as a front-end | + | ==== <span id="gspread-pandas"></span>gspread-pandas ==== |
| − | ** [[Python#pyxll| pyxll]] | + | [https://www.youtube.com/results?search_query=gspread-pandas Youtube search...] |
| + | [https://www.google.com/search?q=gspread-pandas ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/pygsheets/ gspread-pandas] ...Interact with Google Sheet through Pandas DataFrames | ||
| + | |||
| + | ==== <span id="pgsheets"></span>pgsheets ==== | ||
| + | [https://www.youtube.com/results?search_query=pgsheets Youtube search...] | ||
| + | [https://www.google.com/search?q=pgsheets ...Google search] | ||
| + | |||
| + | * [https://pypi.org/project/pgsheets/ pgsheets] ...manipulate Google Sheets Using Pandas DataFrames (independent bidirectional transfer library, using the legacy v3 API, Python 3 only) | ||
| + | |||
| + | === <span id="Python & Excel"></span>Python & Excel === | ||
| + | What is the best library out there for working with [[Excel]] through Python? You can just export to CSV if it's just a table of data that doesn't need any formatting. [[Pandas]] works great for this. You don't need anything else | ||
| + | |||
| + | * [[Excel]] - Data Analysis | [[Microsoft]] | ||
| + | * [[ChatGPT#Microsoft Excel|Excel with ChatGPT]] | ||
| + | * [[Visualization#Python with Excel| Python with Excel Visualization]] | ||
| + | * Building Interactive Python tools with [[Excel]] as a front-end | ||
| + | ** [[Python#pyxll| pyxll]] | ||
** [[Python#xlwings| xlwings]] | ** [[Python#xlwings| xlwings]] | ||
| − | ** [ | + | ** [https://www.pyxll.com/blog/tools-for-working-with-excel-and-python/#pywin32 pywin32, win32com and comtypes] |
| − | * Reading and writing Excel workbooks | + | * Reading and writing [[Excel]] workbooks |
** [[Python#openpyxl| openpyxl]] | ** [[Python#openpyxl| openpyxl]] | ||
** [[Python#XlsxWriter| XlsxWriter]] | ** [[Python#XlsxWriter| XlsxWriter]] | ||
| − | ** [ | + | ** [https://pypi.org/project/xltable/ xltable]... [https://www.pyxll.com/blog/tools-for-working-with-excel-and-python/#xltable notes] |
** [[Python#Pandas|Pandas]] | ** [[Python#Pandas|Pandas]] | ||
| − | ** [ | + | ** [https://www.pyxll.com/blog/tools-for-working-with-excel-and-python/#xlrd xlrd and xlwt] |
| − | + | * pyxll | |
| − | + | * xlwings | |
| − | + | * openpyxl | |
| + | * xlsxWriter | ||
| − | + | ==== <span id="pyxll"></span>pyxll ==== | |
| + | [https://www.youtube.com/results?search_query=pyxll Youtube search...] | ||
| + | [https://www.google.com/search?q=pyxll ...Google search] | ||
| − | PyXLL is an Excel Add-In that enables developers to extend | + | * [https://www.pyxll.com/ pyxll] - Python [[Excel]] Add-In =python(“in [[Excel]]”) |
| + | |||
| + | PyXLL is an [[Excel]] Add-In that enables developers to extend [[Excel]]’s capabilities with Python code. For organizations that want to provide their end users with functionality within [[Excel]], PyXLL makes Python a productive, flexible back-end for [[Excel]] worksheets. With PyXLL, your own Python code runs in [[Excel]] using any Python distribution you like (e.g. [[Anaconda]], Enthought’s Canopy or any other CPython distribution from 2.3 to 3.7). Because PyXLL runs your own full Python distribution you have access to all 3rd party Python packages such as NumPy, Pandas and SciPy and can call them from [[Excel]]. | ||
<youtube>ar_PyW-yPRA</youtube> | <youtube>ar_PyW-yPRA</youtube> | ||
<youtube>ppDt2tIeY0o</youtube> | <youtube>ppDt2tIeY0o</youtube> | ||
| + | ==== <span id="xlwings"></span>xlwings ==== | ||
| + | [https://www.youtube.com/results?search_query=xlwings Youtube search...] | ||
| + | [https://www.google.com/search?q=xlwings ...Google search] | ||
| + | |||
| + | * [https://www.xlwings.org/ xlwings] - Innovative Solutions For [[Microsoft]] [[Excel]] | ||
| + | |||
| + | If you want a user to enter some data in [[Microsoft]] [[Excel]], hand it off to Python, and then show the results to your user in [[Microsoft]] [[Excel]], xlwings is great. | ||
| + | |||
| + | {|<!-- T --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
| + | <youtube>flxjro0S4hg</youtube> | ||
| + | <b>Machine Learning Algorithms in [[Microsoft]] [[Excel]] | Custom [[Excel]] Functions (Part 1 of 2) | ||
| + | </b><br>SATSifaction I this video learn how to add your own custom functions to [[Microsoft]] [[Excel]] using [[Python]]. Integrate both worlds seamlessly. | ||
| + | |} | ||
| + | |<!-- M --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
| + | <youtube>HbWamLW0_wU</youtube> | ||
| + | <b>Machine Learning Algorithms in [[Microsoft]] [[Excel]] | Machine Learning Functions (Part 2 of 2) | ||
| + | </b><br>SATSifaction In this video learn how to add your own Machine Learning Algorithms in [[Excel]] and call them for some simple front end and complex backend calculations | ||
| + | |} | ||
| + | |}<!-- B --> | ||
| + | {|<!-- T --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
| + | <youtube>5iyL9tMw8vA</youtube> | ||
| + | <b>Automate [[Microsoft]] [[Excel]] with Python and xlwings Part 1: Install xlwings and the basic | ||
| + | </b><br>Jie Jenn Buy Me a Coffee? https://www.paypal.me/jiejenn/5 Your donation will support me to continue to make more tutorial videos! xlwings is an open-source library created by Zoomer Analytics to manipulate [[Microsoft]] [[Excel]] with Python. In my opinion, xlwings is probably one of the best libraries out there to interact with Excel application beside win32com. Since you can achieve quiet many things with #xlwings, I construct the tutorial into multiple series. In part 1 of the tutorial, I will be covering just the basic: 1) How to install xlwings, 2) How to create new [[Excel]] file & link existing [[Excel]] file, 3) Write/Read data | ||
| + | |||
| + | Part 2: Run Python Function in [[Excel]] | ||
| + | https://youtu.be/uFJ8wpgoq_E | ||
| + | Part 3: Options and Default Converters | ||
| + | https://youtu.be/lcSgieqjIpY | ||
| − | + | Part 4: Built-in Converters | |
| − | + | https://youtu.be/SuzH6vIwJXc | |
| − | |||
| − | + | Part 5: Access the Missing Features | |
| + | https://youtu.be/_r2zhnUoxIo | ||
| − | + | Visit my website | |
| + | https://LearnDataAnalysis.org | ||
| − | < | + | Supporting my channel so I can continue keep making free content |
| + | |} | ||
| + | |<!-- M --> | ||
| + | | valign="top" | | ||
| + | {| class="wikitable" style="width: 550px;" | ||
| + | || | ||
<youtube>Gk0p3hFRo0k</youtube> | <youtube>Gk0p3hFRo0k</youtube> | ||
| + | <b>The new dynamic arrays in [[Microsoft]] [[Excel]] with Python and xlwings | ||
| + | </b><br>The days of "Ctrl-Shift-Enter" and "You can't change part of an array" are finally over! Say hello to the new dynamic arrays. Dynamic arrays are easily the most revolutionary feature since the introduction of the ribbon and the xlsx format, so make sure to check them out. They also work great with xlwings UDFs (user defined functions). | ||
| + | |} | ||
| + | |}<!-- B --> | ||
| + | ==== <span id="openpyxl"></span>openpyxl ==== | ||
| + | [https://www.youtube.com/results?search_query=openpyxl Youtube search...] | ||
| + | [https://www.google.com/search?q=openpyxl ...Google search] | ||
| − | + | * [https://openpyxl.readthedocs.io/en/stable/ openpyxl] - a Python library for reading and writing [[Excel]] 2010 xlsx/xlsm/xltx/xltm files | |
| − | + | * [https://automatetheboringstuff.com/chapter12/ Working with Excel Spreadsheets | Al Sweigart] - [https://www.amazon.com/gp/product/1593275994/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1593275994&linkCode=as2&tag=playwithpyth-20&linkId=2KIYOE7RFLG7D2RJ Automate the Boring Stuff] | |
| − | |||
| − | |||
| − | * [ | ||
| − | * [ | ||
<youtube>q6Mc_sAPZ2Y</youtube> | <youtube>q6Mc_sAPZ2Y</youtube> | ||
<youtube>A3VRd22fHhA</youtube> | <youtube>A3VRd22fHhA</youtube> | ||
| + | <youtube>Sb0A9i6d320</youtube> | ||
| + | <youtube>7YS6YDQKFh0</youtube> | ||
| + | ==== <span id="XlsxWriter"></span>XlsxWriter ==== | ||
| + | [https://www.youtube.com/results?search_query=XlsxWriter Youtube search...] | ||
| + | [https://www.google.com/search?q=XlsxWriter ...Google search] | ||
| − | + | * [https://pypi.org/project/XlsxWriter/ XlsxWriter] - A Python module for creating [[Excel]] XLSX files. | |
| − | [ | + | * [https://pbpython.com/advanced-excel-workbooks.html Creating Advanced Excel Workbooks with Python | Practical Business Python] |
| − | [ | ||
| − | + | I am just pulling data from an SQL Server, manipulating the results, and then dumping the results into an [[Excel]] spreadsheet. Just working with [[Excel]] cells, and ranges. | |
| − | |||
| − | |||
| − | I am just pulling data from an SQL Server, manipulating the results, and then dumping the results into an Excel spreadsheet. Just working with Excel cells, and ranges. | ||
<youtube>knnhkraHsBg</youtube> | <youtube>knnhkraHsBg</youtube> | ||
| Line 207: | Line 492: | ||
== <span id="PyMC3"></span>PyMC3 == | == <span id="PyMC3"></span>PyMC3 == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=pymc3 Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=pymc3 ...Google search] |
| − | * [ | + | * [https://docs.pymc.io/ PyMC3] - Probabilistic Programming in Python - Bayesian Inference. Fit your model using gradient-based Markov chain [[Monte Carlo]] (MCMC) algorithms like NUTS, using ADVI for fast approximate inference — including minibatch-ADVI for scaling to large datasets — or using Gaussian processes to build Bayesian nonparametric models |
* [[Markov Model (Chain, Discrete Time, Continuous Time, Hidden)]] | * [[Markov Model (Chain, Discrete Time, Continuous Time, Hidden)]] | ||
* [[Markov Decision Process (MDP)]] | * [[Markov Decision Process (MDP)]] | ||
| Line 218: | Line 503: | ||
== <span id="StatsModels"></span>StatsModels == | == <span id="StatsModels"></span>StatsModels == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=statsmodels Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=statsmodels ...Google search] |
| − | * [ | + | * [https://www.statsmodels.org/stable/index.html StatsModels] A module for fitting and estimating many different types of statistical models as well as performing hypothesis testing and exploratory data analysis. It features tools for fitting generalized linear models, survival analyses, and multi-variate statistics. |
<youtube>V86gTgL1FRw</youtube> | <youtube>V86gTgL1FRw</youtube> | ||
| Line 227: | Line 512: | ||
== <span id="OpenCV"></span>OpenCV == | == <span id="OpenCV"></span>OpenCV == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=OpenCV Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=OpenCV ...Google search] |
| − | * [ | + | * [https://opencv.org/ OpenCV] - Open Computer Vision - work with images and/or videos and wish to add a variety of classical and state-of-the-art vision algorithms to their toolbox. |
<youtube>jKtQxvzp1A0</youtube> | <youtube>jKtQxvzp1A0</youtube> | ||
<youtube>sW4CVI51jDY</youtube> | <youtube>sW4CVI51jDY</youtube> | ||
| + | <youtube>WQeoO7MI0Bs</youtube> | ||
== <span id="LibROSA"></span>LibROSA == | == <span id="LibROSA"></span>LibROSA == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=LibROSA Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=LibROSA ...Google search] |
| − | * [ | + | * [https://librosa.github.io/librosa/ LibROSA] - audio and voice processing which can extract various kinds of features from audio segments, such as the rhythm, beats and tempo. |
<youtube>o1OdpUgozs8</youtube> | <youtube>o1OdpUgozs8</youtube> | ||
| Line 245: | Line 531: | ||
== <span id="PyGame"></span>PyGame == | == <span id="PyGame"></span>PyGame == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=PyGame Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=PyGame ...Google search] |
| − | * [https://www.pygame.org/news PyGame] - making multimedia applications like games built on top of the excellent [ | + | * [https://www.pygame.org/news PyGame] - making multimedia applications like games built on top of the excellent [https://www.libsdl.org/ Simple DirectMedia Layer (SDL)] library. |
<youtube>wDIQ17T3sRk</youtube> | <youtube>wDIQ17T3sRk</youtube> | ||
| Line 256: | Line 542: | ||
== Parallel == | == Parallel == | ||
=== <span id="DASK"></span>DASK === | === <span id="DASK"></span>DASK === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=DASK Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=DASK ...Google search] |
| − | * [ | + | * [https://dask.org/ DASK] provides advanced parallelism for [[analytics]], enabling performance at scale for the tools you love - it is developed in coordination with other community projects like [[Python#NumPy |NumPy]], [[Python#Pandas |Pandas]], and [[Python#scikit-learn |scikit-learn]] |
<youtube>tQBovBvSDvA</youtube> | <youtube>tQBovBvSDvA</youtube> | ||
| Line 265: | Line 551: | ||
=== <span id="Joblib"></span>Joblib === | === <span id="Joblib"></span>Joblib === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Joblib Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Joblib ...Google search] |
| − | * [ | + | * [https://joblib.readthedocs.io/en/latest/ Joblib] provide lightweight pipelining |
<youtube>nEyYt-CHRZo</youtube> | <youtube>nEyYt-CHRZo</youtube> | ||
| Line 274: | Line 560: | ||
=== <span id="Tornado"></span>Tornado === | === <span id="Tornado"></span>Tornado === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Tornado+Python+framework Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Tornado+Python+framework ...Google search] |
| − | * [ | + | * [https://www.tornadoweb.org/en/stable/ Tornado] is a web framework and asynchronous networking library. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user. |
<youtube>SkETonolR3U</youtube> | <youtube>SkETonolR3U</youtube> | ||
| Line 283: | Line 569: | ||
== <span id="Numba"></span>Numba == | == <span id="Numba"></span>Numba == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Numba+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Numba+python ...Google search] |
| − | * [ | + | * [https://numba.pydata.org/ Numba] JIT compiler that translates a subset of Python and NumPy code into fast machine code. |
<youtube>NoJr08FNQeg</youtube> | <youtube>NoJr08FNQeg</youtube> | ||
| Line 292: | Line 578: | ||
== <span id="xarray"></span>xarray == | == <span id="xarray"></span>xarray == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=xarray+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=xarray+python ...Google search] |
| − | * [ | + | * [https://xarray.pydata.org/en/stable/ xarray] working with labelled multi-dimensional arrays simple, and efficient. Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like arrays, which allows for a more intuitive, more concise, and less error-prone developer experience. The package includes a large and growing library of domain-agnostic functions for advanced [[analytics]] and visualization with these data structures. Xarray was inspired by and borrows heavily from [[Python#Pandas |Pandas]], the popular data analysis package focused on labelled tabular data. It is particularly tailored to working with netCDF files, which were the source of xarray’s data model, and integrates tightly with [[Python#DASK |DASK]] for parallel computing. |
<youtube>Dgr_d8iEWk4</youtube> | <youtube>Dgr_d8iEWk4</youtube> | ||
| Line 301: | Line 587: | ||
== <span id="IPython Blocks"></span>IPython Blocks == | == <span id="IPython Blocks"></span>IPython Blocks == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=ipythonblocks Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=ipythonblocks ...Google search] |
| − | * [ | + | * [https://www.ipythonblocks.org/ IPython Blocks] a tool for practicing Python in the [[Jupyter]] giving learners a grid of colors to manipulate while practicing for loops, if statements, and other aspects of Python. |
<youtube>L1cCFA13iHo</youtube> | <youtube>L1cCFA13iHo</youtube> | ||
== <span id="Metaflow"></span>Metaflow == | == <span id="Metaflow"></span>Metaflow == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Metaflow+Netflix+AWS Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Metaflow+Netflix+AWS ...Google search] |
| − | * [ | + | * [https://metaflow.org/ Metaflow], Netflix and AWS open source Python library |
<youtube>8jTo3bVni3E</youtube> | <youtube>8jTo3bVni3E</youtube> | ||
| Line 322: | Line 608: | ||
=== <span id="Requests"></span>Requests === | === <span id="Requests"></span>Requests === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Requests+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Requests+python ...Google search] |
| − | * [ | + | * [https://requests.kennethreitz.org/en/master/ Requests.org] simple HTTP library |
| − | * [ | + | * [https://realpython.com/python-requests/ Requests] http library |
<youtube>tb8gHvYlCFs</youtube> | <youtube>tb8gHvYlCFs</youtube> | ||
| Line 332: | Line 618: | ||
=== <span id="Beautiful Soup - bs4"></span>Beautiful Soup - bs4 === | === <span id="Beautiful Soup - bs4"></span>Beautiful Soup - bs4 === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Beautiful+Soup+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Beautiful+Soup+python ...Google search] |
| − | * [ | + | * [https://pypi.org/project/beautifulsoup4 Beautiful Soup Project] for parsing HTML and XML documents. It creates parse trees |
Allows you to import its functions and use them in-line. Therefore, you could even use it in your Jupyter notebooks. | Allows you to import its functions and use them in-line. Therefore, you could even use it in your Jupyter notebooks. | ||
| Line 343: | Line 629: | ||
=== <span id="Scrapy"></span>Scrapy === | === <span id="Scrapy"></span>Scrapy === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Scrapy+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Scrapy+python ...Google search] |
| − | * [ | + | * [https://scrapy.org/ Scrapy] webscraping .. open source and collaborative framework for extracting the data you need from websites |
<youtube>Wp6LRijW9wg</youtube> | <youtube>Wp6LRijW9wg</youtube> | ||
<youtube>OJ8isyws2yw</youtube> | <youtube>OJ8isyws2yw</youtube> | ||
=== <span id="Selenium"></span>Selenium === | === <span id="Selenium"></span>Selenium === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Selenium+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Selenium+python ...Google search] |
| − | * [ | + | * [https://selenium-python.readthedocs.io/ Selenium with Python] a web testing library. It is used to automate browser activities. |
| − | * [ | + | * [https://wiki.saucelabs.com/display/DOCS/Getting+Started+with+Selenium+for+Automated+Website+Testing SauceLabs] |
Initialises a web browser such as Chrome and then simulates all the actions defined in the code; JavaScript functions to e.g. register an account, then log in and get the content after clicking some buttons and links. | Initialises a web browser such as Chrome and then simulates all the actions defined in the code; JavaScript functions to e.g. register an account, then log in and get the content after clicking some buttons and links. | ||
| Line 363: | Line 649: | ||
== <span id="Twisted"></span>Twisted == | == <span id="Twisted"></span>Twisted == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Twisted+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Twisted+python ...Google search] |
| − | * [ | + | * [https://twistedmatrix.com Twisted] an event-driven networking engine |
Twisted has been around a long time in the Python world. Pioneering the Deferred abstraction, which later turned into Promises and found their way into JavaScript, it is a fertile ground for asynchronous I/O experimentation. Through its groundbreaking protocol/transport design that some of us might take for granted these days, and a strict adherence to unit testing and representing things through abstract interfaces, it lets you talk a lot of different network protocols without really having to know everything about them. | Twisted has been around a long time in the Python world. Pioneering the Deferred abstraction, which later turned into Promises and found their way into JavaScript, it is a fertile ground for asynchronous I/O experimentation. Through its groundbreaking protocol/transport design that some of us might take for granted these days, and a strict adherence to unit testing and representing things through abstract interfaces, it lets you talk a lot of different network protocols without really having to know everything about them. | ||
| Line 376: | Line 662: | ||
== <span id="Pipeline"></span>Pipelines == | == <span id="Pipeline"></span>Pipelines == | ||
| − | * [[AIOps]] - Machine Learning (ML) pipelines for SecDevOps | + | * [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] - Machine Learning (ML) pipelines for SecDevOps |
| − | * [ | + | * [https://docs.ansible.com/ansible/latest/dev_guide/developing_python_3.html Ansible and Python 3 | Red Hat] |
| − | * [ | + | * [https://docs.saltstack.com/en/latest/ref/clients/index.html Python Client API | Saltstack] |
| − | Python is one of the most crucial orchestration and infrastructure automation components of AIOps to reduce or almost eliminates disconnect between developers and system admins. AIOps is centered on enabling AI pipelines for continuous integration and continuous deployment (CI/CD) with no downtime. | + | Python is one of the most crucial orchestration and infrastructure automation components of [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] to reduce or almost eliminates disconnect between developers and system admins. [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] is centered on enabling AI pipelines for continuous integration and continuous deployment (CI/CD) with no downtime. |
=== <span id="Vaex"></span>Vaex === | === <span id="Vaex"></span>Vaex === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=vaex+Dataframe+pipeline Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=vaex+Dataframe+pipeline ...Google search] |
| − | * [ | + | * [https://towardsdatascience.com/ml-impossible-train-a-1-billion-sample-model-in-20-minutes-with-vaex-and-scikit-learn-on-your-9e2968e6f385 ML impossible: Train 1 billion samples in 5 minutes on your laptop using Vaex and Scikit-Learn - Make your laptop feel like a supercomputer. | Jovan Veljanoski - Towards Data Science] |
<youtube>ELtjRdPT8is</youtube> | <youtube>ELtjRdPT8is</youtube> | ||
| Line 392: | Line 678: | ||
=== <span id="PyCaret"></span>PyCaret === | === <span id="PyCaret"></span>PyCaret === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=PyCaret+pipeline+AIOp Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=PyCaret+pipeline+AIop ...Google search] |
| − | * [ | + | * [https://towardsdatascience.com/deploy-machine-learning-model-on-google-kubernetes-engine-94daac85108b Deploy Machine Learning Pipeline on Google Kubernetes Engine | Moez Ali - Towards Data Science] |
<youtube>cnxOGWtwdv8</youtube> | <youtube>cnxOGWtwdv8</youtube> | ||
| Line 402: | Line 688: | ||
=== <span id="TPOT"></span>TPOT === | === <span id="TPOT"></span>TPOT === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=TPOT Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=TPOT ...Google search] |
| − | * [ | + | * [https://automl.info/tpot/ TPOT | Randal Olson - University of Pennsylvania] - automatically creates and optimizes full machine learning [[Algorithm Administration#AIOps/MLOps|pipelines]] using genetic programming. The Tree-Based [[Algorithm Administration#AIOps/MLOps|pipeline]] Optimization Tool (TPOT) automates the building of ML [[Algorithm Administration#AIOps/MLOps|pipelines]] by combining a flexible expression tree representation of [[Algorithm Administration#AIOps/MLOps|pipelines]] with stochastic search algorithms such as genetic programming. TPOT makes use of the Python-based [[Python#scikit-learn|scikit-learn]] library as its ML menu. |
| − | + | https://automl.info/wp-content/uploads/2017/07/tpot-pipeline-example-768x361.png | |
<youtube>nDF7_8FOhpI</youtube> | <youtube>nDF7_8FOhpI</youtube> | ||
| Line 413: | Line 699: | ||
=== <span id="ELI5"></span>ELI5 === | === <span id="ELI5"></span>ELI5 === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=ELI5 Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=ELI5 ...Google search] |
| − | * [ | + | * [https://pypi.org/project/eli5/ ELI5] "Explain it like I'm 5" helps to... |
** debug machine learning classifiers and explain their predictions. | ** debug machine learning classifiers and explain their predictions. | ||
| − | *** [[Python#scikit-learn| scikit-learn]] - Currently ELI5 allows to explain weights and predictions of scikit-learn linear classifiers and regressors, print decision trees as text or as SVG, show feature importances and explain predictions of decision trees and tree-based ensembles. ELI5 understands text processing utilities from [[Python#scikit-learn| scikit-learn]] and can highlight text data accordingly. [[AIOps | Pipeline]] and FeatureUnion are supported. It also allows to debug [[Python#scikit-learn| scikit-learn]] [[AIOps | | + | *** [[Python#scikit-learn| scikit-learn]] - Currently ELI5 allows to explain weights and predictions of scikit-learn linear classifiers and regressors, print decision trees as text or as SVG, show feature importances and explain predictions of decision trees and tree-based ensembles. ELI5 understands text processing utilities from [[Python#scikit-learn| scikit-learn]] and can highlight text data accordingly. [[Algorithm Administration#AIOps/MLOps|Pipeline]] and FeatureUnion are supported. It also allows to debug [[Python#scikit-learn| scikit-learn]] [[Algorithm Administration#AIOps/MLOps|pipelines]] which contain HashingVectorizer, by undoing hashing. |
| − | *** [ | + | *** [https://github.com/dmlc/xgboost xgboost] - show feature importances and explain predictions of XGBClassifier, XGBRegressor and xgboost.Booster. |
| − | *** [https://github.com/Microsoft/LightGBM LightGBM] - show feature importances and explain predictions of LGBMClassifier and LGBMRegressor. A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks. It is under the umbrella of the [ | + | *** [[LightGBM]] ...Microsoft's gradient boosting framework that uses tree based learning algorithms ... [https://github.com/Microsoft/LightGBM LightGBM] - show feature importances and explain predictions of LGBMClassifier and LGBMRegressor. A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks. It is under the umbrella of the [https://github.com/microsoft/dmtk Microsoft Distributed Machine Learning Toolkit (DMTK)] project of Microsoft. |
| − | *** [ | + | *** [https://github.com/catboost/catboost CatBoost] - show feature importances of CatBoostClassifier, CatBoostRegressor and catboost.CatBoost. A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on [[Processing Units - CPU, GPU, APU, TPU, VPU, FPGA, QPU|CPU and GPU]] |
| − | *** [ | + | *** [https://github.com/scikit-learn-contrib/lightning lightning] - explain weights and predictions of lightning classifiers and regressors. Large-scale linear classification, regression and ranking in Python |
| − | *** [ | + | *** [https://github.com/TeamHG-Memex/sklearn-crfsuite sklearn-crfsuite] ELI5 allows to check weights of sklearn_crfsuite.CRF models. [https://www.chokkan.org/software/crfsuite/ CRFsuite] is an implementation of Conditional Random Fields (CRFs) for labeling sequential data. |
**ELI5 also implements several algorithms for inspecting black-box models (see Inspecting Black-Box Estimators): | **ELI5 also implements several algorithms for inspecting black-box models (see Inspecting Black-Box Estimators): | ||
| − | *** TextExplainer allows to explain predictions of any text classifier using [ | + | *** TextExplainer allows to explain predictions of any text classifier using [https://arxiv.org/pdf/1602.04938.pdf LIME algorithm]. There are utilities for using [[Python#LIME|LIME]] with non-text data and arbitrary black-box classifiers as well, but this feature is currently experimental. |
*** Permutation importance method can be used to compute feature importances for black box estimators. | *** Permutation importance method can be used to compute feature importances for black box estimators. | ||
| − | Explanation and formatting are separated; you can get text-based explanation to display in console, HTML version embeddable in an IPython notebook or web [[Analytics|dashboards]], a [[Python#Pandas DataFrame|Pandas DataFrame]] object if you want to process results further, or [ | + | Explanation and formatting are separated; you can get text-based explanation to display in console, HTML version embeddable in an IPython notebook or web [[Analytics|dashboards]], a [[Python#Pandas DataFrame|Pandas DataFrame]] object if you want to process results further, or [https://www.w3schools.com/js/js_json.asp JSON] version which allows to implement custom rendering and formatting on a client. |
| − | |||
<youtube>s-yT5Is1G1A</youtube> | <youtube>s-yT5Is1G1A</youtube> | ||
| Line 434: | Line 719: | ||
== <span id="yellowbrick"></span>yellowbrick == | == <span id="yellowbrick"></span>yellowbrick == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=yellowbrick+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=yellowbrick+python ...Google search] |
| − | * [ | + | * [https://www.scikit-yb.org/en/latest/ Yellowbrick: Machine Learning Visualization] |
| − | |||
This library is essentially an extension of the scikit-learn library and provides some really useful and pretty looking visualisations for machine learning models. The visualiser objects, the core interface, are scikit-learn estimators and so if you are used to working with scikit-learn the workflow should be quite familiar. | This library is essentially an extension of the scikit-learn library and provides some really useful and pretty looking visualisations for machine learning models. The visualiser objects, the core interface, are scikit-learn estimators and so if you are used to working with scikit-learn the workflow should be quite familiar. | ||
| Line 446: | Line 730: | ||
== <span id="MLxtend"></span>MLxtend == | == <span id="MLxtend"></span>MLxtend == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=MLxtend+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=MLxtend+python ...Google search] |
* [https://pypi.org/project/mlxtend/ MLxtend] | * [https://pypi.org/project/mlxtend/ MLxtend] | ||
| − | |||
| − | This library contains a host of helper functions for machine learning. This covers things like stacking and voting classifiers, model evaluation, feature extraction and engineering and plotting. | + | This library contains a host of helper functions for machine learning. This covers things like stacking and voting classifiers, model [[evaluation]], feature extraction and engineering and plotting. |
<youtube>axRAVFsMSCw</youtube> | <youtube>axRAVFsMSCw</youtube> | ||
== <span id="LIME"></span>LIME == | == <span id="LIME"></span>LIME == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=LIME+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=LIME+python ...Google search] |
| − | * [ | + | * [https://github.com/marcotcr/lime LIME] (Local Interpretable Model-agnostic Explanations) explains the prediction of any classifier in an interpretable and faithful manner by learning a interpretable model locally around the prediction. |
| − | * [ | + | * [https://blog.dominodatalab.com/shap-lime-python-libraries-part-1-great-explainers-pros-cons/ SHAP and LIME Python Libraries: Part 1 – Great Explainers, with Pros and Cons to Both | Joshua Poduska - Domino] |
| − | * [ | + | * [https://towardsdatascience.com/decrypting-your-machine-learning-model-using-lime-5adc035109b5 Decrypting your Machine Learning model using LIME | Abhishek Sharma - Towards Data Science] |
| − | |||
<youtube>KP7-JtFMLo4</youtube> | <youtube>KP7-JtFMLo4</youtube> | ||
| Line 469: | Line 751: | ||
== <span id="SHAP"></span>SHAP == | == <span id="SHAP"></span>SHAP == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=SHAP+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=SHAP+python ...Google search] |
| − | * [ | + | * [https://pypi.org/project/shap/ SHAP] (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations, uniting several previous methods and representing the only possible consistent and locally accurate additive feature attribution method based on expectations. |
| − | * [ | + | * [https://github.com/slundberg/shap Shapley Additive Explanations (SHAP)] |
| − | * [ | + | * [https://medium.com/civis-analytics/demystifying-black-box-models-with-shap-value-analysis-3e20b536fc80 Demystifying Black-Box Models with SHAP Value Analysis | Peter Cooman] |
| − | |||
| − | + | https://raw.githubusercontent.com/slundberg/shap/master/docs/artwork/shap_diagram.png | |
<youtube>xwl8WhtJNs0</youtube> | <youtube>xwl8WhtJNs0</youtube> | ||
<youtube>C80SQe16Rao</youtube> | <youtube>C80SQe16Rao</youtube> | ||
| + | |||
| + | = Leveraging Large Language Models (LLM) = | ||
| + | [https://www.youtube.com/results?search_query=Applications+Python+Model+LLM Youtube search...] | ||
| + | [https://www.google.com/search?q=Applications+Python+Model+LLM ...Google search] | ||
| + | |||
| + | * [[LangChain]] | ||
| + | * [https://www.pinecone.io/ Pinecone] | ||
| + | * [https://github.com/bentoml/OpenLLM OpenLLM] | ||
| + | * [https://github.com/guardrails-ai/guardrails Guardrails] | ||
| + | * [https://github.com/oughtinc/ice Interactive Composition Explorer (ICE)] | ||
| + | * [https://pypi.org/project/marvin/ Marvin] | ||
| + | * [https://www.infoq.com/presentations/llm-structured-data/ LLMs in the Real World: Structuring Text with Declarative NLP | Adam Azzam - InfoQ] | ||
| + | |||
| + | == Pydantic == | ||
| + | [https://www.youtube.com/results?search_query=Pydantic+Python Youtube search...] | ||
| + | [https://www.google.com/search?q=Pydantic+Python ...Google search] | ||
| + | |||
| + | * [https://pydantic.dev/ Pydantic] | ||
| + | * [https://thenewstack.io/using-pydantic-to-validate-json-documents-with-couchbase/ Using Pydantic to Validate JSON Documents with Couchbase] | ||
| + | |||
| + | Pydantic is a data validation library for Python that leverages type hints to define the structure and constraints of data models. It provides a simple and intuitive interface for creating and validating data models, making it a popular choice for a wide range of applications. | ||
| + | |||
| + | Key Features of Pydantic: | ||
| + | * <b>Data Validation</b>: Pydantic ensures the integrity of data by validating its type, format, and constraints. | ||
| + | * <b>Type Annotations</b>: Pydantic utilizes Python type hints to define the structure of data models, making the code self-documenting and enhancing developer experience. | ||
| + | * <b>Data Serialization and Deserialization</b>: Pydantic seamlessly converts data models into various formats, including JSON, YAML, and TOML. | ||
| + | * <b>Customizable Validation</b>: Pydantic empowers developers to create custom validation rules for specific use cases. | ||
| + | * <b>Seamless Integration</b>: Pydantic integrates smoothly with popular Python frameworks like FastAPI, Django, and Flask. | ||
| + | |||
| + | Benefits of Using Pydantic: | ||
| + | |||
| + | * <b>Improved Code Quality</b>: Pydantic's type annotations and validation checks contribute to cleaner and more reliable code. | ||
| + | * <b>Reduced Errors</b>: Pydantic's validation mechanisms prevent invalid data from entering the application, reducing the risk of errors. | ||
| + | * <b>Enhanced Developer Experience</b>: Pydantic's intuitive syntax and clear error messages make it easy for developers to work with data models. | ||
| + | * <b>Documentation Generation</b>: Pydantic can generate documentation for data models, enhancing code comprehension. | ||
| + | * <b>Integration with Tools</b>: Pydantic's JSON Schema generation enables integration with various tools and libraries. | ||
| + | |||
| + | Common Use Cases for Pydantic: | ||
| + | |||
| + | * <b>API Request and Response Validation</b>: Pydantic validates data exchanged between APIs, ensuring data integrity and consistency. | ||
| + | * <b>Data Configuration Management</b>: Pydantic manages application configuration data, ensuring proper configuration and preventing errors. | ||
| + | * <b>Data Persistence and Storage</b>: Pydantic validates data before persisting it to databases or file systems, maintaining data integrity. | ||
| + | * <b>Data Exchange between Services</b>: Pydantic ensures data consistency when exchanging data between microservices or distributed systems. | ||
| + | * <b>Data Modeling and Abstraction</b>: Pydantic abstracts data structures and provides a consistent way to interact with data, simplifying application development. | ||
| + | |||
| + | <youtube>yj-wSRJwrrc</youtube> | ||
= Python Stack = | = Python Stack = | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Stack+Python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Stack+Python ...Google search] |
| − | * [ | + | * [https://docs.openstack.org/mitaka/user-guide/sdk.html openstack] - open source software for creating private and public clouds |
| − | * [ | + | * [https://www.fullstackpython.com/ Full Stack Python - Book | Matt Makai] |
| − | ** [ | + | ** [https://www.fullstackpython.com/table-of-contents.html Full Stack Python - Table of contents | Matt Makai] |
<youtube>s6NaOKD40rY</youtube> | <youtube>s6NaOKD40rY</youtube> | ||
| Line 494: | Line 821: | ||
=== <span id="Flask"></span>Flask === | === <span id="Flask"></span>Flask === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Flask+Python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Flask+Python ...Google search] |
| − | * [ | + | * [https://www.deploypython.com/deploying-flask-web-apps.html Deploying Flask Web Applications | Matt Makai] [https://gumroad.com/l/python-deployments GumRoad.com to purchase] |
| − | * [ | + | * [https://flask.pocoo.org/ Flask] a microframework for Python. It is classified as a microframework because it does not require particular tools or libraries. It has no database abstraction layer, form validation, or any other components where pre-existing third-party libraries provide common functions. |
| − | ** [ | + | ** [https://github.com/MaxHalford/flask-boilerplate Flask boilerplate | Max Halford] |
| − | ** [ | + | ** [https://github.com/alectrocute/flaskSaaS flaskSaaS | Max Halford] starting point to build your SaaS in Flask & Python, with Stripe subscription billing |
| − | ** [ | + | ** [https://github.com/bboe/flask-image-uploader flask-image-uploader | bboe] |
| − | ** [ | + | ** [https://flask-login.readthedocs.io/en/latest/ Flask-Login] for the user accounts |
| − | ** [ | + | ** [https://flask-sqlalchemy.palletsprojects.com/en/2.x/ Flask-SQLAlchemy] interacting with the database |
| − | ** [ | + | ** [https://flask-wtf.readthedocs.io/en/latest/ Flask-WTF] and [https://wtforms.readthedocs.io/en/latest/ WTForms] for the form handling. |
| − | ** [ | + | ** [https://pythonhosted.org/Flask-Mail/ Flask-Mail] for sending mails. |
| − | ** [ | + | ** [https://flask-bcrypt.readthedocs.io/en/latest/ Flask-Bcrypt] for generating secret user passwords. |
| − | ** [ | + | ** [https://flask-admin.readthedocs.io/en/latest/ Flask-Admin] for building an administration interface. |
| − | ** [ | + | ** [https://flask-script.readthedocs.io/en/latest/ Flask-Script] for managing the app |
| − | ** [ | + | ** [https://github.com/mjhea0/flask-stripe flask-stripe] Stripe Checkout & user registration |
* [[TensorFlow Serving]] | * [[TensorFlow Serving]] | ||
| − | * [ | + | * [[Journalism|News]] Aggregation |
| + | * [https://becominghuman.ai/creating-restful-api-to-tensorflow-models-c5c57b692c10 Creating REST API for TensorFlow models | Vitaly Bezgachev - Medium - Becoming Human] | ||
| − | Flask is considered more Pythonic than the [[Python#Django|Django]] web framework because in common situations the equivalent Flask web application is more explicit. Flask is also easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running. [ | + | Flask is considered more Pythonic than the [[Python#Django|Django]] web framework because in common situations the equivalent Flask web application is more explicit. Flask is also easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running. [https://www.fullstackpython.com/flask.html Flask | Full Stack Python] |
| − | + | https://miro.medium.com/max/700/1*yY0ngG41QQ63ukXuoZM4dQ.png | |
<youtube>Z1RJmh_OqeA</youtube> | <youtube>Z1RJmh_OqeA</youtube> | ||
| Line 522: | Line 850: | ||
<youtube>AsoJL9GPi1k</youtube> | <youtube>AsoJL9GPi1k</youtube> | ||
| − | ==== Flask & [[ | + | ==== Flask & [[JavaScript#React|React]] ==== |
<youtube>YW8VG_U-m48</youtube> | <youtube>YW8VG_U-m48</youtube> | ||
<youtube>TKIHpoF8ZIk</youtube> | <youtube>TKIHpoF8ZIk</youtube> | ||
| Line 530: | Line 858: | ||
<youtube>c6NIsCogMHI</youtube> | <youtube>c6NIsCogMHI</youtube> | ||
| − | ==== Flask, [[ | + | ==== Flask, [[JavaScript#React|React]], & Docker ==== |
| − | * [ | + | * [https://medium.com/@riken.mehta/full-stack-tutorial-flask-react-docker-420da3543c91 Full-stack tutorial: Flask + React + Docker | Riken Mehta - Medium] |
| − | * [ | + | * [https://medium.com/@dummydevops/containerizing-a-flask-react-app-with-docker-compose-5c06ef73cc8f Containerizing a Flask + React app with docker-compose | Devops Dummy - Medium] |
| − | * [ | + | * [https://mherman.org/presentations/microservices-flask-docker/#1 Developing and Testing Microservices with Docker, Flask, and React | Michael Herman] |
| − | * [ | + | * [https://testdriven.io/courses/microservices-with-docker-flask-and-react/?utm_source=fsp Microservices with Docker, Flask, and React | testdriven.io (course)] |
| − | + | https://testdriven.io/static/images/courses/microservices/07_testdriven.png | |
=== <span id="Django"></span>Django === | === <span id="Django"></span>Django === | ||
| − | * [ | + | * [https://www.djangoproject.com/ Django] - a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source. |
| + | * [[News]] Aggregation | ||
| − | Django is a widely-used Python web application framework with a "batteries-included" philosophy. The principle behind batteries-included is that the common functionality for building web applications should come with the framework instead of as separate libraries. [ | + | Django is a widely-used Python web application framework with a "batteries-included" philosophy. The principle behind batteries-included is that the common functionality for building web applications should come with the framework instead of as separate libraries. [https://www.fullstackpython.com/django.html Django | Full Stack Python] |
<youtube>F5mRW0jo-U4</youtube> | <youtube>F5mRW0jo-U4</youtube> | ||
| Line 547: | Line 876: | ||
=== Other Web Frameworks supporting Python === | === Other Web Frameworks supporting Python === | ||
| − | * [ | + | * [https://anvil.works/index-1 Anvil] |
| − | * [ | + | * [https://bobo.readthedocs.io/en/latest/ bobo] |
| − | * [ | + | * [https://bottlepy.org/docs/dev/index.html Bottle] |
| − | * [ | + | * [https://cherrypy.org/ CherryPy] |
| − | * [ | + | * [https://cyclone.io/ Cyclone] |
| − | * [ | + | * [https://falconframework.org/ Falcon] |
| − | * [ | + | * [https://github.com/toastdriven/itty/ Itty-Bitty] |
| − | * [ | + | * [https://klein.readthedocs.io/en/latest/ Klein] |
| − | * [ | + | * [https://morepath.readthedocs.io/en/latest/ Morepath] |
| − | * [ | + | * [https://github.com/klen/muffin Muffin] |
| − | * [ | + | * [https://github.com/aisola/ObjectWeb ObjectWeb] |
| − | * [ | + | * [https://pecan.readthedocs.io/en/latest/index.html Pecan] |
| − | * [ | + | * [https://trypyramid.com/ Pyramid] |
| − | * [ | + | * [https://rayframework.github.io/site/ Ray] |
| − | * [ | + | * [https://github.com/huge-success/sanic Sanic] |
| − | * [ | + | * [https://www.tornadoweb.org/en/stable/ Tornado] |
| − | * [ | + | * [https://www.turbogears.org/ TurboGears] |
| − | * [ | + | * [https://vibora.io/ Vibora] |
| − | * [ | + | * [https://www.web2py.com/ Web2py] |
| − | * [ | + | * [https://pythonhosted.org/wheezy.web/ Wheezy Web] |
| + | |||
| + | = [[Forecasting#Time Series Forecasting|Time Series]] = | ||
| + | [https://www.youtube.com/results?search_query=Time+Series+python Youtube search...] | ||
| + | [https://www.google.com/search?q=Time+Series+python ...Google search] | ||
| + | |||
| + | == <span id="tsfresh"></span>tsfresh == | ||
| + | [https://www.youtube.com/results?search_query=tsfresh+Time+Series+python Youtube search...] | ||
| + | [https://www.google.com/search?q=tsfresh+Time+Series+python ...Google search] | ||
| + | |||
| + | * [https://tsfresh.readthedocs.io/en/latest/ tsfresh] ...python package that automatically calculates a large number of [[Forecasting#Time Series Forecasting|time series]] characteristics, the so called features. Further the package contains methods to evaluate the explaining power and importance of such characteristics for regression or classification tasks. | ||
| + | |||
| + | <youtube>Fm8zcOMJ-9E</youtube> | ||
| + | |||
| + | == <span id="STUMPY"></span>STUMPY == | ||
| + | [https://www.youtube.com/results?search_query=STUMPY+Time+Series+python Youtube search...] | ||
| + | [https://www.google.com/search?q=STUMPY+Time+Series+python ...Google search] | ||
| + | |||
| + | * [https://stumpy.readthedocs.io/en/latest/Tutorial_STUMPY_Basics.html Analyzing Motifs and Anomalies with STUMP] | ||
| + | |||
| + | an open source scientific Python library that implements a novel yet intuitive approach for discovering patterns, anomalies, and other insights from any time series data. STUMPY is a powerful and scalable library that efficiently computes something called the matrix profile, which can be used for a variety of time series data mining tasks such as pattern/motif (approximately repeated subsequences within a longer time series) discovery, anomaly/novelty (discord) discovery, shapelet discovery, semantic segmentation, streaming (on-line) data | ||
| + | fast approximate matrix profiles, time series chains (temporally ordered set of subsequence patterns). [https://stumpy.readthedocs.io/en/latest/index.html STUMPY] | ||
| + | |||
| + | <youtube>5O7GhfZvpnY</youtube> | ||
| + | <youtube>g2jdWfzoCJU</youtube> | ||
= <span id="Visualization with Python"></span>Visualization with Python = | = <span id="Visualization with Python"></span>Visualization with Python = | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Python+Visualization Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Python+Visualization ...Google search] |
| − | * [[Visualization]] | + | * [[Visualization#Python| Python Visualization]] |
| − | * [ | + | * [https://www.anaconda.com/python-data-visualization-2018-why-so-many-libraries/ Python Data Visualization 2018: Why So Many Libraries? | James A. Bednar] |
| − | * [ | + | * [https://pyviz.org/ PyViz.org] meta-initiative for helping users decide on the best open-source Python data visualization tools for their purposes, with links, overviews, comparisons, and examples. |
| − | ** [ | + | ** [https://towardsdatascience.com/pyviz-simplifying-the-data-visualisation-process-in-python-1b6d2cb728f1 PyViz: Simplifying the Data Visualization process in Python | Parul Pandey - Towards Data Science] |
* Pandas .plot() - basic plotting interface uses [[Python#Matplotlib |Matplotlib]] to render static PNGs in a [[Jupyter]] notebook or for exporting from Python, with a command that can be as simple as df.plot() for a DataFrame with two columns. | * Pandas .plot() - basic plotting interface uses [[Python#Matplotlib |Matplotlib]] to render static PNGs in a [[Jupyter]] notebook or for exporting from Python, with a command that can be as simple as df.plot() for a DataFrame with two columns. | ||
| − | * [ | + | * [https://datashader.org/ Datashader] breaks the creation of images into a series of explicit steps that allow computations to be done on intermediate representations. Rasterizing huge datasets quickly as fixed-size images. |
| − | * [ | + | * [https://panel.pyviz.org/?source=post_page--------------------------- Panel] -assembling objects from many different libraries into a layout or app, whether in a [[Jupyter]] notebook or in a standalone serveable [[Analytics|dashboard]] |
| − | * [ | + | * [https://param.pyviz.org/?source=post_page--------------------------- Param] -declaring user-relevant parameters, making it simple to work with widgets inside and outside of a notebook [[context]] |
| − | * [ | + | * [https://www.fullstackpython.com/data-visualization.html Data Visualization |][https://www.fullstackpython.com/table-of-contents.html Full Stack Python] |
| − | + | https://www.anaconda.com/wp-content/uploads/2019/01/PythonVisLandscape.jpg | |
<youtube>FytuB8nFHPQ</youtube> | <youtube>FytuB8nFHPQ</youtube> | ||
| Line 590: | Line 943: | ||
== <span id="Matplotlib"></span>Matplotlib == | == <span id="Matplotlib"></span>Matplotlib == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Matplotlib Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Matplotlib ...Google search] |
| − | * [ | + | * [https://matplotlib.org/ Matplotlib] generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc |
| − | * [ | + | * [https://jakevdp.github.io/PythonDataScienceHandbook/04.00-introduction-to-matplotlib.html Visualization with Matplotlib] | [[Creatives#Jake VanderPlas |Jake VanderPlas]] - [https://tanthiamhuat.files.wordpress.com/2018/04/pythondatasciencehandbook.pdf Python Data Science Handbook] |
* Driving your graphic via [[Jupyter#ipyWidgets|ipyWidgets]] | * Driving your graphic via [[Jupyter#ipyWidgets|ipyWidgets]] | ||
| Line 603: | Line 956: | ||
== <span id="seaborn"></span>seaborn == | == <span id="seaborn"></span>seaborn == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=seaborn+python Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=seaborn+python ...Google search] |
| − | * [ | + | * [https://seaborn.pydata.org/ seaborn] - complements [[Python#Matplotlib |Matplotlib]] and works well with [[Python#Pandas DataFrame |Pandas DataFrame]]s |
<youtube>TLdXM0A7SR8</youtube> | <youtube>TLdXM0A7SR8</youtube> | ||
| Line 612: | Line 965: | ||
== <span id="Plotly"></span>Plotly == | == <span id="Plotly"></span>Plotly == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Plotly Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Plotly ...Google search] |
| − | * [ | + | * [https://plot.ly/python/ Plotly | Plotly] - graphing library makes interactive, publication-quality graphs online. Examples of how to make line plots, scatter plots, area charts, bar charts, error bars, box plots, histograms, heatmaps, subplots, multiple-axes, polar charts, and bubble charts. Plotly is a web-based service by default, but you can use the library offline in Python and upload plots to Plotly's free, public server or paid, private server. From there, you can embed your plots in a web page. All Plotly graphs include tooltips, and you can build custom controls (like sliders and filters) on top of a chart once it's embedded using Plotly's JavaScript API. Plotly.js is based on [https://d3js.org/ D3.js] and [https://get.webgl.org/ WebGL.js]. Another way to work in Plotly and share plots is in Mode [[Notebooks]]. You can pull data with SQL, use the Plotly offline library in the Python Notebook to plot the results of your query, and then add the interactive chart to a report. [https://go.plot.ly/get-pricing Get pricing] |
| − | * [ | + | * [https://www.superdatascience.com/pages/learn-plotly Learn Plotly | SuperDataScience] |
* Driving your graphic via [[Jupyter#ipyWidgets|ipyWidgets]] | * Driving your graphic via [[Jupyter#ipyWidgets|ipyWidgets]] | ||
| − | * [ | + | * [https://towardsdatascience.com/its-2019-make-your-data-visualizations-interactive-with-plotly-b361e7d45dc6 It’s 2019 — Make Your Data Visualizations Interactive with Plotly | Jeff Hale - Towards Data Science] Find the path to make awesome figures quickly with Express and [[Python#Cufflinks |Cufflinks]] |
| − | + | https://miro.medium.com/max/600/1*A8muRMkAljwW8PKWa_OFpg.gif | |
<youtube>j0wvKWb337A</youtube> | <youtube>j0wvKWb337A</youtube> | ||
| Line 626: | Line 979: | ||
=== <span id="Dash"></span>Dash === | === <span id="Dash"></span>Dash === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Plotly+Dash Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Plotly+Dash ...Google search] |
| − | * [ | + | * [https://plot.ly/dash Dash] a framework for building [[Analytics | analytical]] web applications. No [[JavaScript]] required; sits on top of [[Python#Flask |Flask]] |
| − | ** [ | + | ** [https://dash.plot.ly/dash-bio Dash Bio] a web application framework that provides pure Python abstraction around HTML, CSS, and JavaScript. Dash Bio is a suite of bioinformatics components that make it simpler to analyze and visualize bioinformatics data and interact with them in a Dash application. |
<youtube>e4ti2fCpXMI</youtube> | <youtube>e4ti2fCpXMI</youtube> | ||
| Line 638: | Line 991: | ||
=== <span id="Cufflinks"></span>Cufflinks === | === <span id="Cufflinks"></span>Cufflinks === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Cufflinks Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Cufflinks ...Google search] |
| − | * [ | + | * [https://plot.ly/ipython-notebooks/cufflinks/ Cufflinks | Jorge Santos] - a library for easy interactive [[Python#Pandas |Pandas]] charting with [[Python#Plotly |Plotly]]. Cufflinks binds [[Python#Plotly |Plotly]] directly to [[Python#Pandas DataFrame |Pandas DataFrame]]s. |
| − | * [ | + | * [https://towardsdatascience.com/the-next-level-of-data-visualization-in-python-dd6e99039d5e The Next Level of Data Visualization in Python | Will Koehrsen - Towards Data Science] |
| Line 652: | Line 1,005: | ||
=== <span id="plotly.js"></span>plotly.js === | === <span id="plotly.js"></span>plotly.js === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=plotly.js Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=plotly.js ...Google search] |
| − | * [ | + | * [https://plot.ly/javascript/ plotly.js] - built on top of [https://d3js.org/ d3.js] and [https://stack.gl/ stack.gl], plotly.js is a high-level, declarative charting library. plotly.js ships with 20 chart types, including 3D charts, statistical graphs, and [https://www.w3.org/Graphics/SVG/ SVG] maps. |
* [https://redstapler.co/javascript-realtime-chart-plotly/ Create JavaScript Real-Time Chart with Plotly.js | Red Stapler] | * [https://redstapler.co/javascript-realtime-chart-plotly/ Create JavaScript Real-Time Chart with Plotly.js | Red Stapler] | ||
| Line 664: | Line 1,017: | ||
=== <span id="Plotly Chart Studio"></span>Plotly Chart Studio === | === <span id="Plotly Chart Studio"></span>Plotly Chart Studio === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Plotly+Chart+Studio Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Plotly+Chart+Studio ...Google search] |
| − | * [ | + | * [https://plot.ly/online-chart-maker/ Plotly Chart Studio] - editor for creating [https://d3js.org/ d3.js] and [https://get.webgl.org/ WebGL] charts. Chart Studio is built on top of Plotly React, Plotly React Editor, the Plotly Image Server, Sheet.js, Handsontable and many other top-quality, open-source projects. |
== <span id="mpld3"></span>mpld3 == | == <span id="mpld3"></span>mpld3 == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=mpld3 Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=mpld3 ...Google search] |
| − | * [ | + | * [https://mpld3.github.io/ mpld3] | [[Creatives#Jake VanderPlas |Jake VanderPlas]] - brings together [[Python#Matplotlib |Matplotlib]], the popular Python-based graphing library, and [https://d3js.org/ D3js], the popular [[JavaScript]] library for creating interactive data visualizations for the web. The result is a simple API for exporting your [[Python#Matplotlib |Matplotlib]] graphics to HTML code which can be used within the browser, within standard web pages, blogs, or tools such as the IPython notebook. |
<youtube>uPIrPWBOBEg</youtube> | <youtube>uPIrPWBOBEg</youtube> | ||
<youtube>h3EnqnhATQ0</youtube> | <youtube>h3EnqnhATQ0</youtube> | ||
| + | |||
| + | |||
| + | == <span id="Lux"></span>Lux == | ||
| + | [https://www.youtube.com/results?search_query=Lux Youtube search...] | ||
| + | [https://www.google.com/search?q=Lux ...Google search] | ||
| + | |||
| + | * [https://github.com/lux-org/lux lux-org/lux] Python API for Intelligent Visual Data Discovery | ||
| + | |||
| + | * [https://towardsdatascience.com/how-to-create-data-visualizations-on-python-with-one-click-f6bafbd8de54 How to Create Data Visualizations In Python With One Click | Ismael Araujo] Lux is a low-code library that allows us to do a fast and easy data exploration by creating data visualizations with one click. The coolest part is that Lux decides what visualizations are recommended for your dataset without you having to decide. You can install it by typing pip install lux-api in your Terminal. | ||
| + | |||
| + | <youtube>NedCgZQZcwM</youtube> | ||
| + | <youtube>IDhzRghCDJE</youtube> | ||
== <span id="Bokeh"></span>Bokeh == | == <span id="Bokeh"></span>Bokeh == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Bokeh Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Bokeh ...Google search] |
| − | * [ | + | * [https://bokeh.pydata.org/en/latest/ Bokeh | Continuum Analytics] an interactive visualization library that targets modern web browsers for presentation - inspired by the concepts outlined in The Grammar of Graphics. Interactive plotting in web browsers, running JavaScript but controlled by Python. You can layer components on top of one another to create a finished plot—for example, you can start with the axes and then add points, lines, labels, etc. Plots can be output as JSON objects, HTML documents, or interactive web applications. Bokeh does a good job of allowing users to manipulate data in the browser, with sliders and dropdown menus for filtering. Like in mpld3, you can zoom and pan to navigate plots, but you can also focus in on a set of data points with a box or lasso select. |
<youtube>2TR_6VaVSOs</youtube> | <youtube>2TR_6VaVSOs</youtube> | ||
| Line 688: | Line 1,053: | ||
=== <span id="HoloViews"></span>HoloViews === | === <span id="HoloViews"></span>HoloViews === | ||
| − | [ | + | [https://www.youtube.com/results?search_query=HoloViews Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=HoloViews ...Google search] |
| − | * [ | + | * [https://holoviews.org/ HoloViews | J. Stevens, P. Rudiger, and J. Bednar] Declarative objects for instantly visualizable data, building Bokeh plots from convenient high-level specifications |
| − | * [ | + | * [https://github.com/pyviz/holoviews pyviz/holoviews | GitHub] |
<youtube>cMXKE0nB8k4</youtube> | <youtube>cMXKE0nB8k4</youtube> | ||
| Line 698: | Line 1,063: | ||
== <span id="Pygal"></span>Pygal == | == <span id="Pygal"></span>Pygal == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=Pygal Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=Pygal ...Google search] |
| − | * [ | + | * [https://www.pygal.org/en/latest/index.html Pygal | Florian Mounier] for producing beautiful out-of-the-box charts with very few lines of code. Each chart type is packaged into a method (e.g. pygal.Histogram() makes a histogram, pygal.Box() makes a box plot), and there's a variety of colorful default styles. If you want more control, you can configure almost every element of a plot—including sizing, titles, labels, and rendering. You can output charts as SVGs and add them to a web page with an embed tag or by inserting the code directly into the HTML. |
<youtube>SL3qKjLZx-As</youtube> | <youtube>SL3qKjLZx-As</youtube> | ||
== <span id="scikit-image"></span>scikit-image == | == <span id="scikit-image"></span>scikit-image == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=scikit-image Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=scikit-image ...Google search] |
| − | * [ | + | * [https://scikit-image.org/ scikit-image] An image processing library featuring many common operations including convolutional mapping, filtering, edge detection, and image segmentation. |
<youtube>xPrTHRbT1vY</youtube> | <youtube>xPrTHRbT1vY</youtube> | ||
| Line 715: | Line 1,080: | ||
== <span id="Shapely"></span>Shapely == | == <span id="Shapely"></span>Shapely == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=shapely Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=shapely ...Google search] |
| − | * [ | + | * [https://shapely.readthedocs.io/en/latest/ Shapely] - a spatial analysis library which extends Python to work as a fully-featured GIS environmental comparable to commercial software such as ArcGIS. |
<youtube>LwpqA2WMR_8</youtube> | <youtube>LwpqA2WMR_8</youtube> | ||
| Line 724: | Line 1,089: | ||
== <span id="Satellite Imagery"></span>Satellite Imagery == | == <span id="Satellite Imagery"></span>Satellite Imagery == | ||
| − | [ | + | [https://www.youtube.com/results?search_query=pyresample Youtube search...] |
| − | [ | + | [https://www.google.com/search?q=pyresample ...Google search] |
| − | * [ | + | * [[Satellite#Satellite Imagery|Satellite Imagery]] |
| − | * [ | + | * [https://www.openstreetmap.org Open Street Map] a map of the world, created by people |
| − | * [ | + | * [https://gdal.org/ Geospatial Data Abstraction Library (GDAL)] a translator library for raster and vector geospatial data formats |
| − | ** [ | + | * [https://pyresample.readthedocs.io/en/latest/ Pyresample] - re-projecting earth observing satellite data, capable of handling both swath data from polar-orbiting satellites and gridded data from geostationary satellites. |
| − | * [ | + | ** [https://github.com/pytroll/satpy SatPy | GitHub] for earth-observing satellite data processing |
| − | * [ | + | * [https://fiona.readthedocs.io/en/latest/manual.html Fiona] - handle vector data |
| − | * [ | + | * [https://rasterio.readthedocs.io/en/stable/quickstart.html rasterio] - handle raster data |
| − | * [ | + | * [https://pypi.org/project/ pyproj] - transforming spatial reference systems - python interface to [https://proj.org/ PROJ] (cartographic projections and coordinate transformations library). |
| − | * [ | + | * [https://pypi.org/project/folium/ Folium] - creating maps |
| − | * [ | + | * [https://geopandas.org/ GeoPandas] - geospatial analysis; extends the datatypes used by pandas to allow spatial operations on geometric types. Geometric operations are performed by shapely. Geopandas further depends on fiona for file access and descartes and [[Python#matplotlib |matplotlib]] for plotting. |
| + | * [https://geoviews.org/ GeoViews] - visualizable geographic data that that can be mixed and matched with [[Python#HoloViews |HoloViews]] objects | ||
| − | |||
<youtube>G-fz8L9xHIs</youtube> | <youtube>G-fz8L9xHIs</youtube> | ||
<youtube>kJXUUO5M4ok</youtube> | <youtube>kJXUUO5M4ok</youtube> | ||
<youtube>yiiItJYnYEs</youtube> | <youtube>yiiItJYnYEs</youtube> | ||
| − | + | ||
| + | = <span id="Games to Learn Python"></span>Games to Learn Python = | ||
| + | [https://www.youtube.com/results?search_query=Learn+Python+game+gaming YouTube] | ||
| + | [https://www.quora.com/search?q=Learn%20Python%20game%20gaming ... Quora] | ||
| + | [https://www.google.com/search?q=Learn+Python+game+gaming ...Google] | ||
| + | [https://news.google.com/search?q=Learn+Python+game+gaming ...Google News] | ||
| + | [https://www.bing.com/news/search?q=Learn+Python+game+gaming&qft=interval%3d%228%22 ...Bing News] | ||
| + | |||
| + | * [[Game-Based Learning (GBL)]] | ||
| + | * [[JavaScript#Games_to_Learn|Games to Learn JavaScript and CSS]] | ||
| + | * [[Gaming]] | ||
| + | * [https://www.makeuseof.com/tag/best-programming-games/ The 11 Best Coding Games to Build Your Programming Skills | Sahil Kapoor - Make Use Of] | ||
| + | * [https://codecombat.com/ CodeCombat] | ||
| + | |||
| − | + | https://i.imgur.com/QkS2BEF.jpg | |
Latest revision as of 08:59, 23 March 2024
YouTube ... Quora ...Google search ...Google News ...Bing News
- Python ... GenAI w/ Python ... JavaScript ... GenAI w/ JavaScript ... TensorFlow ... PyTorch
- Libraries & Frameworks Overview ... Libraries & Frameworks ... Git - GitHub and GitLab ... Other Coding options
- Agents ... Robotic Process Automation ... Assistants ... Personal Companions ... Productivity ... Email ... Negotiation ... LangChain Python library
- Natural Language libraries, e.g. SpaCy, Natural Language Toolkit (NLTK), CoreNLP, TextBlob, scikit-learn NLP toolkit, fastText, Intel NLP Architect, Gensim
- Other Python-related pages:
- TensorFlow for machine learning model building
- PyTorch authored by Facebook
- Google AutoML automatically build and deploy state-of-the-art machine learning models
- Ludwig - a Python toolbox from Uber that allows to train and test deep learning models
- Cython: blending Python and C/C++ ...thus a superset of programming.
- AWS Lambda & Python
- Notebooks; Jupyter and R Markdown
- How to build your own AlphaZero AI using Python and Keras
- Automate the Boring Stuff with Python
- Best Python Resources | Full Stack Python
- Learn Python Programming, By Example | Real Python
- Top 20 Python AI and Machine Learning Open Source Projects
- Essential Cheat Sheets for Machine Learning and Deep Learning Engineers
- How to Setup a Python Environment for Machine Learning | George Seif - KDnuggets
- Guido Van Rossum; author of Python
- Sphinx is a tool that makes it easy to create intelligent and beautiful documentation | Georg Brandl
- Python programming language: A cheat sheet | James Sanders - TechRepublic explores what it is used for, how it compares to other languages, and building skills resources
- References:
- Quantum Development Algorithms & Kits
- Autonomous Drones
- Environments:
- Development ... Notebooks ... AI Pair Programming ... Codeless, Generators, Drag n' Drop ... AIOps/MLOps ... AIaaS/MLaaS
- Local Machine
- Code completion: works with the top Python editors: Atom, PyCharm, Sublime,Visual Studio (VS) Code and Vim
- Alternative implementations and extensions of Python to address speed & memory usage...
- Python Gets Its Mojo Working for AI | Jessica Wachtel - The New Stack ... Combining the usability of Python with the performance of C, Mojo is a new programming language designed specifically for AI developers.
- CPython written in C and Python is Guido van Rossum's reference version of the Python computing language
- PyPy uses just-in-time compilation
- Cython an optimizing static compiler
- Attention Mechanism ... Transformer ... Generative Pre-trained Transformer (GPT) ... GAN ... BERT
- Artificial General Intelligence (AGI) to Singularity ... Curious Reasoning ... Emergence ... Moonshots ... Explainable AI ... Automated Learning
- Conversational AI ... ChatGPT | OpenAI ... Bing/Copilot | Microsoft ... Gemini | Google ... Claude | Anthropic ... Perplexity ... You ... phind ... Ernie | Baidu
Contents
- 1 Using Python
- 1.1 Python Data Science Handbook
- 1.2 SentenceTransformers
- 1.3 PyScript
- 1.4 NumPy
- 1.5 Pandas
- 1.6 SciPy
- 1.7 SymPy
- 1.8 NetworkX
- 1.9 scikit-learn
- 1.10 Graphical User Interface (GUI)
- 1.11 Spreadsheets
- 1.12 PyMC3
- 1.13 StatsModels
- 1.14 OpenCV
- 1.15 LibROSA
- 1.16 PyGame
- 1.17 Parallel
- 1.18 Numba
- 1.19 xarray
- 1.20 IPython Blocks
- 1.21 Metaflow
- 1.22 Web Automation with Python - Data Gathering
- 1.23 Twisted
- 1.24 Pipelines
- 1.25 yellowbrick
- 1.26 MLxtend
- 1.27 LIME
- 1.28 SHAP
- 2 Leveraging Large Language Models (LLM)
- 3 Python Stack
- 4 Time Series
- 5 Visualization with Python
- 6 Games to Learn Python
Using Python
|
|
|
|
|
|
|
|
Python Data Science Handbook
|
SentenceTransformers
Youtube search... ...Google search
SentenceTransformers is a Python framework for state-of-the-art sentence, text, and image embeddings. It is based on PyTorch and Transformers and offers a large collection of pre-trained models tuned for various tasks. You can use this framework to compute sentence/text embeddings for more than 100 languages. These embeddings can then be compared, for example, with cosine similarity to find sentences with a similar meaning. This can be useful for semantic textual similarity, semantic search, or paraphrase mining. You can install the Sentence Transformers library using pip: pip install -U sentence-transformers
PyScript
Youtube search... ...Google search
- py... run python in your HTML; a framework that allows users to create rich Python applications in the browser using HTML's interface
- PyScript-Use Python Code in HTML | Senthil E - Analytics Vidhya
NumPy
Youtube search... ...Google search
- NumPy -manipulation of numerical arrays. NumPy can also be used as an efficient multi-dimensional container of generic data. Arbitrary data-types can be defined. This allows NumPy to seamlessly and speedily integrate with a wide variety of databases.
- Python Numpy Tutorial | Justin Johnson
Pandas
Youtube search... ...Google search
- Python Data Analysis library - data structures and data analysis tools for the Python programming language. Pandas is a newer package built on top of NumPy, and provides an efficient implementation of a Pandas DataFrame. Pandas DataFrames are essentially multidimensional arrays with attached row and column labels, and often with heterogeneous types and/or missing data. As well as offering a convenient storage interface for labeled data, Pandas implements a number of powerful data operations familiar to users of both database frameworks and spreadsheet programs.
- Python for Data Analysis | Wes McKinney
- Modin accelerates Pandas by automatically distributing the computation across all of the system’s available CPU cores
Pandas DataFrame
Youtube search... ...Google search
- Pandas DataFrame is two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns). A Data frame is a two-dimensional data structure, i.e., data is aligned in a tabular fashion in rows and columns.
- Pandas DataFrame: A lightweight Intro | Daksh Deepak - Towards Data Science
- Joining DataFrames in Pandas | Manish Pathak - Data Camp
SciPy
Youtube search... ...Google search
- SciPy library - one of the core packages that make up the SciPy stack. It provides many user-friendly and efficient numerical routines such as routines for numerical integration, interpolation, optimization, linear algebra and statistics.
SymPy
Youtube search... ...Google search
- SymPy library - a Python library for symbolic mathematics aiming to become a full-featured computer algebra system (CAS)
- mpmath | Fredrik Johansson library for real and complex floating-point arithmetic with arbitrary precision
mpmath
Youtube search... ...Google search
- mpmath | Fredrik Johansson library for real and complex floating-point arithmetic with arbitrary precision. can be used as a library, interactively via the Python interpreter, or from within the SymPy or Sage computer algebra systems which include mpmath as standard component. CoCalc lets you use mpmath directly in the browser. Cocalc or "Collaborative Calculation in the Cloud" enables programming online without the need to install any software.
NetworkX
Youtube search... ...Google search
- NetworkX a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks. Video: Connected: A Social Network Analysis Tutorial with NetworkX ...Social Network Analysis (SNA), the study of the relational structure between actors, is used throughout the social and natural sciences to discover insight from connected entities.
- Social Network Analysis (SNA)
- Twitter Network Analysis with NetworkX - PyCon 2015 | Sarah Guido, Celia La - GitHub
- NetworkX Visualization Powered By Bokeh | Björn Meier
- SageMath graph tools survey ...components
- Network Pattern
scikit-learn
Youtube search... ...Google search
- Scikit-learn library for machine learning in Python built on NumPy, SciPy, and matplotlib. A toolkit implement a wide variety of algorithms for un/supervised machine learning tasks, including regressions, clustering, manifold learning, principal components, density estimation, and much more. It also provides many useful tools to help build “pipelines” for managing modeling tasks such as data processing /normalization, feature engineering, cross-validation, fitting, and prediction. The package scikit-learn is recommended to be installed using pip install scikit-learn but in your code imported using import sklearn.
LazyPredict
Youtube search... ...Google search
Lazy Predict help build a lot of basic models without much code and helps understand which models works better without any parameter tuning
LazyPredict is a low-code machine learning library that allows you to run up to 40 baseline models with two lines of code. LazyPredict uses Sklearn, which allows you to get the models, see what works best for you, and hypertune it as you would usually do. To install, you can type pip install lazypredict in your Terminal. 3 Awesome Python Libraries That You Should Know About | Ismael Araujo - Towards Data Science
Graphical User Interface (GUI)
HiPlot
Youtube search... ...Google search
- HiPlot: High-dimensional interactive plots made easy
- GitHub ...or install with pip install HiPlot.
- Introduction to Best Parallel Plot Python Library: “HiPlot” | Moto DEI - Towards Data Science
HiPlot is Facebook’s interactive visualization tool to help AI researchers discover correlations and patterns in high-dimensional data
Tkinter
Youtube search... ...Google search
- TkInter ... comes with Python already. Tkinter is a Python binding to the Tk GUI toolkit. It is the standard Python interface to the Tk GUI toolkit, and is Python's de facto standard GUI. Tkinter is included with standard Linux, Microsoft Windows and Mac OS X installs of Python. The name Tkinter comes from Tk interface.
- flatplanet/Intro-To-TKinter-Youtube-Course
Kivy
Youtube search... ...Google search
- Kivy ... open source Python library for rapid development of applications that make use of innovative user interfaces, such as multi-touch apps. Kivy bassed on OpenGL, draw in 2D, 3D, meshes, and shaders, on runs on Linux, Windows, OS X, Android, iOS, and Raspberry Pi. You can run the same code on all supported platforms. Kivy is 100% free to use, under an MIT license (starting from 1.7.2) and LGPL 3 for the previous versions.
PyQt
Youtube search... ...Google search
- Kivy ... Qt is set of cross-platform C++ libraries that implement high-level APIs for accessing many aspects of modern desktop and mobile systems. Library implements the QT application development framework and has QTDesigner: drag and drop interface. These include location and positioning services, multimedia, NFC and Bluetooth connectivity, a Chromium based web browser, as well as traditional UI development. PyQt5 is a comprehensive set of Python bindings for Qt v5. It is implemented as more than 35 extension modules and enables Python to be used as an alternative application development language to C++ on all supported platforms including iOS and Android. PyQt5 is released under the GPL v3 license and under a commercial license that allows for the development of proprietary applications.
wxPython
Youtube search... ...Google search
- wxPython a cross-platform GUI toolkit for the Python programming language. It allows Python programmers to create programs with a robust, highly functional graphical user interface, simply and easily. It is implemented as a set of Python extension modules that wrap the GUI components of the popular wxWidgets cross platform library, which is written in C++.Like Python and wxWidgets, wxPython is Open Source, which means that it is free for anyone to use and the source code is available for anyone to look at and modify. And anyone can contribute fixes or enhancements to the project.wxPython is a cross-platform toolkit. This means that the same program will run on multiple platforms without modification. Currently Supported platforms are Microsoft Windows, Mac OS X and macOS, and Linux or other unix-like systems with GTK2 or GTK3 libraries. In most cases the native widgets are used on each platform to provide a 100% native look and feel for the application.
Pyside2
Youtube search... ...Google search
- Pyside2 ...a Python binding of the cross-platform GUI toolkit Qt, currently developed by The Qt Company under the Qt for Python project on porting PySide to work with Qt 5 instead of Qt 4. It is one of the alternatives to the standard library package Tkinter. Like Qt, PySide2 is free software.
Spreadsheets
Python & Google Sheets
gsheets
Youtube search... ...Google search
- gsheets - small wrapper around the Google Sheets API to provide more convenient access to Google Sheets from Python scripts.
gsheets.py
Youtube search... ...Google search
- gsheets - self-containd script to dump all worksheets of a Google Spreadsheet to CSV or convert any subsheet to a pandas DataFrame
gspread
Youtube search... ...Google search
- gspread ...Google Sheets Python API wrapper
- example Jupyter notebook using gspread to fetch a sheet into a Pandas DataFrame
df2gspread
Youtube search... ...Google search
- df2gspread ...transfer data between Google Sheets and Pandas
pygsheets
Youtube search... ...Google search
- pygsheets ...Google Sheets Python API v4 (v4 port of gspread providing further extensions)
gspread-pandas
Youtube search... ...Google search
- gspread-pandas ...Interact with Google Sheet through Pandas DataFrames
pgsheets
Youtube search... ...Google search
- pgsheets ...manipulate Google Sheets Using Pandas DataFrames (independent bidirectional transfer library, using the legacy v3 API, Python 3 only)
Python & Excel
What is the best library out there for working with Excel through Python? You can just export to CSV if it's just a table of data that doesn't need any formatting. Pandas works great for this. You don't need anything else
- Excel - Data Analysis | Microsoft
- Excel with ChatGPT
- Python with Excel Visualization
- Building Interactive Python tools with Excel as a front-end
- Reading and writing Excel workbooks
- pyxll
- xlwings
- openpyxl
- xlsxWriter
pyxll
Youtube search... ...Google search
PyXLL is an Excel Add-In that enables developers to extend Excel’s capabilities with Python code. For organizations that want to provide their end users with functionality within Excel, PyXLL makes Python a productive, flexible back-end for Excel worksheets. With PyXLL, your own Python code runs in Excel using any Python distribution you like (e.g. Anaconda, Enthought’s Canopy or any other CPython distribution from 2.3 to 3.7). Because PyXLL runs your own full Python distribution you have access to all 3rd party Python packages such as NumPy, Pandas and SciPy and can call them from Excel.
xlwings
Youtube search... ...Google search
If you want a user to enter some data in Microsoft Excel, hand it off to Python, and then show the results to your user in Microsoft Excel, xlwings is great.
|
|
|
|
openpyxl
Youtube search... ...Google search
- openpyxl - a Python library for reading and writing Excel 2010 xlsx/xlsm/xltx/xltm files
- Working with Excel Spreadsheets | Al Sweigart - Automate the Boring Stuff
XlsxWriter
Youtube search... ...Google search
- XlsxWriter - A Python module for creating Excel XLSX files.
- Creating Advanced Excel Workbooks with Python | Practical Business Python
I am just pulling data from an SQL Server, manipulating the results, and then dumping the results into an Excel spreadsheet. Just working with Excel cells, and ranges.
PyMC3
Youtube search... ...Google search
- PyMC3 - Probabilistic Programming in Python - Bayesian Inference. Fit your model using gradient-based Markov chain Monte Carlo (MCMC) algorithms like NUTS, using ADVI for fast approximate inference — including minibatch-ADVI for scaling to large datasets — or using Gaussian processes to build Bayesian nonparametric models
- Markov Model (Chain, Discrete Time, Continuous Time, Hidden)
- Markov Decision Process (MDP)
StatsModels
Youtube search... ...Google search
- StatsModels A module for fitting and estimating many different types of statistical models as well as performing hypothesis testing and exploratory data analysis. It features tools for fitting generalized linear models, survival analyses, and multi-variate statistics.
OpenCV
Youtube search... ...Google search
- OpenCV - Open Computer Vision - work with images and/or videos and wish to add a variety of classical and state-of-the-art vision algorithms to their toolbox.
LibROSA
Youtube search... ...Google search
- LibROSA - audio and voice processing which can extract various kinds of features from audio segments, such as the rhythm, beats and tempo.
PyGame
Youtube search... ...Google search
- PyGame - making multimedia applications like games built on top of the excellent Simple DirectMedia Layer (SDL) library.
Parallel
DASK
Youtube search... ...Google search
- DASK provides advanced parallelism for analytics, enabling performance at scale for the tools you love - it is developed in coordination with other community projects like NumPy, Pandas, and scikit-learn
Joblib
Youtube search... ...Google search
- Joblib provide lightweight pipelining
Tornado
Youtube search... ...Google search
- Tornado is a web framework and asynchronous networking library. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user.
Numba
Youtube search... ...Google search
- Numba JIT compiler that translates a subset of Python and NumPy code into fast machine code.
xarray
Youtube search... ...Google search
- xarray working with labelled multi-dimensional arrays simple, and efficient. Xarray introduces labels in the form of dimensions, coordinates and attributes on top of raw NumPy-like arrays, which allows for a more intuitive, more concise, and less error-prone developer experience. The package includes a large and growing library of domain-agnostic functions for advanced analytics and visualization with these data structures. Xarray was inspired by and borrows heavily from Pandas, the popular data analysis package focused on labelled tabular data. It is particularly tailored to working with netCDF files, which were the source of xarray’s data model, and integrates tightly with DASK for parallel computing.
IPython Blocks
Youtube search... ...Google search
- IPython Blocks a tool for practicing Python in the Jupyter giving learners a grid of colors to manipulate while practicing for loops, if statements, and other aspects of Python.
Metaflow
Youtube search... ...Google search
- Metaflow, Netflix and AWS open source Python library
Web Automation with Python - Data Gathering
Write a Python crawler to extract information from websites to identify patterns, both in terms of the URL patterns and XPath patterns. Once these patterns are figured out, these tools can automatically extract the needed information and organize data a usable structure.
Requests
Youtube search... ...Google search
- Requests.org simple HTTP library
- Requests http library
Beautiful Soup - bs4
Youtube search... ...Google search
- Beautiful Soup Project for parsing HTML and XML documents. It creates parse trees
Allows you to import its functions and use them in-line. Therefore, you could even use it in your Jupyter notebooks.
Scrapy
Youtube search... ...Google search
- Scrapy webscraping .. open source and collaborative framework for extracting the data you need from websites
Selenium
Youtube search... ...Google search
- Selenium with Python a web testing library. It is used to automate browser activities.
- SauceLabs
Initialises a web browser such as Chrome and then simulates all the actions defined in the code; JavaScript functions to e.g. register an account, then log in and get the content after clicking some buttons and links.
Twisted
Youtube search... ...Google search
- Twisted an event-driven networking engine
Twisted has been around a long time in the Python world. Pioneering the Deferred abstraction, which later turned into Promises and found their way into JavaScript, it is a fertile ground for asynchronous I/O experimentation. Through its groundbreaking protocol/transport design that some of us might take for granted these days, and a strict adherence to unit testing and representing things through abstract interfaces, it lets you talk a lot of different network protocols without really having to know everything about them.
Pipelines
- AIOps/MLOps - Machine Learning (ML) pipelines for SecDevOps
- Ansible and Python 3 | Red Hat
- Python Client API | Saltstack
Python is one of the most crucial orchestration and infrastructure automation components of AIOps/MLOps to reduce or almost eliminates disconnect between developers and system admins. AIOps/MLOps is centered on enabling AI pipelines for continuous integration and continuous deployment (CI/CD) with no downtime.
Vaex
Youtube search... ...Google search
PyCaret
Youtube search... ...Google search
TPOT
Youtube search... ...Google search
- TPOT | Randal Olson - University of Pennsylvania - automatically creates and optimizes full machine learning pipelines using genetic programming. The Tree-Based pipeline Optimization Tool (TPOT) automates the building of ML pipelines by combining a flexible expression tree representation of pipelines with stochastic search algorithms such as genetic programming. TPOT makes use of the Python-based scikit-learn library as its ML menu.
ELI5
Youtube search... ...Google search
- ELI5 "Explain it like I'm 5" helps to...
- debug machine learning classifiers and explain their predictions.
- scikit-learn - Currently ELI5 allows to explain weights and predictions of scikit-learn linear classifiers and regressors, print decision trees as text or as SVG, show feature importances and explain predictions of decision trees and tree-based ensembles. ELI5 understands text processing utilities from scikit-learn and can highlight text data accordingly. Pipeline and FeatureUnion are supported. It also allows to debug scikit-learn pipelines which contain HashingVectorizer, by undoing hashing.
- xgboost - show feature importances and explain predictions of XGBClassifier, XGBRegressor and xgboost.Booster.
- LightGBM ...Microsoft's gradient boosting framework that uses tree based learning algorithms ... LightGBM - show feature importances and explain predictions of LGBMClassifier and LGBMRegressor. A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks. It is under the umbrella of the Microsoft Distributed Machine Learning Toolkit (DMTK) project of Microsoft.
- CatBoost - show feature importances of CatBoostClassifier, CatBoostRegressor and catboost.CatBoost. A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU
- lightning - explain weights and predictions of lightning classifiers and regressors. Large-scale linear classification, regression and ranking in Python
- sklearn-crfsuite ELI5 allows to check weights of sklearn_crfsuite.CRF models. CRFsuite is an implementation of Conditional Random Fields (CRFs) for labeling sequential data.
- ELI5 also implements several algorithms for inspecting black-box models (see Inspecting Black-Box Estimators):
- TextExplainer allows to explain predictions of any text classifier using LIME algorithm. There are utilities for using LIME with non-text data and arbitrary black-box classifiers as well, but this feature is currently experimental.
- Permutation importance method can be used to compute feature importances for black box estimators.
- debug machine learning classifiers and explain their predictions.
Explanation and formatting are separated; you can get text-based explanation to display in console, HTML version embeddable in an IPython notebook or web dashboards, a Pandas DataFrame object if you want to process results further, or JSON version which allows to implement custom rendering and formatting on a client.
yellowbrick
Youtube search... ...Google search
This library is essentially an extension of the scikit-learn library and provides some really useful and pretty looking visualisations for machine learning models. The visualiser objects, the core interface, are scikit-learn estimators and so if you are used to working with scikit-learn the workflow should be quite familiar.
MLxtend
Youtube search... ...Google search
This library contains a host of helper functions for machine learning. This covers things like stacking and voting classifiers, model evaluation, feature extraction and engineering and plotting.
LIME
Youtube search... ...Google search
- LIME (Local Interpretable Model-agnostic Explanations) explains the prediction of any classifier in an interpretable and faithful manner by learning a interpretable model locally around the prediction.
- SHAP and LIME Python Libraries: Part 1 – Great Explainers, with Pros and Cons to Both | Joshua Poduska - Domino
- Decrypting your Machine Learning model using LIME | Abhishek Sharma - Towards Data Science
SHAP
Youtube search... ...Google search
- SHAP (SHapley Additive exPlanations) is a unified approach to explain the output of any machine learning model. SHAP connects game theory with local explanations, uniting several previous methods and representing the only possible consistent and locally accurate additive feature attribution method based on expectations.
- Shapley Additive Explanations (SHAP)
- Demystifying Black-Box Models with SHAP Value Analysis | Peter Cooman
Leveraging Large Language Models (LLM)
Youtube search... ...Google search
- LangChain
- Pinecone
- OpenLLM
- Guardrails
- Interactive Composition Explorer (ICE)
- Marvin
- LLMs in the Real World: Structuring Text with Declarative NLP | Adam Azzam - InfoQ
Pydantic
Youtube search... ...Google search
Pydantic is a data validation library for Python that leverages type hints to define the structure and constraints of data models. It provides a simple and intuitive interface for creating and validating data models, making it a popular choice for a wide range of applications.
Key Features of Pydantic:
- Data Validation: Pydantic ensures the integrity of data by validating its type, format, and constraints.
- Type Annotations: Pydantic utilizes Python type hints to define the structure of data models, making the code self-documenting and enhancing developer experience.
- Data Serialization and Deserialization: Pydantic seamlessly converts data models into various formats, including JSON, YAML, and TOML.
- Customizable Validation: Pydantic empowers developers to create custom validation rules for specific use cases.
- Seamless Integration: Pydantic integrates smoothly with popular Python frameworks like FastAPI, Django, and Flask.
Benefits of Using Pydantic:
- Improved Code Quality: Pydantic's type annotations and validation checks contribute to cleaner and more reliable code.
- Reduced Errors: Pydantic's validation mechanisms prevent invalid data from entering the application, reducing the risk of errors.
- Enhanced Developer Experience: Pydantic's intuitive syntax and clear error messages make it easy for developers to work with data models.
- Documentation Generation: Pydantic can generate documentation for data models, enhancing code comprehension.
- Integration with Tools: Pydantic's JSON Schema generation enables integration with various tools and libraries.
Common Use Cases for Pydantic:
- API Request and Response Validation: Pydantic validates data exchanged between APIs, ensuring data integrity and consistency.
- Data Configuration Management: Pydantic manages application configuration data, ensuring proper configuration and preventing errors.
- Data Persistence and Storage: Pydantic validates data before persisting it to databases or file systems, maintaining data integrity.
- Data Exchange between Services: Pydantic ensures data consistency when exchanging data between microservices or distributed systems.
- Data Modeling and Abstraction: Pydantic abstracts data structures and provides a consistent way to interact with data, simplifying application development.
Python Stack
Youtube search... ...Google search
- openstack - open source software for creating private and public clouds
- Full Stack Python - Book | Matt Makai
Flask
Youtube search... ...Google search
- Deploying Flask Web Applications | Matt Makai GumRoad.com to purchase
- Flask a microframework for Python. It is classified as a microframework because it does not require particular tools or libraries. It has no database abstraction layer, form validation, or any other components where pre-existing third-party libraries provide common functions.
- Flask boilerplate | Max Halford
- flaskSaaS | Max Halford starting point to build your SaaS in Flask & Python, with Stripe subscription billing
- flask-image-uploader | bboe
- Flask-Login for the user accounts
- Flask-SQLAlchemy interacting with the database
- Flask-WTF and WTForms for the form handling.
- Flask-Mail for sending mails.
- Flask-Bcrypt for generating secret user passwords.
- Flask-Admin for building an administration interface.
- Flask-Script for managing the app
- flask-stripe Stripe Checkout & user registration
- TensorFlow Serving
- News Aggregation
- Creating REST API for TensorFlow models | Vitaly Bezgachev - Medium - Becoming Human
Flask is considered more Pythonic than the Django web framework because in common situations the equivalent Flask web application is more explicit. Flask is also easy to get started with as a beginner because there is little boilerplate code for getting a simple app up and running. Flask | Full Stack Python
Flask & React
Flask & Docker
Flask, React, & Docker
- Full-stack tutorial: Flask + React + Docker | Riken Mehta - Medium
- Containerizing a Flask + React app with docker-compose | Devops Dummy - Medium
- Developing and Testing Microservices with Docker, Flask, and React | Michael Herman
- Microservices with Docker, Flask, and React | testdriven.io (course)
Django
- Django - a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source.
- News Aggregation
Django is a widely-used Python web application framework with a "batteries-included" philosophy. The principle behind batteries-included is that the common functionality for building web applications should come with the framework instead of as separate libraries. Django | Full Stack Python
Other Web Frameworks supporting Python
- Anvil
- bobo
- Bottle
- CherryPy
- Cyclone
- Falcon
- Itty-Bitty
- Klein
- Morepath
- Muffin
- ObjectWeb
- Pecan
- Pyramid
- Ray
- Sanic
- Tornado
- TurboGears
- Vibora
- Web2py
- Wheezy Web
Time Series
Youtube search... ...Google search
tsfresh
Youtube search... ...Google search
- tsfresh ...python package that automatically calculates a large number of time series characteristics, the so called features. Further the package contains methods to evaluate the explaining power and importance of such characteristics for regression or classification tasks.
STUMPY
Youtube search... ...Google search
an open source scientific Python library that implements a novel yet intuitive approach for discovering patterns, anomalies, and other insights from any time series data. STUMPY is a powerful and scalable library that efficiently computes something called the matrix profile, which can be used for a variety of time series data mining tasks such as pattern/motif (approximately repeated subsequences within a longer time series) discovery, anomaly/novelty (discord) discovery, shapelet discovery, semantic segmentation, streaming (on-line) data fast approximate matrix profiles, time series chains (temporally ordered set of subsequence patterns). STUMPY
Visualization with Python
Youtube search... ...Google search
- Python Visualization
- Python Data Visualization 2018: Why So Many Libraries? | James A. Bednar
- PyViz.org meta-initiative for helping users decide on the best open-source Python data visualization tools for their purposes, with links, overviews, comparisons, and examples.
- Pandas .plot() - basic plotting interface uses Matplotlib to render static PNGs in a Jupyter notebook or for exporting from Python, with a command that can be as simple as df.plot() for a DataFrame with two columns.
- Datashader breaks the creation of images into a series of explicit steps that allow computations to be done on intermediate representations. Rasterizing huge datasets quickly as fixed-size images.
- Panel -assembling objects from many different libraries into a layout or app, whether in a Jupyter notebook or in a standalone serveable dashboard
- Param -declaring user-relevant parameters, making it simple to work with widgets inside and outside of a notebook context
- Data Visualization |Full Stack Python
Matplotlib
Youtube search... ...Google search
- Matplotlib generate plots, histograms, power spectra, bar charts, errorcharts, scatterplots, etc
- Visualization with Matplotlib | Jake VanderPlas - Python Data Science Handbook
- Driving your graphic via ipyWidgets
seaborn
Youtube search... ...Google search
- seaborn - complements Matplotlib and works well with Pandas DataFrames
Plotly
Youtube search... ...Google search
- Plotly | Plotly - graphing library makes interactive, publication-quality graphs online. Examples of how to make line plots, scatter plots, area charts, bar charts, error bars, box plots, histograms, heatmaps, subplots, multiple-axes, polar charts, and bubble charts. Plotly is a web-based service by default, but you can use the library offline in Python and upload plots to Plotly's free, public server or paid, private server. From there, you can embed your plots in a web page. All Plotly graphs include tooltips, and you can build custom controls (like sliders and filters) on top of a chart once it's embedded using Plotly's JavaScript API. Plotly.js is based on D3.js and WebGL.js. Another way to work in Plotly and share plots is in Mode Notebooks. You can pull data with SQL, use the Plotly offline library in the Python Notebook to plot the results of your query, and then add the interactive chart to a report. Get pricing
- Learn Plotly | SuperDataScience
- Driving your graphic via ipyWidgets
- It’s 2019 — Make Your Data Visualizations Interactive with Plotly | Jeff Hale - Towards Data Science Find the path to make awesome figures quickly with Express and Cufflinks
Dash
Youtube search... ...Google search
- Dash a framework for building analytical web applications. No JavaScript required; sits on top of Flask
- Dash Bio a web application framework that provides pure Python abstraction around HTML, CSS, and JavaScript. Dash Bio is a suite of bioinformatics components that make it simpler to analyze and visualize bioinformatics data and interact with them in a Dash application.
Cufflinks
Youtube search... ...Google search
- Cufflinks | Jorge Santos - a library for easy interactive Pandas charting with Plotly. Cufflinks binds Plotly directly to Pandas DataFrames.
- The Next Level of Data Visualization in Python | Will Koehrsen - Towards Data Science
Cufflinks --> Plotly --> ployly.js --> D3.js
plotly.js
Youtube search... ...Google search
- plotly.js - built on top of d3.js and stack.gl, plotly.js is a high-level, declarative charting library. plotly.js ships with 20 chart types, including 3D charts, statistical graphs, and SVG maps.
- Create JavaScript Real-Time Chart with Plotly.js | Red Stapler
Plotly Chart Studio
Youtube search... ...Google search
- Plotly Chart Studio - editor for creating d3.js and WebGL charts. Chart Studio is built on top of Plotly React, Plotly React Editor, the Plotly Image Server, Sheet.js, Handsontable and many other top-quality, open-source projects.
mpld3
Youtube search... ...Google search
- mpld3 | Jake VanderPlas - brings together Matplotlib, the popular Python-based graphing library, and D3js, the popular JavaScript library for creating interactive data visualizations for the web. The result is a simple API for exporting your Matplotlib graphics to HTML code which can be used within the browser, within standard web pages, blogs, or tools such as the IPython notebook.
Lux
Youtube search... ...Google search
- lux-org/lux Python API for Intelligent Visual Data Discovery
- How to Create Data Visualizations In Python With One Click | Ismael Araujo Lux is a low-code library that allows us to do a fast and easy data exploration by creating data visualizations with one click. The coolest part is that Lux decides what visualizations are recommended for your dataset without you having to decide. You can install it by typing pip install lux-api in your Terminal.
Bokeh
Youtube search... ...Google search
- Bokeh | Continuum Analytics an interactive visualization library that targets modern web browsers for presentation - inspired by the concepts outlined in The Grammar of Graphics. Interactive plotting in web browsers, running JavaScript but controlled by Python. You can layer components on top of one another to create a finished plot—for example, you can start with the axes and then add points, lines, labels, etc. Plots can be output as JSON objects, HTML documents, or interactive web applications. Bokeh does a good job of allowing users to manipulate data in the browser, with sliders and dropdown menus for filtering. Like in mpld3, you can zoom and pan to navigate plots, but you can also focus in on a set of data points with a box or lasso select.
HoloViews
Youtube search... ...Google search
- HoloViews | J. Stevens, P. Rudiger, and J. Bednar Declarative objects for instantly visualizable data, building Bokeh plots from convenient high-level specifications
- pyviz/holoviews | GitHub
Pygal
Youtube search... ...Google search
- Pygal | Florian Mounier for producing beautiful out-of-the-box charts with very few lines of code. Each chart type is packaged into a method (e.g. pygal.Histogram() makes a histogram, pygal.Box() makes a box plot), and there's a variety of colorful default styles. If you want more control, you can configure almost every element of a plot—including sizing, titles, labels, and rendering. You can output charts as SVGs and add them to a web page with an embed tag or by inserting the code directly into the HTML.
scikit-image
Youtube search... ...Google search
- scikit-image An image processing library featuring many common operations including convolutional mapping, filtering, edge detection, and image segmentation.
Shapely
Youtube search... ...Google search
- Shapely - a spatial analysis library which extends Python to work as a fully-featured GIS environmental comparable to commercial software such as ArcGIS.
Satellite Imagery
Youtube search... ...Google search
- Satellite Imagery
- Open Street Map a map of the world, created by people
- Geospatial Data Abstraction Library (GDAL) a translator library for raster and vector geospatial data formats
- Pyresample - re-projecting earth observing satellite data, capable of handling both swath data from polar-orbiting satellites and gridded data from geostationary satellites.
- SatPy | GitHub for earth-observing satellite data processing
- Fiona - handle vector data
- rasterio - handle raster data
- pyproj - transforming spatial reference systems - python interface to PROJ (cartographic projections and coordinate transformations library).
- Folium - creating maps
- GeoPandas - geospatial analysis; extends the datatypes used by pandas to allow spatial operations on geometric types. Geometric operations are performed by shapely. Geopandas further depends on fiona for file access and descartes and matplotlib for plotting.
- GeoViews - visualizable geographic data that that can be mixed and matched with HoloViews objects
Games to Learn Python
YouTube ... Quora ...Google ...Google News ...Bing News
- Game-Based Learning (GBL)
- Games to Learn JavaScript and CSS
- Gaming
- The 11 Best Coding Games to Build Your Programming Skills | Sahil Kapoor - Make Use Of
- CodeCombat
