Difference between revisions of "Decentralized: Federated & Distributed"

From
Jump to: navigation, search
m (→‎Submarine Scenario)
m
 
(19 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
|title=PRIMO.ai
 
|title=PRIMO.ai
 
|titlemode=append
 
|titlemode=append
−
|keywords=artificial, intelligence, machine, learning, models, algorithms, data, singularity, moonshot, Tensorflow, Google, Nvidia, Microsoft, Azure, Amazon, AWS  
+
|keywords=ChatGPT, artificial, intelligence, machine, learning, GPT-4, GPT-5, NLP, NLG, NLC, NLU, models, data, singularity, moonshot, Sentience, AGI, Emergence, Moonshot, Explainable, TensorFlow, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Hugging Face, OpenAI, Tensorflow, OpenAI, Google, Nvidia, Microsoft, Azure, Amazon, AWS, Meta, LLM, metaverse, assistants, agents, digital twin, IoT, Transhumanism, Immersive Reality, Generative AI, Conversational AI, Perplexity, Bing, You, Bard, Ernie, prompt Engineering LangChain, Video/Image, Vision, End-to-End Speech, Synthesize Speech, Speech Recognition, Stanford, MIT |description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools
−
|description=Helpful resources for your journey with artificial intelligence; videos, articles, techniques, courses, profiles, and tools  
+
 
 +
<!-- Google tag (gtag.js) -->
 +
<script async src="https://www.googletagmanager.com/gtag/js?id=G-4GCWLBVJ7T"></script>
 +
<script>
 +
  window.dataLayer = window.dataLayer || [];
 +
  function gtag(){dataLayer.push(arguments);}
 +
  gtag('js', new Date());
 +
 
 +
  gtag('config', 'G-4GCWLBVJ7T');
 +
</script>
 
}}
 
}}
−
[http://www.youtube.com/results?search_query=Federated+Learning+deep+machine+learning+ML+AI Youtube search...]
+
[https://www.youtube.com/results?search_query=ai+Federated+Learning+deep+machine+learning+ML YouTube]
−
[http://www.google.com/search?q=Federated+Learning+deep+machine+learning+ML+AI ...Google search]
+
[https://www.quora.com/search?q=ai%20Federated%20Learning%20deep%20machine%20learning%20ML ... Quora]
 +
[https://www.google.com/search?q=ai+Federated+Learning+deep+machine+learning+ML ...Google search]
 +
[https://news.google.com/search?q=ai+Federated+Learning+deep+machine+learning+ML ...Google News]
 +
[https://www.bing.com/news/search?q=ai+Federated+Learning+deep+machine+learning+ML&qft=interval%3d%228%22 ...Bing News]
  
−
* [[Case Studies]]
+
* [[Architectures]] for AI ... [[Generative AI Stack]] ... [[Enterprise Architecture (EA)]] ... [[Enterprise Portfolio Management (EPM)]] ... [[Architecture and Interior Design]]
−
** [[Risk, Compliance and Regulation]]
+
* [[Risk, Compliance and Regulation]] ... [[Ethics]] ... [[Privacy]] ... [[Law]] ... [[AI Governance]] ... [[AI Verification and Validation]]
 
* [[Blockchain]]
 
* [[Blockchain]]
−
* [[Privacy]]
+
* [[Memory]]
 
* [[OpenMined]]
 
* [[OpenMined]]
−
* [[Platforms: AI/Machine Learning as a Service (AIaaS/MLaaS)]]
+
* [[Sakana]] ... inspired by the way that fish and other animals work together in groups
−
* [[Computer Networks]]
+
* [[Distributed Deep Reinforcement Learning (DDRL)]]
 +
* [[Development]] ... [[Notebooks]] ... [[Development#AI Pair Programming Tools|AI Pair Programming]] ... [[Codeless Options, Code Generators, Drag n' Drop|Codeless]] ... [[Hugging Face]] ... [[Algorithm Administration#AIOps/MLOps|AIOps/MLOps]] ... [[Platforms: AI/Machine Learning as a Service (AIaaS/MLaaS)|AIaaS/MLaaS]]
 +
* [[Telecommunications]] ... [[Computer Networks]] ... [[Telecommunications#5G|5G]] ... [[Satellite#Satellite Communications|Satellite Communications]] ... [[Quantum Communications]] ... [[Agents#Communication | Agents]] ... [[AI Generated Broadcast Content|AI Broadcast; Radio, Stream, TV]]
 +
 
  
 
= Centralized vs. Decentralized vs. Distributed =
 
= Centralized vs. Decentralized vs. Distributed =
−
[http://www.youtube.com/results?search_query=Decentralized+Federated+Distributed+Learning+deep+machine+learning+AI Youtube search...]
+
[https://www.youtube.com/results?search_query=Decentralized+Federated+Distributed+Learning+deep+machine+learning+AI Youtube search...]
−
[http://www.google.com/search?q=Decentralized+Federated+Distributed+Learning+deep+machine+learning+AI ...Google search]
+
[https://www.google.com/search?q=Decentralized+Federated+Distributed+Learning+deep+machine+learning+AI ...Google search]
  
−
* [http://www.fliphodl.com/social-media-alternatives-series-ep-1-what-you-need-to-know/ Social Media Alternatives Series, EP. 1: What You NEED to Know | Fliphodl]
+
* [https://www.fliphodl.com/social-media-alternatives-series-ep-1-what-you-need-to-know/ Social Media Alternatives Series, EP. 1: What You NEED to Know | Fliphodl]
−
* [http://www.fliphodl.com/exclusive-we-tried-ultra-and-it-will-kill-steam/ Exclusive: We tried Ultra and it will kill Steam | Fliphodl]
+
* [https://www.fliphodl.com/exclusive-we-tried-ultra-and-it-will-kill-steam/ Exclusive: We tried Ultra and it will kill Steam | Fliphodl]
  
−
http://upload.wikimedia.org/wikipedia/commons/b/ba/Centralised-decentralised-distributed.png  
+
https://upload.wikimedia.org/wikipedia/commons/b/ba/Centralised-decentralised-distributed.png  
  
  
 
== Federated ==
 
== Federated ==
−
[http://www.youtube.com/results?search_query=Federated+Learning+deep+machine+learning+ML Youtube search...]
+
[https://www.youtube.com/results?search_query=Federated+Learning+deep+machine+learning+ML Youtube search...]
−
[http://www.google.com/search?q=Federated+Learning+deep+machine+learning+ML ...Google search]
+
[https://www.google.com/search?q=Federated+Learning+deep+machine+learning+ML ...Google search]
  
−
* [http://en.wikipedia.org/wiki/Federated_learning Federated Learning | Wikipedia]
+
* [https://en.wikipedia.org/wiki/Federated_learning Federated Learning | Wikipedia]
−
* [http://arxiv.org/pdf/1902.04885.pdf Federated Machine Learning: Concept and Applications | Q. Yang, Y. Liu, T. Chen and Y. Tong]
+
* [https://arxiv.org/pdf/1902.04885.pdf Federated Machine Learning: Concept and Applications | Q. Yang, Y. Liu, T. Chen and Y. Tong]
−
* [http://medium.com/syncedreview/federated-learning-the-future-of-distributed-machine-learning-eec95242d897 Federated Learning: The Future of Distributed Machine Learning | Mi Zhang - Medium]
+
* [https://medium.com/syncedreview/federated-learning-the-future-of-distributed-machine-learning-eec95242d897 Federated Learning: The Future of Distributed Machine Learning | Mi Zhang - Medium]
−
* [http://www.researchgate.net/publication/329758083_Chained_Anomaly_Detection_Models_for_Federated_Learning_An_Intrusion_Detection_Case_Study Chained Anomaly Detection Models for Federated Learning: An Intrusion Detection Case Study | D. Preuveneers, V. Rimmer, I. Tsingenopoulos, J. Spooren, W. Joosen and E. Ilie-Zudor]
+
* [https://www.researchgate.net/publication/329758083_Chained_Anomaly_Detection_Models_for_Federated_Learning_An_Intrusion_Detection_Case_Study Chained Anomaly Detection Models for Federated Learning: An Intrusion Detection Case Study | D. Preuveneers, V. Rimmer, I. Tsingenopoulos, J. Spooren, W. Joosen and E. Ilie-Zudor]
−
* [http://hackernoon.com/a-beginners-guide-to-federated-learning-b29e29ba65cf A Beginners Guide to Federated Learning | Santanu Bhattacharya]
+
* [https://hackernoon.com/a-beginners-guide-to-federated-learning-b29e29ba65cf A Beginners Guide to Federated Learning | Santanu Bhattacharya]
 
* [[Watch me Build a Cybersecurity Startup]]
 
* [[Watch me Build a Cybersecurity Startup]]
 
* Federated Learning Frameworks:
 
* Federated Learning Frameworks:
−
** [[OpenMined]] Pysyft  ...[http://github.com/OpenMined/PySyft/tree/master/examples/tutorials GitHub]
+
** [[OpenMined]] Pysyft  ...[https://github.com/OpenMined/PySyft/tree/master/examples/tutorials GitHub]
−
** [http://www.tensorflow.org/federated TensorFlow Federated Learning]  ...* [http://venturebeat.com/2019/03/06/tensorflow-federated-allows-machine-learning-models-to-train-on-data-from-different-locations/ TensorFlow Federated] helps train AI models on data from different locations | Kyle Wiggers - VentureBeat     
+
** [https://www.tensorflow.org/federated TensorFlow Federated Learning]  ...* [https://venturebeat.com/2019/03/06/tensorflow-federated-allows-machine-learning-models-to-train-on-data-from-different-locations/ TensorFlow Federated] helps train AI models on data from different locations | Kyle Wiggers - VentureBeat     
−
** [http://www.fedai.org/ Federated AI Ecosystem (FATE) | FEDAI.org]
+
** [https://www.fedai.org/ Federated AI Ecosystem (FATE) | FEDAI.org]
−
** [http://devblogs.nvidia.com/federated-learning-clara/ Clara Federated Learning | NVIDIA]
+
** [https://devblogs.nvidia.com/federated-learning-clara/ Clara Federated Learning | NVIDIA]
  
  
−
<img src="http://questforai.files.wordpress.com/2019/03/federated_learning_animated_labeled.gif" width="400">
+
<img src="https://questforai.files.wordpress.com/2019/03/federated_learning_animated_labeled.gif" width="400">
−
<img src="http://www.searchtechnologies.com/images/federated-search1.png" width="300">
+
<img src="https://www.searchtechnologies.com/images/federated-search1.png" width="300">
  
  
−
<img src="http://cdn-images-1.medium.com/max/2304/1*P2qr6R_VcRE22tnKZpLj_A.png" width="900">
+
<img src="https://cdn-images-1.medium.com/max/2304/1*P2qr6R_VcRE22tnKZpLj_A.png" width="900">
  
 
<youtube>89BGjQYA0uE</youtube>
 
<youtube>89BGjQYA0uE</youtube>
Line 61: Line 76:
  
  
−
* [http://venturebeat.com/2019/10/13/nvidia-uses-federated-learning-to-create-medical-imaging-ai/ NVIDIA (King’s College London) uses federated learning to create medical imaging AI | Khari Johnson - VentureBeat]
+
* [https://venturebeat.com/2019/10/13/nvidia-uses-federated-learning-to-create-medical-imaging-ai/ NVIDIA (King’s College London) uses federated learning to create medical imaging AI | Khari Johnson - VentureBeat]
 
<youtube>Jy7ozgwovgg</youtube>
 
<youtube>Jy7ozgwovgg</youtube>
  
 
== <span id="Distributed"></span>Distributed ==
 
== <span id="Distributed"></span>Distributed ==
−
[http://www.youtube.com/results?search_query=Distributed+Learning+deep+machine+learning+ML Youtube search...]
+
[https://www.youtube.com/results?search_query=Distributed+Learning+deep+machine+learning+ML Youtube search...]
−
[http://www.google.com/search?q=Distributed+Learning+deep+machine+learning+ML ...Google search]
+
[https://www.google.com/search?q=Distributed+Learning+deep+machine+learning+ML ...Google search]
  
 
* [[Distributed Deep Reinforcement Learning (DDRL)]]
 
* [[Distributed Deep Reinforcement Learning (DDRL)]]
 
* [https://learn.microsoft.com/en-us/azure/architecture/example-scenario/ai/training-python-models Distributed hyperparameter tuning for machine learning models | Microsoft]
 
* [https://learn.microsoft.com/en-us/azure/architecture/example-scenario/ai/training-python-models Distributed hyperparameter tuning for machine learning models | Microsoft]
  
−
Distributed machine learning refers to multi-node machine learning algorithms and systems that are designed to improve performance, increase accuracy, and scale to larger input data sizes. Increasing the input data size for many algorithms can significantly reduce the learning error and can often be more effective than using more complex methods [8]. Distributed machine learning allows companies, researchers, and individuals to make informed decisions and draw meaningful conclusions from large amounts of data. Many systems exist for performing machine learning tasks in a distributed environment. These systems fall into three primary categories: database, general, and purpose-built systems. Each type of system has distinct advantages and disadvantages, but all are used in practice depending upon individual use cases, performance requirements, input data sizes, and the amount of implementation effort. | [http://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-8265-9_80647 SpringerLink]
+
Distributed machine learning refers to multi-node machine learning algorithms and systems that are designed to improve performance, increase accuracy, and scale to larger input data sizes. Increasing the input data size for many algorithms can significantly reduce the learning error and can often be more effective than using more complex methods [8]. Distributed machine learning allows companies, researchers, and individuals to make informed decisions and draw meaningful conclusions from large amounts of data. Many systems exist for performing machine learning tasks in a distributed environment. These systems fall into three primary categories: database, general, and purpose-built systems. Each type of system has distinct advantages and disadvantages, but all are used in practice depending upon individual use cases, performance requirements, input data sizes, and the amount of implementation effort. | [https://link.springer.com/referenceworkentry/10.1007%2F978-1-4614-8265-9_80647 SpringerLink]
  
 
{|<!-- T -->
 
{|<!-- T -->
Line 79: Line 94:
 
<youtube>jkkmBpJ-Eeo</youtube>
 
<youtube>jkkmBpJ-Eeo</youtube>
 
<b>Distinguished Lecturer : Eric Xing - Strategies & Principles for Distributed Machine Learning
 
<b>Distinguished Lecturer : Eric Xing - Strategies & Principles for Distributed Machine Learning
−
</b><br>The rise of Big Data has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters that promise adequate capacity to digest massive datasets and offer powerful predictive analytics (such as high-dimensional latent features, intermediate representations, and decision functions) thereupon. In order to run ML algorithms at such scales, on a distributed cluster with 10s to 1000s of machines, it is often the case that significant engineering efforts are required --- and one might fairly ask if such engineering truly falls within the domain of ML research or not. Taking the view that Big ML systems can indeed benefit greatly from ML-rooted statistical and algorithmic insights --- and that ML researchers should therefore not shy away from such systems design --- we discuss a series of principles and strategies distilled from our resent effort on industrial-scale ML solutions that involve a continuum from application, to engineering, and to theoretical research and development of Big ML system and architecture, on how to make them efficient, general, and with convergence and scaling guarantees. These principles concern four key questions which traditionally receive little attention in ML research: How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine communication? How to perform such communication? What should be communicated between machines? By exposing underlying statistical and algorithmic characteristics unique to ML programs but not typical in traditional computer programs, and by dissecting successful cases of how we harness these principles to design both high-performance distributed ML software and general-purpose ML framework, we present opportunities for ML researchers and practitioners to further shape and grow the area that lies between ML and systems. This is joint work with the CMU Petuum Team.  
+
</b><br>The rise of Big Data has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters that promise adequate capacity to digest massive datasets and offer powerful [[Predictive Analytics]] (such as high-dimensional [[latent]] features, intermediate representations, and decision functions) thereupon. In order to run ML algorithms at such scales, on a distributed cluster with 10s to 1000s of machines, it is often the case that significant engineering efforts are required --- and one might fairly ask if such engineering truly falls within the domain of ML research or not. Taking the view that Big ML systems can indeed benefit greatly from ML-rooted statistical and algorithmic insights --- and that ML researchers should therefore not shy away from such systems design --- we discuss a series of principles and strategies distilled from our resent effort on industrial-scale ML solutions that involve a continuum from application, to engineering, and to theoretical research and [[development]] of Big ML system and architecture, on how to make them efficient, general, and with convergence and scaling guarantees. These principles concern four key questions which traditionally receive little attention in ML research: How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine [[Agents#Communication | communication]]? How to perform such [[Agents#Communication | communication]]? What should be communicated between machines? By exposing underlying statistical and algorithmic characteristics unique to ML programs but not typical in traditional computer programs, and by dissecting successful cases of how we harness these principles to design both high-performance distributed ML software and general-purpose ML framework, we present opportunities for ML researchers and practitioners to further shape and grow the area that lies between ML and systems. This is joint work with the CMU Petuum Team.  
 
|}
 
|}
 
|<!-- M -->
 
|<!-- M -->
Line 91: Line 106:
 
Distribution Strategy API:
 
Distribution Strategy API:
 
https://goo.gl/F9vXqQ
 
https://goo.gl/F9vXqQ
−
http://goo.gl/Zq2xvJ
+
https://goo.gl/Zq2xvJ
  
 
ResNet50 Model Garden example with MirroredStrategy API:
 
ResNet50 Model Garden example with MirroredStrategy API:
−
http://goo.gl/3UWhj8
+
https://goo.gl/3UWhj8
  
 
Performance Guides:
 
Performance Guides:
−
http://goo.gl/doqGE7
+
https://goo.gl/doqGE7
−
http://goo.gl/NCnrCn
+
https://goo.gl/NCnrCn
  
 
Commands to set up a GCE instance and run distributed training:
 
Commands to set up a GCE instance and run distributed training:
Line 104: Line 119:
  
 
Multi-machine distributed training with train_and_evaluate:
 
Multi-machine distributed training with train_and_evaluate:
−
http://goo.gl/kyikAC
+
https://goo.gl/kyikAC
  
−
Watch more [[TensorFlow]] sessions from I/O '18 here → http://goo.gl/GaAnBR  
+
Watch more [[TensorFlow]] sessions from I/O '18 here → https://goo.gl/GaAnBR  
−
See all the sessions from Google I/O '18 here → http://goo.gl/q1Tr8x
+
See all the sessions from Google I/O '18 here → https://goo.gl/q1Tr8x
  
−
Subscribe to the [[TensorFlow]] channel → http://goo.gl/ht3WGe
+
Subscribe to the [[TensorFlow]] channel → https://goo.gl/ht3WGe
 
|}
 
|}
 
|}<!-- B -->
 
|}<!-- B -->
Line 118: Line 133:
 
<youtube>xtxxLWZznBI</youtube>
 
<youtube>xtxxLWZznBI</youtube>
 
<b>Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
 
<b>Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
−
</b><br>In this video from 2018 Swiss HPC Conference, Torsten Hoefler from (ETH) Zürich presents: Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. "Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. Specifically, we present trends in DNN architectures and the resulting implications on parallelization strategies. We discuss the different types of concurrency in DNNs; synchronous and asynchronous stochastic gradient descent; distributed system architectures; communication schemes; and performance modeling. Based on these approaches, we extrapolate potential directions for parallelism in deep learning." Learn more: http://hpcadvisorycouncil.com  Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
+
</b><br>In this video from 2018 Swiss HPC Conference, Torsten Hoefler from (ETH) Zürich presents: Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. "[[Neural Network#Deep Neural Network (DNN)|Deep Neural Networks (DNN)]] are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical [[perspective]], followed by approaches for its parallelization. Specifically, we present trends in DNN architectures and the resulting implications on parallelization strategies. We discuss the different types of concurrency in DNNs; synchronous and asynchronous stochastic gradient descent; distributed system architectures; [[Agents#Communication | communication]] schemes; and performance modeling. Based on these approaches, we extrapolate potential directions for parallelism in deep learning." Learn more: https://hpcadvisorycouncil.com  Sign up for our insideHPC Newsletter: https://insidehpc.com/newsletter
 
|}
 
|}
 
|<!-- M -->
 
|<!-- M -->
Line 131: Line 146:
  
 
=== <span id="Peer-to-Peer"></span>Peer-to-Peer ===
 
=== <span id="Peer-to-Peer"></span>Peer-to-Peer ===
−
[http://www.youtube.com/results?search_query=Peer-to-Peer+deep+machine+learning+ML Youtube search...]
+
[https://www.youtube.com/results?search_query=Peer-to-Peer+deep+machine+learning+ML Youtube search...]
−
[http://www.google.com/search?q=Peer-to-Peer+deep+machine+learning+ML ...Google search]
+
[https://www.google.com/search?q=Peer-to-Peer+deep+machine+learning+ML ...Google search]
  
 
* [[Hopfield Network (HN)]]
 
* [[Hopfield Network (HN)]]
 
* [[Loop#Feedback Loop - Peer Learning|Feedback Loop - Peer Learning]]
 
* [[Loop#Feedback Loop - Peer Learning|Feedback Loop - Peer Learning]]
 
* [https://link.springer.com/chapter/10.1007/978-3-642-44958-1_35 Developing Machine Intelligence within P2P Networks Using a Distributed Associative Memory | A. Amir, A. Amin, and A. Khan - SpingerLink]  ...Distributed Associative Memory Tree (DASMET), to deal with multi-feature recognition in a peer-to-peer (P2P)-based system.
 
* [https://link.springer.com/chapter/10.1007/978-3-642-44958-1_35 Developing Machine Intelligence within P2P Networks Using a Distributed Associative Memory | A. Amir, A. Amin, and A. Khan - SpingerLink]  ...Distributed Associative Memory Tree (DASMET), to deal with multi-feature recognition in a peer-to-peer (P2P)-based system.
−
* [http://arxiv.org/abs/1905.06731 BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning | A. G. Roy, S. Siddiqui, S. Pölsterl, N. Navab, and C. Wachinger]
+
* [https://arxiv.org/abs/1905.06731 BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning | A. G. Roy, S. Siddiqui, S. Pölsterl, N. Navab, and C. Wachinger]
−
* [http://blog.imarticus.org/role-of-peer-to-peer-networks-in-creating-transparency-and-increased-usage-of-ai/ Role of Peer to Peer Networks In Creating Transparency And Increased Usage of AI | Imarticus]
+
* [https://blog.imarticus.org/role-of-peer-to-peer-networks-in-creating-transparency-and-increased-usage-of-ai/ Role of Peer to Peer Networks In Creating Transparency And Increased Usage of AI | Imarticus]
−
* [http://mas.cs.umass.edu/Documents/HZHANG_AAMAS07.pdf A Reinforcement Learning based Distributed Search Algorithm For Hierarchical Peer-to-Peer Information Retrieval Systems | Haizheng Zhang and Victor Lesser]  
+
* [https://mas.cs.umass.edu/Documents/HZHANG_AAMAS07.pdf A Reinforcement Learning based Distributed Search Algorithm For Hierarchical Peer-to-Peer Information Retrieval Systems | Haizheng Zhang and Victor Lesser]  
  
  
−
<img src="http://www.henry.pupil.me.uk/Images/Peer%20to%20peer.png" width="300">
+
<img src="https://www.henry.pupil.me.uk/Images/Peer%20to%20peer.png" width="300">
  
  
Line 159: Line 174:
 
<youtube>ie-qRQIQT4I</youtube>
 
<youtube>ie-qRQIQT4I</youtube>
 
<b>What is a Peer to Peer Network? Blockchain P2P Networks Explained
 
<b>What is a Peer to Peer Network? Blockchain P2P Networks Explained
−
</b><br>A peer to peer network, often referred to as p2p network, is one of the key aspects of blockchain technology. In this video, we break down the complexity of peer to peer networks by first defining what a network is and how p2p networks differ from traditional networks. [http://lisk.io/what-is-blockchain Learn more about P2P Networks]  
+
</b><br>A peer to peer network, often referred to as p2p network, is one of the key aspects of blockchain technology. In this video, we break down the complexity of peer to peer networks by first defining what a network is and how p2p networks differ from traditional networks. [https://lisk.io/what-is-blockchain Learn more about P2P Networks]  
 
|}
 
|}
 
|}<!-- B -->
 
|}<!-- B -->
  
 
=== <span id="Proxy"></span>Proxy ===
 
=== <span id="Proxy"></span>Proxy ===
−
[http://www.youtube.com/results?search_query=Proxy+deep+machine+learning+ML+AI Youtube search...]
+
[https://www.youtube.com/results?search_query=Proxy+deep+machine+learning+ML+AI Youtube search...]
−
[http://www.google.com/search?q=Proxy+deep+machine+learning+ML+AI ...Google search]
+
[https://www.google.com/search?q=Proxy+deep+machine+learning+ML+AI ...Google search]
  
 
{|<!-- T -->
 
{|<!-- T -->
Line 186: Line 201:
  
 
=== <span id="Submarine Scenario"></span>Submarine Scenario ===
 
=== <span id="Submarine Scenario"></span>Submarine Scenario ===
−
[http://www.youtube.com/results?search_query=D-DIL+denied+disconnected+intermittent+cloud+machine+learning+ML+AI Youtube search...]
+
[https://www.youtube.com/results?search_query=D-DIL+denied+disconnected+intermittent+cloud+machine+learning+ML+AI Youtube search...]
−
[http://www.google.com/search?q=D-DIL+denied+disconnected+intermittent+cloud+machine+learning+ML+AI ...Google search]
+
[https://www.google.com/search?q=D-DIL+denied+disconnected+intermittent+cloud+machine+learning+ML+AI ...Google search]
  
 
* [https://azure.microsoft.com/en-us/products/azure-stack Azure Stack]   
 
* [https://azure.microsoft.com/en-us/products/azure-stack Azure Stack]   
Line 201: Line 216:
 
**** [https://learn.microsoft.com/en-us/azure/architecture/solution-ideas/articles/ai-at-the-edge-disconnected Disconnected AI at the edge with Azure Stack Hub | ][[Microsoft]]
 
**** [https://learn.microsoft.com/en-us/azure/architecture/solution-ideas/articles/ai-at-the-edge-disconnected Disconnected AI at the edge with Azure Stack Hub | ][[Microsoft]]
 
**** [https://learn.microsoft.com/en-us/azure/architecture/hybrid/deploy-ai-ml-azure-stack-edge Deploy AI and machine learning computing on-premises and to the edge | ][[Microsoft]]
 
**** [https://learn.microsoft.com/en-us/azure/architecture/hybrid/deploy-ai-ml-azure-stack-edge Deploy AI and machine learning computing on-premises and to the edge | ][[Microsoft]]
 +
**** [https://learn.microsoft.com/en-us/azure/machine-learning/reference-machine-learning-cloud-parity Azure Machine Learning feature availability across clouds regions | ][[Microsoft]]
 
** [https://learn.microsoft.com/en-us/azure-stack/hci/concepts/stretched-clusters Stretched clusters overview | ][[Microsoft]]
 
** [https://learn.microsoft.com/en-us/azure-stack/hci/concepts/stretched-clusters Stretched clusters overview | ][[Microsoft]]
 
*** [https://learn.microsoft.com/en-us/azure/architecture/hybrid/azure-stack-hci-dr Use Azure Stack HCI stretched clusters for disaster recovery | ][[Microsoft]]
 
*** [https://learn.microsoft.com/en-us/azure/architecture/hybrid/azure-stack-hci-dr Use Azure Stack HCI stretched clusters for disaster recovery | ][[Microsoft]]
Line 208: Line 224:
 
* [https://dodcio.defense.gov/Portals/0/Documents/DoD-C3-Strategy.pdf Command, Control, and Communications (C3) Modernization Strategy | Department of] [[Defense]]
 
* [https://dodcio.defense.gov/Portals/0/Documents/DoD-C3-Strategy.pdf Command, Control, and Communications (C3) Modernization Strategy | Department of] [[Defense]]
  
−
[[Microsoft]]’s SharePoint, Exchange, and Office 365 products run on Azure and Azure Stack, as do [[Microsoft]]’s database, e-commerce, and software development products. Extend Azure services and capabilities to your environment of choice—from the datacenter to edge locations and remote offices—with Azure Stack. Build, deploy, and run hybrid and edge computing apps consistently across your IT ecosystem, with flexibility for diverse workloads. The Azure Stack Hub architecture lets you provide Azure services at the edge for remote locations or intermittent connectivity, disconnected from the internet. You can create hybrid solutions that process data locally in Azure Stack Hub and then aggregate it in Azure for additional processing and analytics  [https://www.nextplatform.com/2017/09/22/azure-stack-finally-takes-microsoft-public-cloud-private/#:~:text=Microsoft's%20SharePoint%2C%20Exchange%2C%20and%20Office,commerce%2C%20and%20software%20development%20products. Azure Stack Finally Takes Microsoft Public Cloud Private | Paul Teich - The Next Platform]
+
[[Microsoft]]’s SharePoint, Exchange, and Office 365 products run on Azure and Azure Stack, as do [[Microsoft]]’s database, e-commerce, and software [[development]] products. Extend Azure services and capabilities to your environment of choice—from the datacenter to edge locations and remote offices—with Azure Stack. Build, deploy, and run hybrid and edge computing apps consistently across your IT ecosystem, with flexibility for diverse workloads. The Azure Stack Hub architecture lets you provide Azure services at the edge for remote locations or intermittent connectivity, disconnected from the internet. You can create hybrid solutions that process data locally in Azure Stack Hub and then aggregate it in Azure for additional processing and analytics  [https://www.nextplatform.com/2017/09/22/azure-stack-finally-takes-microsoft-public-cloud-private/#:~:text=Microsoft's%20SharePoint%2C%20Exchange%2C%20and%20Office,commerce%2C%20and%20software%20development%20products. Azure Stack Finally Takes Microsoft Public Cloud Private | Paul Teich - The Next Platform]
  
 
<blockquote style="border: 2px solid #666; padding: 10px; background-color: #ccc;"> Sometimes, this kind of environment is also referred to as a 'submarine' scenario. - [[Microsoft]]</blockquote>
 
<blockquote style="border: 2px solid #666; padding: 10px; background-color: #ccc;"> Sometimes, this kind of environment is also referred to as a 'submarine' scenario. - [[Microsoft]]</blockquote>
  
−
In 2021, the [[Defense|DoD]] CIO designated the Department of Navy CIO as the executive agent to lead a cross-service joint working group focused on Denied-Disconnected, Intermittent, and Low bandwidth (D-DIL)... Network server software and hardware exist at the tactical edge to provide critical IT services and data in these DDIL environments, along with a variety of spectrum communications and unclassified & classified network transports leveraging satellite links and low-Earth Orbit (LEO), Wi-Fi, cellular/4G LTE, millimeter wave/5G and others. The working group has teamed up with industry to refine DoD-unique requirements and use cases, resulting in the development of standardized architectures and solutions for the relevant collaboration and productivity tools (email, chat, voice and video, file management). These tools operate as a hybrid capability, which will allow users access to the full feature set when cloud connectivity is available, but remain productive locally within the DDIL environment. [https://www.doncio.navy.mil/chips/ArticleDetails.aspx?ID=15161 DoD working with Industry to Adapt Cloud Tools for the Tactical Edge - DON CIO]
+
In 2021, the [[Defense|DoD]] CIO designated the Department of Navy CIO as the executive agent to lead a cross-service joint working group focused on Denied-Disconnected, Intermittent, and Low bandwidth (D-DIL)... Network server software and hardware exist at the tactical edge to provide critical IT services and data in these DDIL environments, along with a variety of spectrum [[Agents#Communication | communication]]s and unclassified & classified network transports leveraging satellite links and low-Earth Orbit (LEO), Wi-Fi, cellular/4G LTE, millimeter wave/5G and others. The working group has teamed up with industry to refine DoD-unique requirements and use cases, resulting in the development of standardized architectures and solutions for the relevant collaboration and productivity tools (email, chat, voice and video, file management). These tools operate as a hybrid capability, which will allow users access to the full feature set when cloud connectivity is available, but remain productive locally within the DDIL environment. [https://www.doncio.navy.mil/chips/ArticleDetails.aspx?ID=15161 DoD working with Industry to Adapt Cloud Tools for the Tactical Edge - DON CIO]
  
 
Additionally, as [[Defense|DoD]] enterprise IT moves to the cloud, tactical networks must unify access to data and applications from the enterprise level to the tactical edge. This means deploying cloudlike services at the tactical edge of the network, so that data is available at the edge even when WAN connectivity is unavailable.[https://www.army.mil/article/216031/four_future_trends_in_tactical_network_modernization Four future trends In tactical network modernization | US Army]
 
Additionally, as [[Defense|DoD]] enterprise IT moves to the cloud, tactical networks must unify access to data and applications from the enterprise level to the tactical edge. This means deploying cloudlike services at the tactical edge of the network, so that data is available at the edge even when WAN connectivity is unavailable.[https://www.army.mil/article/216031/four_future_trends_in_tactical_network_modernization Four future trends In tactical network modernization | US Army]

Latest revision as of 16:56, 28 April 2024

YouTube ... Quora ...Google search ...Google News ...Bing News


Centralized vs. Decentralized vs. Distributed

Youtube search... ...Google search

Centralised-decentralised-distributed.png


Federated

Youtube search... ...Google search




Distributed

Youtube search... ...Google search

Distributed machine learning refers to multi-node machine learning algorithms and systems that are designed to improve performance, increase accuracy, and scale to larger input data sizes. Increasing the input data size for many algorithms can significantly reduce the learning error and can often be more effective than using more complex methods [8]. Distributed machine learning allows companies, researchers, and individuals to make informed decisions and draw meaningful conclusions from large amounts of data. Many systems exist for performing machine learning tasks in a distributed environment. These systems fall into three primary categories: database, general, and purpose-built systems. Each type of system has distinct advantages and disadvantages, but all are used in practice depending upon individual use cases, performance requirements, input data sizes, and the amount of implementation effort. | SpringerLink

Distinguished Lecturer : Eric Xing - Strategies & Principles for Distributed Machine Learning
The rise of Big Data has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters that promise adequate capacity to digest massive datasets and offer powerful Predictive Analytics (such as high-dimensional latent features, intermediate representations, and decision functions) thereupon. In order to run ML algorithms at such scales, on a distributed cluster with 10s to 1000s of machines, it is often the case that significant engineering efforts are required --- and one might fairly ask if such engineering truly falls within the domain of ML research or not. Taking the view that Big ML systems can indeed benefit greatly from ML-rooted statistical and algorithmic insights --- and that ML researchers should therefore not shy away from such systems design --- we discuss a series of principles and strategies distilled from our resent effort on industrial-scale ML solutions that involve a continuum from application, to engineering, and to theoretical research and development of Big ML system and architecture, on how to make them efficient, general, and with convergence and scaling guarantees. These principles concern four key questions which traditionally receive little attention in ML research: How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine communication? How to perform such communication? What should be communicated between machines? By exposing underlying statistical and algorithmic characteristics unique to ML programs but not typical in traditional computer programs, and by dissecting successful cases of how we harness these principles to design both high-performance distributed ML software and general-purpose ML framework, we present opportunities for ML researchers and practitioners to further shape and grow the area that lies between ML and systems. This is joint work with the CMU Petuum Team.

Distributed TensorFlow training (Google I/O '18)
To efficiently train machine learning models, you will often need to scale your training to multiple GPUs, or even multiple machines. TensorFlow now offers rich functionality to achieve this with just a few lines of code. Join this session to learn how to set this up.

Distribution Strategy API: https://goo.gl/F9vXqQ https://goo.gl/Zq2xvJ

ResNet50 Model Garden example with MirroredStrategy API: https://goo.gl/3UWhj8

Performance Guides: https://goo.gl/doqGE7 https://goo.gl/NCnrCn

Commands to set up a GCE instance and run distributed training: https://goo.gl/xzwN4C

Multi-machine distributed training with train_and_evaluate: https://goo.gl/kyikAC

Watch more TensorFlow sessions from I/O '18 here → https://goo.gl/GaAnBR See all the sessions from Google I/O '18 here → https://goo.gl/q1Tr8x

Subscribe to the TensorFlow channel → https://goo.gl/ht3WGe

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
In this video from 2018 Swiss HPC Conference, Torsten Hoefler from (ETH) Zürich presents: Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. "Deep Neural Networks (DNN) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. Specifically, we present trends in DNN architectures and the resulting implications on parallelization strategies. We discuss the different types of concurrency in DNNs; synchronous and asynchronous stochastic gradient descent; distributed system architectures; communication schemes; and performance modeling. Based on these approaches, we extrapolate potential directions for parallelism in deep learning." Learn more: https://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: https://insidehpc.com/newsletter

Machine Learning Systems for Highly Distributed and Rapidly Growing Data
Microsoft Research The usability and practicality of machine learning are largely influenced by two critical factors: low latency and low cost. However, achieving low latency and low cost is very challenging when machine learning depends on real-world data that are rapidly growing and highly distributed (e.g., training a face recognition model using pictures stored across many data centers globally). In this talk, I will present my work on building low-latency and low-cost machine learning systems that enable efficient processing of real-world, large-scale data. I will describe a system-level approach that is inspired by the general characteristics of machine learning algorithms, machine learning model structures, and machine learning training/serving data. In line with this approach, I will first present a system that provides both low-latency and low-cost machine learning serving (inferencing) over large-scale continuously-growing datasets (e.g. videos). Shifting the focus to model training, I will then present a system that makes machine learning training over geo-distributed datasets as fast as training within a single data center. Finally, I will discuss our ongoing efforts to tackle a fundamental and largely overlooked problem: machine learning training over skewed data partitions (e.g., facial images collected by cameras in different countries).

Peer-to-Peer

Youtube search... ...Google search



Introduction to Decentralized P2P Apps
Most people think peer-to-peer (P2P) networks are just for file sharing, but it turns out you can also build other types of applications on P2P networks with advantages like enhanced privacy and security. We’ll walk through the process of building an increasingly complex P2P cloud storage system (think Dropbox), and touch on the challenges you’d run into and some of their possible solutions. Topics include efficiently locating data within a large network and building a system where we can trust random people on the internet with our personal files. EVENT: SFNode Meetup July 2018 SPEAKER: Dylan Barnard PERMISSIONS: SFNode Meetup Organizer provided Coding Tech with the permission to republish this video.

What is a Peer to Peer Network? Blockchain P2P Networks Explained
A peer to peer network, often referred to as p2p network, is one of the key aspects of blockchain technology. In this video, we break down the complexity of peer to peer networks by first defining what a network is and how p2p networks differ from traditional networks. Learn more about P2P Networks

Proxy

Youtube search... ...Google search

Proxy vs. Reverse Proxy (Explained by Example)
Hussein Nasser In this episode we explain the difference between a Proxy (Forward proxy) and Reverse Proxy by example, and list all the benefits of each server. Online diagram tool used in this video: Http://www.gliffy.com

Proxy vs. Peer-to-Peer (P2P) Connections | remote.it webinar
In this webinar, we will explain how each connection type works, and in what applications you may prefer to use one or the other. You will learn how to use remote.it on Windows or macOS and on mobile (iOS/Android) apps to make P2P connections while we present the advantages of P2P versus traditional proxy connections from the remote.it web portal. All while making port forwardless connections.

Submarine Scenario

Youtube search... ...Google search

Microsoft’s SharePoint, Exchange, and Office 365 products run on Azure and Azure Stack, as do Microsoft’s database, e-commerce, and software development products. Extend Azure services and capabilities to your environment of choice—from the datacenter to edge locations and remote offices—with Azure Stack. Build, deploy, and run hybrid and edge computing apps consistently across your IT ecosystem, with flexibility for diverse workloads. The Azure Stack Hub architecture lets you provide Azure services at the edge for remote locations or intermittent connectivity, disconnected from the internet. You can create hybrid solutions that process data locally in Azure Stack Hub and then aggregate it in Azure for additional processing and analytics Azure Stack Finally Takes Microsoft Public Cloud Private | Paul Teich - The Next Platform

Sometimes, this kind of environment is also referred to as a 'submarine' scenario. - Microsoft

In 2021, the DoD CIO designated the Department of Navy CIO as the executive agent to lead a cross-service joint working group focused on Denied-Disconnected, Intermittent, and Low bandwidth (D-DIL)... Network server software and hardware exist at the tactical edge to provide critical IT services and data in these DDIL environments, along with a variety of spectrum communications and unclassified & classified network transports leveraging satellite links and low-Earth Orbit (LEO), Wi-Fi, cellular/4G LTE, millimeter wave/5G and others. The working group has teamed up with industry to refine DoD-unique requirements and use cases, resulting in the development of standardized architectures and solutions for the relevant collaboration and productivity tools (email, chat, voice and video, file management). These tools operate as a hybrid capability, which will allow users access to the full feature set when cloud connectivity is available, but remain productive locally within the DDIL environment. DoD working with Industry to Adapt Cloud Tools for the Tactical Edge - DON CIO

Additionally, as DoD enterprise IT moves to the cloud, tactical networks must unify access to data and applications from the enterprise level to the tactical edge. This means deploying cloudlike services at the tactical edge of the network, so that data is available at the edge even when WAN connectivity is unavailable.Four future trends In tactical network modernization | US Army

Azure Stack for hybrid compute and disconnected scenarios
Latest updates to hybrid compute using the power of Microsoft's Azure cloud on-premises with Azure Stack. Partner Director Program Manager, Natalia Mackevicius will show you how to run hybrid apps and process data between Azure Stack and your data center and in the Azure public cloud to meet your regulatory and policy requirements. Plus, see how you can use Azure Stack as a powerful control plane for disconnected scenarios to harness insights from your IoT devices running on the Edge.

Introduction to Microsoft 365 | Windows, EMS, Azure, Azure Stack | Versions and Deployment Models
Azure Stack is an Azure addon that may be used in datacenter and hybrid environments. This lets you execute apps on the same platform on the edge, which Microsoft calls hybrid cloud and fully disconnected solutions. It lets you operate cloud applications that connect to on-premises data for regulatory needs while using a single DevOps and maintenance stack. So developers can code for Azure and IT pros can manage it. They have a single platform to manage cloud and on-premises data. Office 365 and Azure both run in the public cloud, whereas private and hybrid clouds employ Azure Stack.

Can I Use Office 365 Offline?
Office 365 is Microsoft's suite of cloud services for office productivity, so you may be thinking it can only be used online. However, Office 365 comes with the offline capability to make sure you can stay productive at any time, whether or not you’re connected to the internet.

Azure AD Connect Sync and Cloud Sync, What’s the Difference?
Many organizations use Azure AD Connect Sync to synchronize hybrid identities from Windows AD to Azure AD DS. Microsoft recently announces a new service, Azure AD Connect Cloud Sync, that also synchronizes Windows AD identities to Azure AD. In this video, we go over how they are similar, features that are different, and when to use one or the other.

Retrieved from "https://primo.ai/index.php?title=Decentralized:_Federated_%26_Distributed&oldid=36967"