Decentralized: Federated & Distributed

Youtube search... ...Google search

• Platforms: Machine Learning as a Service (MLaaS)

Centralized vs. Decentralized vs. Distributed

Youtube search... ...Google search

• Social Media Alternatives Series, EP. 1: What You NEED to Know | Fliphodl
• Exclusive: We tried Ultra and it will kill Steam | Fliphodl

Federated

Youtube search... ...Google search

• Federated Learning | Wikipedia
• Federated Machine Learning: Concept and Applications | Q. Yang, Y. Liu, T. Chen and Y. Tong
• Federated Learning: The Future of Distributed Machine Learning | Mi Zhang - Medium
• Chained Anomaly Detection Models for Federated Learning: An Intrusion Detection Case Study | D. Preuveneers, V. Rimmer, I. Tsingenopoulos, J. Spooren, W. Joosen and E. Ilie-Zudor
• A Beginners Guide to Federated Learning | Santanu Bhattacharya
• Watch me Build a Cybersecurity Startup
• Federated Learning Frameworks:
  • OpenMined Pysyft ...GitHub
  • TensorFlow Federated Learning ...* TensorFlow Federated helps train AI models on data from different locations | Kyle Wiggers - VentureBeat
  • Federated AI Ecosystem (FATE) | FEDAI.org
  • Clara Federated Learning | NVIDIA

• NVIDIA (King’s College London) uses federated learning to create medical imaging AI | Khari Johnson - VentureBeat

Distributed

Youtube search... ...Google search

• Distributed Deep Reinforcement Learning (DDRL)

Distributed machine learning refers to multi-node machine learning algorithms and systems that are designed to improve performance, increase accuracy, and scale to larger input data sizes. Increasing the input data size for many algorithms can significantly reduce the learning error and can often be more effective than using more complex methods [8]. Distributed machine learning allows companies, researchers, and individuals to make informed decisions and draw meaningful conclusions from large amounts of data. Many systems exist for performing machine learning tasks in a distributed environment. These systems fall into three primary categories: database, general, and purpose-built systems. Each type of system has distinct advantages and disadvantages, but all are used in practice depending upon individual use cases, performance requirements, input data sizes, and the amount of implementation effort. | SpringerLink

Distinguished Lecturer : Eric Xing - Strategies & Principles for Distributed Machine Learning The rise of Big Data has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters that promise adequate capacity to digest massive datasets and offer powerful predictive analytics (such as high-dimensional latent features, intermediate representations, and decision functions) thereupon. In order to run ML algorithms at such scales, on a distributed cluster with 10s to 1000s of machines, it is often the case that significant engineering efforts are required --- and one might fairly ask if such engineering truly falls within the domain of ML research or not. Taking the view that Big ML systems can indeed benefit greatly from ML-rooted statistical and algorithmic insights --- and that ML researchers should therefore not shy away from such systems design --- we discuss a series of principles and strategies distilled from our resent effort on industrial-scale ML solutions that involve a continuum from application, to engineering, and to theoretical research and development of Big ML system and architecture, on how to make them efficient, general, and with convergence and scaling guarantees. These principles concern four key questions which traditionally receive little attention in ML research: How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine communication? How to perform such communication? What should be communicated between machines? By exposing underlying statistical and algorithmic characteristics unique to ML programs but not typical in traditional computer programs, and by dissecting successful cases of how we harness these principles to design both high-performance distributed ML software and general-purpose ML framework, we present opportunities for ML researchers and practitioners to further shape and grow the area that lies between ML and systems. This is joint work with the CMU Petuum Team.

Distributed TensorFlow training (Google I/O '18) To efficiently train machine learning models, you will often need to scale your training to multiple GPUs, or even multiple machines. TensorFlow now offers rich functionality to achieve this with just a few lines of code. Join this session to learn how to set this up. Distribution Strategy API: https://goo.gl/F9vXqQ http://goo.gl/Zq2xvJ ResNet50 Model Garden example with MirroredStrategy API: http://goo.gl/3UWhj8 Performance Guides: http://goo.gl/doqGE7 http://goo.gl/NCnrCn Commands to set up a GCE instance and run distributed training: https://goo.gl/xzwN4C Multi-machine distributed training with train_and_evaluate: http://goo.gl/kyikAC Watch more TensorFlow sessions from I/O '18 here → http://goo.gl/GaAnBR See all the sessions from Google I/O '18 here → http://goo.gl/q1Tr8x Subscribe to the TensorFlow channel → http://goo.gl/ht3WGe

Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis In this video from 2018 Swiss HPC Conference, Torsten Hoefler from (ETH) Zürich presents: Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. "Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this talk, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. Specifically, we present trends in DNN architectures and the resulting implications on parallelization strategies. We discuss the different types of concurrency in DNNs; synchronous and asynchronous stochastic gradient descent; distributed system architectures; communication schemes; and performance modeling. Based on these approaches, we extrapolate potential directions for parallelism in deep learning." Learn more: http://hpcadvisorycouncil.com Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Machine Learning Systems for Highly Distributed and Rapidly Growing Data Microsoft Research The usability and practicality of machine learning are largely influenced by two critical factors: low latency and low cost. However, achieving low latency and low cost is very challenging when machine learning depends on real-world data that are rapidly growing and highly distributed (e.g., training a face recognition model using pictures stored across many data centers globally). In this talk, I will present my work on building low-latency and low-cost machine learning systems that enable efficient processing of real-world, large-scale data. I will describe a system-level approach that is inspired by the general characteristics of machine learning algorithms, machine learning model structures, and machine learning training/serving data. In line with this approach, I will first present a system that provides both low-latency and low-cost machine learning serving (inferencing) over large-scale continuously-growing datasets (e.g. videos). Shifting the focus to model training, I will then present a system that makes machine learning training over geo-distributed datasets as fast as training within a single data center. Finally, I will discuss our ongoing efforts to tackle a fundamental and largely overlooked problem: machine learning training over skewed data partitions (e.g., facial images collected by cameras in different countries).

Peer-to-Peer

Youtube search... ...Google search

• Hopfield Network (HN)
• Feedback Loop - Peer Learning
• Developing Machine Intelligence within P2P Networks Using a Distributed Associative Memory | A. Amir, A. Amin, and A. Khan - SpingerLink ...Distributed Associative Memory Tree (DASMET), to deal with multi-feature recognition in a peer-to-peer (P2P)-based system.
• BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning | A. G. Roy, S. Siddiqui, S. Pölsterl, N. Navab, and C. Wachinger
• Role of Peer to Peer Networks In Creating Transparency And Increased Usage of AI | Imarticus
• A Reinforcement Learning based Distributed Search Algorithm For Hierarchical Peer-to-Peer Information Retrieval Systems | Haizheng Zhang and Victor Lesser

Introduction to Decentralized P2P Apps Most people think peer-to-peer (P2P) networks are just for file sharing, but it turns out you can also build other types of applications on P2P networks with advantages like enhanced privacy and security. We’ll walk through the process of building an increasingly complex P2P cloud storage system (think Dropbox), and touch on the challenges you’d run into and some of their possible solutions. Topics include efficiently locating data within a large network and building a system where we can trust random people on the internet with our personal files. EVENT: SFNode Meetup July 2018 SPEAKER: Dylan Barnard PERMISSIONS: SFNode Meetup Organizer provided Coding Tech with the permission to republish this video.

What is a Peer to Peer Network? Blockchain P2P Networks Explained A peer to peer network, often referred to as p2p network, is one of the key aspects of blockchain technology. In this video, we break down the complexity of peer to peer networks by first defining what a network is and how p2p networks differ from traditional networks. Learn more about P2P Networks

Proxy

Youtube search... ...Google search

Proxy vs. Reverse Proxy (Explained by Example) Hussein Nasser In this episode we explain the difference between a Proxy (Forward proxy) and Reverse Proxy by example, and list all the benefits of each server. Online diagram tool used in this video: Http://www.gliffy.com

Proxy vs. Peer-to-Peer (P2P) Connections | remote.it webinar In this webinar, we will explain how each connection type works, and in what applications you may prefer to use one or the other. You will learn how to use remote.it on Windows or macOS and on mobile (iOS/Android) apps to make P2P connections while we present the advantages of P2P versus traditional proxy connections from the remote.it web portal. All while making port forwardless connections.

Privacy Preserving

YouTube search... ...Google search

The availability of massive amounts of data, coupled with high-performance cloud computing platforms, has driven significant progress in artificial intelligence and, in particular, machine learning and optimization. Indeed, much scientific and technological growth in recent years, including in computer vision, natural language processing, transportation, and health, has been driven by large-scale data sets which provide a strong basis to improve existing algorithms and develop new ones. However, due to their large-scale and longitudinal collection, archiving these data sets raise significant privacy concerns. They often reveal sensitive personal information that can be exploited, without the knowledge and/or consent of the involved individuals, for various purposes including monitoring, discrimination, and illegal activities. The AAAI Workshop on Privacy-Preserving Artificial Intelligence

• Case Studies
  • Risk, Compliance and Regulation
• OpenMined
• Facial recognition has to be regulated to protect the public, says AI report | Will Knight - MIT Technology Review
• Screening; Passenger, Luggage, & Cargo
• Other Challenges in Artificial Intelligence

Privacy Preserving - Machine Learning (PPML) Techniques

Youtube search... ...Google search

• Secure and Private AI by Facebook AI | UDACITY introduction of three cutting-edge technologies for privacy-preserving AI: Federated Learning, Differential Privacy, and Encrypted Computation. Learn how to extend PyTorch with the tools necessary to train AI models that preserve user privacy.
• Perfectly Privacy-Preserving AI | Patricia Thaine

Many privacy-enhancing techniques concentrated on allowing multiple input parties to collaboratively train ML models without releasing their private data in its original form. This was mainly performed by utilizing cryptographic approaches, or differentially-private data release (perturbation techniques). Differential privacy is especially effective in preventing membership inference attacks. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Multiparty Computation (MPC) enables computation on data from different providers/parties, such that the other participating parties gain no additional information about each others’ inputs, except what can be learned from the public output of the algorithm. In other words, when we have the parties Alice, Bob and Casper, all three have access to the output. However, it is not possible for, e.g., Alice to know the plain data Bob and Casper provided. Secure Multiparty Computation — Enabling Privacy-Preserving Machine Learning | Florian Apfelbeck - Medium

• DeepSecure DeepSecure: Scalable Provably-Secure Deep Learning | B. Rouhani, M. S. Riazi, and F. Koushanfar
• SecureML SecureML: A System for Scalable Privacy-Preserving Machine Learning | Payman Mohassel & Yupeng Zhang
• MiniONN Oblivious Neural Network Predictions via MiniONN transformations | J. Liu, M. Juuti, Y. Lu and N. Asokan
• ABY3 ABY3: A Mixed Protocol Framework for Machine Learning | Payman Mohassel & Peter Rindal

Cryptographic Approaches

Youtube search... ...Google search

When a certain ML application requires data from multiple input parties, cryptographic protocols could be utilized to perform ML training/testing on encrypted data. In many of these techniques, achieving better efficiency involved having data owners contribute their encrypted data to the computation servers, which would reduce the problem to a secure two/three party computation setting. In addition to increased efficiency, such approaches have the benefit of not requiring the input parties to remain online. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Homomorphic Encryption

Youtube search... ...Google search

Fully homomorphic encryption enables the computation on encrypted data, with operations such as addition and multiplication that can be used as basis for more complex arbitrary functions. Due to the high cost associated with frequently bootstrapping the cipher text (refreshing the cipher text because of the accumulated noise), additive homomorphic encryption schemes were mostly used in PPML approaches. Such schemes only enable addition operations on encrypted data, and multiplication by a plain text. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Garbled Circuits

Youtube search... ...Google search

Assuming a two-party setup with Alice and Bob wanting to obtain the result of a function computed on their private inputs, Alice can convert the function into a garbled circuit, and send this circuit along with her garbled input. Bob obtains the garbled version of his input from Alice without her learning anything about Bob’s private input (e.g., using oblivious transfer). Bob can now use his garbled input with the garbled circuit to obtain the result of the required function (and can optionally share it with Alice). Some PPML approaches combined additive homomorphic encryption with Garbled circuits. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Secret Sharing

Youtube search... ...Google search

A method for distributing a secret among multiple parties, with each one holding a “share” of the secret. Individual shares are of no use on their own; however, when the shares are combined, the secret can be reconstructed. With threshold secret sharing, not all the “shares” are required to reconstruct the secret; but only “t” of them (“t” refers to threshold). In one setting, multiple input parties can generate “shares” of their private data, and send these shares to a set of non-colluding computation servers. Each server could compute a “partial result” from the “shares” it received. Finally, a results’ party (or a proxy) can receive these partial results, and combine them to find the final result. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Secure Processors

Youtube search... ...Google search

While initially introduced to ensure the confidentiality and integrity of sensitive code from unauthorized access by rogue software at higher privilege levels, Intel SGXprocessor are being utilized in privacy-preserving computation. Ohrimenko et al.14 developed a data oblivious ML algorithms for neural networks, SVM, k-means clustering, decision trees and matrix factorization that are based on SGX-processors. The main idea involves having multiple data owners collaborate to perform one of the above mentioned ML tasks with the computation party running the ML task on an SGX-enabled data center. An adversary can control all the hardware and software in the data center except for the SGX-processors used for computation. In this system, each data owner independently establishes a secure channel with the enclave (containing the code and data), authenticates themselves, verifies the integrity of the ML code in the cloud, and securely uploads its private data to the enclave. After all the data is uploaded, the ML task is run by the secure processor, and the output is sent to the results’ parties over secure authenticated channels. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Perturbation Approaches

Youtube search... ...Google search

Differential privacy (DP) techniques resist membership inference attacks by adding random noise to the input data, to iterations in a certain algorithm, or to the algorithm output. While most DP approaches assume a trusted aggregator of the data, local differential privacy allows each input party to add the noise locally; thus, requiring no trusted server. Finally, dimensionally reduction perturbs the data by projecting it to a lower dimensional hyperplane to prevent reconstructing the original data, and/or to restrict inference of sensitive information. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Differential Privacy (DP)

Youtube search... ...Google search

Differential privacy is a powerful tool for quantifying and solving practical problems related to privacy. Its flexible definition gives it the potential to be applied in a wide range of applications, including Machine Learning applications. Understanding Differential Privacy - From Intuitions behind a Theory to a Private AI Application | An Nguyen - Towards Data Science

Differential privacy is a system for publicly sharing information about a dataset by describing the patterns of groups within the dataset while withholding information about individuals in the dataset. Another way to describe differential privacy is as a constraint on the algorithms used to publish aggregate information about a statistical database which limits the disclosure of private information of records whose information is in the database. For example, differentially private algorithms are used by some government agencies to publish demographic information or other statistical aggregates while ensuring confidentiality of survey responses, and by companies to collect information about user behavior while controlling what is visible even to internal analysts. Roughly, an algorithm is differentially private if an observer seeing its output cannot tell if a particular individual's information was used in the computation. Differential privacy is often discussed in the context of identifying individuals whose information may be in a database. Although it does not directly refer to identification and reidentification attacks, differentially private algorithms probably resist such attacks. Differential privacy was developed by cryptographers and thus is often associated with cryptography, and draws much of its language from cryptography. Wikipedia

Local Differential Privacy

Youtube search... ...Google search

When the input parties do not have enough information to train a ML model, it might be better to utilize approaches that rely on local differential privacy (LDP). With LDP, each input party would perturb their data, and only release this obscure view of the data. An old, and well-known version of local privacy is randomized response (Warner 1965), which provided plausible deniability for respondents to sensitive queries. For example, a respondent would flip a fair coin: (a) if “tails”, the respondent answers truthfully, and (b) if “heads”, then flip a second coin, and respond “Yes” if heads, and “No” if tails. RAPPOR 22 is a technology for crowdsourcing statistics from end-user client software by applying RR to Bloom filters with strong 𝜀-DP guarantees. RAPPOR is deployed in Google Chrome web browser, and it permits collecting statistics on client-side values and strings, such as their categories, frequencies, and histograms. By performing RR twice with a memoization step in between, privacy protection is maintained even when multiple responses are collected from the same participant over time. A ML oriented work, AnonML23, utilized the ideas of RR for generating histograms from multiple input parties. AnonML utilizes these histograms to generate synthetic data on which a ML model can be trained. Like other local DP approaches, AnonML is a good option when no input party has enough data to build a ML model on their own (and there is no trusted aggregator). Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Dimensionality Reduction (DR)

Youtube search... ...Google search

perturbs the data by projecting it to a lower dimensional hyperplane. Such transformation is lossy, and it was suggested by Liu et al.24 that it would enhance the privacy, since retrieving the exact original data from a reduced dimension version would not be possible (the possible solutions are infinite as the number of equations is less than the number of unknowns). Hence, Liu et al.24 proposed to use a random matrix to reduce the dimensions of the input data. Since a random matrix might decrease the utility, other approaches used both unsupervised and supervised DR techniques such as principal component analysis (PCA), discriminant component analysis (DCA), and multidimensional scaling (MDS). These approaches try to find the best projection matrix for utility purposes, while relying on the reduced dimensionality aspect to enhance the privacy. Since an approximation of the original data can still be obtained from the reduced dimensions, some approaches, e.g. Jiang et al.25, combined dimensionality reduction with DP to achieve differentially-private data publishing. While some entities might seek total hiding of their data, DR has another benefit for privacy. For datasets that have samples with two labels: a utility label and a privacy label, Kung26 proposes a DR method to enable the data owner to project her data in a way that enables maximizing the accuracy of learning for the utility labels, while decreasing the accuracy for learning the privacy labels. Privacy Preserving Machine Learning: Threats and Solutions | Mohammad Al-Rubaie - Iowa State University

Blockchain

• Case Studies
  • Agriculture
  • Healthcare
  • Law
  • Banking
• 4 Different Types of Blockchain Technology & Networks | 101 Bloackchains

General Data Protection Regulations (GDPR)

YouTube search... ...Google search

• General Data Protection Regulation (GDPR)