Lambda

YouTube ... Quora ...Google search ...Google News ...Bing News

• AWS Lambda & Python
• Python ... GenAI w/ Python ... JavaScript ... GenAI w/ JavaScript ... TensorFlow ... PyTorch
• Development ... Notebooks ... AI Pair Programming ... Codeless, Generators, Drag n' Drop ... AIOps/MLOps ... AIaaS/MLaaS
• Gaming ... Game-Based Learning (GBL) ... Security ... Generative AI ... Games - Metaverse ... Quantum ... Game Theory
• Immersive Reality ... Metaverse ... Omniverse ... Transhumanism ... Religion
• Telecommunications ... Computer Networks ... 5G ... Satellite Communications ... Quantum Communications ... Communication Agents ... Smart Cities ... Digital Twin ... Internet of Things (IoT)
• Product page...
• Lambda Dashboard
• CloudWatch - Lambda logs
• AWS IoT Button
• DeepLens - deep learning enabled video camera
• Amazon AWS
• Kinesis
• Watch me Build a Cybersecurity Startup | Siraj Raval

Lets you run code without provisioning or managing servers. You can run code for virtually any type of application or backend service - all with zero administration. Just upload your code and Lambda takes care of everything required to run and scale your code with high availability. You can set up your code to automatically trigger from other AWS services or call it directly from any web or mobile app.

...with Alexa

• Amazon Developer Portal

Lambda Queue

• Creating a Machine Learning Data Pipeline in AWS Lambda | Alec Hewitt @ Medium

Implement a data pipeline that ingests and processes information and create a prediction of whether the devices are performing well or not and then send that prediction back to the device manager. The data on a Microsoft Azure Queue. A scheduled cloud watch event was created that would trigger a Lambda function once a minute that’s sole purpose was to poll the Azure queue for the number of messages and then send that many notifications to a Simple Notification Service (SNS) topic. Data ingestion would then be scaled off this SNS topic rather than the Azure queue itself. Our application was broken into three stages which mapped to three Lambda functions.

• Poll Queue: Singleton Lambda function polling Azure queue once a minute, determining number of messages in the queue and then sending that many notifications to a SNS topic.

• Data Persist: Collate and persist to our DB. Trigger a prediction if enough data to do so.

• Predict: Predict the performance of devices, then send the performance status back to the device managers.

Few sampling challenges:

• To solve a multiple sampling challenge, a semaphore was implemented using a locking system with Redis ElastiCache. When the data persist function deemed that a device was valid for a prediction it would try to obtain a lock. The key for the lock contained the device id and dates the prediction was for. If it did not succeed in obtaining a lock it meant that another data persist function had already triggered a prediction that was yet to run, so there was no need to trigger a prediction. The lock was released by the predict function, deleting the key in Redis, upon successfully creating and persisting a prediction for the device and time period.

• Some results were close to switch point and cause toggling of the results. A rolling average was relatively trivial to implement for existing devices, and when data was always sent and persisted in order. However whenever we got new device it posed some problems. When a new device was registered we would be sent up to a year’s worth of data for the device, but we had no guarantee of what order the data was going to be sent, as well as having no guarantee upon which data persist function would complete first. This made it difficult to apply a rolling average without knowing whether more data would be coming that pre dated the current data. To solve this problem we made an assumption that all data would have been persisted and propagated through the queue within 45 mins of seeing the last piece of data. To implement this we used a Redis sorted set. A Redis sorted set is a non repeating collection of Strings in which each member of the set is associated with a score. This score is used to order the set. We then had another Lambda function that was polling the sorted set for all the devices whose score was more than 45 mins ago, giving us those who haven’t had any data persisted for at least 45 mins and are therefore ready to have a rolling average calculated.

Developing and managing deployments: Our code was all in Python and to simplify and manage our code we used the Serverless framework. One problem the Serverless framework does not provide a solution to is managing third party dependencies. When packaging python libraries for Lambda, if any of them rely on compiled C code or shared libraries the libraries must be compiled on the Amazon Linux OS, as this is the OS that Lambda uses. One solution to this is to provision an EC2 instance with the Amazon Linux AMI to compile all of the libraries. We decided to use a docker image of the Amazon Linux AMI. Using docker made the deployment pipeline more flexible as we could compile on any platform and get the same result.

Problems Encountered:

• Lambda has a limit for each function deployment size to be less than 50MB. To overcome this we linked the libraries that had the same dependencies to share those dependencies.

• AWS Lambda functions are supposed to be idempotent, meaning that a single event can trigger a function many times without having unintended consequences. For example one notification from SNS can trigger two invocations of a Lambda function rather than just one. For us this posed a problem. These duplicate function were using up valuable resources both on our database and Lambda execution time. To resolve this, we turned to Redis again. We utilised the unique identifier of the event trigger and set it in Redis upon successfully completing execution of the function. Then at the beginning of each function we checked for the existence of this key in Redis. If it already existed we had already successfully processed this event once, therefore there was no need to execute again so the function immediately returned. An expiry was set on the key in order that Redis didn’t get full up with all of these keys.

Lambda is often billed as a magic bullet where you just write the logic of your program then don’t have to think about anything else. Faced with our alternative, setting up and maintaining an Autoscaling group of EC2 instances, it certainly removed a lot of the effort that can entail. However, we found that with Lambda you don’t just write the logic of single functions and then expect everything to work. You still have to think of the entire system as a whole. Rather than just thinking about the isolated logic of an individual function you have to think about how the system will work with many instances of your function running simultaneously.