#AmazonSageMaker

[govindhtech]

AWS Amplify Features For Building Scalable Full-Stack Apps

AWS Amplify features

Build

Summary

Create an app backend using Amplify Studio or Amplify CLI, then connect your app to your backend using Amplify libraries and UI elements.

Verification

With a fully-managed user directory and pre-built sign-up, sign-in, forgot password, and multi-factor auth workflows, you can create smooth onboarding processes. Additionally, Amplify offers fine-grained access management for web and mobile applications and enables login with social providers like Facebook, Google Sign-In, or Login With Amazon. Amazon Cognito is used.

Data Storage

Make use of an on-device persistent storage engine that is multi-platform (iOS, Android, React Native, and Web) and driven by GraphQL to automatically synchronize data between desktop, web, and mobile apps and the cloud. Working with distributed, cross-user data is as easy as working with local-only data thanks to DataStore’s programming style, which leverages shared and distributed data without requiring extra code for offline and online scenarios. Utilizing AWS AppSync.

Analysis

Recognize how your iOS, Android, or online consumers behave. Create unique user traits and in-app analytics, or utilize auto tracking to monitor user sessions and web page data. To increase customer uptake, engagement, and retention, gain access to a real-time data stream, analyze it for customer insights, and develop data-driven marketing plans. Amazon Kinesis and Amazon Pinpoint are the driving forces.

API

To access, modify, and aggregate data from one or more data sources, including Amazon DynamoDB, Amazon Aurora Serverless, and your own custom data sources with AWS Lambda, send secure HTTP queries to GraphQL and REST APIs. Building scalable apps that need local data access for offline situations, real-time updates, and data synchronization with configurable conflict resolution when devices are back online is made simple with Amplify. powered by Amazon API Gateway and AWS AppSync.

Functions

Using the @function directive in the Amplify CLI, you can add a Lambda function to your project that you can use as a datasource in your GraphQL API or in conjunction with a REST API. Using the CLI, you can modify the Lambda execution role policies for your function to gain access to additional resources created and managed by the CLI. You may develop, test, and deploy Lambda functions using the Amplify CLI in a variety of runtimes. After choosing a runtime, you can choose a function template for the runtime to aid in bootstrapping your Lambda function.

GEO

In just a few minutes, incorporate location-aware functionalities like maps and location search into your JavaScript online application. In addition to updating the Amplify Command Line Interface (CLI) tool with support for establishing all necessary cloud location services, Amplify Geo comes with pre-integrated map user interface (UI) components that are based on the well-known MapLibre open-source library. For greater flexibility and sophisticated visualization possibilities, you can select from a variety of community-developed MapLibre plugins or alter embedded maps to fit the theme of your app. Amazon Location Service is the driving force.

Interactions

With only one line of code, create conversational bots that are both interactive and captivating using the same deep learning capabilities that underpin Amazon Alexa. When it comes to duties like automated customer chat support, product information and recommendations, or simplifying routine job chores, chatbots can be used to create fantastic user experiences. Amazon Lex is the engine.

Forecasts

Add AI/ML features to your app to make it better. Use cases such as text translation, speech creation from text, entity recognition in images, text interpretation, and text transcription are all simply accomplished. Amplify makes it easier to orchestrate complex use cases, such as leveraging GraphQL directives to chain numerous AI/ML activities and uploading photos for automatic training. powered by Amazon Sagemaker and other Amazon Machine Learning services.

PubSub

Transmit messages between your app’s backend and instances to create dynamic, real-time experiences. Connectivity to cloud-based message-oriented middleware is made possible by Amplify. Generic MQTT Over WebSocket Providers and AWS IoT services provide the power.

Push alerts

Increase consumer interaction by utilizing analytics and marketing tools. Use consumer analytics to better categorize and target your clientele. You have the ability to customize your content and interact via a variety of channels, such as push alerts, emails, and texts. Pinpoint from Amazon powers this.

Keeping

User-generated content, including images and movies, can be safely stored on a device or in the cloud. A straightforward method for managing user material for your app in public, protected, or private storage buckets is offered by the AWS Amplify Storage module. Utilize cloud-scale storage to make the transition from prototype to production of your application simple. Amazon S3 is the power source.

Ship

Summary

Static web apps can be hosted using the Amplify GUI or CLI.

Amplify Hosting

Fullstack web apps may be deployed and hosted with AWS Amplify’s fully managed service, which includes integrated CI/CD workflows that speed up your application release cycle. A frontend developed with single page application frameworks like React, Angular, Vue, or Gatsby and a backend built with cloud resources like GraphQL or REST APIs, file and data storage, make up a fullstack serverless application. Changes to your frontend and backend are deployed in a single workflow with each code commit when you simply connect your application’s code repository in the Amplify console.

Manage and scale

Summary

To manage app users and content, use Amplify Studio.

Management of users

Authenticated users can be managed with Amplify Studio. Without going through verification procedures, create and modify users and groups, alter user properties, automatically verify signups, and more.

Management of content

Through Amplify Studio, developers may grant testers and content editors access to alter the app data. Admins can render rich text by saving material as markdown.

Override the resources that are created

Change the fine-grained backend resource settings and use CDK to override them. The heavy lifting is done for you by Amplify. Amplify, for instance, can be used to add additional Cognito resources to your backend with default settings. Use amplified override auth to override only the settings you desire.

Personalized AWS resources

In order to add custom AWS resources using CDK or CloudFormation, the Amplify CLI offers escape hatches. By using the “amplify add custom” command in your Amplify project, you can access additional Amplify-generated resources and obtain CDK or CloudFormation placeholders.

Get access to AWS resources

Infrastructure-as-Code, the foundation upon which Amplify is based, distributes resources inside your account. Use Amplify’s Function and Container support to incorporate business logic into your backend. Give your container access to an existing database or give functions access to an SNS topic so they can send an SMS.

Bring in AWS resources

With Amplify Studio, you can incorporate your current resources like your Amazon Cognito user pool and federated identities (identity pool) or storage resources like DynamoDB + S3 into an Amplify project. This will allow your storage (S3), API (GraphQL), and other resources to take advantage of your current authentication system.

Hooks for commands

Custom scripts can be executed using Command Hooks prior to, during, and following Amplify CLI actions (“amplify push,” “amplify api gql-compile,” and more). During deployment, customers can perform credential scans, initiate validation tests, and clear up build artifacts. This enables you to modify Amplify’s best-practice defaults to satisfy the operational and security requirements of your company.

Infrastructure-as-Code Export

Amplify may be integrated into your internal deployment systems or used in conjunction with your current DevOps processes and tools to enforce deployment policies. You may use CDK to export your Amplify project to your favorite toolchain by using Amplify’s export capability. The Amplify CLI build artifacts, such as CloudFormation templates, API resolver code, and client-side code generation, are exported using the “amplify export” command.

Tools

Amplify Libraries

Flutter >> JavaScript >> Swift >> Android >>

To create cloud-powered mobile and web applications, AWS Amplify provides use case-centric open source libraries. Powered by AWS services, Amplify libraries can be used with your current AWS backend or new backends made with Amplify Studio and the Amplify CLI.

Amplify UI components

An open-source UI toolkit called Amplify UI Components has cross-framework UI components that contain cloud-connected workflows. In addition to a style guide for your apps that seamlessly integrate with the cloud services you have configured, AWS Amplify offers drop-in user interface components for authentication, storage, and interactions.

The Amplify Studio

Managing app content and creating app backends are made simple with Amplify Studio. A visual interface for data modeling, authorization, authentication, and user and group management is offered by Amplify Studio. Amplify Studio produces automation templates as you develop backend resources, allowing for smooth integration with the Amplify CLI. This allows you to add more functionality to your app’s backend and establish multiple testing and team collaboration settings. You can give team members without an AWS account access to Amplify Studio so that both developers and non-developers can access the data they require to create and manage apps more effectively.

Amplify CLI toolchain

A toolset for configuring and maintaining your app’s backend from your local desktop is the Amplify Command Line Interface (CLI). Use the CLI’s interactive workflow and user-friendly use cases, such storage, API, and auth, to configure cloud capabilities. Locally test features and set up several environments. Customers can access all specified resources as infrastructure-as-code templates, which facilitates improved teamwork and simple integration with Amplify’s continuous integration and delivery process.

Amplify Hosting

Set up CI/CD on the front end and back end, host your front-end web application, build and delete backend environments, and utilize Amplify Studio to manage users and app content.

Read more on Govindhtech.com

[cizotech]

🚀 Discover the Best No-Code AI Tools for 2024! 🚀

Dive into our latest blog to explore how no-code AI tools like Akkio, ChatGPT, Canva, and more are revolutionizing the tech world. Perfect for businesses, designers, and hobbyists looking to harness the power of AI without writing a single line of code.

🔍 Find out which tool is right for you! 📈 Boost your productivity and creativity! 📱 Plus, learn how our expert team can create high-quality, user-friendly mobile apps tailored to your unique needs. Offering top-notch iOS and Android app development services.

👉 Read the full blog here: https://cizotech.com/best-no-code-ai-tools-for-2024-a-comprehensive-guide/

[vlruso]

Robust time series forecasting with MLOps on Amazon SageMaker

📢 I just published a new blog post on the importance of time series forecasting and how to achieve robust forecasts using MLOps on Amazon SageMaker! 🚀 In this post, I dive into the significance of accurate time series forecasting for effective decision-making. I also explore a robust time series forecasting model trained using Amazon SageMaker, showcasing the use of MLOps infrastructure to automate the entire model development process. From training to deploying the model, we cover it all! If you're interested in learning more about the steps involved in training and deploying the model, the solution's architecture, and the effectiveness of the Spliced Binned Pareto (SBP) distribution, this blog post is for you. 📚 Check it out here: [Robust Time Series Forecasting with MLOps on Amazon SageMaker](https://ift.tt/4Eta6jr) Let me know your thoughts and feel free to share with others interested in time series forecasting and MLOps. Happy reading! 😊 #TimeSeriesForecasting #MLOps #AmazonSageMaker #DataDrivenDecisionMaking List of Useful Links: AI Scrum Bot - ask about AI scrum and agile Our Telegram @itinai Twitter -  @itinaicom

[zoomtecnologico]

AWS anuncia nuevas capacidades para Amazon SageMaker

AWS anuncia nuevas capacidades para Amazon SageMaker

En AWS re:Invent, Amazon Web Services, anunció ocho nuevas capacidades para Amazon SageMaker, su solución integral de aprendizaje automático (ML). Los desarrolladores, científicos de datos y analistas de negocios utilizan Amazon SageMaker para crear, entrenar e implementar modelos de aprendizaje automático de forma rápida y sencilla mediante su infraestructura, herramientas y flujos de trabajo…

[aikungfu]

Amazon SageMaker is a fully managed service that provides every machine learning (ML) developer and data scientist with the ability to build, train, and deploy ML models quickly. Amazon SageMaker Studio is a web-based, integrated development environment (IDE) for ML that lets you build, train, debug, deploy, and monitor your ML models. Amazon SageMaker Studio provides all the tools you need to take your models from experimentation to production while boosting your productivity. You can write code, track experiments, visualize data, and perform debugging and monitoring within a single, integrated visual interface.

This post outlines how to configure access control for teams or groups within Amazon SageMaker Studio using attribute-based access control (ABAC). ABAC is a powerful approach that you can utilize to configure Studio so that different ML and data science teams have complete isolation of team resources.

We provide guidance on how to configure Amazon SageMaker Studio access for both AWS Identity and Access Management (IAM) and AWS Single Sign-On (AWS SSO) authentication methods. This post helps you set up IAM policies for users and roles using ABAC principals. To demonstrate the configuration, we set up two teams as shown in the following diagram and showcase two use cases:

Use case 1 – Only User A1 can access their studio environment; User A2 can’t access User A1’s environment, and vice versa

Use case 2 – Team B users cannot access artifacts (experiments, etc.) created by Team A members

You can configure policies according to your needs. You can even include a project tag in case you want to further restrict user access by projects within a team. The approach is very flexible and scalable.

Authentication

Amazon SageMaker Studio supports the following authentication methods for onboarding users. When setting up Studio, you can pick an authentication method that you use for all your users:

IAM – Includes the following:

IAM users – Users managed in IAM

AWS account federation – Users managed in an external identity provider (IdP)

AWS SSO – Users managed in an external IdP federated using AWS SSO

Data science user personas

The following table describes two different personas that interact with Amazon SageMaker Studio resources and the level of access they need to fulfill their duties. We use this table as a high-level requirement to model IAM roles and policies to establish desired controls based on resource ownership at the team and user level.

User Personas Permissions Admin User

Create, modify, delete any IAM resource.

Create Amazon SageMaker Studio user profiles with a tag.

Sign in to the Amazon SageMaker console.

Read and describe Amazon SageMaker resources.

Data Scientists or Developers

Launch an Amazon SageMaker Studio IDE assigned to a specific IAM or AWS SSO user.

Create Amazon SageMaker resources with necessary tags. For this post, we use the team tag.

Update, delete, and run resources created with a specific tag.

Sign in to the Amazon SageMaker console if an IAM user.

Read and describe Amazon SageMaker resources.

Solution overview

We use the preceding requirements to model roles and permissions required to establish controls. The following flow diagram outlines the different configuration steps:

Applying your policy to the admin user

You should apply the following policy to the admin user who creates Studio user profiles. This policy requires the admin to include the studiouserid tag. You could use a different name for the tag if need be. The Studio console doesn’t allow you to add tags when creating user profiles, so we use the AWS Command Line Interface (AWS CLI).

For admin users managed in IAM, attach the following policy to the user. For admin users managed in an external IdP, add the following policy to the rule that the user assumes upon federation. The following policy enforces the studiouserid tag to be present when the sagemaker:CreateUserProfile action is invoked.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "CreateSageMakerStudioUserProfilePolicy", "Effect": "Allow", "Action": "sagemaker:CreateUserProfile", "Resource": "*", "Condition": { "ForAnyValue:StringEquals": { "aws:TagKeys": [ "studiouserid" ] } } } ] }

AWS SSO doesn’t require this policy; it performs the identity check.

Assigning the policy to Studio users

The following policy limits Studio access to the respective users by requiring the resource tag to match the user name for the sagemaker:CreatePresignedDomainUrl action. When a user tries to access the Amazon SageMaker Studio launch URL, this check is performed.

For IAM users, attach the following policy to the user. Use the user name for the studiouserid tag value.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AmazonSageMakerPresignedUrlPolicy", "Effect": "Allow", "Action": [ "sagemaker:CreatePresignedDomainUrl" ], "Resource": "*", "Condition": { "StringEquals": { "sagemaker:ResourceTag/studiouserid": "${aws:username}" } } } ] }

For AWS account federation, attach the following policy to role that the user assumes after federation:

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AmazonSageMakerPresignedUrlPolicy", "Effect": "Allow", "Action": [ "sagemaker:CreatePresignedDomainUrl" ], "Resource": "*", "Condition": { "StringEquals": { "sagemaker:ResourceTag/studiouserid": "${aws:PrincipalTag/studiouserid}" } } } ] }

Add the following statement to this policy in the Trust Relationship section. This statement defines the allowed transitive tag.

"Statement": [ { --Existing statements }, { "Sid": "IdentifyTransitiveTags", "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::<account id>:saml-provider/<identity provider>" }, "Action": "sts:TagSession", "Condition": { "ForAllValues:StringEquals": { "sts:TransitiveTagKeys": [ "studiouserid" ] } } ]

For users managed in AWS SSO, this policy is not required. AWS SSO performs the identity check.

Creating roles for the teams

To create roles for your teams, you must first create the policies. For simplicity, we use the same policies for both teams. In most cases, you just need one set of policies for all teams, but you have the flexibility to create different policies for different teams. In the second step, you create a role for each team, attach the policies, and tag the roles with appropriate team tags.

Creating the policies

Create the following policies. For this post, we split them into three policies for more readability, but you can create them according to your needs.

Policy 1: Amazon SageMaker read-only access

The following policy gives privileges to List and Describe Amazon SageMaker resources. You can customize this policy according to your needs.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AmazonSageMakerDescribeReadyOnlyPolicy", "Effect": "Allow", "Action": [ "sagemaker:Describe*", "sagemaker:GetSearchSuggestions" ], "Resource": "*" }, { "Sid": "AmazonSageMakerListOnlyPolicy", "Effect": "Allow", "Action": [ "sagemaker:List*" ], "Resource": "*" }, { "Sid": "AmazonSageMakerUIandMetricsOnlyPolicy", "Effect": "Allow", "Action": [ "sagemaker:*App", "sagemaker:Search", "sagemaker:RenderUiTemplate", "sagemaker:BatchGetMetrics" ], "Resource": "*" }, { "Sid": "AmazonSageMakerEC2ReadOnlyPolicy", "Effect": "Allow", "Action": [ "ec2:DescribeDhcpOptions", "ec2:DescribeNetworkInterfaces", "ec2:DescribeRouteTables", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcEndpoints", "ec2:DescribeVpcs" ], "Resource": "*" }, { "Sid": "AmazonSageMakerIAMReadOnlyPolicy", "Effect": "Allow", "Action": [ "iam:ListRoles" ], "Resource": "*" } ] }

Policy 2: Amazon SageMaker access for supporting services

The following policy gives privileges to create, read, update, and delete access to Amazon Simple Storage Service (Amazon S3), Amazon Elastic Container Registry (Amazon ECR), and Amazon CloudWatch, and read access to AWS Key Management Service (AWS KMS). You can customize this policy according to your needs.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AmazonSageMakerCRUDAccessS3Policy", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:CreateBucket", "s3:ListBucket", "s3:PutBucketCORS", "s3:ListAllMyBuckets", "s3:GetBucketCORS", "s3:GetBucketLocation" ], "Resource": "<S3 BucketName>" }, { "Sid": "AmazonSageMakerReadOnlyAccessKMSPolicy", "Effect": "Allow", "Action": [ "kms:DescribeKey", "kms:ListAliases" ], "Resource": "*" }, { "Sid": "AmazonSageMakerCRUDAccessECRPolicy", "Effect": "Allow", "Action": [ "ecr:Set*", "ecr:CompleteLayerUpload", "ecr:Batch*", "ecr:Upload*", "ecr:InitiateLayerUpload", "ecr:Put*", "ecr:Describe*", "ecr:CreateRepository", "ecr:Get*", "ecr:StartImageScan" ], "Resource": "*" }, { "Sid": "AmazonSageMakerCRUDAccessCloudWatchPolicy", "Effect": "Allow", "Action": [ "cloudwatch:Put*", "cloudwatch:Get*", "cloudwatch:List*", "cloudwatch:DescribeAlarms", "logs:Put*", "logs:Get*", "logs:List*", "logs:CreateLogGroup", "logs:CreateLogStream", "logs:ListLogDeliveries", "logs:Describe*", "logs:CreateLogDelivery", "logs:PutResourcePolicy", "logs:UpdateLogDelivery" ], "Resource": "*" } ] }

Policy 3: Amazon SageMaker Studio developer access

The following policy gives privileges to create, update, and delete Amazon SageMaker Studio resources. It also enforces the team tag requirement during creation. In addition, it enforces start, stop, update, and delete actions on resources restricted only to the respective team members.

The team tag validation condition in the following code makes sure that the team tag value matches the principal’s team. Refer to the bolded code for specifcs.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "AmazonSageMakerStudioCreateApp", "Effect": "Allow", "Action": [ "sagemaker:CreateApp" ], "Resource": "*" }, { "Sid": "AmazonSageMakerStudioIAMPassRole", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": "*" }, { "Sid": "AmazonSageMakerInvokeEndPointRole", "Effect": "Allow", "Action": [ "sagemaker:InvokeEndpoint" ], "Resource": "*" }, { "Sid": "AmazonSageMakerAddTags", "Effect": "Allow", "Action": [ "sagemaker:AddTags" ], "Resource": "*" }, { "Sid": "AmazonSageMakerCreate", "Effect": "Allow", "Action": [ "sagemaker:Create*" ], "Resource": "*", "Condition": { "ForAnyValue:StringEquals": { "aws:TagKeys": [ "team" ] }, "StringEqualsIfExists": { "aws:RequestTag/team": "${aws:PrincipalTag/team}" } } }, { "Sid": "AmazonSageMakerUpdateDeleteExecutePolicy", "Effect": "Allow", "Action": [ "sagemaker:Delete*", "sagemaker:Stop*", "sagemaker:Update*", "sagemaker:Start*", "sagemaker:DisassociateTrialComponent", "sagemaker:AssociateTrialComponent", "sagemaker:BatchPutMetrics" ], "Resource": "*", "Condition": { "StringEquals": { "aws:PrincipalTag/team": "${sagemaker:ResourceTag/team}" } } } ] }

Creating and configuring the roles

You can now create a role for each team with these policies. Tag the roles on the IAM console or with the CLI command. The steps are the same for all three authentication types. For example, tag the role for Team A with the tag key= team and value = “<Team Name>”.

Creating the Amazon SageMaker Studio user profile

In this step, we add the studiouserid tag when creating Studio user profiles. The steps are slightly different for each authentication type.

IAM users

For IAM users, you create Studio user profiles for each user by including the role that was created for the team the user belongs to. The following code is a sample CLI command. As of this writing, including a tag when creating a user profile is available only through AWS CLI.

aws sagemaker create-user-profile --domain-id <domain id> --user-profile-name <unique profile name> --tags Key=studiouserid,Value=<aws user name> --user-settings ExecutionRole=arn:aws:iam::<account id>:role/<Team Role Name>

AWS account federation

For AWS account federation, you create a user attribute (studiouserid) in an external IdP with a unique value for each user. The following code shows how to configure the attribute in Okta:

Example below shows how to add “studiouserid” attribute in OKTA. In OKTA’s SIGN ON METHODS screen, configure following SAML 2.0 attributes, as shown in the image below. Attribute 1: Name: https://aws.amazon.com/SAML/Attributes/PrincipalTag:studiouserid Value: user.studiouserid Attribute 2: Name: https://aws.amazon.com/SAML/Attributes/TransitiveTagKeys Value: {"studiouserid"}

The following screenshot shows the attributes on the Okta console.

Next, create the user profile using the following command. Use the user attribute value in the preceding step for the studiouserid tag value.

aws sagemaker create-user-profile --domain-id <domain id> --user-profile-name <unique profile name> --tags Key=studiouserid,Value=<user attribute value> --user-settings ExecutionRole=arn:aws:iam::<account id>:role/<Team Role Name>

AWS SSO

For instructions on assigning users in AWS SSO, see Onboarding Amazon SageMaker Studio with AWS SSO and Okta Universal Directory.

Update the Studio user profile to include the appropriate execution role that was created for the team that the user belongs to. See the following CLI command:

aws sagemaker update-user-profile --domain-id <domain id> --user-profile-name <user profile name> --user-settings ExecutionRole=arn:aws:iam::<account id>:role/<Team Role Name> --region us-west-2

Validating that only assigned Studio users can access their profiles

When a user tries to access a Studio profile that doesn’t have studiouserid tag value matching their user name, an AccessDeniedException error occurs. You can test this by copying the link for Launch Studio on the Amazon SageMaker console and accessing it when logged in as a different user. The following screenshot shows the error message.

Validating that only respective team members can access certain artifacts

In this step, we show how to configure Studio so that members of a given team can’t access artifacts that another team creates.

In our use case, a Team A user creates an experiment and tags that experiment with the team tag. This limits access to this experiment to Team A users only. See the following code:

import sys !{sys.executable} -m pip install sagemaker !{sys.executable} -m pip install sagemaker-experiments import time import sagemaker from smexperiments.experiment import Experiment demo_experiment = Experiment.create(experiment_name = "USERA1TEAMAEXPERIMENT1", description = "UserA1 experiment", tags = [{'Key': 'team', 'Value': 'TeamA'}])

If a user who is not in Team A tries to delete the experiment, Studio denies the delete action. See the following code:

#command run from TeamB User Studio Instance import time from smexperiments.experiment import Experiment experiment_to_cleanup = Experiment.load(experiment_name="USERA1TEAMAEXPERIMENT1") experiment_to_cleanup.delete() [Client Error] An error occurred (AccessDeniedException) when calling the DeleteExperiment operation: User: arn:aws:sts:: :<AWS Account ID>::assumed-role/ SageMakerStudioDeveloperTeamBRole/SageMaker is not authorized to perform: sagemaker:DeleteExperiment on resource: arn:aws:sagemaker:us-east-1:<AWS Account ID>:experiment/usera1teamaexperiment1

Conclusion

In this post, we demonstrated how to isolate Amazon SageMaker Studio access using the ABAC technique. We showcased two use cases: restricting access to a Studio profile to only the assigned user (using the studiouserid tag) and restricting access to Studio artifacts to team members only. We also showed how to limit access to experiments to only the members of the team using the team tag. You can further customize policies by applying more tags to create more complex hierarchical controls.

Try out this solution for isolating resources by teams or groups in Amazon SageMaker Studio. For more information about using ABAC as an authorization strategy, see What is ABAC for AWS?

About the Authors

Vikrant Kahlir is Senior Solutions Architect in the Solutions Architecture team. He works with AWS strategic customers product and engineering teams to help them with technology solutions using AWS services for Managed Databases, AI/ML, HPC, Autonomous Computing, and IoT.

      Rakesh Ramadas is an ISV Solution Architect at Amazon Web Services. His focus areas include AI/ML and Big Data.

        Rama Thamman is a Software Development Manager with the AI Platforms team, leading the ML Migrations team.

from AWS Machine Learning Blog https://ift.tt/38vljnT via A.I .Kung Fu

[phungthaihy]

Threat Stack: Proactive Risk Identification and Real-time Threat Detection across AWS http://ehelpdesk.tk/wp-content/uploads/2020/02/logo-header.png [ad_1] In this video, we outline an ing... #amazonec2 #amazonemr #amazons3 #amazonsagemaker #amazonwebservices #aws #awscertification #awscertifiedcloudpractitioner #awscertifieddeveloper #awscertifiedsolutionsarchitect #awscertifiedsysopsadministrator #awscloud #awscloudtrail #ciscoccna #cloud #cloudcomputing #comptiaa #comptianetwork #comptiasecurity #cybersecurity #ethicalhacking #it #kubernetes #linux #microsoftaz-900 #microsoftazure #networksecurity #software #windowsserver

[govindhtech]

CellProfiler And Cell Painting Batch(CPB) In Drug Discovery

Cell Painting

Enhancing Drug Discovery with high-throughput AWS Cell Painting. Are you having trouble processing cell images? Let’s see how AWS’s Cell Painting Batch offering has revolutionized cell analysis for life sciences clients.

Introduction

The analysis of microscope-captured cell pictures is a key component in the area of drug development. To comprehend cellular activities and phenotypes, a novel method for high-content screening called “Cell Painting” has surfaced. Prominent biopharma businesses have begun using technologies like the Broad Institute’s CellProfiler software, which is designed for cell profiling.

On the other hand, a variety of imaging methods and exponential data expansion provide formidable obstacles. Here, they will discover how AWS has been used by life sciences clients to create a distributed, scalable, and effective cell analysis system.

Current Circumstance

Scalable processing and storage are needed for cell painting operations in order to support big file sizes and high-throughput picture analysis. These days, scientists employ open-source tools such as CellProfiler, but to run automated pipelines without worrying about infrastructure maintenance, they need scalable infrastructure.

In addition to handling massive amounts of data on microscopic images and infrastructure provisioning, scientists are attempting to conduct scientific research. It is necessary for researchers to work together safely and productively across labs using user-friendly tools. The cornerstone of research is scientific reproducibility, which requires scientists to duplicate other people’s discoveries when publishing in highly regarded publications or even examining data from their own labs.

CellProfiler software

Obstacles

Customers in the life sciences sector encountered the following difficulties while using stand-alone instances of technologies such as CellProfiler software:

Difficulties in adjusting to workload fluctuations.

Problems with productivity in intricate, time-consuming tasks.

Problems in teamwork across teams located around the company.

Battles to fulfill the need for activities requiring a lot of computing.

Cluster capacity problems often result in unfinished work, delays, and inefficiencies.

The lack of a centralized data hub results in problems with data access.

Cellprofiler Pipeline

Solution Overview

AWS solution architects collaborated with life sciences clients to create a novel solution known as Cell Painting Batch (CPB) in order to solve these issues. CellProfiler Pipelines are operated on AWS in a scalable and distributed architecture by CPB using the Broad Institute’s CellProfiler image. With CPB, researchers may analyze massive amounts of images without having to worry about the intricate details of infrastructure management. Furthermore, the AWS Cloud Development Kit (CDK), which simplifies infrastructure deployment and management, is used in the construction of the CPB solution.

The whole procedure is automated; upon uploading a picture, an Amazon Simple Queue Service (SQS) message is issued that starts the image processing and ends with the storing of the results. This gives researchers a scalable, automated, and effective way to handle large-scale image processing requirements.Image credit to AWS

This figure illustrates how to dump photos from microscopes into an Amazon S3 bucket. AWS Lambda is triggered by user SQS messages. Lambda submits AWS Batch tasks utilizing container images from the Amazon Elastic Container Registry. Photos are processed using AWS Batch, and the results are sent to Amazon S3 for analysis.

Workflow

The goal of the Cell Painting Batch (CPB) solution on AWS is to simplify the intricate process of processing cell images so that researchers may concentrate on what really counts extrapolating meaning from the data. This is a detailed explanation of how the CPB solution works:

Images are obtained by researchers using microscopes or other means.

Then, in order to serve as an image repository, these photos are uploaded to a specific Amazon Simple Storage Service (S3) bucket.

After storing the photos, researchers send a message to Amazon Simple Queue Service (SQS) specifying the location of the images and the CellProfiler pipeline they want to use. In essence, this message is a request for image processing that is delivered to the SQS service.

An automated AWS Lambda function is launched upon receiving a SQS message. The main responsibility of this function is to start the AWS Batch job for the specific image processing request.

Amazon Batch assesses the needs of the task. AWS Batch dynamically provisioned the required Amazon Elastic Compute Cloud (EC2) instances based on the job.

It retrieves the designated container image that is kept in the Amazon Elastic Container Registry (ECR). This container runs the specified CellProfiler pipeline inside AWS Batch. The integration of Amazon FSx for Lustre with the S3 bucket guarantees that containers may access data quickly.

The picture is processed by the CellProfiler program within the container using a predetermined pipeline. This may include doing image processing operations such as feature extraction and segmentation.

Following CellProfiler post-processing, the outcomes are stored once again to the assigned S3 bucket at the address mentioned in the SQS message.

Scholars use the S3 bucket to get and examine data for their investigations.

Image Credit To AWS

Because the workflow is automated, the solution will begin analyzing images and storing the findings as soon as a picture is uploaded and a SQS message is issued. This gives researchers a scalable, automated, and effective way to handle large-scale image processing requirements.

Safety

For cell painting datasets and workflows, AWS’s Cell Painting Batch (CPB) provides a strong security architecture. The solution offers top-notch data safety with encrypted data storage at rest and in transit, controlled access via AWS Identity and Access Management (IAM), and improved network security via an isolated VPC. Furthermore, the security posture is strengthened by ongoing monitoring using security technologies like Amazon Cloud Watch.

It is advisable to implement additional mitigations, such as version control for system configurations, strong authentication with multi-factor authentication (MFA), protection against excessive resource usage with Amazon Cloud Watch and AWS Service Quotas, cost monitoring with AWS Budgets, and container scanning with Amazon Inspector, in order to further strengthen security.

Life Sciences Customer Success Stories

Customers in the biological sciences have changed drastically after switching to CPB. This streamlined processing pipelines, sped up photo processing, and fostered collaboration. The system’s built-in scalability can manage larger datasets to hasten medication development, making these enterprises future-proof.

Customizing the Solution

CPB may be integrated with other AWS services due to its modularity. Options include AWS Step Functions for efficient process orchestration, Amazon AppStream for browser-based access to scientific equipment, AWS Service Catalog for self-service installs, and Amazon SageMaker for machine learning workloads. Github code has a parameters file for instance class, timeout duration, and other tweaks.

In summary

The cell painting batch approach may boost researcher productivity by eliminating infrastructure management. This method allows scalable and fast image analysis, speeding therapy development. It also lets researchers self-manage processing and distribution, reducing infrastructure administration needs.

The AWS CPB solution has transformed biopharmaceutical cell image processing and helped life sciences companies. A unique approach that combines scalability, automation, and efficiency allows life sciences organizations to easily handle large cell imaging workloads and accelerate drug development.

Read more on Govindhtech.com

[govindhtech]

How BT Group’s GenAI Gateway Advancing AI With AWS

The “GenAI Gateway” platform, powered by AWS, is launched by the Digital Unit of BT Group, hastening the company’s safe and widespread deployment of generative AI. Built in partnership with AWS, the GenAI Gateway is a generative AI enablement platform that gives BT Group safe and secure access to Large Language Models (LLMs) from a scalable number of foundation model suppliers, allowing them to harness the potential of GenAI.

The platform facilitates timely security, central privacy controls, use case-based charging, enterprise search, and the use of diverse corporate data sources, allowing the organization to exercise flexibility and accountability when implementing the many AI models that are necessary. Model exploitation risk is reduced by incorporated safeguards that promote ethics and performance (also known as “jailbreak risk”).

BT Group

As BT Group expands and speeds up the use of generative AI, a single, unified Group platform minimizes duplication of work and resources since APIs, security settings, infrastructure management, etc., can be controlled centrally, lowering the risk of mistake along the way.

The Digital Unit of BT Group has declared the opening of a cutting-edge internal platform that would enable the organization to use large language models (LLMs) from suppliers including Anthropic, Meta, Claude, Cohere, and Amazon. The GenAI Gateway is a vital tool that BT Group will use as it integrates AI into its operations. It was developed in partnership with AWS and makes use of AWS Professional Services, Amazon Bedrock, and Amazon SageMaker to provide secure, private access to a variety of natural-language processing and large language models.

While ad hoc use of LLMs is acceptable for testing and development, it is not a good fit for large-scale implementation; greater attention to cost control, security, and privacy is required. Additionally, LLM performance must be watched for unanticipated mistakes (such “hallucinations”) and model degradation over time (the point at which LLMs cease acting in a predictable manner). In the event that further problems arise, the GenAI Gateway also protects BT Group from being “locked in” to a certain LLM. Because the GenAI Gateway platform allows for per-use case budget monitoring, it will incentivize BT Group engineers to choose the appropriate model for the given use case at the most competitive price.

By consolidating platforms, BT Group can save redundant effort and resources while expanding the use of generative AI. Centralized administration of infrastructure, security configuration, and application programming interfaces (APIs) lowers the chance of mistake and the expense of keeping different LLMs for each use case.

The AWS-deployed GenAI Gateway, like every element of BT Group’s modular digital architecture, can only be accessed via secure APIs.

GenAI Gateway leverages two fully managed services

Amazon Bedrock, which provides a single API access to a selection of high-performing foundation models from top AI firms such as AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon; and Amazon SageMaker, which combines a wide range of tools to enable high-performance, low-cost machine learning for any use case.

Enterprise search, chat history, FinOps charging by use case, fast security, and the use of many corporate data sources are all supported by the platform. Central privacy measures secure data in accordance with Group policy and applicable regulations. These include distinct tenants for each use case, the use of Personal Identifiable Information filters, the placement of the data inside the UK, and the isolation of trained models from one another.

The GenAI Gateway has built-in guardrails that reduce the possibility of jailbreaks and harmful interactions. These guardrails block away queries that are not relevant to a particular application, ensuring that ethical and performance constraints are included from the beginning.

It will also use the “data fabric” data management platform to help enforce governing policies for how data can be used, as well as to manage access control and data sovereignty restrictions. GenAI Gateway is one of several key enablers AWS are deploying to enable BT Group as an AI-enabled enterprise.

The first beta use cases for GenAI Gateway are available right now. A pilot being conducted by Openreach aims to streamline procedures and increase productivity for its teams and clients of communications providers by compiling technical notes on Ethernet and full fiber installations. There is also a live second use case that supports contract analysis for the business, legal, and procurement departments of the group.

Businesses can employ generative AI rapidly and effectively at scale with the BT Group GenAI Gateway. Collaborating and working backwards from the client to create a mechanism to speed the deployment of generative AI use cases into production with integrated security and compliance has been a fantastic, innovative opportunity. The generative AI adoption flywheel effect will be sparked by the GenAI Gateway, giving BT Group and its clients faster outcomes.

AI is assisting it in rethinking the company’s future. So think that in cases when your data is consistent, the LLMs must be flexible. People can now access this formidable new suite of technologies at scale in a secure, responsible, adaptable, and scalable manner with GenAI Gateway, fulfilling the objective of using AI to unleash the potential of every person inside the BT Group both now and in the future.

Read more on Govindhtech.com

[govindhtech]

Redshift Amazon With RDS For MySQL zero-ETL Integrations

With the now broadly available Amazon RDS for MySQL zero-ETL interface with Amazon Redshift, near real-time analytics are possible.

For comprehensive insights and the dismantling of data silos, zero-ETL integrations assist in integrating your data across applications and data sources. Petabytes of transactional data may be made accessible in Redshift Amazon in only a few seconds after being written into Amazon Relational Database Service (Amazon RDS) for MySQL thanks to their completely managed, no-code, almost real-time solution.

- Advertisement -

As a result, you may simplify data input, cut down on operational overhead, and perhaps even decrease your total data processing expenses by doing away with the requirement to develop your own ETL tasks. They revealed last year that Amazon DynamoDB, RDS for MySQL, and Aurora PostgreSQL-Compatible Edition were all available in preview as well as the general availability of zero-ETL connectivity with Redshift Amazon for Amazon Aurora MySQL-Compatible Edition.

With great pleasure, AWS announces the general availability of Amazon RDS for MySQL zero-ETL with Redshift Amazon. Additional new features in this edition include the option to setup zero-ETL integrations in your AWS Cloud Formation template, support for multiple integrations, and data filtering.

Data filtration

The majority of businesses, regardless of size, may gain from include filtering in their ETL tasks. Reducing data processing and storage expenses by choosing just the portion of data required for replication from production databases is a common use case. Eliminating personally identifiable information (PII) from the dataset of a report is an additional step. For instance, when duplicating data to create aggregate reports on recent patient instances, a healthcare firm may choose to exclude sensitive patient details.

In a similar vein, an online retailer would choose to provide its marketing division access to consumer buying trends while keeping all personally identifiable information private. On the other hand, there are other situations in which you would not want to employ filtering, as when providing data to fraud detection teams who need all of the data in almost real time in order to draw conclusions. These are just a few instances; We urge you to explore and find more use cases that might be relevant to your company.

- Advertisement -

Zero-ETL Integration

You may add filtering to your zero-ETL integrations in two different ways: either when you construct the integration from scratch, or when you alter an already-existing integration. In any case, this option may be found on the zero-ETL creation wizard’s “Source” stage.

Entering filter expressions in the format database.table allows you to apply filters that include or exclude databases or tables from the dataset. Multiple expressions may be added, and they will be evaluated left to right in sequence.

If you’re changing an existing integration, Redshift Amazon will remove tables that are no longer included in the filter and the new filtering rules will take effect once you confirm your modifications.

Since the procedures and ideas are fairly similar, we suggest reading this blog article if you want to dig further. It goes into great detail on how to set up data filters for Amazon Aurora zero-ETL integrations.

Amazon Redshift Data Warehouse

From a single database, create several zero-ETL integrations

Additionally, you can now set up connectors to up to five Redshift Amazon data warehouses from a single RDS for MySQL database. The only restriction is that you can’t add other integrations until the first one has successfully completed its setup.

This enables you to give other teams autonomy over their own data warehouses for their particular use cases while sharing transactional data with them. For instance, you may use this in combination with data filtering to distribute distinct data sets from the same Amazon RDS production database to development, staging, and production Redshift Amazon clusters.

One further intriguing use case for this would be the consolidation of Redshift Amazon clusters via zero-ETL replication to several warehouses. Additionally, you may exchange data, train tasks in Amazon SageMaker, examine your data, and power your dashboards using Amazon Redshift materialized views.

In summary

You may duplicate data for near real-time analytics with RDS for MySQL zero-ETL connectors with Redshift Amazon, eliminating the need to create and maintain intricate data pipelines. With the ability to implement filter expressions to include or exclude databases and tables from the duplicated data sets, it is already widely accessible. Additionally, you may now construct connections from many sources to combine data into a single data warehouse, or set up numerous connectors from the same source RDS for MySQL database to distinct Amazon Redshift warehouses.

In supported AWS Regions, this zero-ETL integration is available for Redshift Amazon Serverless, Redshift Amazon RA3 instance types, and RDS for MySQL versions 8.0.32 and later.

Not only can you set up a zero-ETL connection using the AWS Management Console, but you can also do it with the AWS Command Line Interface (AWS CLI) and an official AWS SDK for Python called boto3.

Read more on govindhtech.com

[govindhtech]

Amazon SageMaker HyperPod Presents Amazon EKS Support

Amazon SageMaker HyperPod

Cut the training duration of foundation models by up to 40% and scale effectively across over a thousand AI accelerators.

We are happy to inform you today that Amazon SageMaker HyperPod, a specially designed infrastructure with robustness at its core, will enable Amazon Elastic Kubernetes Service (EKS) for foundation model (FM) development. With this new feature, users can use EKS to orchestrate HyperPod clusters, combining the strength of Kubernetes with the robust environment of Amazon SageMaker HyperPod, which is ideal for training big models. By effectively scaling across over a thousand artificial intelligence (AI) accelerators, Amazon SageMaker HyperPod can save up to 40% of training time.

- Advertisement -

SageMaker HyperPod: What is it?

The undifferentiated heavy lifting associated with developing and refining machine learning (ML) infrastructure is eliminated by Amazon SageMaker HyperPod. Workloads can be executed in parallel for better model performance because it is pre-configured with SageMaker’s distributed training libraries, which automatically divide training workloads over more than a thousand AI accelerators. SageMaker HyperPod occasionally saves checkpoints to guarantee your FM training continues uninterrupted.

You no longer need to actively oversee this process because it automatically recognizes hardware failure when it occurs, fixes or replaces the problematic instance, and continues training from the most recent checkpoint that was saved. Up to 40% less training time is required thanks to the robust environment, which enables you to train models in a distributed context without interruption for weeks or months at a time. The high degree of customization offered by SageMaker HyperPod enables you to share compute capacity amongst various workloads, from large-scale training to inference, and to run and scale FM tasks effectively.

Advantages of the Amazon SageMaker HyperPod

Distributed training with a focus on efficiency for big training clusters

Because Amazon SageMaker HyperPod comes preconfigured with Amazon SageMaker distributed training libraries, you can expand training workloads more effectively by automatically dividing your models and training datasets across AWS cluster instances.

Optimum use of the cluster’s memory, processing power, and networking infrastructure

Using two strategies, data parallelism and model parallelism, Amazon SageMaker distributed training library optimizes your training task for AWS network architecture and cluster topology. Model parallelism divides models that are too big to fit on one GPU into smaller pieces, which are then divided among several GPUs for training. To increase training speed, data parallelism divides huge datasets into smaller ones for concurrent training.

- Advertisement -

Robust training environment with no disruptions

You can train FMs continuously for months on end with SageMaker HyperPod because it automatically detects, diagnoses, and recovers from problems, creating a more resilient training environment.

Customers may now use a Kubernetes-based interface to manage their clusters using Amazon SageMaker HyperPod. This connection makes it possible to switch between Slurm and Amazon EKS with ease in order to optimize different workloads, including as inference, experimentation, training, and fine-tuning. Comprehensive monitoring capabilities are provided by the CloudWatch Observability EKS add-on, which offers insights into low-level node metrics on a single dashboard, including CPU, network, disk, and other. This improved observability includes data on container-specific use, node-level metrics, pod-level performance, and resource utilization for the entire cluster, which makes troubleshooting and optimization more effective.

Since its launch at re:Invent 2023, Amazon SageMaker HyperPod has established itself as the go-to option for businesses and startups using AI to effectively train and implement large-scale models. The distributed training libraries from SageMaker, which include Model Parallel and Data Parallel software optimizations to assist cut training time by up to 20%, are compatible with it. With SageMaker HyperPod, data scientists may train models for weeks or months at a time without interruption since it automatically identifies, fixes, or replaces malfunctioning instances. This frees up data scientists to concentrate on developing models instead of overseeing infrastructure.

Because of its scalability and abundance of open-source tooling, Kubernetes has gained popularity for machine learning (ML) workloads. These benefits are leveraged in the integration of Amazon EKS with Amazon SageMaker HyperPod. When developing applications including those needed for generative AI use cases organizations frequently rely on Kubernetes because it enables the reuse of capabilities across environments while adhering to compliance and governance norms. Customers may now scale and maximize resource utilization across over a thousand AI accelerators thanks to today’s news. This flexibility improves the workflows for FM training and inference, containerized app management, and developers.

With comprehensive health checks, automated node recovery, and work auto-resume features, Amazon EKS support in Amazon SageMaker HyperPod fortifies resilience and guarantees continuous training for big-ticket and/or protracted jobs. Although clients can use their own CLI tools, the optional HyperPod CLI, built for Kubernetes settings, can streamline job administration. Advanced observability is made possible by integration with Amazon CloudWatch Container Insights, which offers more in-depth information on the health, utilization, and performance of clusters. Furthermore, data scientists can automate machine learning operations with platforms like Kubeflow. A reliable solution for experiment monitoring and model maintenance is offered by the integration, which also incorporates Amazon SageMaker managed MLflow.

In summary, the HyperPod service fully manages the HyperPod service-generated Amazon SageMaker HyperPod cluster, eliminating the need for undifferentiated heavy lifting in the process of constructing and optimizing machine learning infrastructure. This cluster is built by the cloud admin via the HyperPod cluster API. These HyperPod nodes are orchestrated by Amazon EKS in a manner akin to that of Slurm, giving users a recognizable Kubernetes-based administrator experience.

Important information

The following are some essential details regarding Amazon EKS support in the Amazon SageMaker HyperPod:

Resilient Environment: With comprehensive health checks, automated node recovery, and work auto-resume, this integration offers a more resilient training environment. With SageMaker HyperPod, you may train foundation models continuously for weeks or months at a time without interruption since it automatically finds, diagnoses, and fixes errors. This can result in a 40% reduction in training time.

Improved GPU Observability: Your containerized apps and microservices can benefit from comprehensive metrics and logs from Amazon CloudWatch Container Insights. This makes it possible to monitor cluster health and performance in great detail.

Scientist-Friendly Tool: This release includes interaction with SageMaker Managed MLflow for experiment tracking, a customized HyperPod CLI for job management, Kubeflow Training Operators for distributed training, and Kueue for scheduling. Additionally, it is compatible with the distributed training libraries offered by SageMaker, which offer data parallel and model parallel optimizations to drastically cut down on training time. Large model training is made effective and continuous by these libraries and auto-resumption of jobs.

Flexible Resource Utilization: This integration improves the scalability of FM workloads and the developer experience. Computational resources can be effectively shared by data scientists for both training and inference operations. You can use your own tools for job submission, queuing, and monitoring, and you can use your current Amazon EKS clusters or build new ones and tie them to HyperPod compute.

Read more on govindhtech.com

[phungthaihy]

SyntheticGestalt is Accelerating Scientific Research Using Machine Learning http://ehelpdesk.tk/wp-content/uploads/2020/02/logo-header.png [ad_1] Designing a new drug is an extre... #amazonsagemaker #amazonstagemaker #amazonwebservices #aws #awscertification #awscertifiedcloudpractitioner #awscertifieddeveloper #awscertifiedsolutionsarchitect #awscertifiedsysopsadministrator #awscloud #ciscoccna #cloud #cloudcomputing #comptiaa #comptianetwork #comptiasecurity #cybersecurity #drugdiscovery #drugresearch #ethicalhacking #it #kubernetes #linux #machinelearning #microsoftaz-900 #microsoftazure #networksecurity #reinforcementlearning #software #windowsserver