#EMRnotebooks

[govindhtech]

EMR Notebooks Security Within AWS Dashboard & EMR Studio

Security for EMR Notebooks

Recent Amazon EMR documentation highlights numerous built-in options to increase EMR Notebook security that are now available in the AWS dashboard as EMR Studio Workspaces. These capabilities are aimed to give users precise control so that only authorised users may access and interact with these notebooks and, most crucially, use the notebook editor to run code on linked clusters.

The security measures for Amazon EMR and its clusters complement those for EMR Notebooks. Tiered security allows for additional thoroughness. Many important processes for restricting access and securing notebook environments are mentioned in the documentation:

AWS IAM Integration: Integrated Identity and Access Management is crucial. Use IAM policy statements. In these policies, AWS defines permissions, including who can access what resources and do what. The documentation suggests using policy statements with notebook tags to restrict access.

This solution lets you tag EMR notebooks with key-value labels and build IAM policies that allow or deny access based on these tags. These extracts do not include the tagging methods, however this allows more granular control than providing access to all notebooks. Certain projects, teams, or data sensitivity levels may allow access control.

Amazon EC2 security groups are highlighted. They function as virtual firewalls. They control network traffic between the notebook editor and the cluster's primary instance in EMR Notebooks.

This basic network security solution restricts network connectivity between the real computing resources (the principal instance of the EMR cluster), where code execution begins, and the notebook environment, where the user interacts. According to the documentation, customers can adjust EMR Notebook security groups to meet their network isolation needs or use the default settings. EMR Notebook EC2 security group configuration instructions are available.

An AWS Service Role is utilised for setup. Highlights your responsibility to define this job. This Service Role is necessary to grant EMR notebooks authorisation to communicate with other AWS services. This Service Role allows notebook code to interface with databases, access S3 data, and call other AWS APIs.

The least privilege principle requires that a position only have the access needed to complete their tasks.

AWS console access requires additional permissions to access EMR Notebooks. Console users can access EMR Notebooks as EMR Studio Workspaces. You require extra IAM role rights to access or create these Workspaces. Use of the “Create Workspace” button requires this. This adds access control to the console interface, unlike the notebook's execution permissions or Service Role for communicating with other services. It indicates that basic EMR console rights and console access to EMR Studio Workspaces are covered elsewhere.

Together, EC2 security groups act as virtual firewalls to regulate network traffic, IAM policies with notebook tags limit access, a specific AWS Service Role defines interaction permissions with other services, and additional IAM permissions for console access to EMR Studio Workspaces allow administrators to customise the security posture of their EMR Notebook environments.

These rules restrict network connections and cross-service rights for notebook operations and ensure that only authorised users can work with notebooks and run programs. According to the documentation, these functionalities complement the Amazon EMR security architecture by providing a multidimensional approach to notebook-based data processing workflow security.

[govindhtech]

What Are The Programmatic Commands For EMR Notebooks?

EMR Notebook Programming Commands

Programmatic Amazon EMR Notebook interaction.

How to leverage execution APIs from a script or command line to control EMR notebook executions outside the AWS UI. This lets you list, characterise, halt, and start EMR notebook executions.

The following examples demonstrate these abilities:

AWS CLI: Amazon EMR clusters on Amazon EC2 and EMR Notebooks clusters (EMR on EKS) with notebooks in EMR Studio Workspaces are shown. An Amazon S3 location-based notebook execution sample is also provided. The displayed instructions can list executions by start time or start time and status, halt an ongoing execution, and describe a notebook execution.

Boto3 SDK (Python): Demo.py uses boto3 to interface with EMR notebook execution APIs. The script explains how to initiate a notebook execution, get the execution ID, describe it, list all running instances, and stop it after a short pause. Status updates and execution IDs are shown in this script's output.

Ruby SDK: Sample Ruby code shows notebook execution API calls and Amazon EMR connection setup. Example: describe execution, print information, halt notebook execution, start notebook execution, and get execution ID. Predicted Ruby notebook run outcomes are also shown.

Programmatic command parameters

Important parameters in these programming instructions are:

EditorId: EMR Studio workspace.

relative-path or RelativePath: The notebook file's path to the workspace's home directory. Pathways include my_folder/python3.ipynb and demo_pyspark.ipynb.

execution-engine or ExecutionEngine: EMR cluster ID (j-1234ABCD123) or EMR on EKS endpoint ARN and type to choose engine.

The IAM service role, such as EMR_Notebooks_DefaultRole, is defined.

notebook-params or notebook_params: Allows a notebook to receive multiple parameter values, eliminating the need for multiple copies. Typically, parameters are JSON strings.

The input notebook file's S3 bucket and key are supplied.

The S3 bucket and key where the output notebook will be stored.

notebook-execution-name: Names the performance.

This identifies an execution when describing, halting, or listing.

–from and –status: Status and start time filters for executions.

The console can also access EMR Notebooks as EMR Studio Workspaces, according to documentation. Workspace access and creation require additional IAM role rights. Programmatic execution requires IAM policies like StartNotebookExecution, DescribeNotebookExecution, ListNotebookExecutions, and iam:PassRole. EMR Notebooks clusters (EMR on EKS) require emr-container permissions.

The AWS Region per account maximum is 100 concurrent executions, and executions that last more than 30 days are terminated. Interactive Amazon EMR Serverless apps cannot execute programs.

You can plan or batch EMR notebook runs using AWS Lambda and Amazon CloudWatch Events, or Apache Airflow or Amazon Managed Workflows for Apache Airflow (MWAA).

[govindhtech]

Working With EMR Notebooks AWS Using Jupyter Notebook

Working with AWS EMR Notebooks

Amazon EMR Notebooks, renamed EMR Studio Workspaces, simplify data processing cluster interaction. They use the popular open-source Jupyter Notebook or JupyterLab editors and are available from Amazon EMR. This may be more efficient than EMR cluster notebooks. Users with suitable IAM rights can open the editor in the console.

Notebook statuses

When and how to communicate with EMR Notebooks requires knowing their status. The numerous states you may encounter are listed below:

The notebook is being produced and connected to the cluster. Launching, stopping, removing, or changing the editor's cluster is currently impossible. It starts rapidly but can take longer if a cluster forms.

You can access the fully prepared notebook in the notebook editor. Stop or remove the notebook in this state. Stop the notebook before altering the cluster. A Ready notebook will shut down after a long inactivity.

The notebook has been produced, however cluster integration may require resource provisioning or additional steps. In this case, you can launch the notebook editor in local mode, but cluster-dependent code will fail.

Stopping: Laptop or cluster shutdown. Like the ‘Starting’ state, the editor cannot be opened, stopped, deleted, or clusters altered while stopping.

The laptop shut down successfully. You can delete the laptop, swap clusters, or restart it on the same cluster (assuming the cluster is still operating).

Notebook is being removed from console list. Even after the notebook entry is erased, Amazon S3 will charge for the notebook file (NotebookName.ipynb). To retrieve the latest status, reload the console's notebook list.

Working in Notebook Editor

The notebook editor starts when the notebook is Ready or Pending. You choose Open in JupyterLab or Jupyter after choosing the notebook from the list. This opens a new browser tab with the editor. After opening, select your programming language's kernel from the Kernel menu.

The console-accessible editor's ability to limit EMR notebooks to one user is critical. Opening an already-used notebook will result in an error. Amazon EMR produces a unique pre-signed URL for each session that is only valid for a short time, displaying security.

This URL should not be shared since recipients could inherit your rights and be at risk. IAM permissions policies and granting EMR Notebooks service role access to the Amazon S3 location are two strategies to control access.

Preserving Work

While editing, your notebook cells and output are automatically and occasionally saved to the Amazon S3 notebook file. When there are no modifications since the last save, the editor displays “autosaved,” and otherwise, “unsaved.” You can manually save the notebook by pressing CTRL+S or choosing Save and Checkpoint from File. Manual saves create a checkpoint file (NotebookName.ipynb) in the notebook's principal Amazon S3 folder's checkpoints folder. This site stores only the latest checkpoint.

Attached Cluster Change

Switching the cluster to which an EMR notebook is linked without affecting its content is useful. Only Stopped notebooks can accomplish this. The approach involves selecting the paused notebook, viewing its data, selecting the Change cluster, and then choosing an existing Hadoop, Spark, and Livy cluster or creating a new one. Finally, select the security group and click Change cluster and start laptop to confirm.

Delete Notebooks and Files

The Amazon EMR interface lets you remove an EMR notebook from your list. Importantly, this approach does not delete Amazon S3 notebook files. These S3 data continue to accrue storage fees.

To remove the notebook entry and files, delete the notebook from the console and note its Amazon S3 location (in the notebook details). The AWS CLI or Amazon S3 interface must be used to manually remove the folder and its contents from the S3 location. An example CLI command removes the notebook directory and its contents.

Share and Use Notebook Files

Every EMR notebook has a NotebookName.ipynb file in Amazon S3. If it works with EMR Notebook Jupyter Notebook, you can open a notebook file as an EMR notebook. Saving the.ipynb file locally and uploading it to Jupyter or JupyterLab makes using a notebook file from another user straightforward. This method can recover a console-erased notebook or work with publicly published Jupyter notebooks if you have the file.

A new EMR notebook can be created by replacing the S3 notebook file. Stop all running EMR notebooks and close any open editor sessions.

Create a new EMR notebook with the precise name you want for the new file, record its S3 location and Notebook ID, stop it, and.Using the AWS CLI, copy and change the ipynb file at that S3 location, making sure the file name matches the notebook's name. This technique is shown using an AWS CLI command.

[govindhtech]

Amazon EMR Studio Workspace Creation and Launching in AWS

Design and customise workspaces in an EMR studio to organise and operate notebooks. This section covers workspace construction and use.

Helpful EMR Studio Workspace topics:

Make an EMR Studio Workspace

Start an EMR Studio Workspace

Learn EMR Studio's Workspace UI.

See EMR Studio notebook examples.

Save EMR Studio Workspace content.

EMR Studio Workspace and notebook deletion

Know workspace status

Fix Workspace connectivity.

Make an EMR Studio Workspace

Create EMR Studio Workspaces to run notebook code.

To create an EMR studio workplace

Log into EMR Studio.

Select “Create a Workspace.”

Enter a workspace description and name. Naming workspaces helps find them on the Workspaces page.

Workspace collaboration allows Studio users to collaborate in real time on this workspace. Create collaborators after starting the Workspace.

Joining a cluster to a workspace requires expanding Advanced setup. You can add a cluster later. Refer to Attach CPU to EMR Studio Workspace for information.

Provisioning a new cluster requires administrator access.

After choosing a workspace cluster, attach the cluster.

Click Create a Workspace at the bottom.

After creating a workspace, EMR Studio opens the Workspaces page. The freshly established workspace is listed with a green success banner at the top.

Any Studio user can see shared workspaces by default. However, only one individual can utilise a workstation. You can collaborate with other users in EMR Studio using workspace collaboration.

Launch of EMR Studio Workspace

The notebook editor in a Workspace lets you deal with notebook files. The Workspaces page of a studio lists all accessible workspaces, along with their Name, Status, Creation time, and Last Modified.

Note

Your EMR notebooks from the previous Amazon EMR console may be in the console as EMR Studio Workspaces. IAM role rights are needed to access or create Workspaces in EMR Notebooks. You may need to refresh the Workspaces list to see a notebook you made in the last terminal.

To create a notepad and editing workspace

Your Studio's Workspaces page has the workspace. Keywords and column values can filter the list.

Select the workspace name to open it in a new browser tab. It may take several minutes to open the workspace if idle. Click Launch Workspace after selecting the Workspace row.

These launch options are available:

Quick launch: Use default settings to launch your workspace. Select Quick launch to connect clusters to JupyterLab.

Start your workstation with customisable settings. Launch Jupyter or JupyterLab, connect to an EMR cluster, and select security groups.

Note

Working in a workspace is limited to one user. EMR Studio alerts you when you try to open a specified Workspace that is in use. The Workspaces page shows the workspace user in the User column.

[govindhtech]

Amazon EMR Notebooks For Enhanced Big Data Exploration

Amazon EMR Notebooks

EMR Notebooks: AWS Simplifies Spark Cluster Data Analysis

Amazon Web Services (AWS) makes big data management more flexible and integrated for data scientists and analysts. Amazon EMR Notebooks offer a familiar interactive interface that connects Apache Spark-powered Amazon EMR clusters. The new feature streamlines data searches, model creation, and result visualisation.

Amazon EMR users can access EMR Notebooks as EMR Studio Workspaces. The console interface's “Create Workspace” button simplifies notebook creation. Users need extra IAM role permissions to create or access these Workspaces.

EMR notebooks are “serverless” interfaces. The equations, queries, models, code, and narrative text you write are client-side in the notebook interface, while a kernel on the Amazon EMR cluster executes your commands. This configuration directly uses your EMR system's scalable computing capability for interactive analysis sessions.

Designing to protect your valuable work from computing cluster transience is crucial. EMR notebook contents are automatically stored on Amazon S3. Your notes, code, and analysis are separated from the cluster's data, allowing flexible notebook reuse and durability (your work continues even if the cluster is shut down).

The flexibility of laptop cluster connections is a major benefit. Users can establish an EMR cluster, connect their notebook for analysis, then shutdown the cluster when they're done for cost-effective, on-demand computing. Closing a notebook connected to one cluster to another lets you shift environments or work with data on another cluster fast.

Multiple users can connect their notebooks to the same EMR cluster at once, and notebook files are hosted on Amazon S3, making sharing easy. It is stated that these features will reduce notebook reset time for diverse datasets and clusters.

Interactive console or programmatic use of EMR Notebooks. Headless execution lets users run an EMR notebook over the Amazon EMR API without using the UI. This involves marking a cell in the EMR notebook with “parameters” to enable. When an external script is launched programmatically, this cell acts as a gateway to feed the notebook new data.

This is useful when creating parameterised notebooks that can be reused with different input values without requiring extra copies. Every time an API-executed parameterised notebook is launched, Amazon EMR generates and stores the output notebook on S3. This functionality can be developed using example API instructions.

EMR Notebooks support 5.18.0 and newer clusters. AWS recommends EMR Notebooks with Amazon EMR 5.30.0, 5.32.0, or 6.2.0 clusters for optimum performance. In these latter versions, the Jupyter kernels that run your code run directly on the cluster, thus this guidance is crucial. Direct cluster execution is said to boost performance and kernel and library modification.

Customers considering Amazon EMR Notebooks should consider the costs. As expected, Amazon S3 storage for notebook data will cost. Standard fees will also apply to connected Amazon EMR clusters utilised for notebook instructions.

Finally, Amazon EMR Notebooks provide a comfortable, adaptable, and interactive environment for data professionals to analyse and develop data directly connected to their Amazon EMR Spark clusters. S3 storing, adjustable cluster attachment, multi-user access, and powerful headless execution make them a compelling AWS large data alternative.

[govindhtech]

How To Create EMR Notebook In Amazon EMR Studio

How to Make EMR Notebook?

Amazon Web Services (AWS) has incorporated Amazon EMR Notebooks into Amazon EMR Studio Workspaces on the new Amazon EMR interface. Integration aims to provide a single environment for notebook creation and massive data processing. However, the new console's “Create Workspace” button usually creates notebooks.

Users must visit the Amazon EMR console at the supplied web URL and complete the previous console's procedures to create an EMR notebook. Users usually select “Notebooks” and “Create notebook” from this interface.

When creating a Notebook, users choose a name and a description. The next critical step is connecting the notebook to an Amazon EMR cluster to run the code.

There are two basic ways users associate clusters:

Select an existing cluster

If an appropriate EMR cluster is already operating, users can click “Choose,” select it from a list, and click “Choose cluster” to confirm. EMR Notebooks have cluster requirements, per documentation. These prerequisites, EMR release versions, and security problems are detailed in specialised sections.

Create a cluster

Users can also “Create a cluster” to have Amazon EMR create a laptop-specific cluster. This method lets users name their clusters. This workflow defaults to the latest supported EMR release version and essential apps like Hadoop, Spark, and Livy, however some configuration variables, such as the Release version and pre-selected apps, may not be modifiable.

Users can customise instance parameters by selecting EC2 Instance and entering the appropriate number of instances. A primary node and core nodes are identified. The instance type determines the maximum number of notebooks that can connect to the cluster, subject to constraints.

The EC2 instance profile and EMR role, which users can choose custom or default roles for, are also defined during cluster setup. Links to more information about these service roles are supplied. An EC2 key pair for cluster instance SSH connections can also be chosen.

Amazon EMR versions 5.30.0 and 6.1.0 and later allow optional but helpful auto-termination. For inactivity, users can click the box to shut down the cluster automatically. Users can specify security groups for the primary instance and notebook client instance, use default security groups, or use custom ones from the cluster's VPC.

Cluster settings and notebook-specific configuration are part of notebook creation. Choose a custom or default AWS Service Role for the notebook client instance. The Amazon S3 Notebook location will store the notebook file. If no bucket or folder exists, Amazon EMR can create one, or users can choose their own. A folder with the Notebook ID and NotebookName and.ipynb extension is created in the S3 location to store the notebook file.

If an encrypted Amazon S3 location is used, the Service role for EMR Notebooks (EMR_Notebooks_DefaultRole) must be set up as a key user for the AWS KMS key used for encryption. To add key users to key policies, see AWS KMS documentation and support pages.

Users can link a Git-based repository to a notebook in Amazon EMR. After selecting “Git repository” and “Choose repository”, pick from the list.

Finally, notebook users can add Tags as key-value pairs. The documentation includes an Important Note about a default tag with the key creatorUserID and the value set to the user's IAM user ID. Users should not change or delete this tag, which is automatically applied for access control, because IAM policies can use it. After configuring all options, clicking “Create Notebook” finishes notebook creation.

Users should note that these instructions are for the old console, while the new console now uses EMR Notebooks as EMR Studio Workspaces. To access existing notebooks as Workspaces or create new ones using the “Create Workspace” option in the new UI, EMR Notebooks users need extra IAM role rights. Users should not change or delete the notebook's default access control tag, which contains the creator's user ID. No notebooks can be created with the Amazon EMR API or CLI.

The thorough construction instructions in some current literature match the console interface, however this transition symbolises AWS's intention to centralise notebook creation in EMR Studio.

[govindhtech]

EMR Studio Features Requirements and Limits AWS

Amazon EMR Studio features, specs, and limitations:

Amazon EMR Studio describes an IDE for data preparation and visualisation, departmental collaboration, and application debugging. When utilising EMR Studio, consider tool usage, cluster demands, known issues, feature constraints, service limits, and regional availability.

Features of Amazon EMR Studio

Service Catalogue lets administrators connect EMR Studio to cluster templates. This lets users create Amazon EC2 EMR clusters for workspaces. Administrators can grant or deny Studio users access to cluster templates.

The Amazon EMR service role is needed to define access permissions to Amazon S3 notebook files or AWS Secrets Manager secrets because session policies do not allow them.

Multiple EMR Studios can control access to EMR clusters in different VPCs.

Use the AWS CLI to configure Amazon EMR on EKS clusters. Connect these clusters to Workspaces via a controlled API in Studio to run notebook jobs.

Amazon EMR and EMR Studio use trusted identity propagation, which has extra considerations. IAM Identity Centre and trusted identity propagation are required for EMR Studio to connect to EMR clusters that use it.

To secure Amazon EMR off-console applications, application hosting domains list their apps in the Public Suffix List (PSL). Examples are emrappui-prod.us-east-1.amazonaws.com, emrnotebooks-prod.us-east-1.amazonaws.com, and emrstudio-prod.us-east-1.amazonaws.com. For sensitive cookies in the default domain name, a __Host- prefix can prevent CSRF and add security.

EMR Studio Workspaces and Persistent UI endpoints use FIPS 140-certified cryptographic modules for encryption-in-transit, making the service suitable for regulated workloads.

Amazon EMR Studio requirements and compatibility

EMR Studio supports Amazon EMR Software versions 5.32.0 and 6.2.0.

EMR clusters using IAM Identity Centre with trusted identity propagation must use it.

Before setting up a Studio, disable browser proxy control applications like FoxyProxy or SwitchyOmega. Active proxies can cause Studio creation network failures.

Amazon EMR Studio restrictions and issues

EMR Studio does not support Python magic commands %alias, %alias_magic, %automagic, %macro, %%js, and %%javascript. Changing KERNEL_USERNAME or proxy_user using %env or %set_env or %configure is not supported.

Amazon EMR on EKS clusters does not support SparkMagic commands in EMR Studio.

All multi-line Scala statements in notebook cells must end with a period except the last.

Amazon EMR kernels on EKS clusters may timeout and fail to start. Should this happen, restart the kernel and close and reopen the notebook file. The Restart kernel operation requires restarting the Workspace, and EMR on EKS clusters may not work.

If a workspace is not connected to a cluster, starting a notebook and choosing a kernel fails. Choose a kernel and attach the workspace to run code, but ignore this error.

With Amazon EMR 6.2.0 security, the Workspace interface may be blank. For security-configured EMRFS S3 authorisation or data encryption, choose a different supported version. Troubleshooting EMR on EC2 tasks may disable on-cluster Spark UI connectivity. Run %%info in a new cell to regenerate these links.

5.32.0, 5.33.0, 6.2.0, and 6.3.0 Amazon EMR primary nodes do not have idle kernels cleaned away by Jupyter Enterprise Gateway. This may drain resources and crash long-running clusters. A script in the sources configures idle kernel cleanup for certain versions.

If the auto-termination policy is enabled on Amazon EMR versions 5.32.0, 5.33.0, 6.2.0, or 6.3.0, a cluster with an active Python3 kernel may be designated as inactive and terminated since it does not submit a Spark task. Amazon EMR 6.4.0 or later is recommended for Python3 kernel auto-termination.

Displaying a Spark DataFrame using %%display may truncate wide tables. Create a scrollable view by right-clicking the output and selecting Create New View for Output.

If you interrupt a running cell in a Spark-based kernel (PySpark, Spark, SparkR), the Spark task stays running. The on-cluster Spark UI is needed to end the job.

EMR Studio Workspaces as the root user in an AWS account causes a 403: Forbidden error because Jupyter Enterprise Gateway settings disallow root user access. Instead of root, employ alternate authentication methods for normal activities.

EMR Studio does not support Amazon EMR features:

connecting to and running tasks on Kerberos-secured clusters.

multi-node clusters.

AWS Graviton2-based EC2 clusters for EMR 6.x releases below 6.9.0 and 5.x releases below 5.36.1.

A studio utilising trusted identity propagation cannot provide these features:

Building EMR clusters without templates using serverless applications.

Amazon EMR launches on EKS clusters.

Use a runtime role.

Supporting SQL Explorer or Workspace collaboration.

Limited Amazon EMR Studio Service

Service Restriction The sources list EMR Studio service limits:

EMR Studios:

Each AWS account can have 100 max.

Maximum five subnets per EMR Studio.

IAM Identity Centre Groups are limited to five per EMR Studio.

EMR Studios can have 100 IAM Identity Centre users.