#apifunctions

[govindhtech]

Dataplex data Catalog Now Offers A Fresh Catalog Experience

Explore a whole new catalog experience with Dataplex, which is currently widely accessible.

Organisations are finding that they require a central catalog for their data assets as a result of the ever-growing volumes and varieties of data. Whether your resources are on-premises or in Google Cloud, Dataplex Catalog, Google  Cloud’s next-generation data asset inventory platform, offers a uniform inventory for all of your metadata. It is currently broadly available.

Dataplex Data catalog

A comprehensive inventory of both on-premises and Google  Cloud resources, including BigQuery, is offered by Dataplex Data catalog. You add metadata for third-party resources into Dataplex Catalog, and metadata for Google Cloud resources is automatically collected. You can add more commercial and technical metadata to your inventory using Dataplex Catalog in order to fully capture the context and knowledge about your resources. You can enable data governance over your data assets and search and find your data throughout the organisation with Dataplex Data catalog.

The following tasks can be completed with Dataplex Catalog:

Find out about and comprehend your data. Throughout the company, Dataplex Data catalog gives you visibility over your data resources. It facilitates finding pertinent resources for your needs when consuming data. It gives data resources context, which enables you to judge whether or not they are appropriate for the requirements of your data consumers. Make data management and governance possible. Your data governance and management capabilities can be strengthened and informed by the metadata provided by Dataplex Catalog.

Keep your metadata in a complete, expandable library. You may access and save metadata that is automatically gathered from your Google Cloud resources using Dataplex Data catalog. Your own metadata from non-Google  Cloud systems can be integrated. Technical and commercial metadata annotations can be added to enhance any metadata.

The operation of Dataplex Catalog

The following ideas form the foundation of Dataplex Catalog:

Entry A data asset is represented by an entry. Aspects inside an entry describe the majority of the metadata. This is comparable to Data Catalog entries. See Entries for further details.

Aspect: Within an entry, an aspect is a collection of connected metadata fields. An aspect might be thought of as extra metadata attached to an entry, or as one of its building blocks. This is comparable to Data Catalog tags, except the aspects are contained in the entries rather than existing as separate resources. See Aspects for additional details.

Aspect type: An aspect type is an aspect template that can be used again. Each aspect is an example of a certain aspect type. This is comparable to Data Catalog’s tag templates. Go to Aspect types for further details.

Entry group: An entry group is a unit of management for entries, acting as a container for them. You can set up IAM access control, project attribution, or location for the entries in an entry group, for instance, using an entry group. This reminds me of Data Catalog entry groupings. Refer to Entry groups for further details.

An entry type is a template that can be used to create entries. It lays forth the necessary metadata components, which are described as a set of conditions for this kind of entry. See Entry types for additional information.

What is it that Dataplex Catalog can do for you?

You can search and find your data throughout the company with Dataplex Data catalog. You can also enable data governance over your data assets, gain a better understanding of the context of your data, and capture context and knowledge about your data domain by adding more business and technical metadata to your data.

How Dataplex Data catalog may assist you with daily data discovery and governance inquiries is as follows:

You can go through related metadata and seek data resources all around the company as a business analyst or data analyst.

You can annotate your data resources as a data producer or governor by adding more technical, semantic, and business metadata.

Establishing the guidelines for annotation and custom resources will help you, as the data owner, steward, or governor, maintain consistency in your metadata.

You have a consolidated inventory of all the resources you have as a data engineer, including resources from Google  Cloud (harvested by Dataplex Catalog automatically) and resources from other systems (harvested by you and ingested into Dataplex Data catalog). A solitary, user-friendly API and a strong metamodel are provided by the Dataplex Catalog.

The following are some advantages of utilising Dataplex Catalog:

You can interact with and store a variety of metadata types, including complicated structures like lists, maps, and arrays, with an expressive metadata structure.

For consistent and efficient ingestion, you can self-configure the metadata schema for your unique resources.

One atomic CRUD operation can be used to interact with all of the metadata associated with an entry, and you can retrieve various metadata annotations linked to search or list responses.

Basic API functions (create, read, update, and delete) and searches conducted against specific Dataplex Data catalog resources are free of charge. The storage of metadata is paid for, nevertheless. Dataplex Data catalog offers complete support for Terraform providers and can be accessed through the google cloud CLI, the console, and an API.

At Google  Cloud, their goal is to simplify their integration process for partners so that google cloud can increase combined value. To expand their data management capabilities into hybrid and multi-cloud systems, google cloud collaborate closely with a wide range of partners. For clients who use Dataplex and Collibra together, Dataplex Data catalog is now linked with Collibra to simplify governance across cloud, on-premises, self-managed, and edge locations. Keep checking back for further details regarding new alliances that will improve their data management skills and benefit their clients even more.

Read more on govindhteh.com

[thinkdash]

Calling all command-line junkies: the new WolframScript is here!

Now you can evaluate Wolfram Language code, call deployed APIs and execute standalone scripts directly from your favorite command-line interface. WolframScript works like any other command-line utility, enabling flexible connections between the Wolfram System and other programs and I/O.

WolframScript comes packaged with Version 11.1 of Mathematica; on Mac, you must run the Extras installer bundled with the Wolfram System. You can also download and install a standalone version from the WolframScript home page.

Once installed, the wolframscript executable can be found in the same folder as your desktop application, and it is added to the PATH so you can call it directly from any command-line interface.

Interactive Scripting

When executed with no options, wolframscript opens a Wolfram Language console interpreter. This interactive shell (sometimes referred to as a REPL or read–eval–print loop) is a convenient way to write and run Wolfram Language code without launching the desktop front end. It also provides an alternative interface for headless servers or embedded computers (for example, a Raspberry Pi).

When running wolframscript in this way, you can simply enter a line of code and press Enter to see the result. Once you’re finished, use Quit to terminate the interactive session.

One-Shot Code Evaluations

To run a single line of code without launching the interactive shell, use the -code option. Commands entered this way are evaluated immediately by the Wolfram Engine, with the result sent to standard output. When evaluation is complete, the Wolfram kernel is terminated. This is convenient for single-use applications, like viewing the contents of a text file using Import. (In some cases you’ll need to escape inner double quotes with the \ character.)

You can also use redirection to supply a file as input through standard input. This incoming data is represented within a script by $ScriptInputString. Adding -linewise uses the standard NewLine character as a delimiter, treating each line of text as a separate input value.

Defining Functions

For more structured scripting, you can indicate a pure function using the -function option and pass in arguments with -args. By default, arguments are interpreted as strings.

With the -signature option, you can specify how arguments should be parsed in each function slot, including any format available to Interpreter—from basic numeric and string types to entities, quantities and many import/export formats. (Keep in mind that some high-level interpreter types require a connection to the Wolfram Cloud.)

Using Cloud Kernels

If you don’t have a local installation of Mathematica, you can run wolframscript in the cloud. Adding the -cloud option to the end of your command sends the computation to an available cloud kernel. You’ll be asked to authenticate the first time you run something in the cloud.

The -cloud option uses a public kernel on the Wolfram Cloud by default. If you’re connected to Wolfram Enterprise Private Cloud, you can specify a different cloud base by passing its URL (e.g. http://ift.tt/2pLtPKq) as an argument directly after -cloud.

You can open and close these connections manually using -auth and -disconnect. Each cloud requires separate authentication, and connection data is stored for use during your session. Cloud authentication is only necessary for sending dedicated computations; it doesn’t affect Wolfram Knowledgebase access.

Calling Packages and APIs

Code from Wolfram Language packages (.wl, .m) can be executed through wolframscript using the -file option. This evaluates each successive line of code in the file, terminating the kernel when finished.

Unlike with interactive scripting, results from -file are not displayed by default unless enclosed in Print, Write or some similar output function. Using the -print option sends the result of the final computation to standard output, and -print all shows intermediate results as well.

You can also call deployed APIs with the -api option. The following API (generated using APIFunction and CloudDeploy) returns a forecast of high temperatures for the next week in a given city. To call the API with wolframscript, you can reference it by URL or by UUID (the last part of the URL). Parameters are passed in by name; in this case, -args is optional.

Output Formats

By default, wolframscript gives a low-level text representation of the result. You can select the type of output you want, including any format understood by Export, using the -format option. For instance, some output may be easier to read in a table format.

When working with non-textual formats (e.g. spreadsheets, audio, video, graphics), it’s often best to write output directly to a file; you can do this using redirection.

Writing and Executing Scripts

Wolfram Language scripts (.wls) are standalone files containing Wolfram Language code that can be executed like any other application. Structurally, scripts are just packages that are launched as programs rather than notebooks by default. You can create a script from Mathematica with the File > New > Script menu item and execute it by typing its name in the command line (prepending ./ on Linux and Mac systems) or by double-clicking its icon in a file explorer.

The shebang line (starting with #!) tells the Unix environment to check the PATH for the wolframscript executable. On Unix-based systems, you can add launch options to this line by opening the script in a text editor. For instance, if you wanted to implement the travel distance function above as a standalone script, you would include the -function and -signature options in this line. (As of this writing, these options are bypassed when running scripts in Windows, but the goal is to eventually have all platforms work the same.)

To access command-line arguments, use $ScriptCommandLine within your script. Arguments are stored as a list of strings, starting with the full path of the script. In most cases, you’ll want to discard that initial value using Rest.

You may need to convert arguments to the correct data type for computations; this can be done using ToExpression. This script also checks for arguments first, printing a message if none are found.

Redirection works both ways when executing scripts, allowing for advanced applications such as the following image processing example. To maintain formatting for non-textual output, use -format when writing to a file.

You can even launch external programs directly from your script. The following will take a fundamental frequency, generate a bell sound using harmonics, export it to a temporary file and play it in your system’s default audio player.

Final Notes

WolframScript makes it easy to access Wolfram kernels from familiar, low-level interfaces for more flexible and universal computations. And with its cloud connectivity, you can access the Wolfram Language even from machines with no Wolfram System installed.

All the scripts demonstrated here are available for direct download as .wls files. You can execute them directly, change code and launch options in a text editor, or open them in Mathematica for standard notebook features like interactive execution, code highlighting and function completion.

For even more ideas, take a look at the WolframScript documentation and our tutorial on writing scripts. These examples barely scratch the surface—with the full functionality of the Wolfram Language available, the possibilities are endless.

So what are you waiting for? Let’s get scripting!

http://ift.tt/2pLyBrr

Think-Dash.com