#FeatureReduction

[learnandgrowcommunity]

Session 15 : What is Dimension Reduction | Explained: A Beginner's Guide for Data Scientists

Welcome to Session 15 of our data science tutorial series! 🚀 In this session, we delve into the fundamental concept of "Dimension Reduction" and provide a comprehensive explanation tailor-made for beginners in the field of data science.

Subscribe to "Learn And Grow Community" Follow #learnandgrowcommunity

[milotec]

Finalmente i Cluster!

Finalmente con la versione 4.14 delle API Javascript ESRI ha inserito il clustering dei punti in mappa in modo nativo (sebbene ancora in beta), senza dover costringere i poveri sviluppatori a trovare soluzioni più o meno azzeccate.

Con le API 3.x è possibile ma è limitato agli Spatial reference; Web Mercator e WGS 84, con le 4.14 questo limite dovrebbe essere superato (per ora è in via sperimentale).

Per maggiori dettaglio fate riferimento dove trovate spiegate le proprietà: clusterRadius e come configurare  popupTemplate 

https://developers.arcgis.com/javascript/latest/sample-code/featurereduction-cluster/index.html?search=clustering

Differenze con lr API 3.x? Tante, anzi migliortamenti!

Cluster più dati: Mentre la 3.x aveva un limite di clustering di un massimo di 50.000 features, la 4.x non ha questo limite. È possibile raggruppare tutti i dati che possono essere caricati nella propria app. Filtraggio più veloce: È possibile utilizzare gli stessi filtri veloci lato client sui livelli clusterizzati come si può fare su livelli non clusterizzati in 4.x. I cluster vengono ricalcolati automaticamente lato client senza bisogno di ricaricare i dati. In 3.x ci si limita a filtrare i dati con la definitionExpression. Questo esegue il refetch dei dati dal servizio, che non ha lo stesso livello di interattività dei filtri lato client. GeoJSON layers: Poiché il livello GeoJSON è un “cittadino di prima classe in 4.x”, ora è possibile abilitare il clustering nello stesso modo in cui lo si farebbe con un livello di funzionalità. Proiezioni multiple: 4.x ha un supporto sperimentale per più proiezioni rispetto a Web Mercator e WGS 84. Questi erano gli unici riferimenti spaziali supportati nell'implementazione del clustering 3.x

[mrrolandtfranco]

Thematic point clustering for data exploration

Extracting meaningful information from large or dense point datasets can be challenging. Sometimes many points aren’t visible because they’re stacked on top of one another. Some datasets contain sparse data in some locations, but very dense data in others. Visualizations of attribute data with color or size in these scenarios can be difficult to digest. Zoom out to smaller scales and all of these problems are compounded. Version 3.22 of the ArcGIS API for JavaScript introduced point clustering to help solve some of these problems. It allows you to reduce and summarize relatively large numbers of data points in the map, providing some insight to geographic patterns in the data. These clusters are scale-dependent, meaning both their size and color change appropriately as the view scale changes.

Perhaps the most compelling piece of the ArcGIS clustering implementation is its separation from the layer’s renderer. Clustering is enabled on the layer, not the renderer, which means that all rendering properties are retained by the layer when clustering is enabled. Once enabled, the popup of each cluster will summarize the attribute used to drive the visualization of each feature. The color and the size of each cluster is determined based on the average or predominant value of the features comprising the cluster. Therefore you can use clustering as a tool for summarizing potential spatial patterns in data exploration apps.

Clustering in the ArcGIS API for JavaScript is enabled outside the renderer, so clusters retain renderer properties and summarize the variable of interest on a cluster-by-cluster basis. In the example above, New York City 311 calls are visualized by the time of day each incident was reported. Cluster colors indicate the predominant time of day incidents were reported among features comprising the cluster.

Exploring 311 calls in New York City

I downloaded 311 data from New York City’s Open Data site and created a small app for exploring the data. While glancing through the available attributes, three date fields caught my attention. They include the date each incident was created, the date it was due, and the date it closed.

Then I wrote three Arcade expressions to dynamically create three new variables of interest, answering the following questions:

Were any incident closures overdue? If so, by how long?

// Days incident closure was overdue var closed = $feature.Closed_Date; var due = $feature.Due_Date; var closureDueDiff = DateDiff(closed, due, "days"); IIF(IsEmpty(closed) || IsEmpty(due), 0, closureDueDiff);

How old were the incidents when closed?

// Incident report age (days) var closed = $feature.Closed_Date; var created = $feature.Created_Date; IIF(!IsEmpty(closed) && !IsEmpty(created), DateDiff(closed, created, "days"), 0);

At what time of day were the incidents reported?

// toEasternTime is defined in a previous function // that converts the date from UTC to eastern time var created = toEasternTime($feature.Created_Date); // Time of day var t = Hour(created); When( t >= 22 || t < 6, "Night", t >= 6 && t < 11, "Morning", t >= 11 && t < 13, "Midday", t >= 13 && t < 17, "Afternoon", t >= 17 && t < 22, "Evening", "It never happened" );

The toEasternTime() function is a custom Arcade function used to determine the hour an event occurred in the Eastern Time Zone. This assumes all data points were collected in the same time zone in the year 2015.

Then I created three renderers, each based on one of the above Arcade expressions, and added a select element to the UI to allow the user to dynamically switch visualizations on the layer. I also added a checkbox allowing the user to view the visualization with clustering enabled or disabled.

Additionally, the app allows you to visualize and filter incidents by type. This filter allows us to explore potential spatial patterns based on location, type, or any of the time variables mentioned above.

Check out the app!

Data observations

After exploring the data, it’s clear how clustering can improve the visualization of points based on a thematic attribute. Take a look at the image comparison below (click the images for a larger view). The renderer depicts the number of days each incident closure was overdue. When clustering is disabled, no obvious pattern emerges. However, once enabled, clustering helps us clearly identify areas where incident closures tend to be overdue more than others. These clustering patterns can be more or less refined as you zoom in and out or adjust the clusterRadius on the layer.

No clustering Clustering enabled

This app also demonstrates how filtering features impacts cluster patterns. As you filter by various complaint types, you will notice not only different spatial patterns emerge, but patterns in the closure times of each incident appear as well. For example, blocked driveways, illegal parking, and noise complaints usually seem to be resolved quickly and on time. Similarly, broken meter complaints tended to be resolved well ahead of schedule.

But graffiti complaints tended to close much later, often well after the assigned due date. Since “graffiti” is the most common incident type, it heavily influences the overall visualization of overdue incident closures.

On average, street condition complaints were resolved before or after the due date depending on the location of the complaint.

The time of day visualizations seemed to follow expectations when filtering by type. For example, noise complaints tended to occur at night, while blocked driveway complaints occurred more often in the morning, evening, and night (when people are more likely to be going in an out the door).

Overall, blocked driveways were reported more often in the mornings, evenings, and nights than at any other time of the day.

Enable clustering in your web apps

Clustering is managed with the featureReduction option in FeatureLayer and CSVLayer. You can enable it by simply setting the type to “cluster” in the layer’s constructor…

var layer = new FeatureLayer(serviceUrl, { featureReduction: { type: "cluster" } });

…or with the setFeatureReduction() method:

layer.setFeatureReduction({ type: "cluster" });

There are several other methods and options available for managing clusters and their children features. See the 3.22 release notes for more details.

Leverage the ArcGIS platform

While the tips mentioned above may help you get started with incorporating clustering in custom data exploration apps, don’t forget to use the ArcGIS platform to your advantage when possible. You can save yourself time by enabling and configuring clustering options in a web map in ArcGIS Online, and loading it into a custom web application. Check out this blog post, which provides a nice summary of the various clustering capabilities available in ArcGIS Online.

A word of caution

Clustering is useful for summarizing and identifying potential spatial patterns in attributes visualized by a layer’s renderer. However, keep in mind that every cartographic visualization includes some level of deception and should be questioned. While the visualizations in this sample show some interesting patterns, they depend solely on my definition of “time of day” and don’t take into account the spread of features as they occur in time.

For example, the time frame for “morning” could be interpreted a number of ways. Does it start at 5 a.m., 6 a.m., or 7 a.m.? When does it end? And regarding the spread of the data, let’s assume that “morning” is defined as (6 am – 11am) and a predominant “morning” cluster contains 500 features. Suppose 250 of the incidents comprising the cluster occurred between 10:50 am – 11:00 am and 240 occurred between 11:00 and 11:10. The cluster would correctly depict the data as predominantly occurring in the morning, but it doesn’t reflect the fact that most features occurred in the late morning, much closer to “midday” than a reasonable person might otherwise assume when viewing this map. Additionally, the visualization doesn’t indicate the strength of the predominant value.

Zooming in and out at various scales and browsing the features of each cluster will help peel away at the inadvertent deception present in the summarized data.

Also note that clustering options available via featureReduction do not perform complex statistical analyses. Therefore, the clustering visualizations described above should not be interpreted as precise, statistically significant “clusters” of data. Rather, they should merely be approached as a nice summary of the data, providing you with a preview to potentially identify spatial patterns that may or may not reveal significant storylines.

If you require more sophisticated spatial analysis, such as attempting to determine statistically significant hot spots and cold spots, or identify clusters of data, then the ArcGIS platform offers several tools that allow you to accomplish this. In ArcGIS Online, these tools include Aggregate Points, Calculate Density, Find Hotspots, and Find Outliers. You can also take advantage of the cluster toolset in ArcGIS Pro for access to additional tools.

Conclusion

Because clustering is a client-side feature reduction solution, it has a number of known limitations (see the list below). Server-side clustering will be implemented in a future release, which will improve performance and allow for more features to be clustered. Clustering will be added to the 4.x series of the ArcGIS API for JavaScript sometime in the near feature. Also check out the samples new to the 3.22 documentation demonstrating various ways to implement clustering in your web apps.

Known Limitations in 3.22

Support is limited to point data in FeatureLayer (from service or FeatureCollection) and CSVLayer.

The map must have a spatial reference of Web Mercator or WGS84.

If the layer contains more than 50,000 features, then only the first 50,000 will be clustered.

A FeatureLayer created from a service URL must point to a service that supports pagination (ArcGIS Server version 10.3.1 or higher).

When editing is initiated with the Editor widget, then feature reduction is disabled until the Editor widget is destroyed.

Feature reduction is disabled when the layer has one of the following renderers: HeatmapRenderer,BlendRenderer, TemporalRenderer, or ScaleDependentRenderer.

from ArcGIS Blog http://ift.tt/2ybJfZf

[mrrolandtfranco]

Mapping art in 3D

I regularly check the Open Data portals because there’s always some new, interesting dataset out there. Recently, I stumbled upon data of independent art galleries in New York City on the NYC OpenData portal and decided to create a scene with them. I was amazed that New York City has so many art galleries…especially in Manhattan. I wanted to give more context to the scene, so I also downloaded the NYC building data. Since this is a scene about art, a natural choice for the basemap was the Watercolor basemap from Stamen. See a live demo here.

After publishing the point and buildings data to ArcGIS Online, I was ready to start with the real fun part: visualizing the art galleries using ArcGIS API for JavaScript.

Visualize buildings

Visualization of the New York buildings was an important step to give a sense of place in the scene. For example, the buildings make it easier to locate art galleries. With this in mind, I shaded in purple buildings that have at least one art gallery. The rest I left beige. To accomplish this, I used a ClassBreaksRenderer:

var renderer = new ClassBreaksRenderer({ field: "NoArtGalleries", defaultSymbol: new MeshSymbol3D({ symbolLayers: [new FillSymbol3DLayer({ material: { color: [255, 235, 190, 0.9] } })] }), classBreakInfos: [{ minValue: 1, maxValue: 30, symbol: new MeshSymbol3D({ symbolLayers: [new FillSymbol3DLayer({ material: { color: [187, 165, 181, 1] } })] }) }] });

Visualize points of interest

With the new features in ArcGIS API 4.4 you can now create really nice visualizations of points of interest where the scene is balanced and easy to read.

First of all, I used a simple renderer to display all the points with the same icon:

var renderer = new SimpleRenderer({ symbol: new PointSymbol3D({ symbolLayers: [new IconSymbol3DLayer({ resource: { href: "./img/icon.png" }, size: 28 })] }) });

In this dataset there are 917 art galleries and many of them are inside buildings. Therefore icons will be hidden by buildings when looking at the scene in a detailed tilted view (see image below on the left). In a smaller scale view of New York, the icons fully cover the scene (image on the right).

Luckily, with the new features of ArcGIS API 4.4 this can be fixed.

Show points inside buildings

If a point of interest is located inside the building, you can’t see it because it is hidden by the building’s geometry.

This can be solved by placing the point of interest on the rooftop of the building. In more technical terms, I can display the point relative to the height of the building scene layer at the location of the point. In the image below the yellow buildings represent scene layer features and the ground is green. With the points aligned to the scene, the art galleries are no longer on the ground, but relative to the building rooftops. To enable this feature, I need to set the elevationInfo mode of the layer to relative-to-scene:

var pointsOfInterest = new FeatureLayer({ ..., elevationInfo: { mode: "relative-to-scene" } });

Relative to scene mode currently only works for point geometries.

Declutter the scene

Now I can see the icons, but I have the opposite problem: in some areas there are so many art galleries that when I zoom out points overlap each other which makes the scene look very crowded.

I can now filter out overlapping icons by using a new layer property called featureReduction. This ensures that all icons are visible at a large city scale, without cluttering the scene at a small scale.

var pointsOfInterest = new FeatureLayer({ ..., featureReduction: { type: "selection" } });

Understand the real location of the point

No matter how good you are at reading maps, in 3D it’s quite hard to tell where an icon is located just by looking at the scene. Have a look at this point:

You’d have to pan and rotate to actually figure out to which building the art gallery belongs. This experience can now be improved by using callout lines that point to the actual location of each point. So now you can clearly see this point is located in the purple building behind:

Callouts are set on a symbol level. First, you need to vertically shift the point with a vertical offset:

verticalOffset: { screenLength: 40, maxWorldLength: 200 }

Next you can set the callout (for now only callouts of type line are supported):

callout: new LineCallout3D({ color: "#534741", size: 1 })

The vertical offset shifts the point in the vertical direction and the callout adds the line corresponding to that offset. Callout lines only work together with the vertical offset.

Improve depth perception

Now my scene looks like this:

You might notice that the icons in the back are smaller than the icons in the front. That’s because we worked on improving depth perception. screenSizePerspectiveEnabled is a property at the layer level that automatically calculates the size of icons based on the distance to the camera. By default it is set to true, so if there are visual variables with size, the setting should be set to false.

Last but not least, I added a search widget to search for features in the art galleries layer and a popup to show more information about each one of the art galleries.

Check the live scene here.

Feel free to drop a comment in case you used any of these features. I’d love to know what you created with them! Raluca

from ArcGIS Blog http://ift.tt/2gJnOtX

[mrrolandtfranco]

Manhattan Skyscraper Explorer – when Open Data and ArcGIS API for JavaScript meet

Open data is an awesome thing nowadays. It’s amazing how many organizations publish their data to allow citizens to learn more about the city they live in. An example of such data is the New York 3D city model data published by the New York City Department of Information Technology and Telecommunications. However, to get to explore the buildings from New York, users should have access to the software that reads and visualizes the 3D models, which is often not the case. Therefore, building a web application based on such data seems like a natural solution to empower citizens and people around the world to find out more about the buildings in New York.

This was also my goal when I started building the Manhattan Skyscraper Explorer. I limited myself to Manhattan, because I was mostly interested in the highest and most popular buildings in New York: when were they built? how high are they? what do they look like? what are they currently used for?

I built an application looking for answers to these questions using ArcGIS API for JavaScript. All the new features in version 4.4 contributed to making this a really fun to use and intuitive app.

View the live application

I started by coloring the buildings based on the time period they were built in. This way I could see where are the areas with newer or older buildings, discover which buildings were built in the same period or see that in some neighbourhoods like Soho most of the buildings are built before 1925. Afterwards I was curious to correlate the construction year with the height, so I built a timeline with Y axis representing the height. In this timeline each building is a circle with the color given by the construction period. It was interesting to notice that less skyscrapers were built during the Great Depression and almost none during the World War II.

The timeline is connected to the map: if a user selects a building in the map, the corresponding element will be highlighted in the timeline. If a user selects a building in the timeline then the app will zoom to that building and display more information about it. Highlighting is a new functionality in the version 4.4 of the JavaScript API. Here is a basic example on how it works.

Immediately spotting the tallest buildings on a 3D map is not very straightforward. That’s why I added a filter to the SceneLayer that would allow me to only display buildings that are higher than a certain level. Let’s say for example I’m only interested in buildings higher than 500 feet. Filtering out buildings smaller than 500 feet leaves only skyscrapers in Lower and Midtown Manhattan.

You can filter features in a FeatureLayer or a SceneLayer by using definitionExpression. Here is a sample on how it works.

Although the original dataset already had some nice attributes like building name, construction year and building height, I was curious about some more information like what’s the current use or how does it look like. That’s why I decided to use the Wikipedia API (called MediaWiki) and the Flickr API. Most of the skyscrapers are very famous and have their own Wikipedia page, so I used MediaWiki to search for it and get the abstract of the page. Not all the buildings have a Wikipedia article so I wanted to mark the buildings that have such information. That’s why I added an extra point layer with information icons that show which buildings have extra information. The problem was that these icons were hidden by the buildings and were overlapping quite a lot. I used several new features among which the relative-to-scene elevationInfo mode and featureReduction to solve these problems. For more information about styling points in a city, have a look at this sample.

I was pretty happy with my result so far, but realized that I needed to add more tools to help users easily find the buildings and to help them orient themselves on the map. In a first phase I decided to add the names of the most important negihbourhoods in Manhattan. Adding labels on a 3D map can be tricky because from most angles they will end up being hidden by buildings. One solution is to set a vertical offset and add a line callout to make sure the location of the label is still clear to the user.

Another very useful feature is the search functionality. Finding the Empire State Building just by looking at the map can be very difficult for users that are not familiar to Manhattan, so searching by the name is what most people would do. So I added the search widget and set the feature layer with the buildings as the source. Here is a sample that explains how to do that.

You can see the live application here. In case you are curious to know more details about a specific implementation process of this app, write it down in the comments below and I’ll write a follow-up blog post on that.

Happy 3D mapping!

Raluca

from ArcGIS Blog http://ift.tt/2uqmnGC