Setup for real time sensors data to Power BI visualisation

Data from Devices/ Sensors can be sent to Azure IoT Hub.

Integration between IoT  Hub and Power BI can be achieved by setting up Azure stream Analytics Job

 Research: Apache Spark Cluster in Azure HDInsight:

In this demo, we will work on creating Spark cluster

a) Go to Azure Portal and search for HDInsight

Fill in cluster name, cluster admin password and region. For Storage with default values.

Configuration with D12v2 nodes selection for lesser pricing and finish review+ Create.

b) Create Jupyter notebook by hitting https://<clustername>.azurehdinsight.net/jupyter

On the sign-in page: enter cluster admin and password. Select New PySpark and run queries.

Select File Menu to Close and Halt notebook and release cluster resources

 Azure Data Explorer with Demo

Azure Data Explorer is a fully-managed big data analytics cloud platform and data-exploration service, developed by Microsoft, that ingests structured, semi-structured and unstructured data. The service then stores this data and answers analytic ad hoc queries on it with seconds of latency. Lets Explore

a) Login to Azure Portal and search Azure data Explorer. On Cluster page as below, select resource group and keep other settings as default. Hit Create.

b) Lets add database

3) Copy the cluster URI on overview page and goto https://dataexplorer.azure.com/home

4) Ingest data from various sources

5) once data ingestion is complete, write Kusto Query Language to query the data

Sample Queries: This will also render Bar Chart

StormEvents

| where isnotempty(EndLocation) 

| summarize event_count=count() by EndLocation

| top 10 by event_count

| render columnchart

 Implement Azure Data Factory Demo: 

In this research, we will go thru steps to create Azure Data Factory to copy data from SQL to Blob. We will create storage account, container, SQL DB table, Data Factory and pipelines to copy data

Azure Data Factory is a Microsoft cloud service that allows data integration from different sources by building hybrid extract-transform-load (ETL/ELT) and data integration pipelines.

a) Go to URL https://portal.azure.com and Login to the account. On the Azure portal menu or from the Home page, select Create a resource and search for Storage Account and click create with default settings and select the resource group.

b) Next go to storage resource and copy the access keys. Next create new container Add.

c) Next step is to create SQL Server Database. For test, set SQL authentication with admin user

Using SSMS, connect to the SQL Server and create new Database and one small table with few rows

d) Create a Data Factory with git configuration as later.

Go to Data Factory resource and open Data factory studio. Create pipeline and copy data.

Define Source data store, Select Blob as destination with below as summary