Got this text back today from our driver at work picking up equipment. ????

Funny text! originally appeared on Shai Perednik.com on January 9, 2013.

Thinking about strategies of how to best edit web content, we will rather quickly agree that the so-called inline or inpage editing is the best way to do it. Inline editing means the editing of web contents right inside their original layout. Still this best way is supported by only very few systems. The project Create.js provides a module for almost every CMS out there to relatively easily implement exactly this, inline editing of any content.

Go to Source

Create.js – InPage Editing for Content Management Systems Using HTML5 originally appeared on Shai Perednik.com on December 29, 2012.

We have come to the end of another fine month. For WordPress users, December 2012 marked the release of WordPress 3.5 Beyond that, many new themes were also released in December, and in this round-up, we shall be taking a look at them.

Go to Source

Fresh and Free WordPress Themes Released In December 2012 originally appeared on Shai Perednik.com on December 25, 2012.

We decided to take our chances and venture out.

Indoor parking? originally appeared on Shai Perednik.com on December 22, 2012.

…this morning. Till we stepped out. I’ll post more soon..

Everything see mind fine originally appeared on Shai Perednik.com on December 22, 2012.

Here’s a good list of siri proxy plugins! Where’s the knock knock joke plugin?
https://github.com/plamoni/SiriProxy/wiki/Plugins

SiriProxy Plugins originally appeared on Shai Perednik.com on November 26, 2012.

Note that an important part of the script is the line commented “#Get all the DPM servers from the DPMServers table.”

SELECT * FROM [DtsDpm].[dbo].[DpmServers]

In this command the script connects to a specified SQL instance and retrieves all the DPM servers.  This table is setup with a server name, dpmdb name, ip, and script group.  So you will need to adjust this portion of the script to collect the DPM servers in your environment.

$SourceInstanceName = 'SQLServerWithDPMServersTable'

Save the below script as Get-SQLData.ps1.  See the help in the script for how to run it.

</pre>
<#

.SYNOPSIS

Runs provided SQL query against DPM servers.

.DESCRIPTION

The Script assumes that you have a SQL Database setup with a table with your DPM servers.  Also requires SQL mgmt studio.

The SQL query must include the following.  If omitted you will not know which data belongs to what DPM server.

SERVERPROPERTY('MachineName') as DPMServerName, GetDate() as DateID

.NOTES

Additional Notes, eg

File Name  : Get-SQLData.ps1

Author     : Shai Perednik shaiss@gmail.com

Requires   : PowerShell V2

.LINK

A hyper link, eg

http://ShaiPerednik.com

.EXAMPLE

-SQLScriptPath "E:\Scripts\SQL\Get-Extents.sql" -csvFilePath "e:\Scripts\SQL\Results\Get-SQLData.csv" -ScriptGroup 4 -verbose

Will run on ScriptGroup 4 only and enable verbose logging.

.INPUTTYPE

Documentary text, eg:

Input type  [Universal.SolarSystem.Planetary.CommonSense]

Appears in -full

.RETURNVALUE

Documentary Text, e,g:

Output type  [Universal.SolarSystem.Planetary.Wisdom]

Appears in -full

.PARAMETER SQLScriptPath

[Required] The path to the .sql file

Example "E:\Scripts\SQL\Get-Agents2.sql"

.PARAMETER DataType

[Optional] What kind of data is being collected?  This is important as it will determine what SQL table the data goes into.

Possible options are Agents, DiskUsage

If this is omitted, nothing will be inserted into SQL and only a csv will be outputted

.PARAMETER csvFilePath

[Optinal] Path to where to output the .csv file with data returned from the query.

Example "e:\Scripts\SQL\Results\Get-DiskUsage.csv"

If excluded output will be saved to current directory as output.csv

.PARAMETER ScriptGroup

[Optional] Will run the query provided in SQLScriptPath against a group of servers defined in the DPMServersTable.

.NOTES

Script uses SQL Server Management Objects (SMO). SMO is installed with SQL Server Management Studio and is available

as a separate download: http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=ceb4346f-657f-4d28-83f5-aae0c5c83d52

Version History

v0.0.5   - Shai Perednik - Initial Release

Know Issues

1.  Invoke-SQLCmd -Inputfile requires a full path including the drive letter.

If full path is not specified, file is looked for in the users document directory, regardless of current path.

2.  No SQL Insert for DiskUsage

3.  Get-RecoveryPoints needs to delete all data in the table first before the insert.

#>

&nbsp;

Param(

[Parameter(Position=0, Mandatory=$true)][String]$SQLScriptPath,

[Parameter(Position=1, Mandatory=$false)][String]$DataType,

[Parameter(Position=2, Mandatory=$false)][String]$csvFilePath,

[Parameter(Position=3, Mandatory=$false)][Int]$ScriptGroup

)

&nbsp;

#Load the SQL PSSnapins

Add-PSSnapin SqlServerCmdletSnapin100 -ErrorAction SilentlyContinue

Add-PSSnapin SqlServerProviderSnapin100 -ErrorAction SilentlyContinue

&nbsp;

#Below trys taken from http://gallery.technet.microsoft.com/ScriptCenter/c193ed1a-9152-4bda-b5c0-acd044e68b2c/  "Add-SqlTable"

try {add-type -AssemblyName "Microsoft.SqlServer.ConnectionInfo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" -EA Stop}

catch {add-type -AssemblyName "Microsoft.SqlServer.ConnectionInfo"}

&nbsp;

try {add-type -AssemblyName "Microsoft.SqlServer.Smo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91" -EA Stop}

catch {add-type -AssemblyName "Microsoft.SqlServer.Smo"}

&nbsp;

#*=============================================================================

#* FUNCTION LISTINGS

#*=============================================================================

# Function: funInsertIntoSQL

# Created: [10/22/2012]

# Author: Shai Perednik

# Arguments:

# =============================================================================

# Purpose: Runs provided query against SQL

#

#

# =============================================================================

&nbsp;

Function funInsertIntoSQL

{

Param ([string]$query, [int]$ResultsCount, [string]$SQLServerInstanceName, [int]$i)

Write-Verbose -Message "Runing query $query"

Write-Verbose -Message "Inserting record $i of $ResultsCount"

$InsertResults = Invoke-Sqlcmd -Query $query -ServerInstance $SQLServerInstanceName

$ResultsRemaining = $ResultsCount - $i

Write-Verbose -Message "$ResultsRemaining records remaining to insert"

}

&nbsp;

####________________End Functions________________####

&nbsp;

####________________Start Script________________####

&nbsp;

#Get all the DPM servers from the DPMServers table.

$SourceInstanceName = 'SQLServerWithDPMServersTable'

if ($ScriptGroup) {

$query = "SELECT * FROM [msdbdpm].[dbo].[DpmServers] WHERE [ScriptGroup] = $ScriptGroup"

} else {

$query = "SELECT * FROM [msdbdpm].[dbo].[DpmServers]"

}

$instanceNameList = Invoke-Sqlcmd -Query $query -ServerInstance $SourceInstanceName

&nbsp;

$query = get-content $SQLScriptPath

$results = @()

$i = 1

&nbsp;

foreach($instanceName in $instanceNameList)

{

$DPMServerName = $instanceName.DpmServer

#check if host is alive first before quering it's SQL.

write-verbose "Pinging $DPMServerName before running SQL on it"

if (test-connection -computername $DPMServerName -quiet -Count 1)

{

$instanceName = $instanceName.DpmServer + "\" + $instanceName.sqlinstancename

write-verbose -Message "Executing query against server: $instanceName"

write-verbose -Message "Query $Query"

$cmd= Invoke-Sqlcmd -InputFile $SQLScriptPath -ServerInstance $instanceName -QueryTimeout 5

if ($cmd -ne $null) {

$results += $cmd

}

}

}

&nbsp;

if ($csvFilePath) {

# Output to CSV if CSV path provided

write-verbose "Saving Query Results in CSV format to $csvFilePath..."

$results | export-csv  $csvFilePath -NoTypeInformation

} else{

# Output to output.csv to current directory since no custom path was provided

write-verbose "Saving Query Results in CSV format to output.csv..."

$results | export-csv  output.csv -NoTypeInformation

}

&nbsp;

#if a $DataType is provided this insert into SQL, otherwise skip over this step.

if ($DataType){

$ResultsCount = $results.Count

#Use select case to determine where/how to insert the data

switch ($DataType) {

"Agent" {

$QueryTable = 'vm_DPM_Agents'

Write-Verbose -Message "Writing Data to $QueryTable on $instanceName"

foreach ($item in $results) {

$query = "INSERT INTO [msdbdpm].[dbo].[$QueryTable]`

([AgentID]`

,[DPMServerName]`

,[DateID]`

,[ServerName]`

,[Version])`

VALUES`

(NewID()`

,`'" + $item.DPMServerName + "`'`

,`'" + $item.DateID + "`'`

,`'" + $item.ServerName + "`'`

,`'" + $item.Version + "`')`

GO"

funInsertIntoSQL -query $query -ResultsCount $ResultsCount -SQLServerInstanceName $SourceInstanceName -i $i

#                                                             Write-Verbose -Message "Runing query $query"

#                                                             Write-Verbose -Message "Inserting record $i of $ResultsCount"

#                                                             $InsertResults = Invoke-Sqlcmd -Query $query -ServerInstance $SourceInstanceName

#                                                             $ResultsRemaining = $ResultsCount - $i

#                                                             Write-Verbose -Message "$ResultsRemaining records remaining to insert"

$i++

}

break

}

"DiskUsage" {

break

}

"RecoverPoints" {

#format should be "RecoveryPointID","DPMServerName","DateID","DatasourceName","serverName1","NewestRecoveryPoint","OldestRecoveryPoint","RecoveryPoints","ServerId","ServerName","NetbiosName","DomainName","IsRG"

#NewID(),"DPMServerA","10/22/2012 5:05:06 PM","C:\","ClientServerA.example.com","10/20/2012 5:06:22 PM","9/23/2012 5:06:57 AM","55","944d9cca-6995-4f7d-97bc-040baa454dd4","ClientServerA.example.com","ClientServerA","example.com","False"

#insert into vw_DPM_RecoveryPointDisk values(NewID(),'DPMServerA','10/22/2012 11:42:44 PM','C:\','ClientServerA.example.com','10/20/2012 5:06:22 PM','9/23/2012 5:06:57 AM','55','944d9cca-6995-4f7d-97bc-040baa454dd4','ClientServerA.example.com','ClientServerA','example.com','False')

$QueryTable = 'vw_DPM_RecoveryPointDisk'

Write-Verbose -Message "Writing Data to $QueryTable on $instanceName"

foreach ($item in $results) {

$query ="INSERT INTO [msdbdpm].[dbo].[$QueryTable]`

([RecoveryPointID]`

,[DPMServerName]`

,[DateID]`

,[DatasourceName]`

,[serverName1]`

,[NewestRecoveryPoint]`

,[OldestRecoveryPoint]`

,[RecoveryPoints]`

,[ServerId]`

,[ServerName]`

,[NetbiosName]`

,[DomainName]`

,[IsRG])`

VALUES`

(NewID()`

,`'" + $item.DPMServerName + "`'`

,`'" + $item.DateID + "`'`

,`'" + $item.DatasourceName + "`'`

,`'" + $item.serverName1 + "`'`

,`'" + $item.NewestRecoveryPoint + "`'`

,`'" + $item.OldestRecoveryPoint + "`'`

,`'" + $item.RecoveryPoints + "`'`

,`'" + $item.ServerId + "`'`

,`'" + $item.ServerName + "`'`

,`'" + $item.NetbiosName + "`'`

,`'" + $item.DomainName + "`'`

,`'" + $item.IsRG + "`')`

"

funInsertIntoSQL -query $query -ResultsCount $ResultsCount -SQLServerInstanceName $SourceInstanceName -i $i

#                                                                             Write-Verbose -Message "Runing query $query"

#                                                                             Write-Verbose -Message "Inserting record $i of $ResultsCount"

#                                                                             $InsertResults = Invoke-Sqlcmd -Query $query -ServerInstance $SourceInstanceName

#                                                                             $ResultsRemaining = $ResultsCount - $i

#                                                                             Write-Verbose -Message "$ResultsRemaining records remaining to insert"

$i++

}

break

}

default {

Write-Host "DataType is Undifined"

break

}

}

}

####________________End Script________________####
<pre>

DPM Reporting Platform – Get-SQLData.ps1 originally appeared on Shai Perednik.com on November 24, 2012.

After working with DPM for the last 2 years I decided to put together this framework for collecting data across DPM servers in an enterprise environment. At the heart of the tool is a PowerShell based script that runs a provided SQL script against a set of servers.  I will post this this script as others I have shortly.  Look for the “Reporting Platform” tag.

Here is a link to the document in docx format thats pasted below.

http://shaiperednik.com/wp-content/uploads/2012/11/Reporting-Platform.docx

Table of Contents

1                 Introduction. 3

2                 Terminology. 3

2.1        Recovery Point Objective (RPO) 3

2.2        Recovery Time Objective (RTO) 3

2.3        Data Protection Manager (DPM) 3

2.4        Data Backup. 3

2.5        Data Archive. 3

2.6        Long Term Protection (LT) 3

2.7        Short Term Protection (ST) 3

2.8        Protection Group (PG) 3

2.9        SQL Server Reporting Services (SSRS) 4

2.10     Internet Information Services (IIS) 4

2.11     Data Protection Manager Database (DPMDB) 4

2.12     Datasource. 4

2.13     PowerGUI 4

3                 Architecture. 4

3.1        Servers required. 4

3.1.1       SQL Server 5

3.1.2       SSRS. 5

3.1.3       Web Server 5

3.1.4       Collector Server 5

3.1.5       Management Server 5

3.2        Components 5

3.2.1       Data Collectors 5

3.2.2       Data Collection Queries 6

3.2.3       Reports 6

3.2.4       Scheduled Tasks 7

3.2.5       Forms 7

3.2.6       SQL Databases 8

3.2.7       SQL Tables 8

3.3        Requirements/Prerequisites 8

3.3.1       Active Directory. 8

3.3.2       SQL Server Authentication. 8

3.3.3       Firewall/Security Rules 8

4                 Deployment 9

4.1        Server setup. 9

4.1.1       SQL Server 9

4.1.2       SSRS. 9

4.1.3       Web Server 9

4.1.4       Collector Server 9

4.1.5       Management Server 9

5                 Appendix. 10

5.1        Notes 10

5.1.1       Versioning. 10

5.1.2       Next Phase. 10

5.2        ToDo’s 10

5.2.1       Create Deployment package. 10

5.2.2       Complete Deployment Section. 10

Table of Figures

Table 1 – DPM Reporting Platform Server Roles 5

Table 2 – Scheduled intervals for data collectors 7

Table 3 – Firewall rules for SQL communication. 9

1         Introduction

The purpose of this document is to detail the components and deployment of the Microsoft Data Protection Manager Reporting Platform.

2         Terminology

In this document the following abbreviations, acronyms and terminology will be used:

2.1       Recovery Point Objective (RPO)

This refers to the point in time that the application should be recovered to.  In other words, how much data can be lost?  If the data is very volatile the RPO might be 15 minutes but if the data changes infrequently the RPO might be 24 hours.

2.2       Recovery Time Objective (RTO)

This refers to amount of time permitted to recover the data or system.  If the system is one of 5 systems that are load balanced the RTO might be 8 hours, but if the system or data is critical the RTO might only be 1 hour.

2.3       Data Protection Manager (DPM)

This is the backup software of choice, the current version is 2012 and the vendor is Microsoft.

2.4       Data Backup

Though a common term it is important to understand that in this document Backup or Data Backup refer to protecting data currently stored and accessible on systems.  This does not refer to data that needs to be retained for legal or other purposes but that does not reside on any systems.

2.5       Data Archive

In this document Archive, Archival etc refers to data that must be retained for longer than 6 months as defined by the business.  This might be backups taken at after year end procedures, Audit files that have been completed or eRooms that are being removed.

2.6       Long Term Protection (LT)

This term is pertinent refers to backups that are taken to tape in MS DPM, and also at times refers to backup jobs being a “long term protection” job or “LT” job, which simply means that the backup job is being done to tape.

2.7       Short Term Protection (ST)

This term is pertinent refers to backups that are taken to disk in MS DPM, and also at times refers to backup jobs being a “short term protection” job or “ST” job, which simply means that the backup job is being done to disk.

2.8       Protection Group (PG)

This term is pertinent to MS DPM 2010, and is the name given to the traditional “Backup Job” it is a collection of servers, files selections on those servers, and polices that are used to back up a given set of data.

2.9       SQL Server Reporting Services (SSRS)

A server-based report generation software system from Microsoft.  Administered via a web interface ,it can be used to prepare and deliver a variety of interactive and printed reports.

2.10  Internet Information Services (IIS)

Web server application and set of feature extension modules created by Microsoft for use with Microsoft Windows . IIS 7.5 supports HTTP, HTTPS, FTP, FTPS, SMTP and NNTP. It is an integral part of the Windows Server family of products, as well as certain editions of Windows XP, Windows Vista and Windows 7. IIS is not turned on by default when Windows is installed.

2.11  Data Protection Manager Database (DPMDB)

This is the default install database holding all DPM data.  While there is no published schema, the data is available and can be mined.

2.12  Datasource

This could be a:

• SQL Database
• SharePoint configuration Database and it’s content databases
• Windows Share
• Local/SAN File system Volume
• System State
• Bare Metal Backup

2.13  PowerGUI

Not a requirement but a recommended tool for developing PowerShell scripts.  Available at http://powergui.org/

3         Architecture

This section details the architecture of the reporting platform.

3.1       Servers required

These are servers that are required for the reporting platform.

Table 1 – DPM Reporting Platform Server Roles

3.1.1    SQL Server

Server should be running Windows Server 2008 Standard R2 SP1 and SQL Server Standard with the latest service pack at minimum.

Server’s role is to serve the SQL databases required to make the platform operational.

Optional, at a later time should resources constraints become an issue on this server, a secondary SQL server may be added to serve as the data warehouse for historical data.

3.1.2    SSRS

Server should be running Windows Server 2008 Standard R2 SP1 at a minimum.

Server’s role will be to display reports using SSRS.

3.1.3    Web Server

Server should be running Windows Server 2008 Standard R2 SP1 at a minimum.

Server’s role is to serve the reports generated from the SQL data and to serve any custom forms or reports that our outside the scope of SSRS.

3.1.4    Collector Server

Server should be running Windows Server 2008 Standard R2 SP1 at a minimum.

Server’s role will be to run collection scripts against DPM servers and either output that data into SQL, using the SQL server described in this section, or to a flat file on a local or remote file system.

3.1.5    Management Server

Server should be running Windows Server 2008 Standard R2 SP1 and latest Visual Studio suite at a minimum.

This server’s role is to hold the development tools needed to build/test new scripts or any other tools.  It will also serve as a central location to administer and monitor the other servers in the platform listed in this section.

3.2       Components

These are the components in the reporting platform.

3.2.1    Data Collectors

Data collectors should be written as PowerShell scripts that can execute a supplied SQL query against a collection of DPM servers.

*For data collection scheduling see “Scheduled Tasks” section 3.2.4 on page 10.

3.2.2    Data Collection Queries

SQL queries validated against individual DPMDB databases should be saved as .sql files and supplied to the data collector script.

It is vital that all SQL SELECT queries begin with the statement below.  This will ensure that all data collected from multiple servers is easily identified both in a flat file or when added to the SQL database.

SELECT SERVERPROPERTY(‘MachineName’) as DPMServerName

,GetDate() as DateID

Some basic required queries are:

3.2.2.1    Agent Information

Collects all data from [DPMDB].[dbo].[vw_DPM_Agents]

3.2.2.2    Disk Information

Collects all data from [DPMDB].[dbo].[tbl_SPM_Disk]

3.2.2.3    Recovery Points

Collects all data from [DPMDB].[dbo].[vw_DPM_RecoveryPointDisk]

Important points when collecting data from this SQL view.  This is a supplied view with DPM and has some caveats.

• CASE WHEN DatasourceName = ‘Non VSS DataSource Writer’ THEN ‘System Protection’ ELSE DatasourceName END
• WHERE ErrorCode=0  and Creationtime > dateadd(month, -1,getdate()) and status=2
  Creationtime is set here as we want to collect the total recovery points and the latest data for the last 30 days.
• GROUP BY ServerName,DatasourceName
  This allows for a rollup of the data.

This is not the ideal collector due to the caveats above and a custom one should be created to collect all actual recovery points as this table also includes failed recovery point attempts which can throw of data.

3.2.2.4    Extents

Collects all data from [DPMDB].[dbo].[tbl_SPM_Extent]

3.2.2.5    Replica Disk Usage

Collects all data from [DPMDB].[dbo].[vw_DPM_Disk_Usage_Replica]

3.2.3    Reports

Reports should be created using SSRS if possible.  Different reports should be created to target different audiences.

Various reports should be created at specific target groups.

3.2.3.1    Application owners/customers

These reports should be easily read so that application owners can easily monitor their server’s backup health.

3.2.3.1.1     Backup Health Report last 24 hours

Report would allow the user to select and subscribe to specific filters:

• Application
• Lifecycle
• Physical Location
• Success/Fail or both

3.2.3.1.2     Datasource health report

User would be able to drill down to a datasource level and see available recovery points for that datasource.

Optional, using the Web Server and forms component the user would then be able to select this datasource, or specific data under it, for restoration.

3.2.3.2    Management

Reports would be more metrics based with historical data and future projections.  The goal is to give management a high level insight into the environments health, requirements, and growth.  This could also be used by other teams and administrators to proactively mitigate possible outages or failures due to current unforeseen problems.

3.2.3.3    Administrators

These reports would give the DPM administrators insights into the environments health and would be based closely on the collectors above with some joined reports to bring specific data into correlation.

Additional reports could be created to present a dashboard style view to help guide administrators to remediating problematic areas first.

3.2.4    Scheduled Tasks

Scheduled tasks will need to be created on the Data Collector server to run the specified queries at specific schedules.  These tasks should be setup to use a service account.

Different data collectors will need to run at different intervals based on the “Data Collection Queries” in section 3.2.2 on page 8.  Some guidance on collection intervals below:

Table 2 – Scheduled intervals for data collectors

3.2.5     Forms

Forms can be created for any action to be taken based on collected data.

An example would be a form that allows submitting a restoration request for specific data with a known available recovery point.

3.2.6    SQL Databases

The SQL Databases should be separated.

3.2.6.1    Platform Database:  Holds tables required for the operation and function of the reporting platform

3.2.6.2    Data Collection Database: Holds all collected data.

3.2.7    SQL Tables

SQL tables are not limited but should at minimum should be:

3.2.7.1    Platform Database

3.2.7.1.1     DPM Servers

Table should provide all the information required to access DPM servers and collect data from them:

• Server Name
• Server IP
• DPM SQL Instance Name
• Grooming period
  • aka how long should collected data be retained
  • This value would act as a server level default and can be overridden by a global default or a specified value provided by a script.

3.2.7.2    Data Collection Database

Tables in this database should mimic the tables they collect data from on the individual DPM server.  This allows for easy identification at a later point of collected data.

All calculations and data manipulation should be done under views.  This will easy any future updates made to DPM database and therefore any collected data.

3.3       Requirements/Prerequisites

3.3.1    Active Directory

A service account will need to be created for the Data Collectors to run under.

3.3.2    SQL Server Authentication

All DPM servers must be reconfigured to allow read-only login to SQL for the service account created in section 3.3.1.

The SQL Server detailed in section 3.1.1, “SQL Server”, should allow read/write access to the service account created in section 3.3.1.

3.3.3    Firewall/Security Rules

Per the Microsoft support article KB287932, http://support.microsoft.com/kb/287932, port 1433 must be opened to allow remote communication.

Port 1433 should be opened per the table below:

Table 3 – Firewall rules for SQL communication

4         Deployment

This section details the deployment of the reporting platform.  This section will be developed as the tool is initial deployed.

4.1       Server setup

Follow “Table 1 – DPM Reporting Platform Server Roles” on page 8 and deploy those servers with the suggested settings in the tables.  Note that these are minimum guidelines and should be changed according to the environment

4.1.1    SQL Server

1. Install SQL 2008 standard or enterprise.  Do not install SSRS
2. If using sample scripts provided in the deployment package, run provided queries to create the necessary tables.
3. SQL service should be set to run as service account setup in section 3.3.1, “Active Directory”
4. Install SSRS from SQL 2008 Installer
5. Install IIS
6. Setup scheduled tasks per the intervals defined in Table 2 “- Scheduled intervals for data collectors” on page 11.
7. Copy any sample scripts from deployment package.
8. Install DPM Management Console.  Will be used for PowerShell collectors are written utilizing DPM Cmdlets
9. Install SQL Management Studio
10. Install DPM Management Console.  Will be used for PowerShell collectors are written utilizing DPM Cmdlets
11. Install PowerGUI (optional)

4.1.2    SSRS

4.1.3    Web Server

4.1.4    Collector Server

4.1.5    Management Server

5         Appendix

5.1       Notes

5.1.1    Versioning

Specific software versions are mentioned throughout the document and may be substituted at deployment time for the latest version available.  Using the latest version would require an update to this document and notations of any deviations from the version used, mentioned below.

5.1.2    Next Phase

While automation using Orchestrator, PowerShell, or SCOM can be built upon this reporting tool, it is outside the scope of this document at the current revision.

5.2       ToDo’s

5.2.1    Create Deployment package

This package should include:

• Sample .sql collectors described in section 3.2.2, “Data Collection Queries”, on page 9.
• CREATE TABLE queries needed for the “Data Collection Queries” insertion of records into the SQL Server.
• Sample PowerShell scripts used to collect data from single servers
• Sample PowerShell script to run a .ps1 file against multiple DPM servers

5.2.2    Complete Deployment Section

Microsoft Data Protection Manager Enterprise Reporting Platform originally appeared on Shai Perednik.com on November 24, 2012.

Intro

While SCOM 2012 allows you to monitor and run commands against these hosts, the focus is still on managing individual DPM servers from a central location. IE if a backup fails on server A and another B, they don’t necessarily have any correlation and have to be managed separately.

The central console is designed for monitoring and not much else. In an enterprise solution the central console should manage and schedule the job to most efficiently adapt the load on the environment. It should also be able to manage storage as a collective whole and reallocate jobs to where resources are available. In short, distributed backups should fashion similar to distributed virtualization where there is no single point of failure and the environment is adaptable to increased load.

Workflow

Managing 8000 datasources is difficult at best. Even worse when filtering the failures, even at 5% is still 400 failures to manage. Datasources are often failures on their own and cannot be correlated to other failures. Each failure must be worked out individually and each one requires deep knowledge of DPM and every underlying component that plays in the backup procedure.

Proposed Solution

Diagram specific steps to be taken and logic paths to resolve common problems. This has been completed and aids new engineers in troubleshooting DPM. However, not all steps can be outlined and require a deeper logic process.

Monitoring

Problem

Monitoring is available on a per server basis where each must be individually managed, configured, and analyzed. SCOM2012 provides a central collection of alerts and states for monitoring, however, the views built in are not practical for our size of environment and will have to be heavily customized for internal administration. The focus on monitoring is still on a per server basis, where the admin must focus on issues that may be affecting an entire server.

Proposed Solution

The SCOM 2012 console will help in monitoring the overall environment, but many custom views will have to be created to properly monitor various aspects of the environment. Customized dashboards should be setup for each DPM engineer allowing to focus on their area of responsibility.

Reporting

Problem

Reporting is available on a per server basis where each must be individually managed, configured, and analyzed.

Proposed Solution

Today we rely on scripts developed internally to collect data from all DPM servers and publish reports the application teams use to track there backups. However, due to firewall rules across domains we are unable to collect data on non-trusted domains. We need to get these firewall rules and ports opened so that we can collect data across all environments. We need to put together a detailed list of reports and their fields that we need to improve administration of DPM and then develop those reports.

Capacity Planning

Problem

DPM has no built in tools to forest usage and analyze historical data. While some built in reports provide current and data dated few weeks back this lacks in two areas. One, the data is collected in each individual DPM server, and two does not have any manipulation tools for forecasting and trending data.

Proposed Solution

I will focus on this area an evaluate the tools we have in place that we can make better projections and plan our build outs more efficiently. Possibility of using Metreon Athene.

Self Healing of Failed Backups

We have consistently found that running a scheduled consistency check address inconsistent backups, yet the built in auto consistency check runs as soon as the DataSource becomes inconsistent. In many cases what caused the inconsistency is still an issue and simply waiting 30 min before running the consistency check would bring the DataSource on track. Microsoft DPM has shared a registry hack that will change this behavior to multiple consistency checks after a failure, but has warned us that this comes at a tedious hit to performance.

Proposed Solution

We will test out the affects of the registry hack provided by Microsoft as an initial option. However, DPM will need to improve it’s failure handling to improve this aspect. DPM should be able to query for VSS writer health, ping for connectivity, and other basic triage steps before requiring manual intervention. When a DPM engineer is engaged, we should be confident that DPM has exhausted all possible automated methods and be provided with a simple log of what was tried and what was failed. DPM also needs to be able to be integrated better into Microsoft and third party tools. IE DPM should be able to read into the SCOM database and know an alerts been raised that will caused backups to fail, wait for the alert to be cleared, then try the backup job. Similar to that, we should have the ability to integrate DPM into Inventory system so that the application is aware of intended reboot due to patching or a decommission and therefore disable backups.

Secondary Protection

Setting up a secondary DPM server involves manually selecting the same datasources on the secondary as exist on the primary. However, when working on the secondary, enumerating available datasources on the primary takes approximately 10 minutes per datasources. A Microsoft ticket has been opened to address this, however it is apparent this is simply a fault of the reliance on the diskshadow enumeration.

Any changes made to the primary must also be made on the secondary. Nothing happens automatically and the admin must consistently ensure that the Primary and Secondary are in sync, therefore increasing the workload twice fold.

Proposed Solution

I am still working out this aspect with the Microsoft DPM Support team, however am confident and have been told by MS Support, that this is an architectural problem as there are minimal issues when replicating a small DPM server with under 20 DataSources. DPM will need to drastically change in this area to be able to automatically adapt the secondary to what’s happened on the primary. We need to have the ability to say everything on server A gets replicated to Server B, but also have the flexibility to get granular and exclude certain datasources from replication. Reporting will also need to be improved so that we have detailed reports of what’s getting replicated and what is not.

Granularity of Detail for FileSystem Backups

Problem

When DPM is set to backup a volume or folder it reports that folder as backed up but provides no detail on the items below it in the built in reports.

Proposed Solution

Since the data is available in the Restore GUI via the catalog it should be exposed in the SQL backend and either Microsoft develop reports or provide us with the SQL schema so that we can develop reports and pass that data to the app teams (they have requested this level of detail).

SharePoint

The SharePoint team has put together articles how to backup/recover SharePoint, http://technet.microsoft.com/en-us/library/cc261687.aspx. However, the VSS writer should expose all the aspects of a service application or web app instead of relying on the SharePoint and DPM admin to ensure all the pieces are in place.

Proposed Solution

See email below:

The SharePoint team has put together articles how to backup/recover SharePoint, http://technet.microsoft.com/en-us/library/cc261687.aspx. However, many are in agreement that much of this work can and should be handled by the SharePoint VSS writer. As an example, the VSS writer should expose all the aspects of a service application or web app. The DPM admin could setup DPM to protect the farm which would then backup all content and applications. Should the SharePoint admin need a restore, they would simply look to DPM to handle a restore in place or to another server. Similar to what we can do with SQL today.

Another simple example is SharePoint configurations which some are stored in databases and some in the file system. Today we backup the entire volume and all SQL databases to ensure we get all of these files. It would be far more efficient if the SP VSS Writer exposed the location of these files. Then DPM could back them up accordingly.

I’ve included a diskshadow writer dump that shows what each writer is exposing. While the SharePoint VSS writer puts out a lot of information, there are three questions.

1. Can the VSS writer be expanded to include more information
2. Can the VSS writer be expanded to allow more control over built in SharePoint functions, IE service app backups.
3. Is DPM fully utilizing all of the information put out by the writer and using it to its full extent.

Hope this provides enough details to get the discussions started. This can only improve the DPM and SharePoint products multifold!

SQL

When backing up SQL databases in native backup mode DPM does not do any checks on the DB. When the DBAs run backups they have a step that runs a DBCC (consistency check) before backing up.

On the backend DPM utilizes SQL server, however, does a poor job of maintaining the backend. A recent check of a fairly new DPM host, ServerA, showed a significant amount of it’s databases as fragmented over the acceptable 30% fragmentation threshold.

Proposed Solution

DPM in it’s maintenance tasks should include online and offline maintenance of the SQL backend. There should be built in tools that allow the DPM admin to run these maintenance tasks and schedule more intensive offline commands to run at specific times.

GUI

There are inconsistencies in the GUI. For example, if you’ve setup system protection, then go back and setup volume protection, DPM will warn that you need to setup system protection.

Proposed Solution

There needs to be more time spent on debugging the GUI and ensuring simple bugs like the above don’t happen.

De-duplication

DPM has no de-duplication built in and as a result we are backing up many of the same files 1000s of times across our environments or the same data day after day. Windows Server 2012 features built in de-duplication, http://blogs.technet.com/b/filecab/archive/2012/05/21/introduction-to-data-deduplication-in-windows-server-2012.aspx which I’ve been told is being investigated for the next major release of DPM.

Proposed Solution

Microsoft will need to work closely with storage and de-duplication vendors to provide a hardware solution that is supported. Since DPM holds the data catalog, there is a fine line that needs to be determined as to which product manages the de-duplication and reconstitution. Most hardware solutions can present a CIFS or NFS share, neither of which DPM is able to manage in it’s built in storage pool.

Reliance on other technologies

DPM heavily relies on other technologies to do the heavy lifting. VSS writers to enumerate data available to backup, Diskshadow to enumerate the contents of the VSS writers, Windows Server Backup, WMI, SAMBA, amongst others. While this allows DPM to be nimble and rely on other products, it also means that troubleshooting DPM involves troubleshooting these other technologies. The engineer must not only be heavily versed in DPM, but all the intricacies of how these Microsoft components work together.

Proposed Solution

The individual components are a strong point to the Windows platform as each is responsible for it’s own. However, to properly manage failed backups DPM needs to do a better job of reporting on the state and managing these individual services, ie restart a service if needed. Since these are all Microsoft products, the DPM team should be able to utilize private APIs allowing the DPM product to better communicate and work with these tools. This can been interpreted also as a lack of communication between internal teams as the DPM team should be pushing other teams to expose more through there VSS writers. Exchange, for example, doesn’t even have a VSS writer.

Reboots Required

Any time an agent is upgraded or reinstalled a pending reboot flag is thrown in the registry and DPM gui. We have been able to clear this pending reboot flag hundreds of times with no impact to backup reliability or performance. Yet this flag if un-cleared will cause all backups and communication to the client to fail.

Proposed Solution

DPM should not be raising pending reboot flags where it’s not needed. The GUI should offer the user a warning and an option to clear the flag.

PowerShell

The DPM PowerShell is a powerful snap in that allows for pulling data out of DPM. However, when trying to do this remotely the DPM SnapIn version must match that of the host, down to the minor version. IE version 3.0.7706.0 will not communicate with 3.0.8158.0 or vise versa. Therefore scripting cross-servers becomes inhibitive as we must maintain a console for each versions.

Adding Agents, creating/modifying protection groups, amongst other CUD, create/update/delete, operations have proven difficult if not nearly impossible without a deep knowledge of PowerShell and DPM innards.

Proposed Solution

The DPM SnapIn should be version agnostic. If using API calls not available due to the target being an old version, the system should throw an error or revert to a deprecated command set or workaround.

Regarding CUD operations , Microsoft needs to better document how to handle batch operations and automate the many manual steps it takes through the GUI. Sample scripts should be provided of common tasks done through the GUI.

Tape Handling

Similar to the lack of central management mentioned above, tape handling is also left to be handled by individual servers. Microsoft has made an attempt to remediate this with library sharing, however this is still a problem area as various DPM server simply share a tape library. DPM server A is aware that it doesn’t own tape A, but it doesn’t know who owns it. Nor can it read and identify the contents of that tape against the shared library.

Proposed Solution

This part of the product will depend on a central management console and commander that will be aware of all the tape jobs and be able to assign them to available media servers. This requires a large shift away from the single node architecture of present DPM and a move towards a centralized platform with a command head.

Agnostic Operation

Along the lines of tape handling above, DPM servers are not aware of what the other one does. Because there is no central management to control each server, it is left up to the DPM admin to best distribute the load across the DPM servers and storage array. Another issue arises when a DPM server is unavailable as all backups on that server fail until that host is fixed. In a centralized environment the central manger would recognize that backups are at risk due to failed hardware and migrate jobs to a healthy server to continue backups. Backups on the storage backend might be lost but at least jobs can seamlessly continue to run. Another problem is when the server is down due to windows issues, the storage pool is inaccessible and although the data is intact, the DPM server that holds the catalog cannot be accessed. While we can build another server and load the original catalog this is additional time that we cannot execute backups or restores.

Proposed Solution

Similar solution to the tape handling above.

Auto-Grow

While the auto-grow feature of DPM2010 was a huge improvement allowing replica and recovery volumes to grow DPM still handles this growth inefficiently and is limited by the OS (Windows Server 2012 will lift this limitation). When a backup job fails due to lack of disk, DPM grows the replica or recovery volume by 20% and then leaves the job in a failed state till the next cycle which might be next hour or next day. Each time a volume grows it consumes the total of extents available at the OS level. As that limit is approached performance takes a exponential impact until the limit is reached and the host rebuilt.

Proposed Solution

When a volume auto-grows the backup job should restart automatically and not wait till the next scheduled job. Another point is when auto-growing it should not be a firm 20% but calculation based on historical growth of the datasource and projected growth.

Remote Desktop Login

Problem

Login takes significant time due to enumeration of volumes

Proposed Solution

Delay the disk enumeration upon login. This is a windows specific issue and not DPM, but as a result DPM administration suffers. This problem can be circumvented in DPM 2012 with the remote console utility.

SQL Self Service Restore

Problem

Setup tool on the DPM server

Each SQL Server Instance must be added to the self-service recovery option by name. There is no dropdown allowing the user to choose existing databases already protected on the DPM server. Also, an entire SQL Server cannot be added with a wildcard.

Using the SQL Server names presented in the GUI fails. Running the suggested Get-Datasources cmdlet produces the same results as the GUI:

Note that various entries above do not work, including all other names in the cluster and excluding/including the SQL Instance name. Since it’s the default MSSQLSERVER instance, the GUI excludes the instance name in various places. Also, permissions have been verified working in SQL MGMT Studio for the DPM Service account and my account.

Recovery Wizard

SQL admin must know which DPM server to connect to for recovery. No way to query by SQL instance since there is no centralized database of SQL ; DPM host attachments.

Welcome page is blank. Shouldn’t the welcome page have text on there guiding the user through the process?

Available Restore path if the user chooses to restore to another SQL server is limited to pre-defined SQL servers setup in the DPM console. User is not able to choose another server.

Proposed Solution

Setup tool on the DPM server

The setup tool should provide the user with an enumerated list of SQL servers they can add. Similar to the enumeration when adding a SQL server into protection or restoring of SQL data.

Restore Wizard

If a generic user is running this process then this is an acceptable behavior. However, if a DBA is running the process and authenticated, since the tool does allow for permissions groups, then the DBA should be allowed to restore to any SQL server. At minimum to any SQL server connected to that DPM server.

Retention Period

Problem

Retention period in the GUI is displayed as X number of days:

However, as we have observed the data is retained in this example as 1 recovery point per day for 30 days. The days can be non consecutive. 2 examples to better illustrate the problem:

1. If backups are missed on some days they do not count towards the 30 number above since there is no recovery point that day. Therefore, in this case you could have data on disk for X number of days missed + the number of days of successful backups.
   1. EX. 30 successful recovery points
   2. 10 days missed
   3. Oldest recovery point would be today minus 40 days.

2. Another option is where retention is set to 30 days, but instead of choosing everyday, you choose Sunday only. Note that this is a recommended setup for system state and other low churn non-critical data. You would therefore have a backup for 30 days, but since there’s only 1 backup per week, that equals 30 weeks of backups!
   1. EX. 30 Successful recovery points
   2. 1 Recovery Point per week ~ 1 recovery point per day once a week
   3. Oldest recovery point 30 weeks old.

While #2 is not a huge problem and requires more logic when setting up, it does pose a similar problem to #1 where if backups are missed, the retention now gets extended.

#1 however is a big problem for 3 reasons:

1. Certain applications are only approved to retain 30 calendar days of data by legal and security teams. So #1 above causes us to regularly break this policy and deviate into violation.

2. Since data is retained longer then its been planned for this often throws of disk utilization projections and causes excess use of data. Since old backups out of SLA are not expiring and new backups require more disk space, we constantly find ourselves adding more disk.
3. There is no way within the GUI or PowerShell to manually expire old backups of a particular datasource. While this might be possible via SQL, it is not supported, nor do we know how to do this.

Proposed Solution

There are several options in which this could be remediated:

1. A GUI/PowerShell option could be provided per data source to manually expire data older than x number of days on a datasource.
2. Change the behavior or allow the user to choose weather retention is:
   1. 30 calendar days <;–Prefered
   2. 30 days of available recovery points

What’s Wrong with Microsoft Data Protection Manager? originally appeared on Shai Perednik.com on November 22, 2012.

*Note the mention of MSAStorageDetails.ps1, this will be posted shortly.

The disk# in DPM matches the disk # in windows. In this case #2

Even after removing it from the pool the dpm cli still sees it: $disk = Get-DPMDisk DPMSERVERNAME| sort netdiskid

But obviously can’t remove it as it’s really not in the pool:

(Note I removed disk #2 from the pool and formatted it with the name F)

In the HP DSM Manager the disk # is still #2 but it’s name is MPIODisk1

Using the MSA collection script @ E:\Scripts\ssh\Get-MSAStorageDetails.ps1 -List .\msalist.csv –XMLFormat and notepad++ to search the results for the LUNID 60CFF0010E35497AECB4E1000

Note that the letters in the middle are what matters: “E35497AECB4E” in this case which matches the exists in the WWN field.

<PROPERTY name=”wwn” type=”string”>600C0FF00010E35497AECB4E01000000</PROPERTY>

And scrolling up in the object we can find the volume name,diskgroup, and use the filename for the MSA the lun is attached to.

You can compare the WWN to the Serial Number

I off-lined the disk on the host and cleared the mapping on the MSA

MPIODisk1 is no longer listed on the HP DSMManager or in the DPM CLI

Disk can now safely be deleted from the MSA

How to Determine what MSA a LUN is Attached to originally appeared on Shai Perednik.com on November 17, 2012.