Quantcast
Channel: PowerShell
Viewing all 1519 articles
Browse latest View live

PowerShell Constrained Language mode and the Dot-Source Operator

$
0
0

PowerShell Constrained Language mode and the Dot-Source Operator

PowerShell works with application control systems, such as AppLocker and Windows Defender Application Control (WDAC), by automatically running in
ConstrainedLanguage mode. ConstrainedLanguage mode restricts some exploitable aspects of PowerShell while still giving you a rich shell to run commands and scripts in. This is different from usual application white listing rules, where an application is either allowed to run or not.

But there are times when the full power of PowerShell is needed, so we allow script files to run in FullLanguage mode when they are trusted by the policy. Trust can be indicated through file signing or other policy mechanisms such as file hash. However, script typed into the interactive shell is always run constrained.

Since PowerShell can run script in both Full and Constrained language modes, we need to protect the boundary between them. We don’t want to leak variables or functions between sessions running in different language modes.

The PowerShell dot-source operator brings script files into the current session scope. It is a way to reuse script. All script functions and variables defined in the script file become part of the script it is dot sourced into. It is like copying and pasting text from the script file directly into your script.

# HelperFn1, HelperFn2 are defined in HelperFunctions.ps1
# Dot-source the file here to get access to them (no need to copy/paste)
. c:\Scripts\HelperFunctions.ps1
HelperFn1
HelperFn2

This presents a problem when language modes are in effect with system application control. If an untrusted script is dot-sourced into a script with full trust then it has access to all those functions that run in FullLanguage mode, which can result in application control bypass through arbitrary code execution or privilege escalation. Consequently, PowerShell prevents this by throwing an error when dot-sourcing is attempted across language modes.

Example 1:

System is in WDAC policy lock down. To start with, neither script is trusted and so both run in ConstrainedLanguage mode. But the HelperFn1 function uses method invocation which isn’t allowed in that mode.

PS> type c:\MyScript.ps1
Write-Output "Dot sourcing MyHelper.ps1 script file"
. c:\MyHelper.ps1
HelperFn1
PS> type c:\MyHelper.ps1
function HelperFn1
{
    "Language mode: $($ExecutionContext.SessionState.LanguageMode)"
    [System.Console]::WriteLine("This can only run in FullLanguage mode!")
}
PS> c:\MyScript.ps1
Dot sourcing MyHelper.ps1 script file
Language mode: ConstrainedLanguage
Cannot invoke method. Method invocation is supported only on core types in this language mode.
At C:\MyHelper.ps1:4 char:5
+     [System.Console]::WriteLine("This cannot run in ConstrainedLangua ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [], RuntimeException
    + FullyQualifiedErrorId : MethodInvocationNotSupportedInConstrainedLanguage

Both scripts are untrusted and run in ConstrainedLanguage mode, so dot-sourcing the MyHelper.ps1 file works. However, the HelperFn1 function performs method invocation that is not allowed in ConstrainedLanguage and fails when run. MyHelper.ps1 needs to be signed as trusted so it can run at FullLanguage.

Next we have mixed language modes. MyHelper.ps1 is signed and trusted, but MyScript.ps1 is not.

PS> c:\MyScript.ps1
Dot sourcing MyHelper.ps1 script file
C:\MyHelper.ps1 : Cannot dot-source this command because it was defined in a different language mode. To invoke this command without importing its contents, omit the '.' operator.
At C:\MyScript.ps1:2 char:1
+ . 'c:\MyHelper.ps1'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [MyHelper.ps1], NotSupportedException
    + FullyQualifiedErrorId : DotSourceNotSupported,MyHelper.ps1
...

And we get a dot-source error because we are trying to dot-source script that has a different language mode than the session it is being dot-sourced into.

Finally, we sign as trusted both script files and everything works.

PS> c:\MyScript.ps1
Dot sourcing MyHelper.ps1 script file
Language mode: FullLanguage
This can only run in FullLanguage mode!

The lesson here is to ensure all script components run in the same language mode on policy locked down systems. If one component must run in FullLanguage mode, then all components should run in FullLanguage mode. This means validating that each component is safe to run in FullLanguage and indicating they are trusted to the application control policy.

So this solves all language mode problems, right? If FullLanguage is not needed then just ensure all script components run untrusted, which is the default condition. If they require FullLanguage then carefully validate all components and mark them as trusted. Unfortuantely, there is one case where this best practice doesn’t work.

PowerShell Profile File

The PowerShell profile file (profile.ps1) is loaded and run at PowerShell start up. If that script requires FullLanguage mode on policy lock down systems, you just validate and sign the file as trusted, right?

Example 2:

PS> type c:\users\<user>\Documents\WindowsPowerShell\profile.ps1
Write-Output "Running Profile"
[System.Console]::WriteLine("This can only run in FullLanguage!")
# Sign file so it is trusted and will run in FullLanguage mode
PS> Set-AuthenticodeSignature -FilePath .\Profile.ps1 -Certificate $myPolicyCert
# Start a new PowerShell session and run the profile script
PS> powershell.exe
Windows PowerShell
Copyright (C) Microsoft Corporation. All rights reserved.
C:\Users\<user>\Documents\WindowsPowerShell\profile.ps1 : Cannot dot-source this command because it was defined in a different language mode. To invoke this command without importing its contents, omit the '.' operator.
At line:1 char:1
+ . 'C:\Users\<user>\Documents\WindowsPowerShell\profile.ps1'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [profile.ps1], NotSupportedException
    + FullyQualifiedErrorId : DotSourceNotSupported,profile.ps1

What gives? The profile.ps1 file was signed and is policy trusted. Why the error?
Well, the issue is that PowerShell dot-sources the profile.ps1 file into the default PowerShell session, which must run in ConstrainedLanguage because of the policy. So we are attempting to dot-source a FullLanguage script into a ConstrainedLanguage session, and that is not allowed. This is a catch 22 because if the profile.ps1 is not signed, it may not run if it needs FullLanguage privileges (e.g., invoke methods). But if you sign it, it still won’t run because of how it is dot-sourced into the current ConstrainedLanguage interactive session.

Unfortunately, the only solution is to keep the profile.ps1 file fairly simple so that it does not need FullLanguage, and refrain from making it trusted. Keep in mind that this is only an issue when running with application control policy. Otherwise, language modes do not come into play and PowerShell profile files run normally.

Paul Higinbotham
Senior Software Engineer
PowerShell Team


DSC Resource Kit Release November 2018

$
0
0

We just released the DSC Resource Kit!

This release includes updates to 9 DSC resource modules. In the past 6 weeks, 61 pull requests have been merged and 67 issues have been closed, all thanks to our amazing community!

The modules updated in this release are:

  • AuditPolicyDsc
  • DFSDsc
  • NetworkingDsc
  • SecurityPolicyDsc
  • SharePointDsc
  • StorageDsc
  • xBitlocker
  • xExchange
  • xHyper-V

For a detailed list of the resource modules and fixes in this release, see the Included in this Release section below.

Our latest community call for the DSC Resource Kit was supposed to be today, November 28, but the public link to the call expired, so the call was cancelled. I will update the link for next time. If there is interest in rescheduling this call, the new call time will be announced on Twitter (@katiedsc or @migreene) The call for the next release cycle is also getting moved a week later than usual to January 9 at 12PM (Pacific standard time). Join us to ask questions and give feedback about your experience with the DSC Resource Kit.

The next DSC Resource Kit release will be on Wednesday, January 9.

We strongly encourage you to update to the newest version of all modules using the PowerShell Gallery, and don’t forget to give us your feedback in the comments below, on GitHub, or on Twitter (@PowerShell_Team)!

Please see our documentation here for information on the support of these resource modules.

Included in this Release

You can see a detailed summary of all changes included in this release in the table below. For past release notes, go to the README.md or CHANGELOG.md file on the GitHub repository page for a specific module (see the How to Find DSC Resource Modules on GitHub section below for details on finding the GitHub page for a specific module).

Module Name Version Release Notes
AuditPolicyDsc 1.3.0.0
  • Update LICENSE file to match the Microsoft Open Source Team standard.
  • Added the AuditPolicyGuid resource.
DFSDsc 4.2.0.0
  • Add support for modifying staging quota size in MSFT_DFSReplicationGroupMembership – fixes Issue 77.
  • Refactored module folder structure to move resource to root folder of repository and remove test harness – fixes Issue 74.
  • Updated Examples to support deployment to PowerShell Gallery scripts.
  • Remove exclusion of all tags in appveyor.yml, so all common tests can be run if opt-in.
  • Added .VSCode settings for applying DSC PSSA rules – fixes Issue 75.
  • Updated LICENSE file to match the Microsoft Open Source Team standard – fixes Issue 79
NetworkingDsc 6.2.0.0
  • Added .VSCode settings for applying DSC PSSA rules – fixes Issue 357.
  • Updated LICENSE file to match the Microsoft Open Source Team standard – fixes Issue 363
  • MSFT_NetIPInterface:
    • Added a new resource for configuring the IP interface settings for a network interface.
SecurityPolicyDsc 2.6.0.0
  • Added SecurityOption – Network_access_Restrict_clients_allowed_to_make_remote_calls_to_SAM
  • Bug fix – Issue 105 – Spelling error in SecurityOption”User_Account_Control_Behavior_of_the_elevation_prompt_for_standard_users”
  • Bug fix – Issue 90 – Corrected value for Microsoft_network_server_Server_SPN_target_name_validation_level policy
SharePointDsc 3.0.0.0
  • Changes to SharePointDsc
    • Added support for SharePoint 2019
    • Added CredSSP requirement to the Readme files
    • Added VSCode Support for running SharePoint 2019 unit tests
    • Removed the deprecated resources SPCreateFarm and SPJoinFarm (replaced in v2.0 by SPFarm)
  • SPBlobCacheSettings
    • Updated the Service Instance retrieval to be language independent
  • SPConfigWizard
    • Fixed check for Ensure=Absent in the Set method
  • SPInstallPrereqs
    • Added support for detecting updated installation of Microsoft Visual C++ 2015/2017 Redistributable (x64) for SharePoint 2016 and SharePoint 2019.
  • SPSearchContentSource
    • Added support for Business Content Source Type
  • SPSearchMetadataCategory
    • New resource added
  • SPSearchServiceApp
    • Updated resource to make sure the presence of the service app proxy is checked and created if it does not exist
  • SPSecurityTokenServiceConfig
    • The resource only tested for the Ensure parameter. Added more parameters
  • SPServiceAppSecurity
    • Added support for specifying array of access levels.
    • Changed implementation to use Grant-SPObjectSecurity with Replace switch instead of using a combination of Revoke-SPObjectSecurity and Grant-SPObjectSecurity
    • Added all supported access levels as available values.
    • Removed unknown access levels: Change Permissions, Write, and Read
  • SPUserProfileProperty
    • Removed obsolete parameters (MappingConnectionName, MappingPropertyName, MappingDirection) and introduced new parameter PropertyMappings
  • SPUserProfileServiceApp
    • Updated the check for successful creation of the service app to throw an error if this is not done correctly The following changes will break v2.x and earlier configurations that use these resources:
  • Implemented IsSingleInstance parameter to force that the resource can only be used once in a configuration for the following resources:
    • SPAntivirusSettings
    • SPConfigWizard
    • SPDiagnosticLoggingSettings
    • SPFarm
    • SPFarmAdministrators
    • SPInfoPathFormsServiceConfig
    • SPInstall
    • SPInstallPrereqs
    • SPIrmSettings
    • SPMinRoleCompliance
    • SPPasswordChangeSettings
    • SPProjectServerLicense
    • SPSecurityTokenServiceConfig
    • SPShellAdmin
  • Standardized Url/WebApplication parameter to default WebAppUrl parameter for the following resources:
    • SPDesignerSettings
    • SPFarmSolution
    • SPSelfServiceSiteCreation
    • SPWebAppBlockedFileTypes
    • SPWebAppClientCallableSettings
    • SPWebAppGeneralSettings
    • SPWebApplication
    • SPWebApplicationAppDomain
    • SPWebAppSiteUseAndDeletion
    • SPWebAppThrottlingSettings
    • SPWebAppWorkflowSettings
  • Introduced new mandatory parameters
    • SPSearchResultSource: Added option to create Result Sources at different scopes.
    • SPServiceAppSecurity: Changed parameter AccessLevel to AccessLevels in MSFT_SPServiceAppSecurityEntry to support array of access levels.
    • SPUserProfileProperty: New parameter PropertyMappings
SharePointDsc 3.1.0.0
  • Changes to SharePointDsc
    • Updated LICENSE file to match the Microsoft Open Source Team standard.
  • ProjectServerConnector
    • Added a file hash validation check to prevent the ability to load custom code into the module.
  • SPFarm
    • Fixed localization issue where TypeName was in the local language.
  • SPInstallPrereqs
    • Updated links in the Readme.md file to docs.microsoft.com.
    • Fixed required prereqs for SharePoint 2019, added MSVCRT11.
  • SPManagedMetadataServiceApp
    • Fixed issue where Get-TargetResource method throws an error when the service app proxy does not exist.
  • SPSearchContentSource
    • Corrected issue where the New-SPEnterpriseSearchCrawlContentSource cmdlet was called twice.
  • SPSearchServiceApp
    • Fixed issue where Get-TargetResource method throws an error when the service application pool does not exist.
    • Implemented check to make sure cmdlets are only executed when it actually has something to update.
    • Deprecated WindowsServiceAccount parameter and moved functionality to new resource (SPSearchServiceSettings).
  • SPSearchServiceSettings
    • Added new resource to configure search service settings.
  • SPServiceAppSecurity
    • Fixed unavailable utility method (ExpandAccessLevel).
    • Updated the schema to no longer specify username as key for the sub class.
  • SPUserProfileServiceApp
    • Fixed issue where localized versions of Windows and SharePoint would throw an error.
  • SPUserProfileSyncConnection
    • Corrected implementation of Ensure parameter.
StorageDsc 4.3.0.0
  • WaitForDisk:
    • Added readonly-property isAvailable which shows the current state of the disk as a boolean – fixes Issue 158.
xBitlocker 1.3.0.0
  • Update appveyor.yml to use the default template.
  • Added default template files .gitattributes, and .vscode settings.
  • Fixes most PSScriptAnalyzer issues.
  • Fix issue where AutoUnlock is not set if requested, if the disk was originally encrypted and AutoUnlock was not used.
  • Add remaining Unit Tests for xBitlockerCommon.
  • Add Unit tests for MSFT_xBLTpm
  • Add remaining Unit Tests for xBLAutoBitlocker
  • Add Unit tests for MSFT_xBLBitlocker
  • Moved change log to CHANGELOG.md file
  • Fixed Markdown validation warnings in README.md
  • Added .MetaTestOptIn.json file to root of module
  • Add Integration Tests for module resources
  • Rename functions with improper Verb-Noun constructs
  • Add comment based help to any functions without it
  • Update Schema.mof Description fields
  • Fixes issue where Switch parameters are passed to Enable-Bitlocker even if the corresponding DSC resource parameter was set to False (Issue 12)
xExchange 1.25.0.0
  • Opt-in for the common test flagged Script Analyzer rules (issue 234).
  • Opt-in for the common test testing for relative path length.
  • Removed the property PSDscAllowPlainTextPassword from all examples so the examples are secure by default. The property PSDscAllowPlainTextPassword was previously needed to (test) compile the examples in the CI pipeline, but now the CI pipeline is using a certificate to compile the examples.
  • Opt-in for the common test that validates the markdown links.
  • Fix typo of the word “Certificate” in several example files.
  • Add spaces between array members.
  • Add initial set of Unit Tests (mostly Get-TargetResource tests) for all remaining resource files.
  • Add WaitForComputerObject parameter to xExchWaitForDAG
  • Add spaces between comment hashtags and comments.
  • Add space between variable types and variables.
  • Fixes issue where xExchMailboxDatabase fails to test for a Journal Recipient because the module did not load the Get-Recipient cmdlet (335).
  • Fixes broken Integration tests in MSFT_xExchMaintenanceMode.Integration.Tests.ps1 (336).
  • Fix issue where Get-ReceiveConnector against an Absent connector causes an error to be logged in the MSExchange Management log.
  • Rename poorly named functions in xExchangeDiskPart.psm1 and MSFT_xExchAutoMountPoint.psm1, and add comment based help.
xHyper-V 3.14.0.0
  • MSFT_xVMHost:
    • Added support to Enable / Disable VM Live Migration. Fixes Issue 155.

How to Find Released DSC Resource Modules

To see a list of all released DSC Resource Kit modules, go to the PowerShell Gallery and display all modules tagged as DSCResourceKit. You can also enter a module’s name in the search box in the upper right corner of the PowerShell Gallery to find a specific module.

Of course, you can also always use PowerShellGet (available starting in WMF 5.0) to find modules with DSC Resources:

# To list all modules that tagged as DSCResourceKit
Find-Module -Tag DSCResourceKit 
# To list all DSC resources from all sources 
Find-DscResource

Please note only those modules released by the PowerShell Team are currently considered part of the ‘DSC Resource Kit’ regardless of the presence of the ‘DSC Resource Kit’ tag in the PowerShell Gallery.

To find a specific module, go directly to its URL on the PowerShell Gallery:
http://www.powershellgallery.com/packages/< module name >
For example:
http://www.powershellgallery.com/packages/xWebAdministration

How to Install DSC Resource Modules From the PowerShell Gallery

We recommend that you use PowerShellGet to install DSC resource modules:

Install-Module -Name < module name >

For example:

Install-Module -Name xWebAdministration

To update all previously installed modules at once, open an elevated PowerShell prompt and use this command:

Update-Module

After installing modules, you can discover all DSC resources available to your local system with this command:

Get-DscResource

How to Find DSC Resource Modules on GitHub

All resource modules in the DSC Resource Kit are available open-source on GitHub.
You can see the most recent state of a resource module by visiting its GitHub page at:
https://github.com/PowerShell/< module name >
For example, for the CertificateDsc module, go to:
https://github.com/PowerShell/CertificateDsc.

All DSC modules are also listed as submodules of the DscResources repository in the DscResources folder and the xDscResources folder.

How to Contribute

You are more than welcome to contribute to the development of the DSC Resource Kit! There are several different ways you can help. You can create new DSC resources or modules, add test automation, improve documentation, fix existing issues, or open new ones.
See our contributing guide for more info on how to become a DSC Resource Kit contributor.

If you would like to help, please take a look at the list of open issues for the DscResources repository.
You can also check issues for specific resource modules by going to:
https://github.com/PowerShell/< module name >/issues
For example:
https://github.com/PowerShell/xPSDesiredStateConfiguration/issues

Your help in developing the DSC Resource Kit is invaluable to us!

Questions, comments?

If you’re looking into using PowerShell DSC, have questions or issues with a current resource, or would like a new resource, let us know in the comments below, on Twitter (@PowerShell_Team), or by creating an issue on GitHub.

Katie Kragenbrink
Software Engineer
PowerShell DSC Team
@katiedsc (Twitter)
@kwirkykat (GitHub)

DSC Resource Kit Release January 2019

$
0
0

We just released the DSC Resource Kit!

This release includes updates to 14 DSC resource modules. In the past 6 weeks, 41 pull requests have been merged and 54 issues have been closed, all thanks to our amazing community!

The modules updated in this release are:

  • ActiveDirectoryCSDsc
  • AuditPolicyDsc
  • CertificateDsc
  • ComputerManagementDsc
  • NetworkingDsc
  • SecurityPolicyDsc
  • SqlServerDsc
  • StorageDsc
  • xActiveDirectory
  • xBitlocker
  • xExchange
  • xFailOverCluster
  • xHyper-V
  • xWebAdministration

Several of these modules were released to remove the hidden files/folders from this issue. This issue should now be fixed for all modules except DFSDsc which is waiting for some fixes to its tests.

For a detailed list of the resource modules and fixes in this release, see the Included in this Release section below.

Our latest community call for the DSC Resource Kit was today, January 9. A recording is available on YouTube here. Join us for the next call at 12PM (Pacific time) on February 13 to ask questions and give feedback about your experience with the DSC Resource Kit.

The next DSC Resource Kit release will be on Wednesday, February 20.

We strongly encourage you to update to the newest version of all modules using the PowerShell Gallery, and don’t forget to give us your feedback in the comments below, on GitHub, or on Twitter (@PowerShell_Team)!

Please see our documentation here for information on the support of these resource modules.

Included in this Release

You can see a detailed summary of all changes included in this release in the table below. For past release notes, go to the README.md or CHANGELOG.md file on the GitHub repository page for a specific module (see the How to Find DSC Resource Modules on GitHub section below for details on finding the GitHub page for a specific module).

Module Name Version Release Notes
ActiveDirectoryCSDsc 3.1.0.0
  • Updated LICENSE file to match the Microsoft Open Source Team standard.
  • Added .VSCode settings for applying DSC PSSA rules – fixes Issue 60.
  • Added fix for two tier PKI deployment fails on initial deployment, not error – fixes Issue 57.
AuditPolicyDsc 1.4.0.0
  • Explicitly removed extra hidden files from release package
CertificateDsc 4.3.0.0
  • Updated certificate import to only use Import-CertificateEx – fixes Issue 161
  • Update LICENSE file to match the Microsoft Open Source Team standard -fixes Issue 164.
  • Opted into Common Tests – fixes Issue 168:
    • Required Script Analyzer Rules
    • Flagged Script Analyzer Rules
    • New Error-Level Script Analyzer Rules
    • Custom Script Analyzer Rules
    • Validate Example Files To Be Published
    • Validate Markdown Links
    • Relative Path Length
  • CertificateExport:
    • Fixed bug causing PFX export with matchsource enabled to fail – fixes Issue 117
ComputerManagementDsc 6.1.0.0
  • Updated LICENSE file to match the Microsoft Open Source Team standard. Fixes Issue 197.
  • Explicitly removed extra hidden files from release package
NetworkingDsc 6.3.0.0
  • MSFT_IPAddress:
    • Updated to allow retaining existing addresses in order to support cluster configurations as well
SecurityPolicyDsc 2.7.0.0
  • Bug fix – Issue 83 – Network_access_Remotely_accessible_registry_paths_and_subpaths correctly applies multiple paths
  • Update LICENSE file to match the Microsoft Open Source Team standard
SqlServerDsc 12.2.0.0
  • Changes to SqlServerDsc
    • During testing in AppVeyor the Build Worker is restarted in the install step to make sure the are no residual changes left from a previous SQL Server install on the Build Worker done by the AppVeyor Team (issue 1260).
    • Code cleanup: Change parameter names of Connect-SQL to align with resources.
    • Updated README.md in the Examples folder.
      • Added a link to the new xADObjectPermissionEntry examples in ActiveDirectory, fixed a broken link and a typo. Adam Rush (@adamrushuk)
    • Change to SqlServerLogin so it doesn”t check properties for absent logins.
StorageDsc 4.4.0.0
  • Refactored module folder structure to move resource to root folder of repository and remove test harness – fixes Issue 169.
  • Updated Examples to support deployment to PowerShell Gallery scripts.
  • Removed limitation on using Pester 4.0.8 during AppVeyor CI.
  • Moved the Code of Conduct text out of the README.md and into a CODE_OF_CONDUCT.md file.
  • Explicitly removed extra hidden files from release package
xActiveDirectory 2.23.0.0
  • Explicitly removed extra hidden files from release package
xBitlocker 1.4.0.0
  • Change double quoted string literals to single quotes
  • Add spaces between array members
  • Add spaces between variable types and variable names
  • Add spaces between comment hashtag and comments
  • Explicitly removed extra hidden files from release package
xExchange 1.26.0.0
  • Add support for Exchange Server 2019
  • Added additional parameters to the MSFT_xExchUMService resource
  • Rename improperly named functions, and add comment based help in MSFT_xExchClientAccessServer, MSFT_xExchDatabaseAvailabilityGroupNetwork, MSFT_xExchEcpVirtualDirectory, MSFT_xExchExchangeCertificate, MSFT_xExchImapSettings.
  • Added additional parameters to the MSFT_xExchUMCallRouterSettings resource
  • Rename improper function names in MSFT_xExchDatabaseAvailabilityGroup, MSFT_xExchJetstress, MSFT_xExchJetstressCleanup, MSFT_xExchMailboxDatabase, MSFT_xExchMailboxDatabaseCopy, MSFT_xExchMailboxServer, MSFT_xExchMaintenanceMode, MSFT_xExchMapiVirtualDirectory, MSFT_xExchOabVirtualDirectory, MSFT_xExchOutlookAnywhere, MSFT_xExchOwaVirtualDirectory, MSFT_xExchPopSettings, MSFT_xExchPowershellVirtualDirectory, MSFT_xExchReceiveConnector, MSFT_xExchWaitForMailboxDatabase, and MSFT_xExchWebServicesVirtualDirectory.
  • Add remaining unit and integration tests for MSFT_xExchExchangeServer.
xFailOverCluster 1.12.0.0
  • Explicitly removed extra hidden files from release package
xHyper-V 3.15.0.0
  • Explicitly removed extra hidden files from release package
xWebAdministration 2.4.0.0
  • Explicitly removed extra hidden files from release package

How to Find Released DSC Resource Modules

To see a list of all released DSC Resource Kit modules, go to the PowerShell Gallery and display all modules tagged as DSCResourceKit. You can also enter a module’s name in the search box in the upper right corner of the PowerShell Gallery to find a specific module.

Of course, you can also always use PowerShellGet (available starting in WMF 5.0) to find modules with DSC Resources:

# To list all modules that tagged as DSCResourceKit
Find-Module -Tag DSCResourceKit 
# To list all DSC resources from all sources 
Find-DscResource

Please note only those modules released by the PowerShell Team are currently considered part of the ‘DSC Resource Kit’ regardless of the presence of the ‘DSC Resource Kit’ tag in the PowerShell Gallery.

To find a specific module, go directly to its URL on the PowerShell Gallery:
http://www.powershellgallery.com/packages/< module name >
For example:
http://www.powershellgallery.com/packages/xWebAdministration

How to Install DSC Resource Modules From the PowerShell Gallery

We recommend that you use PowerShellGet to install DSC resource modules:

Install-Module -Name < module name >

For example:

Install-Module -Name xWebAdministration

To update all previously installed modules at once, open an elevated PowerShell prompt and use this command:

Update-Module

After installing modules, you can discover all DSC resources available to your local system with this command:

Get-DscResource

How to Find DSC Resource Modules on GitHub

All resource modules in the DSC Resource Kit are available open-source on GitHub.
You can see the most recent state of a resource module by visiting its GitHub page at:
https://github.com/PowerShell/< module name >
For example, for the CertificateDsc module, go to:
https://github.com/PowerShell/CertificateDsc.

All DSC modules are also listed as submodules of the DscResources repository in the DscResources folder and the xDscResources folder.

How to Contribute

You are more than welcome to contribute to the development of the DSC Resource Kit! There are several different ways you can help. You can create new DSC resources or modules, add test automation, improve documentation, fix existing issues, or open new ones.
See our contributing guide for more info on how to become a DSC Resource Kit contributor.

If you would like to help, please take a look at the list of open issues for the DscResources repository.
You can also check issues for specific resource modules by going to:
https://github.com/PowerShell/< module name >/issues
For example:
https://github.com/PowerShell/xPSDesiredStateConfiguration/issues

Your help in developing the DSC Resource Kit is invaluable to us!

Questions, comments?

If you’re looking into using PowerShell DSC, have questions or issues with a current resource, or would like a new resource, let us know in the comments below, on Twitter (@PowerShell_Team), or by creating an issue on GitHub.

Katie Kragenbrink
Software Engineer
PowerShell DSC Team
@katiedsc (Twitter)
@kwirkykat (GitHub)

Windows Security change affecting PowerShell

$
0
0

Windows Security change affecting PowerShell

January 9, 2019

The recent (1/8/2019) Windows security patch CVE-2019-0543, has introduced a breaking change for a PowerShell remoting scenario. It is a narrowly scoped scenario that should have low impact for most users.

The breaking change only affects local loopback remoting, which is a PowerShell remote connection made back to the same machine, while using non-Administrator credentials.

PowerShell remoting endpoints do not allow access to non-Administrator accounts by default. However, it is possible to modify endpoint configurations, or create new custom endpoint configurations, that do allow non-Administrator account access. So you would not be affected by this change, unless you explicitly set up loopback endpoints on your machine to allow non-Administrator account access.

Example of broken loopback scenario

# Create endpoint that allows Users group access
PS > Register-PSSessionConfiguration -Name MyNonAdmin -SecurityDescriptorSddl 'O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;BU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)' -Force

# Create non-Admin credential
PS > $nonAdminCred = Get-Credential ~\NonAdminUser

# Create a loopback remote session to custom endpoint using non-Admin credential
PS > $session = New-PSSession -ComputerName localhost -ConfigurationName MyNonAdmin -Credential $nonAdminCred

New-PSSession : [localhost] Connecting to remote server localhost failed with the following error message : The WSMan
service could not launch a host process to process the given request.  Make sure the WSMan provider host server and
proxy are properly registered. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:1
+ New-PSSession -ComputerName localhost -ConfigurationName MyNonAdmin - ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
   gTransportException
    + FullyQualifiedErrorId : -2146959355,PSSessionOpenFailed

The above example fails only when using non-Administrator credentials, and the connection is made back to the same machine (localhost). Administrator credentials still work. And the above scenario will work when remoting off-box to another machine.

Example of working loopback scenario

# Create Admin credential
PS > $adminCred = Get-Credential ~\AdminUser

# Create a loopback remote session to custom endpoint using Admin credential
PS > $session = New-PSSession -ComputerName localhost -ConfigurationName MyNonAdmin -Credential $adminCred
PS > $session

 Id Name            ComputerName    ComputerType    State         ConfigurationName     Availability
 -- ----            ------------    ------------    -----         -----------------     ------------
  1 WinRM1          localhost       RemoteMachine   Opened        MyNonAdmin               Available

The above example uses Administrator credentials to the same MyNonAdmin custom endpoint, and the connection is made back to the same machine (localhost). The session is created successfully using Administrator credentials.

The breaking change is not in PowerShell but in a system security fix that restricts process creation between Windows sessions. This fix is preventing WinRM (which PowerShell uses as a remoting transport and host) from successfully creating the remote session host, for this particular scenario. There are no plans to update WinRM.

This affects Windows PowerShell and PowerShell Core 6 (PSCore6) WinRM based remoting.

This does not affect SSH remoting with PSCore6.

This does not affect JEA (Just Enough Administration) sessions.

A workaround for a loopback connection is to always use Administrator credentials.

Another option is to use PSCore6 with SSH remoting.

Paul Higinbotham
Senior Software Engineer
PowerShell Team

Parsing Text with PowerShell (1/3)

$
0
0

This is the first post in a three part series.

  • Part 1:
    • Useful methods on the String class
    • Introduction to Regular Expressions
    • The Select-String cmdlet
  • Part 2:
    • The -split operator
    • The -match operator
    • The switch statement
    • The Regex class
  • Part 3:
    • A real world, complete and slightly bigger, example of a switch-based parser

A task that appears regularly in my workflow is text parsing. It may be about getting a token from a single line of text or about turning the text output of native tools into structured objects so I can leverage the power of PowerShell.

I always strive to create structure as early as I can in the pipeline, so that later on I can reason about the content as properties on objects instead of as text at some offset in a string. This also helps with sorting, since the properties can have their correct type, so that numbers, dates etc. are sorted as such and not as text.

There are a number of options available to a PowerShell user, and I’m giving an overview here of the most common ones.

This is not a text about how to create a high performance parser for a language with a structured EBNF grammar. There are better tools available for that, for example ANTLR.

.Net methods on the string class

Any treatment of string parsing in PowerShell would be incomplete if it didn’t mention the methods on the string class.
There are a few methods that I’m using more often than others when parsing strings:

Name Description
Substring(int startIndex) Retrieves a substring from this instance. The substring starts at a specified character position and continues to the end of the string.
Substring(int startIndex, int length) Retrieves a substring from this instance. The substring starts at a specified character position and has a specified length.
IndexOf(string value) Reports the zero-based index of the first occurrence of the specified string in this instance.
IndexOf(string value, int startIndex) Reports the zero-based index of the first occurrence of the specified string in this instance. The search starts at a specified character position.
LastIndexOf(string value) Reports the zero-based index of the last occurrence of the specified string in this instance. Often used together with Substring.
LastIndexOf(string value, int startIndex) Reports the zero-based index position of the last occurrence of a specified string within this instance. The search starts at a specified character position and proceeds backward toward the beginning of the string.

This is a minor subset of the available functions. It may be well worth your time to read up on the string class since it is so fundamental in PowerShell.
Docs are found here.

As an example, this can be useful when we have very large input data of comma-separated input with 15 columns and we are only interested in the third column from the end. If we were to use the -split ',' operator, we would create 15 new strings and an array for each line. On the other hand, using LastIndexOf on the input string a few times and then SubString to get the value of interest is faster and results in just one new string.

function parseThirdFromEnd([string]$line){
    $i = $line.LastIndexOf(",")             # get the last separator
    $i = $line.LastIndexOf(",", $i - 1)     # get the second to last separator, also the end of the column we are interested in
    $j = $line.LastIndexOf(",", $i - 1)     # get the separator before the column we want
    $j++                                    # more forward past the separator
    $line.SubString($j,$i-$j)               # get the text of the column we are looking for
}

In this sample, I ignore that the IndexOf and LastIndexOf returns -1 if they cannot find the text to search for. From experience, I also know that it is easy to mess up the index arithmetics.
So while using these methods can improve performance, it is also more error prone and a lot more to type. I would only resort to this when I know the input data is very large and performance is an issue. So this is not a recommendation, or a starting point, but something to resort to.

On rare occasions, I write the whole parser in C#. An example of this is in a module wrapping the Perforce version control system, where the command line tool can output python dictionaries. It is a binary format, and the use case was complicated enough that I was more comfortable with a compiler checked implementation language.

Regular Expressions

Almost all of the parsing options in PowerShell make use of regular expressions, so I will start with a short intro of some regular expression concepts that are used later in these posts.

Regular expressions are very useful to know when writing simple parsers since they allow us to express patterns of interest and to capture text that matches those patterns.

It is a very rich language, but you can get quite a long way by learning a few key parts. I’ve found regular-expressions.info to be a good online resource for more information. It is not written directly for the .net regex implementation, but most of the information is valid across the different implementations.

Regex Description
* Zero or more of the preceding character. a* matches the empty string, a, aa, etc, but not b.
+ One or more of the preceding character. a+ matches a, aa, etc, but not the empty string or b.
. Matches any character
[ax1] Any of a,x,1
a-d matches any of a,b,c,d
\w The \w meta character is used to find a word character. A word character is a character from a-z, A-Z, 0-9, including the _ (underscore) character. It also matches variants of the characters such as ??? and ???.
\W The inversion of \w. Matches any non-word character
\s The \s meta character is used to find white space
\S The inversion of \s. Matches any non-whitespace character
\d Matches digits
\D The inversion of \d. Matches non-digits
\b Matches a word boundary, that is, the position between a word and a space.
\B The inversion of \b. . er\B matches the er in verb but not the er in never.
^ The beginning of a line
$ The end of a line
(<expr>) Capture groups

Combining these, we can create a pattern like below to match a text like:

Text Pattern
" 42,Answer" ^\s+\d+,.+

The above pattern can be written like this using the x (ignore pattern whitespace) modifier.

Starting the regex with (?x) ignores whitespace in the pattern (it has to be specified explicitly, with \s) and also enables the comment character #.

(?x)  # this regex ignores whitespace in the pattern. Makes it possible do document a regex with comments.
^     # the start of the line
\s+   # one or more whitespace character
\d+   # one or more digits
,     # a comma
.+    # one or more characters of any kind

By using capture groups, we make it possible to refer back to specific parts of a matched expression.

Text Pattern
" 42,Answer" ^\s+(\d+),(.+)
(?x)  # this regex ignores whitespace in the pattern. Makes it possible to document a regex with comments.
^     # the start of the line
\s+   # one or more whitespace character
(\d+) # capture one or more digits in the first group (index 1)
,     # a comma
(.+)  # capture one or more characters of any kind in the second group (index 2)

Naming regular expression groups

There is a construct called named capturing groups, (?<group_name>pattern), that will create a capture group with a designated name.

The regex above can be rewritten like this, which allows us to refer to the capture groups by name instead of by index.

^\s+(?<num>\d+),(?<text>.+)

Different languages have implementation specific solutions to accessing the values of the captured groups. We will see later on in this series how it is done in PowerShell.

The Select-String cmdlet

The Select-String command is a work horse, and is very powerful when you understand the output it produces.
I use it mainly when searching for text in files, but occasionally also when looking for something in command output and similar.

The key to being efficient with Select-String is to know how to get to the matched patterns in the output. In its internals, it uses the same regex class as the -match and -split operator, but instead of populating a global variable with the resulting groups, as -match does, it writes an object to the pipeline, with a Matches property that contains the results of the match.

Set-Content twitterData.txt -value @"
Lee, Steve-@Steve_MSFT,2992
Lee Holmes-13000 @Lee_Holmes
Staffan Gustafsson-463 @StaffanGson
Tribbiani, Joey-@Matt_LeBlanc,463400
"@

# extracting captured groups
Get-ChildItem twitterData.txt |
    Select-String -Pattern "^(\w+) ([^-]+)-(\d+) (@\w+)" |
    Foreach-Object {
        $first, $last, $followers, $handle = $_.Matches[0].Groups[1..4].Value   # this is a common way of getting the groups of a call to select-string
        [PSCustomObject] @{
            FirstName = $first
            LastName = $last
            Handle = $handle
            TwitterFollowers = [int] $followers
        }
    }
FirstName LastName   Handle       TwitterFollowers
--------- --------   ------       ----------------
Lee       Holmes     @Lee_Holmes             13000
Staffan   Gustafsson @StaffanGson              463

Support for Multiple Patterns

As we can see above, only half of the data matched the pattern to Select-String.

A technique that I find useful is to take advantage of the fact that Select-String supports the use of multiple patterns.

The lines of input data in twitterData.txt contain the same type of information, but they’re formatted in slightly different ways.
Using multiple patterns in combination with named capture groups makes it a breeze to extract the groups even when the positions of the groups differ.

$firstLastPattern = "^(?<first>\w+) (?<last>[^-]+)-(?<followers>\d+) (?<handle>@.+)"
$lastFirstPattern = "^(?<last>[^\s,]+),\s+(?<first>[^-]+)-(?<handle>@[^,]+),(?<followers>\d+)"
Get-ChildItem twitterData.txt |
     Select-String -Pattern $firstLastPattern, $lastFirstPattern |
    Foreach-Object {
        # here we access the groups by name instead of by index
        $first, $last, $followers, $handle = $_.Matches[0].Groups['first', 'last', 'followers', 'handle'].Value
        [PSCustomObject] @{
            FirstName = $first
            LastName = $last
            Handle = $handle
            TwitterFollowers = [int] $followers
        }
    }
FirstName LastName   Handle        TwitterFollowers
--------- --------   ------        ----------------
Steve     Lee        @Steve_MSFT               2992
Lee       Holmes     @Lee_Holmes              13000
Staffan   Gustafsson @StaffanGson               463
Joey      Tribbiani  @Matt_LeBlanc           463400

Breaking down the $firstLastPattern gives us

(?x)                # this regex ignores whitespace in the pattern. Makes it possible do document a regex with comments.
^                   # the start of the line
(?<first>\w+)       # capture one or more of any word characters into a group named 'first'
\s                  # a space
(?<last>[^-]+)      # capture one of more of any characters but `-` into a group named 'last'
-                   # a '-'
(?<followers>\d+)   # capture 1 or more digits into a group named 'followers'
\s                  # a space
(?<handle>@.+)      # capture a '@' followed by one or more characters into a group named 'handle'

The second regex is similar, but with the groups in different order. But since we retrieve the groups by name, we don’t have to care about the positions of the capture groups, and multiple assignment works fine.

Context around Matches

Select-String also has a Context parameter which accepts an array of one or two numbers specifying the number of lines before and after a match that should be captured. All text parsing techniques in this post can be used to parse information from the context lines.
The result object has a Context property, that returns an object with PreContext and PostContext properties, both of the type string[].

This can be used to get the second line before a match:

# using the context property
Get-ChildItem twitterData.txt |
    Select-String -Pattern "Staffan" -Context 2,1 |
    Foreach-Object { $_.Context.PreContext[1], $_.Context.PostContext[0] }
Lee Holmes-13000 @Lee_Holmes
Tribbiani, Joey-@Matt_LeBlanc,463400

To understand the indexing of the Pre- and PostContext arrays, consider the following:

Lee, Steve-@Steve_MSFT,2992                  <- PreContext[0]
Lee Holmes-13000 @Lee_Holmes                 <- PreContext[1]
Staffan Gustafsson-463 @StaffanGson          <- Pattern matched this line
Tribbiani, Joey-@Matt_LeBlanc,463400         <- PostContext[0]

The pipeline support of Select-String makes it different from the other parsing tools available in PowerShell, and makes it the undisputed king of one-liners.

I would like stress how much more useful Select-String becomes once you understand how to get to the parts of the matches.

Summary

We have looked at useful methods of the string class, especially how to use Substring to get to text at a specific offset. We also looked at regular expression, a language used to describe patterns in text, and on the Select-String cmdlet, which makes heavy use of regular expression.

Next time, we will look at the operators -split and -match, the switch statement (which is surprisingly useful for text parsing), and the regex class.

Staffan Gustafsson, @StaffanGson, github

Thanks to Jason Shirk, Mathias Jessen and Steve Lee for reviews and feedback.

Announcing the PowerShell Preview Extension in VSCode

$
0
0

Preview builds of the PowerShell extension are now available in VSCode

We are excited to announce the PowerShell Preview extension in the VSCode marketplace!
The PowerShell Preview extension allows users on Windows PowerShell 5.1, Powershell 6.0, and all newer versions to get and test the latest updates to the PowerShell extension and comes with some exciting features. The PowerShell Preview extension is a substitute for the PowerShell extension so both the PowerShell extension and the PowerShell Preview
extension should not be enabled at the same time.

Features of the PowerShell Preview extension

The PowerShell Preview extension is built on .NET Standard thereby enabling simplification of our code and dependency structure.

The PowerShell Preview extension also contains PSReadLine support in the integrated console for Windows behind a
feature flag. PSReadLine provides a consistent and rich interactive experience, including syntax coloring and
multi-line editing and history, in the PowerShell console, in Cloud Shell, and now in VSCode terminal.
For more information on the benefits of PSReadLine, check out their documentation.

To enable PSReadLine support in the Preview version on Windows, please add the following to your user settings:

"powershell.developer.featureFlags": [ "PSReadLine" ]

HUGE thanks to @SeeminglyScience for all his amazing work getting PSReadLine working in PowerShell Editor Services!

Why we created the PowerShell Preview extension

By having a preview channel, which supports Windows Powershell 5.1 and PowerShell Core 6, in addition to our existing stable channel, we can get new features out faster. PSReadLine support for the VSCode integrated console is a great
example of a feature that the preview build makes possible. Having a preview channel also allows us to get more feedback
on new features and to iterate on changes before they arrive in our stable channel.

How to Get/Use the PowerShell Preview extension

If you dont already have VSCode, start here.

Once you have VSCode open, click Clt+Shift+X to open the extensions marketplace.
Next, type PowerShell Preview in the search bar.
Click Install on the PowerShell Preview page.
Finally, click Reload in order to refresh VSCode.

If you already have the PowerShell extension please disable it to use the Powershell Preview extension.
To disable the PowerShell extension find it in the Extensions sidebar view, specifically under the list of Enabled extensions, Right-click on the PowerShell extension and select Disable. Please note that it is important to only have either the
PowerShell extension or the PowerShell Preview extension endabled at one time.

Breaking Changes

As stated above, this version of the PowerShell extension only works with Windows PowerShell versions 5.1 and
PowerShell Core 6.

Reporting Feedback

An important benefit of a preview extension is getting feedback from users.
To report issues with the extension use our GitHub repo.
When reporting issues be sure to specify the version of the extension you are using.

Sydney Smith
Program Manager
PowerShell Team

The PowerShell-Docs repositories have been moved

$
0
0

The PowerShell-Docs repositories have been moved from the PowerShell organization to the MicrosoftDocs organization in GitHub.

The tools we use to build the documentation are designed to work in the MicrosoftDocs org. Moving the repository lets us build the foundation for future improvements in our documentation experience.

Impact of the move

During the move there may be some downtime. The affected repositories will be inaccessible during
the move process. Also, the documentation processes will be paused. After the move, we need to test
access permissions and automation scripts.
After these tasks are complete, access and operations will return to normal. GitHub automatically
redirects requests to the old repo URL to the new location.
For more information about transferring repositories in GitHub, see About repository transfers

If the transferred repository has any forks, then those forks will remain associated with the
repository after the transfer is complete.
  • All Git information about commits, including contributions, are preserved.
  • All of the issues and pull requests remain intact when transferring a repository.
  • All links to the previous repository location are automatically redirected to the new location.


When you use git clone, git fetch, or git push on a transferred repository, these commands will r
edirect to the new repository location or URL.

However, to avoid confusion, we strongly recommend updating any existing local clones to point to
the new repository URL.
For more information, see Changing a remote’s URL.

The following example shows how to change the “upstream” remote to point to the new location:

[Wed 06:08PM] [staging =]
PS C:\Git\PS-Docs\PowerShell-Docs> git remote -v
origin  https://github.com/sdwheeler/PowerShell-Docs.git (fetch)
origin  https://github.com/sdwheeler/PowerShell-Docs.git (push)
upstream        https://github.com/PowerShell/PowerShell-Docs.git (fetch)
upstream        https://github.com/PowerShell/PowerShell-Docs.git (push)

[Wed 06:09PM] [staging =]
PS C:\Git\PS-Docs\PowerShell-Docs> git remote set-url upstream https://github.com/MicrosoftDocs/PowerShell-Docs.git

[Wed 06:10PM] [staging =]
PS C:\Git\PS-Docs\PowerShell-Docs> git remote -v
origin  https://github.com/sdwheeler/PowerShell-Docs.git (fetch)
origin  https://github.com/sdwheeler/PowerShell-Docs.git (push)
upstream        https://github.com/MicrosoftDocs/PowerShell-Docs.git (fetch)
upstream        https://github.com/MicrosoftDocs/PowerShell-Docs.git (push)


Which repositories were moved?

 

The following repositories were transferred:

  • PowerShell/PowerShell-Docs
  • PowerShell/powerShell-Docs.cs-cz
  • PowerShell/powerShell-Docs.de-de
  • PowerShell/powerShell-Docs.es-es
  • PowerShell/powerShell-Docs.fr-fr
  • PowerShell/powerShell-Docs.hu-hu
  • PowerShell/powerShell-Docs.it-it
  • PowerShell/powerShell-Docs.ja-jp
  • PowerShell/powerShell-Docs.ko-kr
  • PowerShell/powerShell-Docs.nl-nl
  • PowerShell/powerShell-Docs.pl-pl
  • PowerShell/powerShell-Docs.pt-br
  • PowerShell/powerShell-Docs.pt-pt
  • PowerShell/powerShell-Docs.ru-ru
  • PowerShell/powerShell-Docs.sv-se
  • PowerShell/powerShell-Docs.tr-tr
  • PowerShell/powerShell-Docs.zh-cn
  • PowerShell/powerShell-Docs.zh-tw

Call to action

If you have a fork that you cloned, change your remote configuration to point to the new upstream URL.
Help us make the documentation better.
  • Submit issues when you find a problem in the docs.
  • Suggest fixes to documentation by submitting changes through the PR process.
Sean Wheeler
Senior Content Developer for PowerShell
https://github.com/sdwheeler

Parsing Text with PowerShell (2/3)

$
0
0

This is the second post in a three-part series.

  • Part 1:
    • Useful methods on the String class
    • Introduction to Regular Expressions
    • The Select-String cmdlet
  • Part 2:
    • the -split operator
    • the -match operator
    • the switch statement
    • the Regex class
  • Part 3:
    • a real world, complete and slightly bigger, example of a switch-based parser

The -split operator

The -split operator splits one or more strings into substrings.

The first example is a name-value pattern, which is a common parsing task. Note the usage of the Max-substrings parameter to the -split operator.
We want to ensure that it doesn’t matter if the value contains the character to split on.

$text = "Description=The '=' character is used for assigning values to a variable"
$name, $value = $text -split "=", 2

@"
Name  =  $name
Value =  $value
"@
Name  =  Description
Value =  The '=' character is used for assigning values to a variable

When the line to parse contains fields separated by a well known separator, that is never a part of the field values, we can use the -split operator in combination with multiple assignment to get the fields into variables.

$name, $location, $occupation = "Spider Man,New York,Super Hero" -split ','

If only the location is of interest, the unwanted items can be assigned to $null.

$null, $location, $null = "Spider Man,New York,Super Hero" -split ','

$location
New York

If there are many fields, assigning to null doesn’t scale well. Indexing can be used instead, to get the fields of interest.

$inputText = "x,Staffan,x,x,x,x,x,x,x,x,x,x,Stockholm,x,x,x,x,x,x,x,x,11,x,x,x,x"
$name, $location, $age = ($inputText -split ',')[1,12,21]

$name
$location
$age
Staffan
Stockholm
11

It is almost always a good idea to create an object that gives context to the different parts.

$inputText = "x,Steve,x,x,x,x,x,x,x,x,x,x,Seattle,x,x,x,x,x,x,x,x,22,x,x,x,x"
$name, $location, $age = ($inputText -split ',')[1,12,21]
[PSCustomObject] @{
    Name = $name
    Location = $location
    Age = [int] $age
}
Name  Location Age
----  -------- ---
Steve Seattle   22

Instead of creating a PSCustomObject, we can create a class. It’s a bit more to type, but we can get more help from the engine, for example with tab completion.

The example below also shows an example of type conversion, where the default string to number conversion doesn’t work.
The age field is handled by PowerShell’s built-in type conversion. It is of type [int], and PowerShell will handle the conversion from string to int,
but in some cases we need to help out a bit. The ShoeSize field is also an [int], but the data is hexadecimal,
and without the hex specifier (‘0x’), this conversion fails for some values, and provides incorrect results for the others.

class PowerSheller {
    [string] $Name
    [string] $Location
    [int] $Age
    [int] $ShoeSize
}

$inputText = "x,Staffan,x,x,x,x,x,x,x,x,x,x,Stockholm,x,x,x,x,x,x,x,x,33,x,11d,x,x"
$name, $location, $age, $shoeSize = ($inputText -split ',')[1,12,21,23]
[PowerSheller] @{
    Name = $name
    Location = $location
    Age = $age
    # ShoeSize is expressed in hex, with no '0x' because reasons :)
    # And yes, it's in millimeters.
    ShoeSize = [Convert]::ToInt32($shoeSize, 16)
}
Name    Location  Age ShoeSize
----    --------  --- --------
Staffan Stockholm  33      285

The split operator’s first argument is actually a regex (by default, can be changed with options).
I use this on long command lines in log files (like those given to compilers) where there can be hundreds of options specified. This makes it hard to see if a certain option is specified or not, but when split into their own lines, it becomes trivial.
The pattern below uses a positive lookahead assertion.
It can be very useful to make patterns match only in a given context, like if they are, or are not, preceded or followed by another pattern.

$cmdline = "cl.exe /D Bar=1 /I SomePath /D Foo  /O2 /I SomeOtherPath /Debug a1.cpp a3.cpp a2.cpp"

$cmdline -split "\s+(?=[-/])"
cl.exe
/D Bar=1
/I SomePath
/D Foo
/O2
/I SomeOtherPath
/Debug a1.cpp a2.cpp

Breaking down the regex, by rewriting it with the x option:

(?x)      # ignore whitespace in the pattern, and enable comments after '#'
\s+       # one or more spaces
(?=[-/])  # only match the previous spaces if they are followed by any of '-' or '/'.

Splitting with a scriptblock

The -split operator also comes in another form, where you can pass it a scriptblock instead of a regular expression.
This allows for more complicated logic, that can be hard or impossible to express as a regular expression.

The scriptblock accepts two parameters, the text to split and the current index. $_ is bound to the character at the current index.

function SplitWhitespaceInMiddleOfText {
    param(
        [string]$Text,
        [int] $Index
    )
    if ($Index -lt 10 -or $Index -gt 40){
        return $false
    }
    $_ -match '\s'
}

$inputText = "Some text that only needs splitting in the middle of the text"
$inputText -split $function:SplitWhitespaceInMiddleOfText
Some text that
only
needs
splitting
in
the middle of the text

The $function:SplitWhitespaceInMiddleOfText syntax is a way to get to content (the scriptblock that implements it) of the function, just as $env:UserName gets the content of an item in the env: drive.
It provides a way to document and/or reuse the scriptblock.

The -match operator

The -match operator works in conjunction with the $matches automatic variable. Each time a -match or a -notmatch succeeds, the $matches variable is populated so that each capture group gets its own entry. If the capture group is named, the key will be the name of the group, otherwise it will be the index.

As an example:

if ('a b c' -match '(\w) (?<named>\w) (\w)'){
    $matches
}
Name                           Value
----                           -----
named                          b
2                              c
1                              a
0                              a b c

Notice that the indices only increase on groups without names. I.E. the indices of later groups change when a group is named.

Armed with the regex knowledge from the earlier post, we can write the following:

PS> "    10,Some text" -match '^\s+(\d+),(.+)'
True
PS> $matches
Name                           Value
----                           -----
2                              Some text
1                              10
0                              10,Some text

or with named groups

PS> "    10,Some text" -match '^\s+(?<num>\d+),(?<text>.+)'
True
PS> $matches
Name                           Value
----                           -----
num                            10
text                           Some text
0                              10,Some text

The important thing here is to put parentheses around the parts of the pattern that we want to extract. That is what creates the capture groups that allow us to reference those parts of the matching text, either by name or by index.

Combining this into a function makes it easy to use:

function ParseMyString($text){
    if ($text -match '^\s+(\d+),(.+)') {
        [PSCustomObject] @{
            Number = [int] $matches[1]
            Text    = $matches[2]
        }
    }
    else {
        Write-Warning "ParseMyString: Input `$text` doesn't match pattern"
    }
}

ParseMyString "    10,Some text"
Number  Text
------- ----
     10 Some text

Notice the type conversion when assigning the Number property. As long as the number is in range of an integer, this will always succeed, since we have made a successful match in the if statement above. ([long] or [bigint] could be used. In this case I provide the input, and I have promised myself to stick to a range that fits in a 32-bit integer.)
Now we will be able to sort or do numerical operations on the Number property, and it will behave like we want it to – as a number, not as a string.

The switch statement

Now we’re at the big guns 🙂

The switch statement in PowerShell has been given special functionality for parsing text.
It has two flags that are useful for parsing text and files with text in them. -regex and -file.

When specifying -regex, the match clauses that are strings are treated as regular expressions. The switch statement also sets the $matches automatic variable.

When specifying -file, PowerShell treats the input as a file name, to read input from, rather than as a value statement.

Note the use of a ScriptBlock instead of a string as the match clause to determine if we should skip preamble lines.

class ParsedOutput {
    [int] $Number
    [string] $Text

    [string] ToString() { return "{0} ({1})" -f $this.Text, $this.Number }
}

$inputData =
    "Preamble line",
    "LastLineOfPreamble",
    "    10,Some Text",
    "    Some other text,20"

$inPreamble = $true
switch -regex ($inputData) {

    {$inPreamble -and $_ -eq 'LastLineOfPreamble'} { $inPreamble = $false; continue }

    "^\s+(?<num>\d+),(?<text>.+)" {  # this matches the first line of non-preamble input
        [ParsedOutput] @{
            Number = $matches.num
            Text = $matches.text
        }
        continue
    }

    "^\s+(?<text>[^,]+),(?<num>\d+)" { # this matches the second line of non-preamble input
        [ParsedOutput] @{
            Number = $matches.num
            Text = $matches.text
        }
        continue
    }
}
Number  Text
------ ----
    10 Some Text
    20 Some other text

The pattern [^,]+ in the text group in the code above is useful. It means match anything that is not a comma ,. We are using the any-of construct [], and within those brackets, ^ changes meaning from the beginning of the line to anything but.

That is useful when we are matching delimited fields. A requirement is that the delimiter cannot be part of the set of allowed field values.

The regex class

regex is a type accelerator for System.Text.RegularExpressions.Regex. It can be useful when porting code from C#, and sometimes when we want to get more control in situations when we have many matches of a capture group. It also allows us to pre-create the regular expressions which can matter in performance sensitive scenarios, and to specify a timeout.

One instance where the regex class is needed is when you have multiple captures of a group.

Consider the following:

Text Pattern
a,b,c, (\w,)+

If the match operator is used, $matches will contain

Name                           Value
----                           -----
1                              c,
0                              a,b,c,

The pattern matched three times, for a,, b, and c,. However, only the last match is preserved in the $matches dictionary.
However, the following will allow us to get to all the captures of the group:

[regex]::match('a,b,c,', '(\w,)+').Groups[1].Captures
Index Length Value
----- ------ -----
    0      2 a,
    2      2 b,
    4      2 c,

Below is an example that uses the members of the Regex class to parse input data

class ParsedOutput {
    [int] $Number
    [string] $Text

    [string] ToString() { return "{0} ({1})" -f $this.Text, $this.Number }
}

$inputData =
    "    10,Some Text",
    "    Some other text,20"  # this text will not match

[regex] $pattern = "^\s+(\d+),(.+)"

foreach($d in $inputData){
    $match = $pattern.Match($d)
    if ($match.Success){
        $number, $text = $match.Groups[1,2].Value
        [ParsedOutput] @{
            Number = $number
            Text = $text
        }
    }
    else {
        Write-Warning "regex: '$d' did not match pattern '$pattern'"
    }
}
WARNING: regex: '    Some other text,20' did not match pattern '^\s+(\d+),(.+)'
Number Text
------ ----
    10 Some Text

It may surprise you that the warning appears before the output. PowerShell has a quite complex formatting system at the end of the pipeline, which treats pipeline output different than other streams. Among other things, it buffers output in the beginning of a pipeline to calculate sensible column widths. This works well in practice, but sometimes gives strange reordering of output on different streams.

Summary

In this post we have looked at how the -split operator can be used to split a string in parts, how the -match operator can be used to extract different patterns from some text, and how the powerful switch statement can be used to match against multiple patterns.

We ended by looking at how the regex class, which in some cases provides a bit more control, but at the expense of ease of use. This concludes the second part of this series. Next time, we will look at a complete, real world, example of a switch-based parser.

Thanks to Jason Shirk, Mathias Jessen and Steve Lee for reviews and feedback.

Staffan Gustafsson, @StaffanGson, powercode@github

Staffan works at DICE in Stockholm, Sweden, as a Software Engineer and has been using PowerShell since the first public beta.
He was most seriously pleased when PowerShell was open sourced, and has since contributed bug fixes, new features and performance improvements.
Staffan is a speaker at PSConfEU and is always happy to talk PowerShell.


Parsing Text with PowerShell (3/3)

$
0
0

This is the third and final post in a three-part series.

  • Part 1:
    • Useful methods on the String class
    • Introduction to Regular Expressions
    • The Select-String cmdlet
  • Part 2:
    • the -split operator
    • the -match operator
    • the switch statement
    • the Regex class
  • Part 3:
    • a real world, complete and slightly bigger, example of a switch-based parser
      • General structure of a switch-based parser
      • The real world example

In the previous posts, we looked at the different operators what are available to us in PowerShell.

When analyzing crashes at DICE, I noticed that some of the C++ runtime binaries where missing debug symbols. They should be available for download from Microsoft’s public symbol server, and most versions were there. However, due to some process errors at DevDiv, some builds were released publicly without available debug symbols.
In some cases, those missing symbols prevented us from debugging those crashes, and in all cases, they triggered my developer OCD.

So, to give actionable feedback to Microsoft, I scripted a debugger (cdb.exe in this case) to give a verbose list of the loaded modules, and parsed the output with PowerShell, which was also later used to group and filter the resulting data set. I sent this data to Microsoft, and 5 days later, the missing symbols were available for download. Mission accomplished!

This post will describe the parser I wrote for this task (it turned out that I had good use for it for other tasks later), and the general structure is applicable to most parsing tasks.

The example will show how a switch-based parser would look when the input data isn’t as tidy as it normally is in examples, but messy – as the real world data often is.

General Structure of a switch Based Parser

Depending on the structure of our input, the code must be organized in slightly different ways.

Input may have a record start that differs by indentation or some distinct token like

Foo                    <- Record start - No whitespace at the beginning of the line
    Prop1=Staffan      <- Properties for the record - starts with whitespace
    Prop3 =ValueN
Bar
    Prop1=Steve
    Prop2=ValueBar2

If the data to be parsed has an explicit start record, it is a bit easier than if it doesn’t have one.
We create a new data object when we get a record start, after writing any previously created object to the pipeline.
At the end, we need to check if we have parsed a record that hasn’t been written to the pipeline.

The general structure of a such a switch-based parser can be as follows:

$inputData = @"
Foo
    Prop1=Value1
    Prop3=Value3
Bar
    Prop1=ValueBar1
    Prop2=ValueBar2
"@ -split '\r?\n'   # This regex is useful to split at line endings, with or without carriage return

class SomeDataClass {
    $ID
    $Name
    $Property2
    $Property3
}

# map to project input property names to the properties on our data class
$propertyNameMap = @{
    Prop1 = "Name"
    Prop2 = "Property2"
    Prop3 = "Property3"
}

$currentObject = $null
switch -regex ($inputData) {

    '^(\S.*)' {
        # record start pattern, in this case line that doesn't start with a whitespace.
        if ($null -ne $currentObject) {
            $currentObject                   # output to pipeline if we have a previous data object
        }
        $currentObject = [SomeDataClass] @{  # create new object for this record
            Id = $matches.1                  # with Id like Foo or Bar
        }
        continue
    }

    # set the properties on the data object
    '^\s+([^=]+)=(.*)' {
        $name, $value = $matches[1, 2]
        # project property names
        $propName = $propertyNameMap[$name]
        if ($propName = $null) {
            $propName = $name
        }
        # assign the parsed value to the projected property name
        $currentObject.$propName = $value
        continue
    }
}

if ($currentObject) {
    # Handle the last object if any
    $currentObject # output to pipeline
}
ID  Name      Property2 Property3
--  ----      --------- ---------
Foo Value1              Value3
Bar ValueBar1 ValueBar2

Alternatively, we may have input where the records are separated by a blank line, but without any obvious record start.

commitId=1234                         <- In this case, a commitId is first in a record
description=Update readme.md
                                      <- the blank line separates records
user=Staffan                          <- For this record, a user property comes first
commitId=1235
description=Fix bug.md

In this case the structure of the code looks a bit different. We create an object at the beginning, but keep track of if it’s dirty or not.
If we get to the end with a dirty object, we must output it.

$inputData = @"

commit=1234
desc=Update readme.md

user=Staffan
commit=1235
desc=Bug fix

"@ -split "\r?\n"

class SomeDataClass {
    [int] $CommitId
    [string] $Description
    [string] $User
}

# map to project input property names to the properties on our data class
# we only need to provide the ones that are different. 'User' works fine as it is.
$propertyNameMap = @{
    commit = "CommitId"
    desc   = "Description"
}

$currentObject = [SomeDataClass]::new()
$objectDirty = $false
switch -regex ($inputData) {
    # set the properties on the data object
    '^([^=]+)=(.*)$' {
        # parse a name/value
        $name, $value = $matches[1, 2]
        # project property names
        $propName = $propertyNameMap[$name]
        if ($null -eq $propName) {
            $propName = $name
        }
        # assign the projected property
        $currentObject.$propName = $value
        $objectDirty = $true
        continue
    }

    '^\s*$' {
        # separator pattern, in this case any blank line
        if ($objectDirty) {
            $currentObject                           # output to pipeline
            $currentObject = [SomeDataClass]::new()  # create new object
            $objectDirty = $false                    # and mark it as not dirty
        }
    }
    default {
        Write-Warning "Unexpected input: '$_'"
    }
}

if ($objectDirty) {
    # Handle the last object if any
    $currentObject # output to pipeline
}
CommitId Description      User
-------- -----------      ----
    1234 Update readme.md
    1235 Bug fix          Staffan

The Real World Example

I have adapted this sample slightly so that I get the loaded modules from a running process instead of from my crash dumps. The format of the output from the debugger is the same.
The following command launches a command line debugger on notepad, with a script that gives a verbose listing of the loaded modules, and quits:

# we need to muck around with the console output encoding to handle the trademark chars
# imagine no encodings
# it's easy if you try
# no code pages below us
# above us only sky
[Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding("iso-8859-1")

$proc = Start-Process notepad -passthru
Start-Sleep -seconds 1
$cdbOutput = cdb -y 'srv*c:\symbols*http://msdl.microsoft.com/download/symbols' -c ".reload -f;lmv;q" -p $proc.ProcessID

The output of the command above is here for those who want to follow along but who aren’t running windows or don’t have cdb.exe installed.

The (abbreviated) output looks like this:

Microsoft (R) Windows Debugger Version 10.0.16299.15 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.

*** wait with pending attach

************* Path validation summary **************
Response                         Time (ms)     Location
Deferred                                       srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols
Executable search path is:
ModLoad: 00007ff6`e9da0000 00007ff6`e9de3000   C:\Windows\system32\notepad.exe
...
ModLoad: 00007ffe`97d80000 00007ffe`97db1000   C:\WINDOWS\SYSTEM32\ntmarta.dll
(98bc.40a0): Break instruction exception - code 80000003 (first chance)
ntdll!DbgBreakPoint:
00007ffe`9cd53050 cc              int     3
0:007> cdb: Reading initial command '.reload -f;lmv;q'
Reloading current modules
.....................................................
start             end                 module name
00007ff6`e9da0000 00007ff6`e9de3000   notepad    (pdb symbols)          c:\symbols\notepad.pdb\2352C62CDF448257FDBDDA4081A8F9081\notepad.pdb
    Loaded symbol image file: C:\Windows\system32\notepad.exe
    Image path: C:\Windows\system32\notepad.exe
    Image name: notepad.exe
    Image was built with /Brepro flag.
    Timestamp:        329A7791 (This is a reproducible build file hash, not a timestamp)
    CheckSum:         0004D15F
    ImageSize:        00043000
    File version:     10.0.17763.1
    Product version:  10.0.17763.1
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        1.0 App
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft??? Windows??? Operating System
    InternalName:     Notepad
    OriginalFilename: NOTEPAD.EXE
    ProductVersion:   10.0.17763.1
    FileVersion:      10.0.17763.1 (WinBuild.160101.0800)
    FileDescription:  Notepad
    LegalCopyright:   ??? Microsoft Corporation. All rights reserved.
...
00007ffe`9ccb0000 00007ffe`9ce9d000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\B8AD79538F2730FD9BACE36C9F9316A01\ntdll.pdb
    Loaded symbol image file: C:\WINDOWS\SYSTEM32\ntdll.dll
    Image path: C:\WINDOWS\SYSTEM32\ntdll.dll
    Image name: ntdll.dll
    Image was built with /Brepro flag.
    Timestamp:        E8B54827 (This is a reproducible build file hash, not a timestamp)
    CheckSum:         001F20D1
    ImageSize:        001ED000
    File version:     10.0.17763.194
    Product version:  10.0.17763.194
    File flags:       0 (Mask 3F)
    File OS:          40004 NT Win32
    File type:        2.0 Dll
    File date:        00000000.00000000
    Translations:     0409.04b0
    CompanyName:      Microsoft Corporation
    ProductName:      Microsoft??? Windows??? Operating System
    InternalName:     ntdll.dll
    OriginalFilename: ntdll.dll
    ProductVersion:   10.0.17763.194
    FileVersion:      10.0.17763.194 (WinBuild.160101.0800)
    FileDescription:  NT Layer DLL
    LegalCopyright:   ??? Microsoft Corporation. All rights reserved.
quit:

The output starts with info that I’m not interested in here. I only want to get the detailed information about the loaded modules. It is not until the line

start             end                 module name

that I care about the output.

Also, at the end there is a line that we need to be aware of:

quit:

that is not part of the module output.

To skip the parts of the debugger output that we don’t care about, we have a boolean flag initially set to true.
If that flag is set, we check if the current line, $_, is the module header in which case we flip the flag.

    $inPreamble = $true
    switch -regex ($cdbOutput) {

        { $inPreamble -and $_ -eq "start             end                 module name" } { $inPreamble = $false; continue }

I have made the parser a separate function that reads its input from the pipeline. This way, I can use the same function to parse module data, regardless of how I got the module data. Maybe it was saved on a file. Or came from a dump, or a live process. It doesn’t matter, since the parser is decoupled from the data retrieval.

After the sample, there is a breakdown of the more complicated regular expressions used, so don’t despair if you don’t understand them at first.
Regular Expressions are notoriously hard to read, so much so that they make Perl look readable in comparison.

# define an class to store the data
class ExecutableModule {
    [string]   $Name
    [string]   $Start
    [string]   $End
    [string]   $SymbolStatus
    [string]   $PdbPath
    [bool]     $Reproducible
    [string]   $ImagePath
    [string]   $ImageName
    [DateTime] $TimeStamp
    [uint32]   $FileHash
    [uint32]   $CheckSum
    [uint32]   $ImageSize
    [version]  $FileVersion
    [version]  $ProductVersion
    [string]   $FileFlags
    [string]   $FileOS
    [string]   $FileType
    [string]   $FileDate
    [string[]] $Translations
    [string]   $CompanyName
    [string]   $ProductName
    [string]   $InternalName
    [string]   $OriginalFilename
    [string]   $ProductVersionStr
    [string]   $FileVersionStr
    [string]   $FileDescription
    [string]   $LegalCopyright
    [string]   $LegalTrademarks
    [string]   $LoadedImageFile
    [string]   $PrivateBuild
    [string]   $Comments
}

<#
.SYNOPSIS Runs a debugger on a program to dump its loaded modules
#>
function Get-ExecutableModuleRawData {
    param ([string] $Program)
    $consoleEncoding = [Console]::OutputEncoding
    [Console]::OutputEncoding = [System.Text.Encoding]::GetEncoding("iso-8859-1")
    try {
        $proc = Start-Process $program -PassThru
        Start-Sleep -Seconds 1  # sleep for a while so modules are loaded
        cdb -y srv*c:\symbols*http://msdl.microsoft.com/download/symbols -c ".reload -f;lmv;q" -p $proc.Id
        $proc.Close()
    }
    finally {
        [Console]::OutputEncoding = $consoleEncoding
    }
}

<#
.SYNOPSIS Converts verbose module data from windows debuggers into ExecutableModule objects.
#>
function ConvertTo-ExecutableModule {
    [OutputType([ExecutableModule])]
    param (
        [Parameter(ValueFromPipeline)]
        [string[]] $ModuleRawData
    )
    begin {
        $currentObject = $null
        $preamble = $true
        $propertyNameMap = @{
            'File flags'      = 'FileFlags'
            'File OS'         = 'FileOS'
            'File type'       = 'FileType'
            'File date'       = 'FileDate'
            'File version'    = 'FileVersion'
            'Product version' = 'ProductVersion'
            'Image path'      = 'ImagePath'
            'Image name'      = 'ImageName'
            'FileVersion'     = 'FileVersionStr'
            'ProductVersion'  = 'ProductVersionStr'
        }
    }
    process {
        switch -regex ($ModuleRawData) {

            # skip lines until we get to our sentinel line
            { $preamble -and $_ -eq "start             end                 module name" } { $preamble = $false; continue }

            #00007ff6`e9da0000 00007ff6`e9de3000   notepad    (deferred)
            #00007ffe`9ccb0000 00007ffe`9ce9d000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\B8AD79538F2730FD9BACE36C9F9316A01\ntdll.pdb
            '^([0-9a-f`]{17})\s([0-9a-f`]{17})\s+(\S+)\s+\(([^\)]+)\)\s*(.+)?' {
                # see breakdown of the expression later in the post
                # on record start, output the currentObject, if any is set
                if ($null -ne $currentObject) {
                    $currentObject
                }
                $start, $end, $module, $pdbKind, $pdbPath = $matches[1..5]
                # create an instance of the object that we are adding info from the current record into.
                $currentObject = [ExecutableModule] @{
                    Start        = $start
                    End          = $end
                    Name         = $module
                    SymbolStatus = $pdbKind
                    PdbPath      = $pdbPath
                }
                continue
            }
            '^\s+Image was built with /Brepro flag.' {
                $currentObject.Reproducible = $true
                continue
            }
            '^\s+Timestamp:\s+[^\(]+\((?<timestamp>.{8})\)' {
                # see breakdown of the regular  expression later in the post
                # Timestamp:        Mon Jan  7 23:42:30 2019 (5C33D5D6)
                $intValue = [Convert]::ToInt32($matches.timestamp, 16)
                $currentObject.TimeStamp = [DateTime]::new(1970, 01, 01, 0, 0, 0, [DateTimeKind]::Utc).AddSeconds($intValue)
                continue
            }
            '^\s+TimeStamp:\s+(?<value>.{8}) \(This' {
                # Timestamp:        E78937AC (This is a reproducible build file hash, not a timestamp)
                $currentObject.FileHash = [Convert]::ToUInt32($matches.value, 16)
                continue
            }
            '^\s+Loaded symbol image file: (?<imageFile>[^\)]+)' {
                $currentObject.LoadedImageFile = $matches.imageFile
                continue
            }
            '^\s+Checksum:\s+(?<checksum>\S+)' {
                $currentObject.Checksum = [Convert]::ToUInt32($matches.checksum, 16)
                continue
            }
            '^\s+Translations:\s+(?<value>\S+)' {
                $currentObject.Translations = $matches.value.Split(".")
                continue
            }
            '^\s+ImageSize:\s+(?<imageSize>.{8})' {
                $currentObject.ImageSize = [Convert]::ToUInt32($matches.imageSize, 16)
                continue
            }
            '^\s{4}(?<name>[^:]+):\s+(?<value>.+)' {
                # see breakdown of the regular expression later in the post
                # This part is any 'name: value' pattern
                $name, $value = $matches['name', 'value']

                # project the property name
                $propName = $propertyNameMap[$name]
                $propName = if ($null -eq $propName) { $name } else { $propName }

                # note the dynamic property name in the assignment
                # this will fail if the property doesn't have a member with the specified name
                $currentObject.$propName = $value
                continue
            }
            'quit:' {
                # ignore and exit
                break
            }
            default {
                # When writing the parser, it can be useful to include a line like the one below to see the cases that are not handled by the parser
                # Write-Warning "missing case for '$_'. Unexpected output format from cdb.exe"

                continue # skip lines that doesn't match the patterns we are interested in, like the start/end/modulename header and the quit: output
            }
        }
    }
    end {
        # this is needed to output the last object
        if ($null -ne $currentObject) {
            $currentObject
        }
    }
}


Get-ExecutableModuleRawData Notepad |
    ConvertTo-ExecutableModule |
    Sort-Object ProductVersion, Name
    Format-Table -Property Name, FileVersion, Product_Version, FileDescription
Name               FileVersionStr                             ProductVersion FileDescription
----               --------------                             -------------- ---------------
PROPSYS            7.0.17763.1 (WinBuild.160101.0800)         7.0.17763.1    Microsoft Property System
ADVAPI32           10.0.17763.1 (WinBuild.160101.0800)        10.0.17763.1   Advanced Windows 32 Base API
bcrypt             10.0.17763.1 (WinBuild.160101.0800)        10.0.17763.1   Windows Cryptographic Primitives Library
...
uxtheme            10.0.17763.1 (WinBuild.160101.0800)        10.0.17763.1   Microsoft UxTheme Library
win32u             10.0.17763.1 (WinBuild.160101.0800)        10.0.17763.1   Win32u
WINSPOOL           10.0.17763.1 (WinBuild.160101.0800)        10.0.17763.1   Windows Spooler Driver
KERNELBASE         10.0.17763.134 (WinBuild.160101.0800)      10.0.17763.134 Windows NT BASE API Client DLL
wintypes           10.0.17763.134 (WinBuild.160101.0800)      10.0.17763.134 Windows Base Types DLL
SHELL32            10.0.17763.168 (WinBuild.160101.0800)      10.0.17763.168 Windows Shell Common Dll
...
windows_storage    10.0.17763.168 (WinBuild.160101.0800)      10.0.17763.168 Microsoft WinRT Storage API
CoreMessaging      10.0.17763.194                             10.0.17763.194 Microsoft CoreMessaging Dll
gdi32full          10.0.17763.194 (WinBuild.160101.0800)      10.0.17763.194 GDI Client DLL
ntdll              10.0.17763.194 (WinBuild.160101.0800)      10.0.17763.194 NT Layer DLL
RMCLIENT           10.0.17763.194 (WinBuild.160101.0800)      10.0.17763.194 Resource Manager Client
RPCRT4             10.0.17763.194 (WinBuild.160101.0800)      10.0.17763.194 Remote Procedure Call Runtime
combase            10.0.17763.253 (WinBuild.160101.0800)      10.0.17763.253 Microsoft COM for Windows
COMCTL32           6.10 (WinBuild.160101.0800)                10.0.17763.253 User Experience Controls Library
urlmon             11.00.17763.168 (WinBuild.160101.0800)     11.0.17763.168 OLE32 Extensions for Win32
iertutil           11.00.17763.253 (WinBuild.160101.0800)     11.0.17763.253 Run time utility for Internet Explorer

Regex pattern breakdown

Here is a breakdown of the more complicated patterns, using the ignore pattern whitespace modifier x:

([0-9a-f`]{17})\s([0-9a-f`]{17})\s+(\S+)\s+\(([^\)]+)\)\s*(.+)?

# example input: 00007ffe`9ccb0000 00007ffe`9ce9d000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\B8AD79538F2730FD9BACE36C9F9316A01\ntdll.pdb

(?x)                # ignore pattern whitespace
^                   # the beginning of the line
([0-9a-f`]{17})     # capture expression like 00007ff6`e9da0000 - any hex number or backtick, and exactly 17 of them
\s                  # a space
([0-9a-f`]{17})     # capture expression like 00007ff6`e9da0000 - any hex number or backtick, and exactly 17 of them
\s+                 # skip any number of spaces
(\S+)               # capture until we get a space - this would match the 'ntdll' part
\s+                 # skip one or more spaces
\(                  # start parenthesis
    ([^\)])         # capture anything but end parenthesis
\)                  # end parenthesis
\s*                 # skip zero or more spaces
(.+)?               # optionally capture any symbol file path

Breakdown of the name-value pattern:

^\s+(?<name>[^:]+):\s+(?<value>.+)

# example input:  File flags:       0 (Mask 3F)

(?x)                # ignore pattern whitespace
^                   # the beginning of the line
\s+                 # require one or more spaces
(?<name>[^:]+)      # capture anything that is not a `:` into the named group "name"
:                   # require a comma
\s+                 # require one or more spaces
(?<value>.+)        # capture everything until the end into the name group "value"

Breakdown of the timestamp pattern:

^\s{4}Timestamp:\s+[^\(]+\((?<timestamp>.{8})\)

#example input:     Timestamp:        Mon Jan  7 23:42:30 2019 (5C33D5D6)

(?x)                # ignore pattern whitespace
^                   # the beginning of the line
\s+                 # require one or more spaces
Timestamp:          # The literal text 'Timestamp:'
\s+                 # require one or more spaces
[^\(]+              # one or more of anything but a open parenthesis
\(                  # a literal '('
(?<timestamp>.{8})  # 8 characters of anything, captured into the group 'timestamp'
\)                  # a literal ')'

Gotchas – the Regex Cache

Something that can happen if you are writing a more complicated parser is the following:
The parser works well. You have 15 regular expressions in your switch statement and then you get some input you haven’t seen before, so you add a 16th regex.
All of a sudden, the performance of your parser tanks. WTF?

The .net regex implementation has a cache of recently used regexs. You can check the size of it like this:

PS> [regex]::CacheSize
15

# bump it
[regex]::CacheSize = 20

And now your parser is fast(er) again.

Bonus tip

I frequently use PowerShell to write (generate) my code:

Get-ExecutableModuleRawData pwsh |
    Select-String '^\s+([^:]+):' |       # this pattern matches the module detail fields
    Foreach-Object {$_.matches.groups[1].value} |
    Select-Object -Unique |
    Foreach-Object -Begin   { "class ExecutableModuleData {" }`
                   -Process { "    [string] $" + ($_ -replace "\s.", {[char]::ToUpperInvariant($_.Groups[0].Value[1])}) }`
                   -End     { "}" }

The output is

class ExecutableModuleData {
    [string] $LoadedSymbolImageFile
    [string] $ImagePath
    [string] $ImageName
    [string] $Timestamp
    [string] $CheckSum
    [string] $ImageSize
    [string] $FileVersion
    [string] $ProductVersion
    [string] $FileFlags
    [string] $FileOS
    [string] $FileType
    [string] $FileDate
    [string] $Translations
    [string] $CompanyName
    [string] $ProductName
    [string] $InternalName
    [string] $OriginalFilename
    [string] $ProductVersion
    [string] $FileVersion
    [string] $FileDescription
    [string] $LegalCopyright
    [string] $Comments
    [string] $LegalTrademarks
    [string] $PrivateBuild
}

It is not complete – I don’t have the fields from the record start, some types are incorrect and when run against some other executables a few other fields may appear.
But it is a very good starting point. And way more fun than typing it 🙂

Note that this example is using a new feature of the -replace operator – to use a ScriptBlock to determine what to replace with – that was added in PowerShell Core 6.1.

Bonus tip #2

A regular expression construct that I often find useful is non-greedy matching.
The example below shows the effect of the ? modifier, that can be used after * (zero or more) and + (one or more)

# greedy matching - match to the last occurrence of the following character (>)
if("<Tag>Text</Tag>" -match '<(.+)>') { $matches }
Name                           Value
----                           -----
1                              Tag>Text</Tag
0                              <Tag>Text</Tag>
# non-greedy matching - match to the first occurrence of the the following character (>)
if("<Tag>Text</Tag>" -match '<(.+?)>') { $matches }
Name                           Value
----                           -----
1                              Tag
0                              <Tag>

See Regex Repeat for more info on how to control pattern repetition.

Summary

In this post, we have looked at how the structure of a switch-based parser could look, and how it can be written so that it works as a part of the pipeline.
We have also looked at a few slightly more complicated regular expressions in some detail.

As we have seen, PowerShell has a plethora of options for parsing text, and most of them revolve around regular expressions.
My personal experience has been that the time I’ve invested in understanding the regex language was well invested.

Hopefully, this gives you a good start with the parsing tasks you have at hand.

Thanks to Jason Shirk, Mathias Jessen and Steve Lee for reviews and feedback.

Staffan Gustafsson, @StaffanGson, github

Staffan works at DICE in Stockholm, Sweden, as a Software Engineer and has been using PowerShell since the first public beta.
He was most seriously pleased when PowerShell was open sourced, and has since contributed bug fixes, new features and performance improvements.
Staffan is a speaker at PSConfEU and is always happy to talk PowerShell.

PowerShell ForEach-Object Parallel Feature

$
0
0

PowerShell ForEach-Object Parallel Feature

PowerShell 7.0 Preview 3 is now available with a new ForEach-Object Parallel Experimental feature. This feature is a great new tool for parallelizing work, but like any tool, it has its uses and drawbacks.

This article describes this new feature, how it works, when to use it and when not to.

What is ForEach-Object -Parallel?

ForEach-Object -Parallel is a new parameter set added to the existing PowerShell ForEach cmdlet.

ForEach-Object -Parallel <scriptblock> [-InputObject <psobject>] [-ThrottleLimit <int>]
[-TimeoutSeconds <int>] [-AsJob] [-WhatIf] [-Confirm] [<CommonParameters>]

Normally, when you use the ForEach-Object cmdlet, each object piped to the cmdlet is processed sequentially.

1..5 | ForEach-Object { "Hello $_"; sleep 1 }
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5

(Measure-Command {
    1..5 | ForEach-Object { "Hello $_"; sleep 1 } 
}).Seconds
5

But with the new ForEach-Object -Parallel parameter set, you can run all script in parallel for each piped input object.

1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 
Hello 1 
Hello 3 
Hello 2 
Hello 4 
Hello 5 

(Measure-Command {
    1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 
}).Seconds
1

Because each script block in the ForEach-Object example above takes 1 second to run, running all five in parallel takes only one second instead of 5 seconds when run sequentially.

Since the script blocks are run in parallel for each of the 1-5 piped input integers, the order of execution is not guaranteed. The -ThrottleLimit parameter limits the number of script blocks running in parallel at a given time, and its default value is 5.

This new feature also supports jobs, where you can choose to have a job object returned instead of having results written to the console.

$Job = 1..5 | ForEach-Object -Parallel { "Hello $_"; sleep 1; } -ThrottleLimit 5 -AsJob 
$job | Wait-Job | Receive-Job 
Hello 1 
Hello 2 
Hello 3 
Hello 5 
Hello 4

ForEach-Object -Parallel is not the same as the foreach language keyword

Don’t confuse ForEach-Object cmdlet with PowerShell’s foreach keyword. The foreach keyword does not handle piped input but instead iterates over an enumerable object. There is currently no parallel support for the foreach keyword.

foreach ($item in (1..5)) { "Hello $item" }
Hello 1
Hello 2
Hello 3
Hello 4
Hello 5

How does it work?

The new ForEach-Object -Parallel parameter set uses existing PowerShell APIs for running script blocks in parallel. These APIs have been around since PowerShell v2, but are cumbersome and difficult to use correctly. This new feature makes it much easier to run script blocks in parallel. But there is a fair amount of overhead involved and many times there is no gain in running scripts in parallel, and in fact it can end up being significantly slower than running ForEach-Object normally.

PowerShell currently supports parallelism in three main categories.

  1. PowerShell remoting. Here PowerShell sends script to external machines to run, using PowerShell’s remoting system.
  2. PowerShell jobs. This is the same as remoting except that script is run in separate processes on the local machine, rather than on external machines.
  3. PowerShell runspaces. Here script is run on the local machine within the same process but on separate threads.

This new feature uses the third method for running scripts in parallel. It has the least overhead of the other two methods and does not use the PowerShell remoting system. So it is generally much faster than the other two methods.

However, there is still quite a bit of overhead to run script blocks in parallel. Script blocks run in a context called a PowerShell runspace. The runspace context contains all of the defined variables, functions and loaded modules. So initializing a runspace for script to run in takes time and resources. When scripts are run in parallel they must be run within their own runspace. And each runspace must load whatever module is needed and have any variable be explicitly passed in from the calling script. The only variable that automatically appears in the parallel script block is the piped in object. Other variables are passed in using the $using: keyword.

$computers = 'computerA','computerB','computerC','computerD' 
$logsToGet = 'LogA','LogB','LogC' 

# Read specified logs on each machine, using custom module
$logs = $computers | ForEach-Object -ThrottleLimit 10 -Parallel {
    Import-Module MyLogsModule 
    Get-Logs -ComputerName $_ -LogName $using:logsToGet 
}

Given the overhead required to run scripts in parallel, the -ThrottleLimit becomes very useful to prevent the system from being overwhelmed. There are some cases where running a lot of script blocks in parallel makes sense, but also many cases where it does not.

When should it be used?

There are two primary reasons to run script blocks in parallel with the ForEach-Object -Parallel feature (keeping in mind that this feature runs the script on separate system threads).

  1. Highly compute intensive script. If your script is crunching a lot of data over a significant period of time and the scripts can be run independently, then it is worthwhile to run them in parallel. But only if the machine you are running on has multiple cores that can host the script block threads. In this case the -ThrottleLimit parameter should be set approximately to the number of available cores. If you are running on a VM with a single core, then it makes little sense to run high compute script blocks in parallel since the system must serialize them anyway to run on the single core.
  2. Script that must wait on something. If you have script that can run independently and performs long running work that requires waiting for somethings to complete, then it makes sense to run these tasks in parallel. If you have 5 scripts that take 5 minutes each to run but spend most of the time waiting, you can have them all run/wait at the same time, and complete all 5 tasks in 5 minutes instead of 25 minutes. Scripts that do a lot of file operations, or perform operations on external machines can benefit by running in parallel. Since the running script cannot use all of the machine cores, it makes sense to set the -ThrottleLimit parameter to something greater than the number of cores. If one script execution waits many minutes to complete, you may want to allow tens or hundreds of scripts to run in parallel.
$logNames.count 
10

Measure-Command { 
    $logs = $logNames | ForEach-Object -Parallel {
        Get-WinEvent -LogName $_ -MaxEvents 5000 2>$null
    } -ThrottleLimit 10
}

TotalMilliseconds : 115994.3 (1 minute 56 seconds)
$logs.Count
50000


Measure-Command {
    $logs = $logNames | ForEach-Object {
        Get-WinEvent -LogName $_ -MaxEvents 5000 2>$null
    } 
}

TotalMilliseconds : 229768.2364 (3 minutes 50 seconds)
$logs.Count
50000

The script above collects 50,000 log entries on the local machine from 10 system log names. Running this in parallel is almost twice as fast as running sequentially, because it involves some relatively slow disk access and can also take advantage of the machine multiple cores as it processes the log entries.

When should it be avoided?

ForEach-Object -Parallel should not be thought as something that will always speed up script execution. And in fact it can significantly slow down script execution if used heedlessly. For example, if your script block is executing trivial script then running in parallel adds a huge amount of overhead and will run much slower.

(Measure-Command {
    1..1000 | ForEach-Object -Parallel { "Hello: $_" } 
}).TotalMilliseconds
10457.962


(Measure-Command {
    1..1000 | ForEach-Object { "Hello: $_" } 
}).TotalMilliseconds
18.4473

The above example, a trivial script block is run 1000 times. The ThrottleLimit is 5 by default so only 5 runspace/threads are created at a time, but still a runspace and thread is created 1000 times to do a simple string evaluation. Consequently, it takes over 10 seconds to complete. But removing the -Parallel parameter and running the ForEach-Object cmdlet normally, results in completion in about 18 milliseconds.

So, it is important to use this feature wisely.

Implementation details

As previously mentioned, the new ForEach-Object -Parallel feature uses existing PowerShell functionality to run script blocks concurrently. The primary addition is the ability to limit the number of concurrent scripts running at a given time with the -ThrottleLimit parameter. Throttling is accomplished by a PSTaskPool class that holds running tasks (running scripts), and has a settable size limit which is set to the throttle limit value. An Add method allows tasks to be added to the pool, but if it is full then the method blocks until a new slot becomes available. Adding tasks to the task pool was initially performed on the ForEach-Object cmdlet piped input processing thread. But that turned out to be a performance bottleneck, and now a dedicated thread is used to add tasks to the pool.

PowerShell itself imposes conditions on how scripts run concurrently, based on its design and history. Scripts have to run in runspace contexts and only one script thread can run at a time within a runspace. So in order to run multiple scripts simultaneously multiple runspaces must be created. The current implementation of ForEach-Object -Parallel creates a new runspace for each script block execution instance. It may be possible to optimize this by re-using runspaces from a pool, but one concern in doing this is leaking state from one script execution to another.

Runspace contexts are an isolation unit for running scripts, and generally do not allow sharing state between themselves. However, variables can be passed at the beginning of script execution through the $using: keyword, from the calling script to the parallel script block. This was borrowed from the remoting layer which uses the keyword for the same purpose but over a remote connection. But there is a big difference when using the $using: keyword in ForEach-Object -Parallel. And that is for remoting, the variable being passed is a copy sent over the remoting connection. But with ForEach-Object -Parallel, the actual object reference is being passed from one script to another, violating normal isolation restrictions. So it is possible to have a non thread-safe variable used in two scripts running on different threads, which can lead to unpredictable behavior.

# This does not throw an error, but is not guaranteed to work since the dictionary object is not thread safe 
$threadUnSafeDictionary = [System.Collections.Generic.Dictionary[string,object]]::new()
Get-Process | ForEach-Object -Parallel {
    $dict = $using:threadUnSafeDictionary
    $dict.TryAdd($_.ProcessName, $_)
}
# This *is* guaranteed to work because the passed in concurrent dictionary object is thread safe
$threadSafeDictionary = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()
Get-Process | ForEach-Object -Parallel {
    $dict = $using:threadSafeDictionary
    $dict.TryAdd($_.ProcessName, $_)
}

$threadSafeDictionary["pwsh"]

NPM(K) PM(M) WS(M) CPU(s) Id SI ProcessName
------ ----- ----- ------ -- -- -----------
112 108.25 124.43 69.75 16272 1 pwsh

Conclusion

This feature can greatly improve your life for many work load scenarios. As long as you understand how it works and what its limitations are, you can experiment with parallelism and make real performance improvements with your scripts.

Paul Higinbotham
Senior Software Engineer
PowerShell Team

The post PowerShell ForEach-Object Parallel Feature appeared first on PowerShell.

Release of PowerShell Script Analyzer (PSScriptAnalyzer) 1.18.2

$
0
0

In keeping with the tradition of releasing improvements to PSScriptAnalyzer more often, we’re happy to announce that 1.18.12 is now available! As a dependency of PowerShell Editor Services (a module used by editor extensions like the PowerShell Visual Studio Code extension), this release is motivated by a desire to further stabilize our editor experience. At the moment, the Visual Studio Code PowerShell extension still ships with PSScriptAnalyzer 1.18.0. After fixing some undesirable edge cases between 1.18.1 and 1.18.2, we intend to ship an update to the Visual Studio Code extension that will include 1.18.2.

The blocking issue that it resolves is quite technical and should not concern end-users, but for those who are interested: starting with1.18.1, a performance optimization was added whereby we started to share and cache a PowerShell runspace pool instead of creating a new one for every command invocation. However, it turns out that there is an edge case where, when dealing with specific commands from thePackageManagementmodule, the runspace pool can get into a deadlock, which causes the execution of PSScriptAnalyzer to hang indefinitely. This is due to a bug inPackageManagementitself (a very unfortunate asynchronous API call that leads to the deadlock) but also PowerShell itself, which should be able to handle bad scenarios like this. Therefore, a workaround for this had to be implemented in PSScriptAnalyzer by blacklisting the PackageManagement commands.

Given that the other changes in this release are mainly fixes and small enhancements, we decided to not bump the minor version number. We ask that the community participate in testing and giving feedback on this update before it ships by default in the Visual Studio Code extension. You can make this new update with the Visual Studio Code extension start by executing the following command:

Install-Module -Name PSScriptAnalyzer -Repository PSGallery -Scope CurrentUser

Should you find that there are changes that you are not happy with, please report them here.

Optionally, you can roll back to the default included version of PSScriptAnalyzer by running Uninstall-Module -Name PSScriptAnalyzer.

In this release, we’ve made the following fixes

  • PipelineIndentation: More edge cases when using non-default values of this setting (NoIndentation in the Visual Studio Code extension) were fixed. This feature was only introduced in1.18.0and we hope the be closer to a state now where we could potentially change the default.
  • New compatibility rule profiles were added for non-Windows OSs on PowerShell 7 (preview). Additionally, fixes were made to profile generation to support macOS and Linux.
  • A fix was made to PSCloseBrace to correctly flag the closing brace of a one-line hashtable, correcting some broken formatting.

Enhancements were made in the following areas

  • When using settings files, error messages are now much more actionable.

PS> Invoke-ScriptAnalyzer -Settings /tmp/MySettings.psd1 -ScriptDefinition 'gci'

Invoke-ScriptAnalyzer : includerule is not a valid key in the settings hashtable.
Valid keys are CustomRulePath, ExcludeRules, IncludeRules, IncludeDefaultRules,
RecurseCustomRulePath, Rules and Severity. 
...

  • PSScriptAnalyzer has a logo now thanks to the community member @adilio
  • The formatter was enhanced to also take commented lines into account in multi-line commands
  • The formatter was enhanced to optionally allow correction of aliases as well. With this change, a setting in the Visual Studio Code extension will soon be made available to configure this. By default, this setting will not be on for the moment. We are open to feedback: while there are very likely a few people that would love for it to be enabled, it may upset others.
  • UseDeclaredVarsMoreThanAssignmentsnow also takes into account the usage of Get-Variable with an array of variables and usage of the named parameter -Name

We’ve also made some changes in our GitHub repository and changed the default branch from development to master to simplify the development workflow and be consistent with other repositories in the PowerShell organization. If you have a fork of the project, you will need to make this change in your fork as well or remember to use master as a base and open pull requests against master. This also means that the next version of the Visual Studio Code extension will point tomasterfor the documentation of PSScriptAnalyzer’s rules.

The Changelog has more details if you want to dig further.

Future Directions

We are thinking of following an approach similar to the Visual Studio Code extension where we make a version 2.0 at that drops support for PowerShell version 3 and 4. One of the next changes could be to improve how PowerShellEditorServices calls into PSScriptAnalyzer: currently, Editor Services uses the PSScriptAnalyzer PowerShell cmdlets which means that we have to create an entire instance of PowerShell for these invocations. Knowing that bothPowerShellEditorServicesandPSScriptAnalyzerare binary .NET modules, we could directly call into PSScriptAnalyzer’s .NET code by publishing a NuGet package of PSScriptAnalyzer with suitable public APIs. Given that PSScriptAnalzyer currently performs a conditional compilation for each PowerShell version (3, 4, 5, and 6+), dropping support for version 3 and 4 could help make the aforementioned move to an API model much easier to implement. Please give feedback if your use case ofPSScriptAnalyzerwould be impacted by this.

On behalf of the Script Analyzer team,

Christoph Bergmeister, Project Maintainer from the community, BJSS
Jim Truher, Senior Software Engineer, Microsoft

The post Release of PowerShell Script Analyzer (PSScriptAnalyzer) 1.18.2 appeared first on PowerShell.

Updating Help for older versions of PowerShell

$
0
0

It turns out, maintaining Updateable Help is difficult. The steps required to build and host the updated files are mostly manual. We made improvements to our build automation in the release of PowerShell 6 and now host the help files in Azure blob storage. However, for various internal reasons, we haven’t been able to change the way we distribute the help files for previous versions. Because of that, the downloadable help files for version 5.1 and older have not been updated since PowerShell 6.0 released.

Manually update local help from source

The good news is we have been updating the content. You can see that on the Docs site. And, since the documentation is open source, you can build the updated help content locally and install it on any machine you want. To make it easier to build a specific version, I have created a script that creates the files you need. The process is simple:

  1. Clone or download the PowerShell-Docs repository
  2. Run build-updatedhelp.ps1 to build version of the documentation you want to update
  3. Run Update-Help to install the newly built help

Prerequisites

Almost all the documentation on the Microsoft Docs platform is written in Markdown. This is the case for PowerShell documentation. However, the help files used by PowerShell are plain text files for About_ topics and Microsoft Assistance Markup Language (MAML) files for cmdlet reference. MAML is XML conforming to a well defined schema. The MAML schema describes the structure of the help content. There is no layout or style information within the MAML files. Rendering this help content in PowerShell is managed by the Get-Help cmdlet. To build the content for Get-Help, we must convert the Markdown source files to plain text and MAML. This is done using the following tools:

  • PlatyPS – an open source tool that creates PowerShell help content – Converts Markdown to MAML.
  • Pandoc – an open source tool that converts documents to or from many different formats – Converts markdown to plain text for About topics.
  • build-updatedhelp.ps1 – This script downloads PlatyPS and Pandoc, then builds CAB files that can be installed using Update-Help.

Step by step instructions

  1. Clone or download the PowerShell-Docs repository

    cd C:\temp
    git clone https://github.com/MicrosoftDocs/PowerShell-Docs.git --depth 1
     Output
    Cloning into 'PowerShell-Docs'...
    remote: Enumerating objects: 3588, done.
    remote: Counting objects: 100% (3588/3588), done.
    remote: Compressing objects: 100% (2169/2169), done.
    remote: Total 3588 (delta 2270), reused 1754 (delta 1403), pack-reused 0
    Receiving objects: 100% (3588/3588), 8.63 MiB | 14.70 MiB/s, done.
    Resolving deltas: 100% (2270/2270), done.
    Updating files: 100% (3905/3905), done.

    NOTE: Cloning the repository using --depth 1 minimizes the history that is downloaded. This reduces the download size. This is the quickest way to get the latest content. If you intend to contribute changes to the documentation, you should first create a fork then clone your fork.

  2. Run build-updatedhelp.ps1 to build version of the documentation you want to update. This script is in the tools folder of the PowerShell-Docs repository. This is script is based on thebuild.ps1 script that is used to build the the documentation today. This script downloads PlatyPS and Pandoc then builds the documentation in the target folder. For example, run the following command to build the help content for PowerShell v5.1:

    C:\temp\PowerShell-Docs\tools\build-updatedhelp.ps1 -sourceFolder C:\temp\PowerShell-Docs\reference\5.1 -Verbose

    The script builds the updateable help content for all Markdown source files in each module subfolder.

  3. Run Update-Help to install the newly built help

    This command must be run within the version of PowerShell that is being updated. Update-Help overwrites the local help content with whatever content you target.

    Installing help in PowerShell 5.1 (and older) requires administrative rights. You must run the following command from an elevated session:

    Update-Help -SourcePath c:\temp\updatablehelp\5.1 -Recurse -Force

    When the Update-Help command completes, you may receive an error message similar to this:

    Update-Help : Failed to update Help for the module(s) 'ActiveDirectory, AppBackgroundTask,
    AppLocker, AppvClient, Appx, AssignedAccess, BestPractices, BitLocker, BitsTransfer, BranchCache,
    ConfigCI, Defender, DirectAccessClientComponents, Dism, DnsClient, EventTracingManagement,
    HgsClient, HgsDiagnostics, HostNetworkingService, Hyper-V, IISAdministration, International,
    iSCSI, IscsiTarget, Kds, MMAgent, MsDtc, NetAdapter, NetConnection, NetEventPacketCapture,
    NetLbfo, NetLldpAgent, NetNat, NetQos, NetSecurity, NetSwitchTeam, NetTCPIP,
    NetworkConnectivityStatus, NetworkControllerDiagnostics, NetworkSwitchManager, NetworkTransition,
    NFS, PcsvDevice, PKI, PnpDevice, PrintManagement, ProcessMitigations, Provisioning,
    RemoteDesktop, ScheduledTasks, SecureBoot, ServerManager, ServerManagerTasks, SmbShare,
    SmbWitness, StartLayout, Storage, TLS, TroubleshootingPack, TrustedPlatformModule, UEV,
    VpnClient, Wdac, WebAdministration, Whea, WindowsDeveloperLicense, WindowsErrorReporting,
    WindowsSearch, WindowsUpdate, WindowsUpdateProvider' with UI culture(s) {en-US} : Unable to
    retrieve the HelpInfo XML file for UI culture en-US. Make sure the HelpInfoUri property in the
    module manifest is valid or check your network connection and then try the command again.
    
    At line:1 char:1
    +Update-Help -SourcePath c:\temp\updatablehelp\5.1 -Recurse -Force
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     + CategoryInfo          : ResourceUnavailable: (:) [Update-Help], Exception
     + FullyQualifiedErrorId : UnableToRetrieveHelpInfoXml,Microsoft.PowerShell.Commands.UpdateHel

    This message is warning you that the update did not include help for the modules listed. The content for those modules is also kept up-to-date on Microsoft Docs, but the script we’ve provided doesn’t yet support the windows-powershell-docs repo where that content is hosted. However, it could potentially be extended to support that repo (and others) too, so stay tuned for updates.

NOTE: The build script downloads the supported versions Pandoc and PlatyPS to a temp folder on your computer. It does not overwrite version you may have installed already. The files are left behind in the temp folder.

Further reading

 

The post Updating Help for older versions of PowerShell appeared first on PowerShell.

PowerShell 7 Preview 4

$
0
0

We continue to make progress towards our PowerShell 7 release which currently is targeting December 2019 for a Release Candidate and January 2020 for General Availability and will be our first LTS (Long Term Servicing) release!

Please see the previous blog post on Preview 3 for more details about LTS and also Windows PowerShell compatibility.

Preview 4 contains a number of bug fixes, but also new features which I’ll cover in this blog post.

New Experimental Features in Preview 4

This is just a small part of the entire changelog. New experimental features in this preview from the community and also the PowerShell team:

Ternary Operator

The ternary operator is popular among C# developers due to its terseness which can improve readability if you are familiar with this operator.

This operator is completely opt-in so if you prefer to use if..else instead, you can certainly continue to do that.

gif

Start-Job -WorkingDirectory

Those of you familiar with the Start-Job cmdlet will have encountered that the new PowerShell process started to handle the job will have different working directory on Windows PowerShell and PowerShell Core and it can sometimes be not what you expected. This parameter was added to allow you to specify the working directory of the new job process before your script block runs!

gif

$ErrorActionPreference = “Break”

This feature comes from a well known PowerShell MVP Kirk Munro. Basically, if you set $ErrorActionPreference to Break, then when there is an error it will drop you into the debugger immediately!

gif

Invoke-DscResource

With this change, you can now leverage DSC Resources while by-passing the LCM (Local Configuration Manager). This means that you can author your own LCM or simply leverage existing DSC Resources within your scripts and this also works cross platform!

Note that binary DSC Resources are not supported!

gif

DSC Configuration Compilation

Previously if you authored a DSC Configuration script, you would need to use a Windows machine to compile it to a mof file to deploy onto your managed node. Starting with Preview4, you can now perform DSC compilation on non-Windows systems.

Note that this is work in progress with some known issues.

gif

Testing the MSIX package

Recently, we started publishing a MSIX package for Windows. This will eventually allow us to publish PowerShell 7 to the Windows Store. For now, if you wish to try out this package, you must be in Developer Mode and use Add-AppxPackage to install it. Double clicking it from the Windows Shell will not allow you to install the developer signed package.

Experimental Features

In Preview3 we announced that Experimental Features would be enabled by default. The way this was done is to create a system wide powershell.config.json file listing each of the Experimental Features which would make them enabled. However, if you had previously explicitly enabled Experimental Features, then you would have created a user specific powershell.config.json which would take precedence which means that any features you didn’t explicitly enable would be disabled.

To get all Experimental Features, you can:

Get-ExperimentalFeature | Enable-ExperimentalFeature

 

or delete your user specific powershell.config.json (assuming you didn’t put any other specific configuration settings in that file).

Closing

Although this blog post focuses on new features, this release also contains many bug fixes as well as targeted performance improvements.

You can always get the latest version of PowerShell from https://aka.ms/get-powershell.

Expect more new features from the community and the PowerShell team in future Preview releases!

Steve Lee
PowerShell Team

The post PowerShell 7 Preview 4 appeared first on PowerShell.

DSC Resource Kit Release September 2019

$
0
0

We just released the DSC Resource Kit!

This release includes updates to 15 DSC resource modules. In the past 6 weeks, 160 pull requests have been merged and 68 issues have been closed, all thanks to our amazing community!

The modules updated in this release are:

  • ActiveDirectoryCSDsc 4.1.0.0
  • ActiveDirectoryDsc 4.1.0.0
  • ComputerManagementDsc 7.0.0.0
  • DFSDsc 4.4.0.0
  • NetworkingDsc 7.4.0.0
  • SecurityPolicyDsc 2.10.0.0
  • SqlServerDsc 13.2.0.0
  • xDnsServer 1.15.0.0
  • xExchange 1.29.0.0
  • xFailOverCluster 1.13.0.0
  • xPSDesiredStateConfiguration 8.10.0.0
  • xRemoteDesktopSessionHost 1.9.0.0
  • xSCSMA 2.1.0.0
  • xWebAdministration 2.8.0.0

For a detailed list of the resource modules and fixes in this release, see the Included in this Release section below.

Our latest community call for the DSC Resource Kit was last Wednesday, September 11. A recording of the call is posted on the PowerShell YouTube channel. You can join us for the next call at 12PM (Pacific time) on August 28th to ask questions and give feedback about your experience with the DSC Resource Kit.

The next DSC Resource Kit release will be on Wednesday, October 9.

We strongly encourage you to update to the newest version of all modules using the PowerShell Gallery, and don’t forget to give us your feedback in the comments below, on GitHub, or on Twitter (@PowerShell_Team)!

Please see our documentation here for information on the support of these resource modules.

Included in this Release

You can see a detailed summary of all changes included in this release in the table below. For past release notes, go to the README.md or CHANGELOG.md file on the GitHub repository page for a specific module (see the How to Find DSC Resource Modules on GitHub section below for details on finding the GitHub page for a specific module).

Module Name Version Release Notes
ActiveDirectoryCSDsc 4.1.0.0
  • AdcsCertificationAuthoritySettings:
    • Fix grammar in the resource README.md.
  • Fix minor style issues in statement case.
ActiveDirectoryDsc 4.1.0.0
    • We could not add the change log to the release notes due to the length of the change log. What have change in this release can be found here

https://github.com/PowerShell/ActiveDirectoryDsc/blob/dev/CHANGELOG.md#4100

    .
ComputerManagementDsc 7.0.0.0
  • ScheduledTask:
    • Better compatibility with Group LogonType when passing BuiltIn groups through ExecuteAsCredential
      • Primary use case is “BUILTIN\Users”
      • Use the ExecuteAsCredential property to pass the username The PSCredential needs a non-null that is ignored
    • Delay property not handled properly on AtLogon and AtStartup trigger – Fixes Issue 230
    • Changed Get-ScheduledTask calls to ScheduledTasks\Get-ScheduledTask to avoid name clash with Carbon module. Fixes Issue 248
    • Cast MultipleInstances value returned by Get-TargetResource to string – fixes Issue 255
  • PendingReboot:
    • Migrated xPendingReboot from xPendingReboot and renamed to PendingReboot.
    • Converted to meet HQRM guidelines – Fixes Issue 12.
    • Changed SkipCcmClientSDK parameter to default to $true – Fixes Issue 13.
    • Fixed Test-TargetResource so that if ConfigMgr requires a reboot then the pending reboot will be set – Fixes Issue 26.
    • Refactored Test-TargetResource to reduce code duplication and move to a data driven design.
    • Refactored Get-TargetResource by adding a new function Get-PendingRebootState so that Test-TargetResource no longer needed to use Get-TargetResource. This eliminated the need to include write parameters in Get-TargetResource.
    • Converted the call to Invoke-WmiMethod to Invoke-CimMethod.
    • Deleted the code that removes the regRebootLocations variable at the end of the resource as it appears to serve no purpose.
  • Correct all tests to meet Pester 4.0 standards.
  • RemoteDesktopAdmin:
    • New resource for configuring Remote Desktop for Administration – fixes Issue 224.
  • Updated common function Test-DscParameterState to support ordered comparison of arrays by copying function and tests from NetworkingDsc – fixes Issue 250.
  • BREAKING CHANGE: ScheduledTask:
    • Correct output type of DaysInterval,StartTime,WeeksDaysOfWeek, and WeeksInterval parameters from Get-TargetResource to match MOF.
    • Refactored Get-TargetResource to remove parameters that are not key or required – fixes Issue 249.
    • Added function Test-DateStringContainsTimeZone to determine if a string containing a date time includes a time zone.
    • Enable verbose preference to be passed through to Test-DscParameterState.
    • Changed Test-TargetResource so that StartTime is only compared for trigger types Daily,Weekly or Once.
  • Fix minor style issues in statement case.
DFSDsc 4.4.0.0
  • Fix example publish to PowerShell Gallery by adding gallery_api environment variable to AppVeyor.yml – fixes Issue 91.
  • Fix minor style issues in statement case.
NetworkingDsc 7.4.0.0
  • Added Comment Based Help for New-NotImplementedException common function – fixes Issue 411.
  • Added common function “Format-Win32NetworkADapterFilterByNetConnectionID” to properly accept wild cards for Win32_NetworkAdapter filters.
  • Updated MSFT_Netbios to use “Format-Win32NetworkADapterFilterByNetConnectionID”
  • Corrected minor style and consistency issues in NetworkingDsc.Common.tests.ps1 and NetworkingDsc.Common.ps1.
  • Changed verbose messages in Test-DscParameterState to include full type name.
  • Fixed bug in Test-DscParameterState that causes it to return true when both the current array and desired array is empty.
  • Fix minor style issues in statement case.
SecurityPolicyDsc 2.10.0.0
  • Changes to SecurityPolicyDsc
    • Opt-in to the following DSC Resource Common Meta Tests:
      • Common Tests – Validate Module Files
      • Common Tests – Validate Script Files
      • Common Tests – Validate Markdown Files
      • Common Tests – Required Script Analyzer Rules
      • Common Tests – Flagged Script Analyzer Rules
      • Common Tests – New Error-Level Script Analyzer Rules
      • Common Tests – Custom Script Analyzer Rules
      • Common Tests – Validate Markdown Links
      • Common Tests – Relative Path Length
      • Common Tests – Validate Example Files
      • Common Tests – Validate Example Files To Be Published
    • Fix keywords to lower-case to align with guideline.
SqlServerDsc 13.2.0.0
  • Changes to SqlServerDsc
    • Fix keywords to lower-case to align with guideline.
    • Fix keywords to have space before a parenthesis to align with guideline.
xDnsServer 1.15.0.0
xExchange 1.29.0.0
  • Enable Script Analyzer default rules
  • Fixed keywords in upper case
xFailOverCluster 1.13.0.0
  • Updated the xCluster test method to return true if a node is joined to the cluster but is in a Paused state.
xPSDesiredStateConfiguration 8.10.0.0
  • Changes to xPSDesiredStateConfiguration
    • Fix keywords to lower-case to align with guideline.
  • Added SMB PullServer support for publishing.
xRemoteDesktopSessionHost 1.9.0.0
  • Changes to xRDRemoteApp
    • Fixing typo in parameter name when calling the function ValidateCustomModeParameters (issue 50).
  • Changes to xRDSessionDeployment
    • When RDMS service does not exist the Get-TargetResource will no longer throw an error (issue 47).
  • Rename Tests/Unit folder to use upper case on first letter.
  • Update appveyor.yml to use the default template.
  • Added default template files .codecov.yml, .gitattributes, and .gitignore, and .vscode folder.
  • xRDSessionCollectionConfiguration:
    • Changed CollectionName variable validation max length to 256
  • xRDSessionCollection
    • Changed CollectionName variable validation max length to 256
  • xRDRemoteApp
    • Changed CollectionName variable validation max length to 256
xSCSMA 2.1.0.0
  • Update appveyor.yml to use the default template.
  • Added default template files .codecov.yml, .gitattributes, and .gitignore, and .vscode folder.
  • Closed issue 29 – Web bindings fail due to hardcoded WSE
  • Switched from Get-WmiObject Win32_Product to Get-ItemProperty for identifer number
xWebAdministration 2.8.0.0
  • Fix multiple HTTPS bindings on one xWebsite receiving the first binding”s certificate 332
    • Added unit regression test
  • Changes to xWebsite
    • Added ServerAutoStart (controls website autostart) and changed documentation for ServiceAutoStartEnabled (controls application auto-initialization). Fixes 325.
    • Fix multiple HTTPS bindings on one xWebsite receiving the first binding”s certificate 332
      • Added unit regression test
    • Changes to xWebAppPool
      • Fix false Test-TargetResource failure for logEventOnRecycle if items in the Configuration property are specified in a different order than IIS natively stores them 434
    • Changes to xIisModule
      • Fixed the parameters specification for the internal Get-IISHandler and Remove-IISHandler function

How to Find Released DSC Resource Modules

To see a list of all released DSC Resource Kit modules, go to the PowerShell Gallery and display all modules tagged as DSCResourceKit. You can also enter a module’s name in the search box in the upper right corner of the PowerShell Gallery to find a specific module.

Of course, you can also always use PowerShellGet (available starting in WMF 5.0) to find modules with DSC Resources:

# To list all modules that tagged as DSCResourceKit
Find-Module -Tag DSCResourceKit 
# To list all DSC resources from all sources 
Find-DscResource

Please note only those modules released by the PowerShell Team are currently considered part of the ‘DSC Resource Kit’ regardless of the presence of the ‘DSC Resource Kit’ tag in the PowerShell Gallery.

To find a specific module, go directly to its URL on the PowerShell Gallery:
http://www.powershellgallery.com/packages/< module name >
For example:
http://www.powershellgallery.com/packages/xWebAdministration

How to Install DSC Resource Modules From the PowerShell Gallery

We recommend that you use PowerShellGet to install DSC resource modules:

Install-Module -Name < module name >

For example:

Install-Module -Name xWebAdministration

To update all previously installed modules at once, open an elevated PowerShell prompt and use this command:

Update-Module

After installing modules, you can discover all DSC resources available to your local system with this command:

Get-DscResource

How to Find DSC Resource Modules on GitHub

All resource modules in the DSC Resource Kit are available open-source on GitHub.
You can see the most recent state of a resource module by visiting its GitHub page at:
https://github.com/PowerShell/< module name >
For example, for the CertificateDsc module, go to:
https://github.com/PowerShell/CertificateDsc.

All DSC modules are also listed as submodules of the DscResources repository in the DscResources folder and the xDscResources folder.

How to Contribute

You are more than welcome to contribute to the development of the DSC Resource Kit! There are several different ways you can help. You can create new DSC resources or modules, add test automation, improve documentation, fix existing issues, or open new ones.
See our contributing guide for more info on how to become a DSC Resource Kit contributor.

If you would like to help, please take a look at the list of open issues for the DscResources repository.
You can also check issues for specific resource modules by going to:
https://github.com/PowerShell/< module name >/issues
For example:
https://github.com/PowerShell/xPSDesiredStateConfiguration/issues

Your help in developing the DSC Resource Kit is invaluable to us!

Questions, comments?

If you’re looking into using PowerShell DSC, have questions or issues with a current resource, or would like a new resource, let us know in the comments below, on Twitter (@PowerShell_Team), or by creating an issue on GitHub.

Michael Greene
Principal Program Manager
PowerShell DSC Team
@migreene (Twitter)
@mgreenegit (GitHub)

The post DSC Resource Kit Release September 2019 appeared first on PowerShell.

PowerShell 7 Preview 5

$
0
0

Today we shipped PowerShell 7 Preview5! This release contains a number of new features and many bug fixes from both the community as well as the PowerShell team. See the Release Notes for all the details of what is included in this release.

We are still on track to have one more preview release next month in November. Then, barring any quality concerns, a Release Candidate in December aligned with the .NET Core 3.1 final release. Finally, we expect General Availability of PowerShell 7 in January as our first Long Term Servicing release.

Between the Release Candidate and General Availability, we will only accept critical bug fixes and no new features will be included. For that release, some Experimental Features will be considered design stable and no longer be Experimental. This means that any future design changes for those features will be considered a breaking change.

New Features in Preview 5

This release has a number of new features from both the community as well as the PowerShell team. Remember that preview releases of PowerShell are installed side-by-side with stable versions so you can use both and provide us feedback on previews for bugs and also on experimental features.

You can read about new features in previous preview releases:

There were new features in Preview 1 and Preview 2, but I didn’t blog about them… sorry!

Chain operators

The new Pipeline Chain Operators allow conditional execution of commands depending on whether the previous command succeeded for failed. This works with both native commands as well as PowerShell cmdlets or functions. Prior to this feature, you could already do this by use of if statements along with checking if $? indicated that the last statement succeeded or failed. This new operator makes this simpler and consistent with other shells.

img

Null conditional operators for coalescing and assignment

Often in your scripts, you may need to check if a variable is $null or if a property is $null before using it. The new Null conditional operators makes this simpler.

The new ?? null coalescing operator removes the need for if and else statements if you want to get the value of a statement if it’s not $null or return something else if it is $null. Note that this doesn’t replace the check for a boolean value of true or false, it’s only checking if the returned value is $null.

The new ??= null conditional assignment operator makes it easy to assign a variable a value only if the variable value is $null.

img

New PowerShell version notification

Our telemetry available in our PowerBI Dashboard indicates some set of users are still using older versions (sometimes older previews of released stable versions!). This new feature will inform you on startup if a new preview version is available (if you are using a preview version) or if a new stable version is available to keep you up-to-date on the latest servicing release which may contain security fixes. Because this is new, you won’t see this in action until Preview 6 comes out.

More details of this feature including how to disable it in the Notification on Version Update RFC

img

Tab completion for variable assignment

This new feature will allow you to use tab completion on variable assignment and get allowed values for enums or variables with type constraints like [ValidateSet()]. This makes it easy to change $ErrorActionPreference or the new $ErrorView (detailed below) to valid values without having to type them out.

img

Format-Hex improved formatting

This improvement comes from Joel Sallow making Format-Hex more useful when viewing different types of objects in a pipeline as well as supporting viewing more types of objects.

img

Get-HotFix is back

The Get-HotFix cmdlet only works on Windows and will query the system on what patches have been installed. This was previously unavailable in PowerShell Core 6 because it depended on System.Management namespace which wasn’t available on .NET Core 2.x which PowerShell Core 6.x is built on. However, .NET Core 3.0 which PowerShell 7 is built on brought back this namespace (for Windows only) so we re-enabled this cmdlet.

There is a delay getting results in this example due to the number of patches I have on my Windows 7 VM.

img

Select-String adds emphasis

This was a HackIllinois project by Derek Xia that uses inverse colored text to highlight the text in a string that matches the selection criteria. There is an optional -NoEmphasis switch to suppress the emphasis.

img

ConciseView for errors

Some user feedback we’ve consistently received is about the amount of red text you get when you encounter an error in PowerShell.

The $ErrorView preference variable allows you to change the formatting of errors. Previously, it supported NormalView (the default) as well as a more terse CategoryView. This feature adds a ConciseView where most commands return just the relevant error message. In cases where there is additional contextual information in a script file or the location in a script block, you get the line number, the line of text in question, and a pointer to where the error occurred.

This new view is part of the Update Error View RFC so please provide feedback there!

img

Get-Error cmdlet

While ConciseView gives you more precise, but limited information on errors, we added a new cmdlet Get-Error to get much richer information on errors.

By default, just running Get-Error shows a formatted view of the most recent error including showing specific nested types like Exceptions and ErrorRecords making it easier to diagnose what went wrong.

This new cmdlet is part of the Update Error View RFC so please provide feedback there!

img

Closing

We have one more preview planned for November with a few more features coming from the PowerShell team as well as the PowerShell community!

Steve Lee
PowerShell Team

The post PowerShell 7 Preview 5 appeared first on PowerShell.


DSC Resource Kit Release October 2019

$
0
0

DSC Resource Kit Release

We just released the DSC Resource Kit!

This release includes updates to 9 DSC resource modules. In the past 6 weeks, 91 pull requests have been merged and 41 issues have been closed, all thanks to our amazing community!

Special thanks to everyone who contributed to the Hacktoberfest effort to update xWebAdministration!!! This accounted for 26 of the pull requests closed this month.

The modules updated in this release are:

  • ActiveDirectoryDsc 4.2.0.0
  • ComputerManagementDsc 7.1.0.0
  • SharePointDsc 3.7.0.0
  • StorageDsc 4.9.0.0
  • xDnsServer .16.0.0
  • xDscResourceDesigner .13.0.0
  • xExchange .30.0.0
  • xHyper-V .17.0.0
  • xWebAdministration 3.0.0.0

For a detailed list of the resource modules and fixes in this release, see the Included in this Release section below.

Our latest community call for the DSC Resource Kit was last Wednesday, October 23. A recording of the call is posted on the PowerShell YouTube channel. You can join us for the next call at 12PM (Pacific time) on August 28th to ask questions and give feedback about your experience with the DSC Resource Kit.

Following this resource kit release, maintainers will begin publishing as soon as they are ready rather than holding 6 weeks to do a group release. In the next community call we will discuss progress and whether we need to do a November release or not. Be sure to follow the DSC Community on Twitter for live updates as modules release.

You can find more information about our progress as a community on the DSC Community page.

We strongly encourage you to update to the newest version of all modules using the PowerShell Gallery, and don’t forget to give us your feedback in the comments below, on GitHub, or on Twitter (@PowerShell_Team)!

Please see our documentation here for information on the support of these resource modules.

Included in this Release

You can see a detailed summary of all changes included in this release in the table below. For past release notes, go to the README.md or CHANGELOG.md file on the GitHub repository page for a specific module (see the How to Find DSC Resource Modules on GitHub section below for details on finding the GitHub page for a specific module).

Module Name Version Release Notes
ActiveDirectoryDsc 4.2.0.0
  • Changes to ActiveDirectoryDsc
    • Resolved custom Script Analyzer rules that was added to the test framework.
    • Resolve style guideline violations for hashtables (issue 516).
  • Changes to ADReplicationSite
    • Added “Description” attribute parameter (issue 500).
    • Added Integration testing (issue 355).
    • Correct value returned for RenameDefaultFirstSiteName (issue 502).
  • Changes to ADReplicationSubnet
    • Added “Description” attribute parameter (issue 503)
    • Added Integration testing (issue 357)
  • Changes to ADReplicationSiteLink
    • Added Integration testing (issue 356).
    • Added ability to set “Options” such as Change Notification Replication (issue 504).
  • Changes to ActiveDirectoryDsc.Common
    • Fix Test-DscPropertyState Failing when Comparing $Null and Arrays. (issue 513)
ComputerManagementDsc 7.1.0.0
  • ComputerManagementDsc:
    • Update psd1 description – Fixes Issue 269.
  • Fix minor style issues with missing spaces between param statements and “(“.
  • SmbServerConfiguration:
    • New resource for configuring the SMB Server settings.
    • Added examples for SMB Server Configuration.
  • Minor corrections to CHANGELOG.MD.
  • ScheduledTask:
    • Fixed bug when description has any form of whitespace at beginning or end the resource would not go into state – Fixes Issue 258.
  • SmbShare:
    • Removal of duplicate code in Add-SmbShareAccessPermission helper function fixes Issue 226.
SharePointDsc 3.7.0.0
        • SPConfigWizard
          • Fixed issue with incorrect check for upgrade status of server
        • SPDistributedCacheService
          • Improved error message for inclusion of server name into ServerProvisionOrder parameters when Present or change to Ensure Absent
        • SPFarm
          • Removed SingleServer as ServerRole, since this is an invalid role.
          • Handle case where null or empty CentralAdministrationUrl is passed in
          • Move CentralAdministrationPort validation into parameter definition to work with ReverseDsc
          • Add NotNullOrEmpty parameter validation to CentralAdministrationUrl
          • Fixed error when changing developer dashboard display level.
          • Add support for updating Central Admin Authentication Method
        • SPFarmSolution
          • Fix for Web Application scoped solutions.
        • SPInstall
          • Fixes a terminating error for sources in weird file shares
          • Corrected issue with incorrectly detecting SharePoint after it has been uninstalled
          • Corrected issue with detecting a paused installation
        • SPInstallLanguagePack
          • Fixes a terminating error for sources in weird file shares
        • SPInstallPrereqs
          • Fixes a terminating error for sources in weird file shares
        • SPProductUpdate
          • Fixes a terminating error for sources in weird file shares
          • Corrected incorrect farm detection, added in earlier bugfix
        • SPSite
          • Fixed issue with incorrectly updating site OwnerAlias and SecondaryOwnerAlias
        • SPWebAppAuthentication
          • Fixes issue where Test method return false on NON-US OS.
StorageDsc 4.9.0.0
  • Disk:
    • Added Location as a possible value for DiskIdType. This will select the disk based on the Location property returned by Get-Disk
    • Maximum size calculation now uses workaround so that Test-TargetResource works properly – workaround for Issue 181.
  • DiskAccessPath:
    • Added Location as a possible value for DiskIdType. This will select the disk based on the Location property returned by Get-Disk
  • WaitForDisk:
    • Added Location as a possible value for DiskIdType. This will select the disk based on the Location property returned by Get-Disk
xDnsServer 1.16.0.0
  • Changes to XDnsServerADZone
    • Raise an exception if DirectoryPartitionName is specified and ReplicationScope is not Custom. (issue 110).
    • Enforce the ReplicationScope parameter being passed to Set-DnsServerPrimaryZone if DirectoryPartitionName has changed.
  • xDnsServer:
    • OptIn to the following Dsc Resource Meta Tests:
      • Common Tests – Relative Path Length
      • Common Tests – Validate Markdown Links
      • Common Tests – Custom Script Analyzer Rules
      • Common Tests – Required Script Analyzer Rules
      • Common Tests – Flagged Script Analyzer Rules
xDscResourceDesigner 1.13.0.0
  • Fix Parameter Blocks to conform to Dsc Style Guidlelines issue 79.
  • Fix README.md MarkDownLint Errors and Formatting Issues
xExchange 1.30.0.0
  • Resolved custom Script Analyzer rules that was added to the test framework.
  • Added xExchAcceptedDomain resource
  • Resolved hashtable styling issues
  • Added xExchRemoteDomain resource
xHyper-V 3.17.0.0
  • MSFT_xVMNetworkAdapter:
    • Added NetworkSettings to be able to statically set IPAddress.
    • Added option for Vlan tagging. You can now setup a Network Adapeter as an access switch on a specific Vlan.
xWebAdministration 3.0.0.0
  • Changes to xWebAdministration
    • Changes to PULL_REQUEST_TEMPLATE.md
      • Improving descriptive text around the CHANGELOG.md entry.
      • Adding note that entry in CHANGELOG.md is mandatory for all PRs.
    • Resolved custom Script Analyzer rules that was added to the test framework.
    • Moved change log from README.md to a separate CHANGELOG.md (issue 446).
    • Remove example “Creating the default website using configuration data” from README.md (issue 488).
    • Removed examples README.md as it was obsolete (issue 482).
    • Updated Ensure property description for xIisHandler resource to match schema.mof
    • Moved examples from Readme.md to respective /Examples/Resources/ folders (issue 486).
    • Created new folder structure for examples so that examples will be placed in /Examples/Resources/$resourceName (issue 483).
    • Added a table of contents for the resource list (issue 450).
    • Alphabetized the resource list in the README.md (issue 449).
    • Optimized exporting in the module manifest for best performance (issue 448).
    • Updated hashtables in the repo to adhere to the style guidelines described at https://github.com/PowerShell/DscResources/blob/master/StyleGuidelines.mdcorrect-format-for-hashtables-or-objects (issue 524)
    • Moved example Sample_EndToEndxWebAdministration from readme.md to a separate .ps1 in /examples/ (issue 491)
    • Removed example “Create and configure an application pool” from README.md (issue 489).
  • Changes to xIisHandler
    • Updated schema.mof to include descriptions for each property (issue 453).
    • Moved MSFT_xIisHandler localization strings to strings.psd1 (issue 463).
  • Changes to xWebSite
    • Fix Get-TargetResource so that LogFlags are returned as expected array of strings (one for each flag) rather than an array containing a single comma-separated string of flags” (issue 332).
    • Moved localization strings to strings.psd1 file (issue 475)
    • Updated schema.mof so that each property has an appropriate description (issue 456).
    • Updated schema.mof and README so that SourceType and SourceName properties for MSFT_xLogCustomFieldInformation are associated with the appropriate descriptions and valuemaps/values (issue 456).
    • Move examples from README.md to resource examples folder (issue 487).
    • Fix case of resource name from xWebsite to xWebSite (issue 535).
  • Changes to xIISLogging
    • Fix Get-TargetResource so that LogFlags are returned as expected array of strings (one for each flag) rather than an array containing a single comma-separated string of flags (issue 332).
    • Moved MSFT_xIisLogging localization strings to strings.psd1 (issue 464).
  • Changes to xSslSettings
    • Updated casing of xSslSettings in all file names, folder names, schema, and documentation (issue 461).
    • Updated casing of xSslSettings in all file names, folder names, schema, and documentation (issue 536).
    • Moved MSFT_xSslSettings localization strings to strings.psd1 (issue 467).
  • Changes to xWebConfigKeyValue
    • Updated schema.mof to include a description for the Ensure property (issue 455).
    • Move localization strings to strings.psd1 file (issue 472).
  • Changes to xWebAppPoolDefaults
    • Move localization strings to strings.psd1 file (issue 470).
    • BREAKING CHANGE: Changed ApplyTo key parameter to IsSingleInstance to bring the resource into compliance with published best practices (issue 462).
  • Changes to xWebApplication
    • Move localization strings to strings.psd1 file (issue 468)
    • Add description on class MSFT_xWebApplicationAuthenticationInformation (issue 454).
  • Changes to xIisModule entry
    • Moved xIisModule localization strings to strings.psd1 (issue 466).
  • Changes to xIisMimeTypeMapping
    • Moved MSFT_xIisMimeTypeMapping localization strings to strings.psd1 (issue 465).
  • Changes to xWebVirtualDirectory
    • Moved MSFT_xWebVirtualDirectory localization strings to strings.psd1 (issue 477).
  • Changes to xWebSiteDefaults
    • Move localization strings to strings.psd1 file (issue 475).
    • BREAKING CHANGE: Changed ApplyTo key parameter to IsSingleInstance to bring the resource into compliance with published best practices (issue 457).
    • Fix case of resource name from xWebsiteDefaults to xWebSiteDefaults (issue 535).
  • Changes to xWebConfigProperty
    • Move localization strings to strings.psd1 file (issue 473).
  • Changes to xWebConfigPropertyCollection
    • Move localization strings to strings.psd1 file (issue 474).
  • Changes to xIisFeatureDelegation
    • Moved MSFT_xIisFeatureDelegation localization strings to strings.psd1 (issue 459).
  • Changes to xWebAppPool
    • Moved MSFT_xWebAppPool localization strings to strings.psd1 (issue 469).

How to Find Released DSC Resource Modules

To see a list of all released DSC Resource Kit modules, go to the PowerShell Gallery and display all modules tagged as DSCResourceKit. You can also enter a module’s name in the search box in the upper right corner of the PowerShell Gallery to find a specific module.

Of course, you can also always use PowerShellGet (available starting in WMF 5.0) to find modules with DSC Resources:

# To list all modules that tagged as DSCResourceKit
Find-Module -Tag DSCResourceKit 
# To list all DSC resources from all sources 
Find-DscResource

Please note only those modules released by the PowerShell Team are currently considered part of the ‘DSC Resource Kit’ regardless of the presence of the ‘DSC Resource Kit’ tag in the PowerShell Gallery.

To find a specific module, go directly to its URL on the PowerShell Gallery:
http://www.powershellgallery.com/packages/< module name >
For example:
http://www.powershellgallery.com/packages/xWebAdministration

How to Install DSC Resource Modules From the PowerShell Gallery

We recommend that you use PowerShellGet to install DSC resource modules:

Install-Module -Name < module name >

For example:

Install-Module -Name xWebAdministration

To update all previously installed modules at once, open an elevated PowerShell prompt and use this command:

Update-Module

After installing modules, you can discover all DSC resources available to your local system with this command:

Get-DscResource

How to Find DSC Resource Modules on GitHub

All resource modules in the DSC Resource Kit are available open-source on GitHub.
You can see the most recent state of a resource module by visiting its GitHub page at:
https://github.com/PowerShell/< module name >
For example, for the CertificateDsc module, go to:
https://github.com/PowerShell/CertificateDsc.

All DSC modules are also listed as submodules of the DscResources repository in the DscResources folder and the xDscResources folder.

How to Contribute

You are more than welcome to contribute to the development of the DSC Resource Kit! There are several different ways you can help. You can create new DSC resources or modules, add test automation, improve documentation, fix existing issues, or open new ones.
See our contributing guide for more info on how to become a DSC Resource Kit contributor.

If you would like to help, please take a look at the list of open issues for the DscResources repository.
You can also check issues for specific resource modules by going to:
https://github.com/PowerShell/< module name >/issues
For example:
https://github.com/PowerShell/xPSDesiredStateConfiguration/issues

Your help in developing the DSC Resource Kit is invaluable to us!

Questions, comments?

If you’re looking into using PowerShell DSC, have questions or issues with a current resource, or would like a new resource, let us know in the comments below, on Twitter (@PowerShell_Team), or by creating an issue on GitHub.

Michael Greene
Principal Program Manager
PowerShell DSC Team
@migreene (Twitter)
@mgreenegit (GitHub)

The post DSC Resource Kit Release October 2019 appeared first on PowerShell.

PowerShell Extension Roadmap

$
0
0

Over the last year we have committed to making the PowerShell editing experience in Visual Studio Code a rich and productive cross-platform alternative for the PowerShell ISE. To that end, we have focused on two primary areas: bringing the PSReadLine experience to the Integrated Console, and improving the stability of the extension while editing and debugging. The goal of this blog post is to walk through how we have made efforts in these key areas, and what our next steps are to follow through on these efforts.

Investments in the reliability of PowerShell in Visual Studio Code

Our number one user request for the PowerShell editing experience in Visual Studio Code is to improve the stability of the editor and debugger. Long-standing constraints in the original design of the PowerShell extension made it difficult to improve its robustness through incremental changes. Instead, over the last six months we prioritized work to re-architect the extension with an emphasis on stability.

Using the Omnisharp Project’s Common Language Server Protocol

Based on the maturity of OmniSharp we took advantage of the project’s common language server protocol implementation which is a .NET library. It is important to note that Omnisharp’s common LSP server library is distinct from Omnisharp which is largely synonymous with the C# LSP backend. While we do hope this Omnisharp port will improve your editing experience, do not expect it to provide additional .NET completions. Omnisharp’s architecture is more robust meaning that bugs that might once have been crashes will now be caught and logged. By leveraging this library we were able to greatly simplify our code and are now more compliant with language server protocol.
Ultimately, we believe that these changes will significantly reduce the number crashes of the extension and improve the performance overall. These changes shipped in the November 2019 release of the PowerShell Preview extension and will ship with the January 2020 release of the PowerShell extension.

Other features of the Omnisharp Port

  • Asynchronous message handling for increase in performance
  • CodeLens requests no longer depend on running PowerShell so IntelliSense hangs should reduce
  • Formatting is now handled directly by the language server

Hosted PSScriptAnalyzer

PSScriptAnalyser (PSSA) is a static code checker for PowerShell modules and scripts, which provides services like script diagnostics and formatting in the PowerShell extension. In our analysis of the PowerShell extension we found that PSScriptAnalyzer had a major impact on the performance of the extension overall so we have been investing in changes to how PSSA integrates with the PowerShell extension.
In the current architecture, the PowerShell extension must interface with PSSA through its cmdlets, which re-instantiate the PSSA engine on every invocation. Further, because we must use PowerShell cmdlet invocation for this, the PowerShell extension is forced to manage the overhead of PowerShell runspace management and command resolution to run PSSA. In scenarios like real-time diagnostics and formatting, this overhead has become a significant bottleneck for the extension. Instead, since PSSA is .NET code at heart, we are moving toward a model allowing direct hosting of PSSA in .NET, with management and invocation of the PSSA engine performed through a set of suitable public .NET APIs. Work on these APIs has entered into a validation and integration phase and we expect this improvement to ship with the January 2020 release of the PowerShell extension.

PSReadLine support in the Integrated Console

Full PSReadLine support has long been at the top of our list for feature requests. It has also been among our most difficult problems to solve because at its core it also required architectural changes in how the PowerShell extension manages threading and runspaces.
The additional challenge of trying to support both legacy versions of PowerShell and a range of platform distributions has caused this problem to continually be delayed. In January of 2019 we released a Preview version of the PowerShell extension which was built on .NET Standard thereby enabling us to support PSReadLine in the integrated console for Windows users on PowerShell Version 5.1 and above.

With PowerShell 7 delivering a fix in .NET Core 3.0 for the way POSIX terminal APIs are handled when starting new processes, we are finally able to move the PSReadLine support currently available in the PowerShell Preview extension into the stable PowerShell extension with support across platform distributions. We expect to ship this update in the same time frame as PowerShell 7 (targeted for January 2020).

Dropping support for PowerShell Versions 3 and 4

Support for PSReadLine in the PowerShell extension Integrated Console depends on changes made in PSReadLine since it moved to version 2.0, where it dropped support for PowerShell versions 3 and 4. In turn, we also made the difficult decision to no longer support PowerShell 3 and 4 in future updates of the extension. In making this decision we analyzed the use of these PowerShell versions and found that approximately 1% of PowerShell session in VSCode use one of these versions. In order to accommodate these use cases we will ship a final stable version of the extension with PowerShell version 3 and 4 support which can continue to be used. To use this version of the extension the user will still install the PowerShell extension through the VSCode marketplace. They will then need to use the extension settings to select their desired version.

We plan to release a “release candidate” for this version as our November release of the PowerShell extension and make any necessary bug fixes before our January 2020 release of the extension.

Summarized Release Plan for the VSCode Extension

Please note that this is the currently targeted timeline for release but it subject to change:

  • Novemeber 2019: PowerShell Preview Extension ships with increased stability from the Omnisharp library
  • November 2019: PowerShell Extension ships as a release candidate for the last version supporting PowerShell 3.0 and 4.0
  • December 2019: PowerShell Extension ships with any necessary bug fixes as the last version supporting PowerShell 3.0 and 4.0
  • January 2020: The PowerShell Extension ships with full PSReadLine support, Omnisharp integration, and hosted PowerShell Script Analyzer (and drops support for PowerShell 3.0 and 4.0)

We will then be able to address other outstanding issues in the extension,
and use the PowerShell Preview extension as a means of increasing our release cadence.

What’s Next and Providing Feedback

Once we make these releases we will continue to make investments in the PowerShell editing experience. If you encounter any issues with the PowerShell extension in Visual Studio Code or have feature requests, the best place to get support is through our GitHub repository.

Sydney Smith, Robert Holt, Tyler Leonhardt
PowerShell Team

The post PowerShell Extension Roadmap appeared first on PowerShell.

PowerShell 7 Preview 6

$
0
0

Today we shipped PowerShell 7 Preview.6! This release contains a number of new features and many bug fixes from both the community as well as the PowerShell team. See the Release Notes for all the details of what is included in this release.

This will be the last preview release as we head towards a Release Candidate in December. For the Release Candidate, there will be no more new features although small changes to cmdlets may still be accepted depending on risk of the change. Bug fixes will still be accepted but also accessed for their risk of causing a regression. Finally, we expect General Availability of PowerShell 7 in January as our first Long Term Servicing release.

New Features in Preview 6

This release has a number of new features from both the community as well as the PowerShell team. Remember that preview releases of PowerShell are installed side-by-side with stable versions so you can use both and provide us feedback on previews for bugs and also on experimental features.

You can read about new features in previous preview releases:

There were new features in Preview 1 and Preview 2, but I didn’t blog about them… sorry!

Skip Error Check for Web Cmdlets

Great addition by community member Vincent Damewood to allow skipping the internal HTTP response error check within the web cmdlets. This means that you can now handle the web error yourself including getting the response object as well as the HTTP response headers whereas previously you would have to get it from the resulting error object.

Null Conditional Member Property and Method Access

This is a new language feature that allows you to skip checking if a variable is $null before indexing into the variable, calling a method, or accessing a property. For properties, PowerShell will already allow accesing properties of $null, but this new syntax makes it more clear the intent in the script. Note that because PowerShell allows variable names to end with a question mark, you must use braces around the variable name

Extended Unix Filesystem Information

For users of PowerShell on Linux and macOS, they may miss some of the additional information of the filesystem provided by ls -l (long form of a directory listing). This new feature makes that information available using Get-ChildItem so you are no longer missing any useful information.

Windows PowerShell Compatibility in Preview 6

This release also improves the compatibility of PowerShell 7 with Windows PowerShell 5.1. We’ve brought back many existing cmdlets from Windows PowerShell 5.1 thanks to .NET Core 3! In addition, we have a new feature included with PowerShell 7 that encapsulates the Windows Compatibility Module (more on that below!).

Clipboard Cmdlets

Get-Clipboard and Set-Clipboard are back! Not only are they available again, but they are cross platform compatible which means you can use them on Linux (requires xclip to be installed) and macOS. Note that only text is supported at this time even on Windows.

Performance Counter Cmdlets

Get-Counter is back allowing you to get Windows performance counter information. Note that Import-Counter and Export-Counter cmdlets are not supported in PowerShell 7.

Graphical Tools Cmdlets

With .NET Core 3.0 bringing back WPF support on Windows, we’ve been able to bring back some popular graphical tools. Out-GridView gives a dynamic table view of results in a pipeline with sorting and filtering capabilities and when used with -PassThru can be used interactively within a pipeline to select objects to send back to the pipeline. Show-Command gives a graphical view of a command including parameter sets, parameters, switches, etc… Finally, Get-Help -ShowWindow works again to give a graphical view of PowerShell help content.

Update-List, Out-Printer, and Clear-RecycleBin cmdlets

Update-List allows adding/removing items from a property value that contains a collection of objects within a pipeline. This cmdlet is cross-platform compatible.

Out-Printer sends a PowerShell object to the printer. This cmdlet is only supported on Windows currently.

Clear-RecycleBin empties the recycle bin and is currently only supported on Windows.

Test-Connection Improvements

The original implementation of Test-Connection in Windows PowerShell relied on WMI which made it not cross-platform compatible.

Community member and PowerShell repo maintainer Ilya Sazonov ported the cmdlet to work against .NET Core APIs in PowerShell Core 6 making it work cross-platform. However, this also changed the user experience.

For PowerShell 7, Joel Sallow, another community member made improvements to this cmdlet and also getting back to similar experience as the original cmdlet in Windows PowerShell.

Import Windows PowerShell Modules in PS7

For PowerShell Core 6, we introduced the Windows Compatibility Module to allow importing Windows PowerShell modules that were not compatible with PowerShell Core 6 leveraging WinRM and implicit remoting.

In PowerShell 7, we have included this functionality into Import-Module directly without relying on WinRM, but does rely on Windows PowerShell 5.1 (it won’t work if Windows PowerShell 5.1 is not available).

Basically, for modules in the System32 folder, if the module manifest doesn’t indicate that module is compatible with Core then that module will be loaded in a Windows PowerShell process and using implicit remoting reflected into your PowerShell 7 session.

More details of this feature along with other information regarding Windows PowerShell compatibility in general coming in a separate blog post.

Closing

These are the last of the big changes coming in PowerShell 7 as we heads toward a Release Candidate next month. Please try out this preview and report issues in our GitHub repo as we still have opportunity to fix existing bugs as well as new bugs introduced by these features.

Thanks!

Steve Lee
PowerShell Team

The post PowerShell 7 Preview 6 appeared first on PowerShell.

Improvements in Windows PowerShell Container Images

$
0
0

Beginning with Windows Server 20H1 Insider builds, Windows Server Core Insider images have been reduced in size from ~2.1 GBs to ~1.1 GBs.

How did the Server Core images get over 40% smaller?

Traditionally, Windows 10 and Windows Server have always included a set of .NET native binaries that were pre-compiled using the Native Image Generator tool (Ngen.exe). This native pre-compilation makes these binaries faster on default installations of the OS, but it also makes the image size grow: managed/IL .NET binaries are typically smaller and slower initially (until JIT compilation happens) than their native counterparts (with another tradeoff being that the latter are not portable between platforms and architectures).

For more details, check out corresponding blogs published by the .NET team and the Windows Server team.

What does this mean for me as a PowerShell user?

If you depend on Windows Server container images for usage of Windows PowerShell, and you value performance, you should switch from the windows/servercore images to the dotnet/framework/runtime images. The latter are specifically optimized for .NET Framework workloads like Windows PowerShell.

And in fact, switching to the new dotnet/framework/runtime images will actually provide a greater benefit to startup performance even over the old windows/servercore images. When running Measure-Command { docker run --rm <image> powershell -c "echo 1" } on a Windows box, observe the following differences:

windows/servercore:1903 windows/servercore/insider:10.0.19023.1 dotnet/framework/runtime:4.8-20191008-windowsservercore-1903 dotnet/framework/runtime:4.8-windowsservercore-2004
7.34 sec 5.41 sec 6.8 sec 3.76 sec

 

What if I’m using PowerShell Core instead of Windows PowerShell in my containers?

If have already moved your workloads from Windows PowerShell to PowerShell Core, you should continue to use the windowsservercore images from microsoft/powershell, and when the Windows Server reductions graduate from Insiders you’ll simply enjoy the benefit of smaller image sizes.

That’s it!

Thanks to everyone leveraging PowerShell in Docker containers! And make sure to file any issues you have in our powershell-docker repository.

Thanks,
Joey Aiello
Program Manager, PowerShell

The post Improvements in Windows PowerShell Container Images appeared first on PowerShell.

Announcing General Availability of the Windows Compatibility Module 1.0.0

$
0
0

The Windows Compatibility module (WindowsCompatibility) is a PowerShell module that lets PowerShell Core 6 scripts access Windows PowerShell modules that are not yet natively available on PowerShell Core. (Note: the list of unavailable commands is getting smaller with each new release of PowerShell Core. This module is just for things aren’t natively supported yet.)

You can install the module from the PowerShell Gallery using the command

Install-Module WindowsCompatibility

and the source code is available on GitHub. (This is where you should open issues or make suggestions.)

Once you have WindowsCompatibility installed, you can start using it. The first thing you might want to run is Get-WinModule which will show you the list of available modules. From that list, choose a module, say PKI and and load it. To do this, run the following command:

Import-WinModule PKI

and you’ll have the commands exported by the PKI module in your local session. You can run them just like any other command. For example:

New-SelfSignedCertificate -DnsName localhost

As always, you can see what a module exported by doing:

Get-Command -module PKI

just like any other module.

These are the most important commands but the WindowsCompatibility module provides some others:

  • Invoke-WinCommand allows you to invokes a one-time command in the compatibility session.
  • Add-WinFunction allows you to define new functions that operate implicitly in the compatibility session.
  • Compare-WinModule lets you compare what you have against what’s available.
  • Copy-WinModule will let you copy Window PowerShell modules that are known to work in PowerShell 6 to the PowerShell 6 command path.
  • Initialize-WinSession gives you more control on where and how the compatibility session is created. For example. it will allow you to place the compatibility session on another machine.

(See the module’s command help for more details and examples on how to use the WindowsCompatibility functions.)

How It Works

The WindowsCompatibility module takes advantage of the ‘Implicit Remoting‘ feature that has been available in PowerShell since version 2. Implicit remoting works by retrieving command metadata from a remote session and synthesizing proxy functions in the local session. When you call one of these proxy function, it takes all of the parameters passed to it and forwards them to the real command in the “remote” session. Wait a minute you may be thinking – what does remoting have to do with the WindowsCompatibility module? WindowsCompatibility automatically creates and manages a ‘local remote’ session, called the ‘compatibility session’ that runs with Windows PowerShell on the local machine. It imports the specified module and then creates local proxy functions for all of commands defined in that module.

OK – what about modules that exist in both Windows PowerShell and PowerShell core? Yes – you can import them. After all, there are still a fair number of base cmdlets that aren’t available in PowerShell core yet.

So how does this work? WindowsCompatibility is very careful to not overwrite native PowerShell core commands. It only imports the ones that are available with Windows PowerShell but not with PowerShell Core. For example, the following will import the PowerShell default management module

 Import-WinModule  Microsoft.PowerShell.Management

which contains, among others, the Get-EventLog cmdlet. None of the native PowerShell Core cmdlets get overwritten but now you have Get-EventLog available in your session.

At this point, if you call Get-Module, you will see something a bit strange:

Get-Module | ForEach-Object Name

results in output that looks like:

Microsoft.PowerShell.Management
Microsoft.PowerShell.Management.WinModule
Microsoft.PowerShell.Utility
NetTCPIP

Import-WinModule renames the compatibility module at load time to prevent collisions with identically named modules. This is so the module qualified commands will resolve against the current module. In fact, if you want to see what additional commands were imported, you can run:

Get-Command -Module  Microsoft.PowerShell.Management.WinModule

Limitations

Because WindowsCompatibility is based on implicit remoting, there are a number of significant limitations on the cmdlets imported by the module. First, because everything is done using the remoting protocol, the imported cmdlets will return deserialized objects that only contain properties. Much of the time, this won’t matter because the parameter binder binds by property name rather than by object type. As long as the required properties are present on the object, it doesn’t matter what type the object actually is. There are, however, cases where the cmdlet actually requires that the object be of a specific type or that it have methods. WindowsCompatibility won’t work for these cmdlets.

Windows Forms and other graphical tools

The remoting session is considered non-interactive so graphical tools such as notepad or Winforms scripts will either fail, or worse hang.

Linux and Mac support

This module depends on WinRM and the client libraries on these platforms are known to be unstable and limited. So for this release, only PowerShell Core running on Windows is supported. (This may change in the future. But you’ll still need a Windows machine with Windows PowerShell to host the compatibility session.)

PowerShell 6.1 Dependency

WindowsCompatibility depends on a feature introduced in PowerShell Core 6.1 for keeping the current working directory in both the local and compatibility sessions synchronized. Earlier versions of PowerShell will work with WindowsCompatibility but won’t have this directory synchronization feature. So if you’re running PowerShell Core 6.0, import a command that writes to files, do Set-Location to a new directory, then use that command to write to a file with an unqualified path; it will use the original path from when the module was imported rather than your sessions current working directory. On PowerShell Core 6.1, it will correctly use the current working directory.

Summary

To sum it all up, the WindowsCompatibility module provides a set of commands that allow you to access Window PowerShell modules from PowerShell Core 6. There are however, some limitations that make it unsuitable for all scenarios. Over time, as more and more modules are ported to .NET Core/PowerShell 6 natively there will be less need for this module.

Cheers!
Bruce Payette,
PowerShell Team.

Viewing all 1519 articles
Browse latest View live


<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>