Tag Archives: Invoke-WebRequest

Apartment Hunting with PowerShell

Note: Expect a part two on this post.

I know a guy, and that guy is looking for an apartment. It turns out that apartments are going really fast and inventory is low — maybe you knew this, but it was news to me. Just about as soon as they become available, they are gone. I suggested that I might be able to lend a hand… maybe, who knows. This is not because I know someone in apartments, but rather that if there is a way to use PowerShell here, then there is a good chance I can help. He got lucky and I learned something new.

I started by going to the apartment website where he was interested and found a page that listed each apartment model and whether or not they had any availability. It was a floor plan page. I was not expecting much honestly, but I used the built-in Web Developer Tools in my browser and viewed the page source, and found some exciting news (for me and PowerShell, at least). It was enough good news that I am able to write about this whole experience.

While empty here, this data structure caught my eye. The output I had hoped to gather, was in JSON format; that was huge! Best I can tell, it is generated by a JavaScript file, which then embeds the JSON in the HTML that makes up the webpage. That is not overly important, however, but look at this structure; it is magnificent.

floorplans: [
  {...
  },
  {...
  },
  {...
  },
  {...
  },
  {...
  },
  {...
  }
],
propertyID: 60484,

Inside the floor plans JSON array ([]) are six objects, each in their own set of curly braces. Inside each of those, was a plethora of information regarding each floor plan. These properties included things like Model, Sq.Ft., Beds, Baths, etc. Let’s start by taking a look at the Watch-Apartment PowerShell function I wrote. Just a note, but in order to make this work for yourself, you will need to edit the path in the $ContentPath variable.

function Watch-Apartment {
    $Uri = 'https://theplaceatcreekside.securecafe.com/onlineleasing/the-place-at-creekside/floorplans'
    $WebRequestContent = (Invoke-WebRequest -Uri $Uri).Content
    $ContentPath = 'C:\users\tommymaynard\Documents\tommymaynard.com\Apartment Hunting\WebpageContents.txt'
    Set-Content -Path $ContentPath -Value $WebRequestContent

    $File = Get-Content -Path $ContentPath
    $Pattern = "floorplans:(.*?)propertyID:"
    $ParsedPage = [regex]::Match($File,$Pattern).Groups[1].Value
    $ParsedPage = $ParsedPage.Trim(); $ParsedPage = $ParsedPage.TrimEnd(',')

    $JsonDocument = ConvertFrom-Json -InputObject $ParsedPage
    $JsonDocument |
        Select-Object -Property @{Name='Available';Expression={if ($_.isFullyOccupied -eq 0) {"Yes ($($_.availableCount))"} else {'No'}}},
        @{Name='Model';Expression={$_.name}},
        @{Name='Sq.Ft.';Expression={$_.sqft}},
        @{Name='Beds';Expression={$_.beds}},
        @{Name='Baths';Expression={$_.baths}} |
    Format-Table -AutoSize
}
Watch-Apartment

We will discuss the above function using its line numbers:

Line 1: Declares/creates the Watch-Apartment function.
Line 2: Stores the site’s URI inside the $Uri variable.
Line 3: Invokes an Invoke-WebRequest command using the URI and stores the Contents (as in the Contents property) inside the $WebRequestContent variable.
Line 4: Creates the ContentPath variable to hold a path to a text file that will be created in the next line/command.
Line 5: Takes the content from the webpage and writes it to a text file.

Writing to a file was not a requirement, however, it was my first choice for whatever reason and so I went with it, and then stayed with it.

Line 7: Read in the contents from the file and store them in the $File variable.
Line 8: Create a Regex pattern to allow us to collect all the content between the word “floorplans:” and “propertyID:”.
Line 9: Parse out the data we want and store it in the $ParsedPage variable.
Line 10: Trim off the white space from the beginning and end of the JSON string, and then trim off the trailing comma at the end of the JSON string.

Line 12: Assign the $JsonDocument variable the value assigned to the $ParsedPage variable after it has been converted from JSON by CovertFrom-Json.
Lines 13 – 19: Use Select-Object to select and modify our desired properties.

In the final command, we determine which floor plan is available, how many apartments there are, which model it is, how many square feet that model has, and how many bedrooms and bathrooms it has. Each line/property includes a calculated property and often, just to modify the case of the text.

I edited the friend’s PowerShell profile script and added this function. Not only is the function added to the PowerShell session by the profile script, but it also invokes the function, too. Open PowerShell, and just about instantly know whether anything is available or not.

These were the results the first time it ran back on my machine.

It was a good thing that the Available property included both the isFullyOccupied (“Yes” versus “No”) and the availableCount (# of apartments) information. Take a look at the next image to see why.

In the above image, it still says, “Yes,” but the count is zero. Apparently, my decision to include both values was the right choice, as this information is not all updated at the same time.

Later that same day, it cleared up.

Now, he waits, as my work is done.

Note: As stated at the top of this post, expect a part two. There is more than one apartment complex now.

View Current PowerShell.org Q&A Forum Topics

Note: Update added at the bottom on this post on August, 9, 2016. Please read.

Sometimes you don’t always have the time to finish something you’ve started. For me, it was this function. I pounded this out in a quick few minutes, and while I don’t see myself investing in it any further, I didn’t want to forget the function, and thought I would hang on to it somewhere. Well, that’s why it’s here, especially as someone may find it useful, or helpful.

The function, which I called Get-PowerShell.orgForumTopic, runs out to PowerShell.org and grabs the current topics (page one) from the PowerShell Q&A forum (http://powershell.org/wp/forums/forum/windows-powershell-qa). It only returns the Thread name and the URL, because, well, that’s what seemed useful and relevant at the time I wrote it (which was many months ago).

Function Get-PowerShell.orgForumTopic {
    [CmdletBinding()]
    Param ()

    Begin {
    } # End Begin.

    Process {
        (Invoke-WebRequest -Uri 'http://powershell.org/wp/forums/forum/windows-powershell-qa/' |
            Select-Object -ExpandProperty Links |
            Where-Object {$_.outerHTML -like '*http://powershell.org/wp/forums/topic*'} |
            Select-Object @{N='Thread';E={$_.innerHTML}},@{N='Url';E={$_.href}} |
            Select-Object -First 30)[(0..30 |
                ForEach-Object {
                    If (-not($_ % 2)) {
                        $_
                    }
                 }
            )]
    } # End Process.

    End {
    } # End End.
} # End Function: Get-PowerShell.orgForumTopic

Here’s what the results looked like in the ConsoleHost, near in time to when this post was published.

current-powershell.org-qa-forum-topics-01

While I never added any more to this function, I had some ideas: add the thread status, add the “started by user,” add the user that made the last post, add the number of posts per topic, and allow it to run against other PowerShell.org forum topics. It might’ve also been helpful to include additional pages, if requested by the user of the function, such as adding a -Pages parameter (-Pages 4).

Anyway, here it is. Beside being helpful to see the top PowerShell.org Q&A forum posts at a specific point in time, it’s an interesting example of reading from a webpage, of which I had minimal experience. So yeah, I probably learned something by doing this exercise.

If you want to do the same, then start by running the first command, Invoke-WebRequest -Uri ‘http://powershell.org/wp/forums/forum/windows-powershell-qa/’. Then add the pipe and first Select-Object command and run that. Then add the Where-Object command, and so on. This will allow you to see how I finally got to only returning the “Threads” and “Urls.” Take care.

Update: Since a site redesign at PowerShell.org, this function, no longer functions. I’m not sure that I’ll bother to update it — I don’t get the feeling that it was ever used by anyone — but I’ll keep this post up for anything else it may offer, that may be helpful in learning PowerShell.

Automate App Server (Non Visual) Website Test

This is part II of this post: http://tommymaynard.com/quick-learn-automate-app-server-visual-website-test-2016. It might be beneficial to read that first.

It was minutes before my family needed to leave the house and start our Saturday, when my desire to monitor eight of my application servers hit. In a quick moment, I wrote three commands in three PowerShell consoles and left the house.

Before I show you what was in console one, I have to show and mention a couple functions on which it relied. These functions are a bit of a continuation of the link above. The Watch-MyApp function below, will visually and textually display indicators so that I know if my app servers’ websites are responding, or not. The -Visual parameter will open an instance of Internet Explorer for each server URL, so I can visually check the app servers’ websites (see the link to view this function).  Not using the -Visual parameter, which is how it’s used here, uses Invoke-WebRequest to determine a webpage’s status code. A 200 status code (OK response) means our webpage is up, and responding. If it’s not up, it’ll indicate a timeout warning. It could also produce a different status code, if the page is up, but produces something other than a 200 status code. Other more well-known status codes are the dreaded 404 (page cannot be found), or a 5xx error which indicates a problem with the web server. It may also be helpful to know that the $MyAppServers variable contains the default properties for five servers, returned from the Get-ADComputer cmdlet.

Function Watch-MyApp {
    Param (
        [switch]$Visual
    )
    
    If ($Visual) {
        $MyAppServers.Name | ForEach-Object {
            Open-InternetExplorer -Url "$_.myapp.mydomain.com"
        }
    } Else {
        $MyAppServers.Name | ForEach-Object {
            try {
                $StatusCode = (Invoke-WebRequest -Uri "http://$_.myapp.mydomain.com" -TimeoutSec 10).StatusCode
                "$StatusCode`: $_"
            } catch {
                Write-Warning -Message "$_"
            }    
        }
    }
 }

The first of my three consoles wrapped the function above, Watch-MyApp, in a never ending loop ($true is always true). Every minute it would get the date and time and write it to a log file. Immediately after that, it would run the function to get the status code of each web server, and append it to the same log file. A minute later and it does the same thing. Here’s the command and a sample from the log file it created.

Do {
    (Get-Date).ToString() | Out-File -FilePath 'C:\MyApp.txt' -Append
    Watch-MyApp | Out-File -FilePath 'C:\MyApp.txt' -Append
    Start-Sleep -Seconds 60
} While ($true)
2/21/2016 10:48:29 PM
200: appsrv01
200: appsrv02
200: appsrv03
200: appsrv04
200: appsrv05
2/21/2016 10:49:29 PM
200: appsrv01
200: appsrv02
200: appsrv03
200: appsrv04
200: appsrv05

The second console ran another endless Do-While loop. The difference, is that this loop’s job was to copy the file that was being updated by the command in the first console, to my Dropbox folder every ten minutes. Genius, right? While I could have, I opted to not have my first command write/overwrite the file directly in my Dropbox folder. I didn’t feel it was necessary to update Dropbox each minute for hours on end. When I was back home later, I changed this from ten minutes to 30 minutes, or 1800 seconds, as it was even less important to update Dropbox then.

Do {
    (Get-Date).ToString() | Out-File -FilePath 'C:\MyApp.txt' -Append
    Copy-Item -Path 'C:\MyApp.txt' -Destination 'C:\Users\tommymaynard\Dropbox\MyApp.txt' -Force
    Start-Sleep -Seconds 600
} While ($true)

What this meant is that while I was away, I was able to check Dropbox on my phone and quickly determine if there were any status codes other than a 200. Every ten minutes and there would be an update to my file that would include the status code results for each minute in the last ten.

The final console didn’t do much other than allow me to manually check the file in Dropbox, for something other than a date entry, and status code of 200. So far so good: None of the web apps returned anything more than a 200 status code. Maybe the change at the end of the day on Friday really did fixed things. Here’s the command I used to read the file and exclude lines that had a date and a 200 status code (all the lines unless there was a problem).

Get-Content -Path 'C:\MyApp.txt' | Where-Object {($_ -notlike '*2016*') -and ($_ -notlike '200*')}

But why visually scan the file myself, when PowerShell can do that too!? This third command could’ve also been an endless loop. It could’ve automated scanning the file for me, like I was doing already, and then could’ve been set up to send me an email using the Send-MailMessage cmdlet if it found something, other than the 200 status code. I could’ve avoided looking at the file in Dropbox and just watched my email. One better, and I could’ve had my command do email to text with a specially crafted email address. Here’s what I mean for US Verizon customers: http://www.verizonwireless.com/news/article/2013/06/computer-to-phone-text-messaging.html. Many days, it feels like PowerShell is only limited by what you can think do with it.

There have been times in my life where my wife will ask me, “How would people that don’t know computers, know how to do this?” I usually answer by saying, that “They wouldn’t,” or “I don’t know, I’m not that person.” The same thing applies here: “How would a Windows System Administrator know what to do if they hadn’t already learned PowerShell?” Those three commands might take someone new with PowerShell half a day to consider, and then write. Learn it while you don’t think you need it, because one day you will.

So, to recap, while I was away doing the family thing, my computer sat on the kitchen table at home, checked on eight app servers for me and updated a file in Dropbox, that I was able to check at times while we were away. PowerShell has a purpose, and without it, you’re losing yours.

Save External Dynamic IP to Dropbox

There are times when I’m away from the house, where I need to be able to reach my home computer via Remote Desktop. Because of this occasional need, I have my home router set up to allow this external connection. To do this required a static (internal) IP assignment for my home computer (I actually use a DHCP reservation), my router listening on port 3389 on the outside IP, and port forwarding that routes this external traffic to the home computer’s internal IP address. I used to connect by name, but the application I was using to sync my IP with my hostname, doesn’t seem to be working consistently. Therefore, I’ve been using my fallback option a lot lately: PowerShell and Dropbox.

Here’s what happens: I have a scheduled task on my home computer that runs a PowerShell script at 12 a.m., 6 a.m., 12 p.m, and 6 p.m. each day. The purpose of the script is to get my current, outside IP and write it, as well as the date and time, to a text file in Dropbox. That file is then snyced to my other computer and phone. It’s simple.

In line one, below, we create a variable, $UseableDate, to hold the string I use for the date and time. For today’s date, it might look something like this: D2015-08-19_T09-02-43-PM. This has long been my preferred way to store the date and time, especially when used in file names, as it will keep everything in a sort able order by year, month, and then day, and doesn’t include any invalid characters when used in a file path.

$UseableDate = Get-Date -Format 'Dyyyy-MM-dd_Thh-mm-ss-tt'

Next we need to collect the outside IP address. This is an important distinction — I don’t want my internal, NAT’d IP address. Line two, below, which checks in with dyndns.com, was borrowed from Aman Dhally. I’ve included a couple other options that can be used to populate the same $IPAddress variable (note: ifconfig.me has always seemed to be slower than the rest). Notice that using something other than dyndns will not require using any Regex, just some trimming to remove some white space surrounding the IP address.

$IPAddress = (Invoke-WebRequest -Uri ‘http://wtfismyip.com/text’).Content.Trim()
$IPAddress = (Invoke-WebRequest -Uri ‘http://ifconfig.me/ip’).Content.Trim()

$UseableDate = Get-Date -Format 'Dyyyy-MM-dd_Thh-mm-ss-tt'
$IPAddress = (Invoke-WebRequest -Uri 'http://checkip.dyndns.com').Content -replace "[^\d\.]"

The third line in the script, joins the date and time with the IP address and writes (appends) it to the file inside my Dropbox folder. Again, this is running from my home computer. Once in Dropbox at home, my work computer and phone will have the newest version of the file. In fact, it’s become a pretty good reminder that it is lunch time at work, as Dropbox pops up a Notification Area balloon that my file has been updated at 12 p.m.

$UseableDate = Get-Date -Format 'Dyyyy-MM-dd_Thh-mm-ss-tt'
$IPAddress = (Invoke-WebRequest -Uri 'http://checkip.dyndns.com').Content -replace "[^\d\.]"
"$UseableDate -- $IPaddress" | Out-File -FilePath 'C:\Users\tommymaynard\Dropbox\IP\homecomputer-IP.txt' -Append

To be as complete as possible, and save as many seconds as I can per day, I have a little function in my profile on my work computer to grab the IP address from the file. First, here’s an example of my homecomputer-IP.txt file (with hashtags taking the place of the digits):

...
D2015-08-18_T06-00-03-PM -- ##.###.##.###
D2015-08-19_T12-00-06-AM -- ##.###.##.###
D2015-08-19_T06-00-07-AM -- ##.###.##.###
D2015-08-19_T12-00-06-PM -- ##.###.##.###
D2015-08-19_T06-00-03-PM -- ##.###.##.###

The function below will extract the last IP address added to the homecomputer-IP.txt file. With that, I can add it to the Remote Desktop command line executable and get connected to the home computer from anywhere.

Function Get-HomeIP {
    $IPFile = 'C:\Users\tommymaynard\Dropbox\IP\homecomputer-IP.txt'
    (Get-Content -Path $IPFile | Select-Object -Last 1).Split(' -- ')[-1]
}
PS> Get-HomeIP
##.###.##.###
PS> mstsc.exe /v (Get-HomeIP)

That’s it — I hope this might be helpful to someone, someday. It’s saved me a time, or two now.

Update [02/09/2017]: I recently made a couple changes to the above function. The problem was that sometimes the IP wasn’t written to the file, for whatever reason. In this instance, my function ends up returning nothing. Therefore, I now sanitize the file first, so that any rows that don’t contain an IP address are excluded. Take a look.

Function Get-HomeIP {
    $IPFile = 'C:\Users\$env:USERNAME\Dropbox\IP\homecomputer-IP.txt'
    $SantizedIPs = Get-Content -Path $IPFile | Where-Object -FilterScript {$_.Length -gt 34}
    ($SantizedIPs | Select-Object -Last 1).Split(' -- ')[-1]
}