Edventures in Normalcy

A Troubleshooters Guide to Process Monitor (ProcMon)

Posted by Rose Bush on December 3rd, 2015

Process Monitor, by SysInternals under Microsoft, shows real-time file system, Registry and process/thread activity. To do this it registers itself with the Event Tracing for Windows to receive activity reports from both the file system and the windows registry.

It can be found here: Windows Sysinternals Process Monitor.


Display Overview

Some basics of what to expect to see, here is a resized window, giving all of the default information I want to go over:
The Process Monitor GUI.

Process Monitor Capture Icon Capture (Ctrl+E): Enables/Disables capturing activity.
Process Monitor Autoscroll Icon Autoscroll (Ctrl+E): Enables/Disables scrolling of display as activity is shown. I find it best to disable Autoscroll until enough filters are in place, ymmv.
Process Monitor Clear Icon Clear (Ctrl+X): Clears/flushes captured activity.
Process Monitor Filter Icon Filter (Ctrl+L): Shortcut to the filter screen.
Process Monitor Highlight Icon Highlight (Ctrl+H): Enable highlighting of particular captured activity.
Process Monitor Include Process From Window Icon Include Process From Window: You see a program you want to watch to see what it does, these cross-hairs will help you zero in on just it.


Process Monitor Registry Activity Icon Registry Activity: I use this secondarily and have sparse experience with it, I tend to disable it on startup.  Results include SUCCESS, NAME NOT FOUND, BUFFER OVERFLOW, REPARSE, NO MORE ENTRIES, ACCESS DENIED, and BUFFER TOO SMALL.

Process Monitor File System Activity Icon File System Activity: My bread and butter of this tool.  Almost always will be using this.  The seemingly more useful results include SUCCESS, ACCESS DENIED, NAME COLLISION, PATH NOT FOUND, END OF FILE, and BUFFER OVERFLOW, while the plethora of other results are NO MORE FILES, NAME NOT FOUND, FILE LOCKED WITH WRITERS, FILE LOCKED WITH ONLY READERS, NOT REPARSE POINT, PRIVILEGE NOT HELD, IS DIRECTORY, INVALID PARAMETER, NO SUCH FILE, RANGE NOT LOCKED, SHARING VIOLATION, OPLOCK NOT GRANTED, INVALID DEVICE REQUEST, and FAST IO DISALLOWED.  This may not be the full list, but what I could generate.

Process Monitor Network Activity Icon Network Activity: This would be my 3rd most used, and last as I don’t use the next two.  I like this to see if a connection was made out/in and to/from what host.

Process Monitor Process and Thread Activity Icon Process and Thread Activity

Process Monitor Profiling Events Icon Profiling Events

*A note on Fast IO
“Fast IO indicators in a trace have to do with how the windows file cache works. Process Monitor provides a default filter that removes most of the Fast IO events, by doing an exclude on events that have an Operation starting with the string “FASTIO_”. This leaves “FAST IO DISALLOWED” events captured and displayed. You often see a “FAST IO DISALLOWED” entry on a file followed by the normal path attempt to open the file which succeeds. Adding a filter to exclude Operations start with “FAST IO” eliminates these red herrings.


Starting up Process Monitor

When starting up the application, the output is overwhelming. Expect it, much like running Wireshark and showing all traffic on an interface, you are going to see more than you really need.

Filters
The default filter:

Process Monitor Filter GUI.

This strips out actions by the Process Monitor application itself. I use this as a base, and once the program starts showing results, exclude the imports I am not looking for. I rarely use the registry watch, and I typically stick to just the file monitor.

So for the default that loaded on the last version I downloaded, I would start the application, Process Monitor Capture Icon to stop the capture, disable Process Monitor Registry Activity IconProcess Monitor Network Activity Icon, and processandthreadactivity.  I am then left with just Process Monitor File System Activity Icon enabled.  Now start the capture again, Process Monitor Capture Icon.

For the slew of what is left, we have two options.

Include only the process by name or PID as a filter.

Pros: It is much cleaner and more direct if you know exactly what to look for. If I am trying to watch a single IIS Application pool, I can find its PID in the task manager, and add an Include for the matching PID.

Cons: This can exclude an item you had not anticipated, leaving a surprise to find out later.

Exclude Noise by Process name

Pros: Easier to start with when learning.  Can reveal conflicting applications.

Cons: Time cost, this can be a 30+ list in some cases.  Excluding that number of applications can take numerous runs of Process Monitor, starting and stopping captures to catch as much noise as one can.


Exclude Noise by Process name

I start excluding process’s by name for items I know I am not looking for.

Process Names’s I have been known to exclude:
In General:
System
Explorer.EXE
services.exe
scrnsave.scr
svchost.exe
mmc.exe
rundll32.exe

Desktop Environments:
trillian.exe
lync.exe
chrome.exe
firefox.exe
plugin-container.exe
FlashPlayerPlugin_11_5_502_135.exe
Nexus.exe
NOTEPAD.EXE
pn.exe
OUTLOOK.EXE
vmware-usbarbitrator.exe
FlashPlayerUpdateService.exe
GoogleUpdate.exe
putty.exe

Server Environments:
SSSvc.exe


Include only the process by name or PID as a filter.

TBD, As I run into further examples, I will flesh this out better.


Tips Tricks Notes

  • Don’t record/track when you don’t need to be. The application can lock up and/or run up resources. To be as clear as I can, I had to restart my machine after leaving it running overnight. I was gathering examples by running the software, and failed to follow my own directions. Windows literally told me to restart my applications.
  • When troubleshooting when layers/applications, I find it helpful to keep the browser on the local server as being listed in my output so that I can use it as a marker for when I am testing. It helps to narrow down what’s going on in the underlying moments when a request goes awry.
  • Even with good filters, there still is a lot of stuff to wade through in a trace. For example, when the application loads a dll, there may be several attempts to open the file (CreateFile) under different folders, until the file is found (keep in mind that you don’t necessarily want to go throwing a looked for file in the first place the application looks for. Saw this be very bad in a ColdFusion example).

DOS commands for IP

Posted by Rose Bush on December 1st, 2015

Below, you’ll find a list of the most common IP commands for Windows and DOS. These include ipconfig, trace route, netstat, arp, route, hostname, control netconnections, and other popular DOS and Windows IP commands.

MSSQL Backup all Non System Databases Scripted

Posted by Rose Bush on November 24th, 2015

Microsoft SQL Server backups, MSSQL Backup, process

  • Note: You may need to go to Tools > Options > Query Results > SQL Server > Results to Text in the Studio Manager and change the “Maximum number of characters displayed in each column to 1024” (default is 256). Also make sure you are outputting the results to text. If you do not do this prior, you run into the chance of either set of commands not outputting the full commands needed for the next step.
  • Also, you may need to open a New Query as the change ‘seems’ to only affect newly opened queries. The second pertinent setting, Open a new query window and ensure it has the focus, then go to Query > 249521. Here you can see the setting is located in 249521 > 249521. Change the Maximum number of characters displayed in each column the fields maximum of 8192.

The following command will generate a SQL script to back up all non system SQL databases. Be sure to replace the restore path. Make sure you set the results to text output.

Once the above query has been run, copy the output and run it as a new query on the source server (I migrate a lot, hence the terminology used).

SCP for practical uses

Posted by Rose Bush on November 9th, 2015

A basic example:

SCP
The scp command allows you to copy files over ssh connections. This is pretty useful if you want to transport files between computers, for example to backup something. The scp command uses the ssh command and they are very much alike. However, there are some important differences.

The scp command can be used in three* ways: to copy from a (remote) server to your computer, to copy from your computer to a (remote) server, and to copy from a (remote) server to another (remote) server. In the third case, the data is transferred directly between the servers; your own computer will only tell the servers what to do. These options are very useful for a lot of things that require files to be transferred, so let’s have a look at the syntax of this command:

Looks quite familiar, right? But there are differences. The command above will transfer the file “examplefile” to the directory “/home/yourusername/” at the server “yourserver”, trying to get ssh acces with the username “yourusername”. That’s quite a lot information, but scp really needs it all. Well, almost all of it. You could leave out the “yourusername@” in front of “yourserver”, but only if you want to login on the server with your current username on your own computer. Let’s have a closer look at the end of the command. There’s a colon over there, with a directory after it. Just like Linux’s normal cp command, scp will need to know both the source file(s) and the target directory (or file). For remote hosts, the file(s)/directory are given to the scp command is this way.

You can also copy a file (or multiple files) from the (remote) server to your own computer. Let’s have a look at an example of that:

Note: The dot at the end means the current local directory. This is a handy trick that can be used about everywhere in Linux. Besides a single dot, you can also type a double dot ( .. ), which is the parent directory of the current directory.

This will copy the file “/home/yourusername/examplefile” to the current directory on your own computer, provided that the username and password are correct and that the file actually exists.

You probably already guessed that the following command copies a file from a (remote) server to another (remote) server:

Please note that, to make the above command work, the servers must be able to reach each other, as the data will be transferred directly between them. If the servers somehow can’t reach each other (for example, if port 22 is not open on one of the sides) you won’t be able to copy anything. In that case, copy the files to your own computer first, then to the other host. Or make the servers able to reach each other (for example by opening the port).

Well, those are the main uses of scp. We’ll now go a bit more in-depth about the differences between ssh and scp.

*: Actually you can also use it just like the normal cp command, withhout any ssh connections in it, but that’s quite useless. It requires you to type an extra ‘s’ =).

Specifying a port with scp
The scp command acts a little different when it comes to ports. You’d expect that specifying a port should be done this way:

However, that will not work. You will get an error message like this one:

cp: cannot stat `yourport’: No such file or directory
This is caused by the different architecture of scp. It aims to resemble cp, and cp also features the -p option. However, in cp terms it means ‘preserve’, and it causes the cp command to preserve things like ownership, permissions and creation dates. The scp command can also preserve things like that, and the -p option enables this feature. The port specification should be done with the -P option. Therefore, the following command will work:

Also note that the -P option must be in front of the (remote) server. The ssh command will still work if you put -p yourport behind the host syntax, but scp won’t. Why? Because scp also supports copying between two servers and therefore needs to know which server the -P option applies to.

Another difference between scp and ssh
Unlike ssh, scp cannot be used to run a command on a (remote) server, as it already uses that feature of ssh to start the scp server on the host. The scp command does have an option that accepts a program (the -S option), but this program will then be used instead of ssh to establish the encrypted connection, and it will not be executed on the remote host.

Tips & Tricks with ssh and scp
Quite a handy thing about scp is that it supports asterisks. You can copy all files in a remote directory in a way like this:

And you can also just copy a whole directory by specifying the -r (recursive) option:

 

Both of these also work when copying to a (remote) server or copying between a (remote) server and another (remote) server.

The ssh command can come in handy if you don’t know the exact location of the file you want to copy with scp. First, ssh to the (remote) server:
ssh yourusername@yourserver
Then browse to the right directory with cd. This is essential Linux terminal knowledge, so I won’t explain it here. When you’re in the right directory, you can get the full path with this command:

Note: pwd is an abbreviation of Print Working Directory, which is a useful way to remember the command.

You can then copy this output, leave the ssh shell by pressing Ctrl + D, and then paste the full directory path in your scp command. This saves a lot of remembering and typing!

You can also limit the bandwidth scp may use when copying. This is very useful if you’re wanting to copy a huge amount of data without suffering from slow internet for a long time. Limiting bandwidth is done this way:

The bandwidth is specified in Kbit/sec. What does this mean? Eight bits is one byte. If you want to copy no faster than 10 Kbyte/sec, set the limit to 80. If you want to copy no faster than 80 Kbyte/sec, set the limit to 640. Get it? You should set the limit to eight times the maximum Kbyte/sec you want it to be. I’d recommend to set the -l option with all scp’ing you do on a connection that other people need to use, too. A big amount of copying can virtually block a whole 10 Mbit network if you’re using hubs.

FusionReactor and Forwarded Headers

Posted by Rose Bush on November 6th, 2015

In some cases, FusionReactor may be behind a load balancer or CDN, or any other type of proxy that updates the source IP as seen by the server. It helps to see where the requests are actually originating from. For that, FusionReactor has an option for this:
http://www.fusion-reactor.com/support/kb/frs-351/

See the Requests>Settings>Proxy, which lets you tell FR that you need it to use some alternative header for the “real” ip address..

You will see that in the “proxy header” field there is a drop-down, but it doesn’t show your specific header, rather a couple of the most common alternatives. Just paste the name of your header into that field and save the configuration changes.

Once you have saved this change, you should be able to check the Requests>History page to confirm that you now see the real IP’s showing up.

Installing ColdFusion 11 Under cPanel

Posted by Rose Bush on November 3rd, 2015

First, you need your installation defaults, find the installer.properties example below:

With this file, you can install ColdFusion 11 with the following:

The output in the log file should look something like this:

Chown the CFIDE:

Backup your license.properties:

Add the service:

If the above does not exist, and well hey, it didn’t for me, I did write another article with the file needed.  Check that Article out.

 

sconfig wants the “real” hostname or the connector won’t install.

Turn the listen flag to false in the license.propterties:

Start ColdFusion:

Perform the wsconfig:

In my example, I happened to run the command twice, with varied setups.  I am documenting them both and will clean up at next install:

After the above, my CF11Installer.log added the following:

Performing the wsconfig generated the file I needed to move forward, /usr/local/apache/conf/mod_jk.conf to which the contents are below:

I then distilled the changes to ensure cPanel was copacetic with them, regenerated the conf and restarted apache:

During the last step, I did open for editing /usr/local/apache/conf/httpd.conf twice and /usr/local/apache/conf/mod_jk.conf once before distilling and restarting again.

 

Edit the neo-security.xml and throw in some good defaults.

set rds.security.usesinglerdsp to false

set allowconcurrentadminlogin to false

set admin.userid.required to true

set allowedAdminIPList to 127.0.0.1 and an any others of importance

set secureprofile.enabled to true

 

The admin.userid.root.salt string may need to be updated, idk.

Restart the CF’s

/opt/coldfusion11/cfusion/bin/coldfusion stop
/opt/coldfusion11/cfusion/bin/coldfusion start

Switching SSLs when on Multiple HTTPS Bindings

Posted by Rose Bush on November 2nd, 2015

I had to switch out a certificate on a server and got the below error messages, documented here to make this familiar int he future. I was worried that it would break the other sites on the server, 2 others using the old certificate. Instead the server updated the certificate on all 3 and all 3 remained up and running. I could swear that is not how it happened in the past, but this may have been a different case.

EditSiteBindingSSL1

Edit Site Binding

At least one other site is using the same HTTPS binding and the binding is configured with a different certificate.  Are you sure that you want to reuse this HTTPS binding and reassign the other site or sites to use the new certificate?

EditSiteBindingSSL2

Edit Site Binding

The certificate is associated with this binding is also assigned to another site’s binding.  Editing this binding will cause the HTTPS binding of the other site to be unusable.  Do you still want to continue?

ColdFusion 11 init.d script

Posted by Rose Bush on September 28th, 2015

I recently installed ColdFusion 11 on RHEL, but the cfinit script did not exist. I then pulled up a ColdFusion 10 init script, updated the CF10-CF11 bits, corrected the coding error for the CFSTATUS section particular to my OS and have the script below:

I then ensured permissions on the file and made sure it would add to startup correctly:

 

cPanel License Verification and Update Script

Posted by Rose Bush on September 17th, 2015

You can run the above to re sync the local status/key.  To verify your machine is licensed, browse to the following and enter in the external IP address:

https://verify.cpanel.net/

Personal use case scenarios for needing to run the aforementioned script are changing the IP on the machine, or if the license has been upgrades, from trial fro example.

Sauce: https://forums.cpanel.net/threads/cpanel-license-activation.97409/

 

After a recent installation of dotDefender, the following error came up:
“Internal Server Error

500

No response from subprocess (/usr/local/cpanel/whostmgr/docroot/cgi/addon_dotDefender.cgi): The subprocess exited with statu s 2 (ENOENT).”

The log file for dotDefender, /usr/local/APPCure-full/log/dotDefender_bpd.log, gives further detail:

The fix was to install Crypt::Passwd::XS as such:

 

Update 1-29-2016

This error popped up again, and seemingly coincided to an upgrade in cPanel.  The upgraded version is WHM 54.0 (build 8).  The error was almost identical to the one above, but instead of outputting the error to the dotDefender log, it was going to the cPanel log (/usr/local/cpanel/logs/error_log) as such:

~

Can’t locate cPanel/PublicAPI.pm in @INC (@INC contains: /usr/local/cpanel /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/local/cpanel/Cpanel/Accounting.pm line 32.

BEGIN failed–compilation aborted at /usr/local/cpanel/Cpanel/Accounting.pm line 32.

Compilation failed in require at /usr/local/cpanel/whostmgr/docroot/cgi/addon_dotDefender.cgi line 5.

BEGIN failed–compilation aborted at /usr/local/cpanel/whostmgr/docroot/cgi/addon_dotDefender.cgi line 5.

[2016-01-28 17:35:38 -0500] info [cpsrvd] Internal Server Error: “GET /cpsess122967374/cgi/addon_dotDefender.cgi HTTP/1.1” 500 No response from subprocess (/usr/local/cpanel/whostmgr/docroot/cgi/addon_dotDefender.cgi): The subprocess reported error number 2 when it ended.

~~

Speaking with cPanel support,

This should be fixed. After further review we didn’t acutally rename all of these packages, it looks to maybe only be some API calls rather than the package. This was failing because the following file was modified.

[21:37:25 cp root@7449479 ~]cPs# head /usr/local/cpanel/Cpanel/Accounting.pm

ypackage cPanel::Accounting;

It should have been this looking at another server.

[21:40:42 cp root@7449479 ~]cPs# head Accounting.pm package Cpanel::Accounting;

Once I corrected that I could load the page. I did have a token error and had to login and then got an access error but I believe as root you should have no issue now.

~~~

 

The updated file correcting the issue is below:

 

 

Copyright © Edventures in Normalcy. All rights reserved.