I wanted to start this blog by posting this program I’ve been working on. The function is very simple: take input from a CSV file with URL’s of executable files, download, hash, check Virus Total for infections, and report it.
It was created for a group of colleagues that needed to perform this very same task manually on a regular basis. The source of the data (CSV file) is as NIDS signature created by IBM’s X-Force called “HTTP_Executable_Transfer“. To make use of this program you’ll need:
1. Create a filter/view with the HTTP_Executable_Transfer signature in SiteProtector, and export the contents to CSV. The following columns are required:
Time,Source IP,Target IP,Object Name,:URL,:server,:arg
If you don’t use Site Protector and still want to use this program, here’s the data mapping for the columns:
Time = Self-explanatory, date format: YYYY-MM-DD HH:MM:SS TIME_ZONE (2013-04-22 08:48:34 EDT)
Source IP = Self-explanatory
Target IP = Self-explanatory
Object Name = Destination port, i.e: 80
:URL = The path of the URL, needs to have the “/”. Example: /CodeStuff/PDFStreamDumper_Setup.exe
:server = The server where the file is, for example file.google.com or http://www.google.com. Don’t add HTTP/HTTPs or slashes
:arg = Some URLs use arguments. Do not add question mark, the program will do that for you. Example of argument: file=updates/early_up_20130331.exe
Example of a properly formatted CSV file:
Time,Source IP,Target IP,Object Name,:URL,:server,:arg
“2013-04-04 08:58:41 EDT”,”10.10.1.1″,”22.214.171.124″,”80″,”/fileDownloader.php”,”www.styleadvisor.com”,”file=updates/early_up_20130331.exe”
2. A VirusTotal API number. The program will make connections to Virus Total via its public API, so you’ll need to get one.
3. Internet connection – Well, yes, you need one….as mentioned before, the program will download the file for you from the URL provided.
The program will NOT submit the file to VT automatically. It will simply submit the HASH (MD5) and get the information back. No files are submitted AT ALL. (we plan to add the option later).
This is the 0.1 version, so there’s still a lot to do. Read the information below for more details on usage or the read me file included in ZIP file.
By posting it here, we hope other people with a similar requirement can make use of it. If you like the program and have some other use cases, please let us know. We want to continue the development based on what other users need.
The program was created in Python 2.7.X, however it is a static file. That means you don’t need to have Python installed. It will run on Windows (tested on Windows 8/7) and Linux (tested on Ubuntu 64-bit). It may work on other versions of Windows/Linux but I have not tested them.
All you need to do is extract the ZIP file and run the executable from the command-line. Do not double-click from Windows, you need to go to command prompt and execute from there.
This program uses SQLite. All information obtained will be recorded in the following files:
db\http_exec.db – Contains the data related to files reported as infected by VT, and other
things. It will have all of the records that couldn’t be processed for whatever reason.
db\temp.db – Temporary DB used to record the data from the CSV file. You may delete this file after the program finished using it.
The report provided is basic. Feel free to explore the DB files and see what you can come up with.
WHITE LIST FILE
Inside the directory config there’s a file named white-list.txt
There you should have the list of domains you trust and don’t want to check.
It’s one domain per line. Example:
FILES & DOWNLOADS
The program will only download files of size 10MB or less. Anything over the 10MB limit,
the download will be interrupted and the temporary file deleted. This information is written
to the discarded table inside the db\http_exec.db file.
When the program can’t download a file, for whatever reason, that information is recorded
in the discarded table inside the db\http_exec.db file.
Http_exec automatically creates the following sub-directories relative to the location
of the program:
temp – Contains the files while it’s downloading. Files found to not have any detection in Virus Total will be deleted.
unknown – Contains files that were unknown by Virus Total. In other words, nobody previously submitted those files. You have to manually submit them.
malware – Contains files detected by AV engines from Virus Total.
reports – Where the report.csv file will be written to. (default)
A sample CSV file (http_exec.csv) file is provided with the package.
Basic usage: http_exec.exe -a <VT API>
Use with -v for a more detailed information regarding this program.
For help, use with the -h option
-h, –help show this help message and exit
-i SRCFILE Specify source CSV file. Default: http_exec.csv
-w WHITELISTFILE Specify white list file. One domain per line. Default:
-o REPORTFILE Specify name of report CSV file. Default:
-a VT_API Virus Total API Number
-V, –verbose Debug mode for logs. Default is to log errors and info
-q, –quiet Only log errors to log file.
-c, –continue Continue where it stopped. Won’t process any new records.
-r, –report Use this to simply generate a report on whatever is in the
-v, –version Show program’s information and exit.
If using all the default settings, all you need to do is:
http_exec.exe -a <VT API>
The -c|–continue option can be used if the CSV file was already imported into the temp.db file and all you need is for the program to process whatever is in that DB and that has not been processed by the program yet. There is a column in that DB that flags whether the row was already processed or not. If you simply want to import everything again from a CSV and process everything, delete the temp.db file and proceed as if this was the first time importing it. Caution: You will get duplicated entries in your main http_exec.db if you import the same CSV files over and over.
The -r|–report can be used if all you need is to generate the report. Nothing is processed, no CSV file is required, no connections or downloads are performed, only the report is generated by looking into the http_exec.db file.
The log is located inside the \logs directory. If you use the -V option, it will
be more verbose. Default is INFO level. Please send this file along in case you need
support. The log can also provide information for failed download or VT calls.
Author: Elias Silva
Email: http_exec [ @ ] securityops [ . ] ca
Copyright: Elias Silva. All rights reserved.
Program being executed:
LICENSE & DISCLAIMER
THIS PROGRAM IS FREE TO USE AND DISTRIBUTE. THE SOURCE IS NOT AVAILABLE AT THIS MOMENT.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS;
OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.