Overview

Previously we featured the 404Project https://isc.sans.edu/diary/ISC+Feature+of+the+Week+The+404Project/12415 As we mentioned, the main purpose of this project is to trend the web pages crawlers and automated bots are trying to access.

We've had a good number of submitters add this script to their error page and have been collecting data for a while now. We made a few summary reports to get started trending the information. The project has been moved to its own space but the old pages should still get you to https://isc.sans.edu/404project/.

Report Information

The summary reports can be viewed at https://isc.sans.edu/404project/reports.html. The page is generated once a day for the previous day's data. The tables have descriptions for each field so I will just list them and summarize here.

Daily Totals

• Complete summary totals for the given date

Top 10 Submitted URLs

• Note the percentage is based on the max a particular URL has been submitted meaning if the submissions that particular day have been the most ever, the graphic will be out at 100%

Top 10 User Agents Submitting

• User Agents with counts and unique submitters and URLs

API Information

We've also added a couple API interfaces if you'd like to view previous data.

https://isc.sans.edu/api/#daily404summary

• Each days totals. Accepts date and limit.

https://isc.sans.edu/api/#daily404detail

• Each days details. Accepts date and limit. (Look for more fields output in the future)

Let us know in the section below if you have suggestion or feedback about these preliminary reports or send us any questions or comments in the contact form at https://isc.sans.edu/contact.html

--
Adam Swanger, Web Developer (GWEB)
Internet Storm Center (http://isc.sans.edu)