apache-logs-to-mysql download

Apache Log Parser and Data Normalization Application

Python handles File Processing & MySQL handles Data Processing

ApacheLogs2MySQL consists of two Python Modules & one MySQL Schema to automate importing Access & Error files and normalizing data into database designed for reports & data analysis.

Runs on Windows, Linux and MacOS & tested with MySQL versions 8.0.39, 8.4.3, 9.0.0 & 9.1.0.

4 LogFormats & 2 ErrorLogFormats can be loaded and 5 MySQL Stored Procedures can be processed in a single Python `ProcessLogs function` execution.

Database system designed to accommodate unlimited domains. Step-by-step guide for easy installation.

Web interface with Drill Down Capability and apache/echarts Log Visualization integration in development.

MySQL apache_logs schema currently has 49 Tables, 853 Columns, 168 Indexes, 66 Views, 7 Stored Procedures and 43 Functions to process Apache Access log in 4 formats & Apache Error log in 2 formats. Database normalization at work!

Features

Creates MySQL schema - apache_logs
Includes 2 Python modules
Imports Apache Access Logs - common, combined and vhost_combined
Imports Apache Error Logs
Complete Apache Log Data Normalization
MySQL does all Data Manipulation & Processing
Python handles polling of log file folders
Python executes MySQL Database LOAD DATA statements
Python executes Stored Procedures & Functions and SQL Statements
Requires installation of 4 Python modules - requirements.txt included

Project Samples

MySQL View - apache_logs.access_ua_browser_list - data from LogFormat: combined & extended

MySQL View - apache_logs.access_requri_list - data from LogFormat: combined & extended

MySQL View - apache_logs.error_apachemessage_list

Stored Procedures & Functions in apache_logs Schema

Project Activity

See All Activity >

License

Apache License V2.0

Follow apache-logs-to-mysql

apache-logs-to-mysql Web Site

Other Useful Business Software

EHS Software and Management System

ERA offers the only full EHS&Q platform with advanced automation to drive your complete compliance.

ERA Environmental Software Solutions develops web-based EHS management software for small, medium, and large manufacturers needing to comply with federal, provincial, and state regulations, monitor their air, water, and waste emissions and other environmental outputs, author and manage Safety Data Sheets (SDS) in more than 40 languages, or standardize their Health and Safety procedures for incident and inspection tracking, training delivery, and audit management. The platform also supports comprehensive reporting for programs like TRI, Tier II, Title V, NEI, and NPRI. Companies across the automotive, aerospace, general manufacturing, and paints and coatings industries, to name a few, rely on ERA’s all-in-one, SOC 2 Type II certified SaaS for complete coverage of their EHS needs.

Learn More

Rate This Project

User Reviews

Be the first to post a review of apache-logs-to-mysql!

Additional Project Details

Operating Systems

Linux, Mac, Windows

Intended Audience

Developers, Information Technology, Management, System Administrators

User Interface

Command-line, Console/Terminal

Programming Language

Python

Database Environment

Python Database API, SQL-based

Related Categories

Python Log Analysis Software, Python Log Management Tool, Python Log Parsers

Registered

2024-11-19

Similar Business Software

Windocks

Windocks is a leader in cloud native database DevOps, recognized by Gartner as a Cool Vendor, and as an innovator by Bloor research in Test Data Management. Novartis, DriveTime, American Family Insurance, and other enterprises rely on Windocks for on-demand database environments for development,...

See Software
Google Cloud SQL

Fully managed relational database service for MySQL, PostgreSQL, and SQL Server with rich extension collections, configuration flags, and developer ecosystems. New customers get $300 in free credits to spend on Cloud SQL. You won’t be charged until you upgrade. Reduce maintenance costs with...

See Software
pCloud Business

pCloud is a cloud-based digital asset management platform and cloud storage that provides access to all your digital content including images, video, audio, docs, and more- anytime, anywhere, on any device. Keep all of your important files safe and centralized in one place. You can share with...

See Software
Files.com

Files.com is a cloud-native Managed File Transfer (MFT) platform that unifies file transfers, sharing, and automation across any cloud, protocol, or partner. It connects 50+ storage systems — including Amazon S3, Azure, Google Drive, SharePoint, Dropbox, and Box — presenting them as a single...

See Software
JAMS

JAMS is an automation orchestration and job scheduling solution that works across applications, APIs, and scripting languages. Run, monitor, and manage critical IT processes—from simple batch jobs to cross-platform workflows—from a single pane of glass. JAMS can automate jobs on any...

See Software
Cerberus FTP Server

Cerberus FTP Server provides a secure and reliable file transfer solution for the demanding IT professional in any industry. Supporting SFTP and SCP, FTP/S, and HTTP/S, Cerberus is able to authenticate against Active Directory and LDAP, run as a Windows service, has native x64 support, includes...

See Software