This project does NS1 Log analysis using statistics and ML.
- Linux shell + some RAM
- Java 17 for Spark (Java 21+ won't work)
- Spark 3.4+ and its required Java version
- Python 3.12+ (required by
full_analysis.sh) - Pip packages from
requirements.txt
Use these when running locally on macOS/Linux:
export SPARK_LOCAL_IP=127.0.0.1
export JAVA_HOME="$(/usr/libexec/java_home -v 17)"Why these are needed:
SPARK_LOCAL_IPForces Spark to bind to localhost. This avoids hostname/network interface resolution problems (common on laptops and VPN setups) that can cause Spark startup failures or flaky local runs.JAVA_HOMEPins Spark to Java 17 explicitly. Spark in this project is expected to run with Java 17; pointing to newer JVMs can fail at startup or runtime.PYTHON_LOGGING_LEVEL(optional) Controls Python logging verbosity in the analysis scripts viaea_common.py. Example:
export PYTHON_LOGGING_LEVEL=DEBUGAccepted values follow standard Python logging levels, for example DEBUG, INFO, WARNING, ERROR, CRITICAL.
For full-run, execute in a *nix terminal:
bash ./full_analysis.sh