Skip to content

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

37 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Natural Selection Skill Ranking and Data Analysis

This project does NS1 Log analysis using statistics and ML.

Requirements:

  • Linux shell + some RAM
  • Java 17 for Spark (Java 21+ won't work)
  • Spark 3.4+ and its required Java version
  • Python 3.12+ (required by full_analysis.sh)
  • Pip packages from requirements.txt

Environment variables

Use these when running locally on macOS/Linux:

export SPARK_LOCAL_IP=127.0.0.1
export JAVA_HOME="$(/usr/libexec/java_home -v 17)"

Why these are needed:

  • SPARK_LOCAL_IP Forces Spark to bind to localhost. This avoids hostname/network interface resolution problems (common on laptops and VPN setups) that can cause Spark startup failures or flaky local runs.
  • JAVA_HOME Pins Spark to Java 17 explicitly. Spark in this project is expected to run with Java 17; pointing to newer JVMs can fail at startup or runtime.
  • PYTHON_LOGGING_LEVEL (optional) Controls Python logging verbosity in the analysis scripts via ea_common.py. Example:
export PYTHON_LOGGING_LEVEL=DEBUG

Accepted values follow standard Python logging levels, for example DEBUG, INFO, WARNING, ERROR, CRITICAL.

For full-run, execute in a *nix terminal:

bash ./full_analysis.sh

About

ENSL Data Analysis

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages