Flickering Flame

Network detection script for Odroid C2/N2, Raspberry Pi Zero

Rajat -

I got 3 single-board servers at home - "trusty" Odroid C2, "powerful" Odroid N2 and my "nibble" Raspberry Pi Zero. Odroid N2 is my media center (does anyone still use one in 2020!) running CoreElec

All 3 perform different functions and run different versions of OSs.

I was trying to solve specific problems with each of them -

  • Odroid C2 - Wanted to auto mount my network shares, if they were reachable.
  • Odroid N2 & Raspberry Pi Zero - Due to unreliable power supply, these servers during a power recycle failed to connect to the router/LAN or couldn't connect to the internet.

Only solution which worked was to manually power off/on the devices. So I thought I would quickly mash up a BASH script which could detect and resolve these issues. 6 hours later, after multiple rewrites and refactoring, I finally managed to get it working. Implementing a script on 3 devices which was running 3 different versions of OS, different network configurations and using no external dependencies was quite hairy!

BASH script can currently do the following -

  • Check if the device is connected to the router
  • Check if the DHCP is configured as expected on device
  • If remote share is online, auto-mount remote shares
  • Verify if DNS resolution is working using multiple methods. If ICMP is blocked, then PING will not work so we will try Netcat
  • Verify if internet connectivity is working
  • Verify if server can be pinged
  • Verify if HTTP requests can be made. Can be configured to use a Proxy.
  • Automatic log cleanup when it reaches 100MB (configurable)
  • Placeholder notification function which can perform a custom action on failure
  • Finally, scheduled restart of device if it's not connected to Router/LAN (To deal with specific issue with CoreElec)

The script can be configured to run periodically using cron. It will generate a log like -

-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Timestamp: 1585131300
Pinging gateway 192.168.2.1 to verify LAN connectivity
Gateway reachable. Proceeding with additional checks.
Name resolution of 192.168.2.1 using first DNS server in resolv.conf
DNS resolution successful for 192.168.2.1.
Pinging google.com to check for internet connection.
google.com pingable. Internet up.
Scan of google.com on port 443
Success. google.com is available.
Verify HTTP connectivity
HTTP connectivity is up

The script is available here - https://gist.github.com/RajatNair/b510934d08bb9e18e1f3040c9e6696e4

#!/bin/bash

# Author: Rajat Nair (aunlead.com)
# Description: This script will periodically check -
#  - if device is on LAN
#  - if remote share is online
#  - Automount remote shares
#  - Verify if DNS resolution is working
#  - Verify if internet connectivity is working
#  - Verify if server can be pinged
#  - Verify if HTTP requests can be made
#  - Automatic log cleanup when it reaches 100MB
#  - Restart of device if it's not connected to LAN
#
# Usage -
# Edit Cron using
# crontab -e
# Copy script and give it execute permissions. To run every 30 minutes-
# */30 * * * * /opt/network/network-detection-script.sh

#
# Configurations
#
GATEWAY_CONFIGURED=$(/sbin/ip route | awk '/default/ { print $3 }' | awk 'NR == 1 {print; exit}')
GATEWAY_EXPECTED=192.168.2.1
VERIFY_DNS=$(cat /etc/resolv.conf | awk '/nameserver/ {print $2}' | awk 'NR == 1 {print; exit}')
VERIFY_DOMAIN=google.com
# Change your mountpoint according to your requirement
CUSTOM_MOUNTPOINT=/mnt
NETWORK_SHARE=192.168.2.3

#
# Logs
#
LOG_FILE=/var/log/network-script.log
TIMESTAMP=$(date +%s)
# 100 MB
LOG_SIZE=100000000

#
# When check fails do..
#
verificationFailure()
{
  # Placeholder to implement your custom notification
  # exit 1
  echo "Verification failed."
}


#
# Netcat to domain on HTTPS port
#
netcatScan()
{
  echo "Scan of $VERIFY_DOMAIN on port 443" >>${LOG_FILE}
  if nc -zw1 $VERIFY_DOMAIN 443; then
    echo "Success. $VERIFY_DOMAIN is available." >>${LOG_FILE}
  else
    echo "Failed. $VERIFY_DOMAIN is unreachable." >>${LOG_FILE}
    verificationFailure
  fi
}

#
# Ping domain
#
internetAccessVerification()
{
  echo "Pinging $VERIFY_DOMAIN to check for internet connection." >>${LOG_FILE}
  ping $VERIFY_DOMAIN -c 4
  if [ $? -eq 0 ]; then
    echo "$VERIFY_DOMAIN pingable. Internet up." >>${LOG_FILE}
  else
    echo "Internet down." >>${LOG_FILE}
    verificationFailure
    #exit 1
  fi
}

#
# Verify DNS resolution
#
dnsVerification()
{
  if [ "$VERIFY_DNS" != "run" ]; then
      echo "Name resolution of $VERIFY_DNS using first DNS server in resolv.conf" >>${LOG_FILE}
      ping "$VERIFY_DNS" -c 4
      if [ $? -eq 0 ]; then
        echo "DNS resolution successful for $VERIFY_DNS." >>${LOG_FILE}
      else
        echo "DNS resolution failed for $VERIFY_DNS." >>${LOG_FILE}
        verificationFailure
        #exit 1
      fi
    else
      echo "DNS resolution skipped" >>${LOG_FILE}
  fi
}

#
# Verify HTTP connection
#
httpVerification()
{
  echo "Verify HTTP connectivity" >>${LOG_FILE}
  case "$(curl -s --max-time 2 -I $VERIFY_DOMAIN | sed 's/^[^ ]*  *\([0-9]\).*/\1/; 1q')" in
  [23]) echo "HTTP connectivity is up" >>${LOG_FILE} ;;
  5)
    echo "Proxy error" >>${LOG_FILE}
    verificationFailure
    ;;
  *)
    echo "HTTP connectivity is down" >>${LOG_FILE}
    verificationFailure
    ;;
  esac
  #  exit 0
}

#
# Mount network shared from fstab
# if they are online
#
mountNetworkShares()
{
  ISMOUNTED=$(df -k | grep $CUSTOM_MOUNTPOINT | wc -l)
  if [ "$ISMOUNTED" -gt 0 ]; then
    echo "$CUSTOM_MOUNTPOINT has been mounted." >>${LOG_FILE}
  else
    echo "$CUSTOM_MOUNTPOINT is not mounted. Attempting to mount." >>${LOG_FILE}
    ping "$NETWORK_SHARE" -c 4
    if [ $? -eq 0 ]; then
      echo "$NETWORK_SHARE is available." >>${LOG_FILE}
      # mount -a will mount drives from /etc/fstab
      /bin/mount -a
    else
      echo "$NETWORK_SHARE is unreachable." >>${LOG_FILE}
      #exit 1
    fi
    #exit 1
  fi
}

#
# Clean up log
#
logCleanup()
{
  size="$(wc -c <"$LOG_FILE")"
  if [ "$size" -gt $LOG_SIZE ]; then
    cat /dev/null >$LOG_FILE
  fi
}


#
# Unrecoverable issue detected. Attempt reboot in 10 minutes.
# To prevent reboot loop, login within 10 minutes
# and run `/sbin/shutdown -c` to cancel it
#
catastrophicFailure()
{
  #/bin/sleep 5m
  /sbin/shutdown -r +10
}

echo "-=-=-=-=-=-=-=-=-=-=-=-=-=-=" >>${LOG_FILE}
echo "Timestamp: $TIMESTAMP" >>${LOG_FILE}
echo "Pinging gateway $GATEWAY_CONFIGURED to verify LAN connectivity" >>${LOG_FILE}
if [ "$GATEWAY_CONFIGURED" = "" ]; then
  echo "No gateway detected" >>${LOG_FILE}
  catastrophicFailure
  #exit 1
fi

if [ "$GATEWAY_CONFIGURED" != "$GATEWAY_EXPECTED" ]; then
  echo "DHCP failed." >>${LOG_FILE}
  catastrophicFailure
  #exit 1
fi

ping "$GATEWAY_CONFIGURED" -c 4

if [ $? -eq 0 ]; then
  echo "Gateway reachable. Proceeding with additional checks." >>${LOG_FILE}
  mountNetworkShares
  dnsVerification
  internetAccessVerification
  netcatScan
  httpVerification
  logCleanup
  exit 0
else
  echo "Gateway unreachable." >>${LOG_FILE}
  catastrophicFailure
  exit 1
fi